diff --git a/_build_cfg.yml b/_build_cfg.yml index 5faa6ecbec70..f8f2811f1a62 100644 --- a/_build_cfg.yml +++ b/_build_cfg.yml @@ -217,6 +217,8 @@ Topics: File: sti - Name: Testing S2I Images File: sti_testing + - Name: Custom Builder + File: custom --- Name: Using Images diff --git a/architecture/core_objects/builds.adoc b/architecture/core_objects/builds.adoc index adc82fa3faae..57a515b5189c 100644 --- a/architecture/core_objects/builds.adoc +++ b/architecture/core_objects/builds.adoc @@ -10,33 +10,38 @@ toc::[] == Overview -A build is the end result of transforming input parameters, typically source code, into a resulting object, typically a runnable image. +A build is the process of transforming input parameters, typically source code, +into a resulting object, typically a runnable image. For a list of build commands, see the link:../../dev_guide/builds.html[Developer's Guide]. == Build Strategies -There are three available link:openshift_model.html#build-strategies[build strategies]: +There are three link:openshift_model.html#build-strategies[build strategies] available: * link:#docker-build[Docker build] -* link:#source-build[Source-to-Image build] +* link:#source-build[Source-to-Image (S2I) build] * link:#custom-build[Custom build] -The resulting object depends on the builder used to create the image. +The resulting object depends on the builder used to create it, for Docker and S2I builds +those will be runnable images, for Custom build - whatever the builder image author +wants. [#docker-build] === Docker Build -Docker builds invoke the plain https://docs.docker.com/reference/commandline/cli/#build[docker build command], and therefore expect a repository with a *_Dockerfile_* and all required directories. +Docker build invokes the plain https://docs.docker.com/reference/commandline/cli/#build[docker build] +command, and therefore expects a repository with a *_Dockerfile_* and all required +artifacts in it to produce a runnable image. [#source-build] === Source-to-Image Build link:../../creating_images/sti.html[Source-to-Image (S2I)] is a tool for building reproducible Docker images. It produces ready-to-run images by injecting a user source into a docker image and assembling a new docker image. -The new image incorporates the base image and built source, and is ready to use -with the `docker run` command. S2I supports incremental builds, which re-use -previously downloaded dependencies, previously built artifacts, etc. +The new image incorporates the base image (the builder) and built source, and is +ready to use with the `docker run` command. S2I supports incremental builds, which +re-use previously downloaded dependencies, previously built artifacts, etc. The advantages of S2I include: @@ -49,8 +54,8 @@ source, so the image needs to be able to process tarred content. Speed:: With S2I, the assemble process can perform a large number of complex operations without creating a new layer at each step, resulting in a fast process. In addition, S2I scripts can be written to re-use artifacts stored in a -previous version of the application image rather than having to download or -build them each time the build is run. +previous version of the application image, rather than having to download or +build them, each time the build is run. Patchability:: S2I allows you to rebuild the application consistently if an underlying image needs a patch due to a security issue. @@ -63,7 +68,7 @@ Operational security:: Building an arbitrary *_Dockerfile_* exposes the host system to root privilege escalation. This can be exploited by a malicious user because the entire docker build process is run as a user with docker privileges. S2I restricts the operations performed as a root user, and can run the scripts -as an non-root user. +as a non-root user. User efficiency:: S2I prevents developers from performing arbitrary `yum install` type operations during their application build, which results in slow @@ -74,7 +79,7 @@ best practices for your applications. [#custom-build] === Custom Build -Custom builds allow developers to define the specific builder image responsible -for the entire build process. The custom builder image is a plain Docker image -with embedded build process logic, such as building RPMs or building base Docker -images. +link:../../creating_images/custom.html[Custom build] allows developers to define +a specific builder image, responsible for the entire build process. The custom +builder image is a plain Docker image with embedded build process logic, such as +building RPMs or building base Docker images. diff --git a/creating_images/custom.adoc b/creating_images/custom.adoc new file mode 100644 index 000000000000..9fdad8db5085 --- /dev/null +++ b/creating_images/custom.adoc @@ -0,0 +1,40 @@ += Custom builder +{product-author} +{product-version} +:data-uri: +:icons: +:experimental: +:toc: macro +:toc-title: + +toc::[] + +== Overview +link:../architecture/core_objects/builds.html#custom-build[Custom build] is designed +to fill the gap that was created when everybody jumped into creating docker images. +Still there is a requirement to produce packages with all the software being installed +and most importantly one needs to be able to build base docker images. This is +where Custom build is the perfect match to fill in that gap. + +To fully utilize the power of Custom build one needs to be understand how to +create a builder image that will be capable of building desired objects. + +== Custom builder image +The builder image upon invocation receives following environment variables with +the information needed to proceed with the build: + +.Custom Builder Environment Variables +[cols="4a,6a",options="header"] +|=== + +|Variable name |Description + +|`*BUILD*` +|This variable specifies the entire serialized link:../rest_api/openshift_v1.html#v1-build[Build] object. + +|`*SOURCE_REPOSITORY*` +|This variable specifies the URL to a repository with sources to build. + +|`*DOCKER_SOCKET*` +|This variable specifies the path to docker socket, if exposing docker socket was enabled on BuildConfig. +|=== diff --git a/creating_images/sti.adoc b/creating_images/sti.adoc index 301e15bf3f2d..c15e52cd42d2 100644 --- a/creating_images/sti.adoc +++ b/creating_images/sti.adoc @@ -16,11 +16,12 @@ code as an input and produce a new image that runs the assembled application as output. The main advantage of using S2I for building reproducible Docker images is the -ease of use for developers. As a builder image author, you must be aware of the -two basic requirements for the best possible S2I performance: the required image -contents and S2I scripts. +ease of use for developers. As a builder image author, you must understand two +basic concepts in order for your images to provide the best possible S2I performance: +link:#build-process[the build process] and link:#s2i-scripts[S2I scripts]. -== Required Image Contents +[#build-process] +== Build Process The build process consists of the following three fundamental elements, which are combined into a final Docker image: @@ -36,11 +37,11 @@ location specified with the `--destination` flag or the `*io.s2i.destination*` label from the builder image, with the default location being the *_/tmp_* directory. -For this *tar* process to happen, your image must supply the *tar* archiving +For this process to happen, your image must supply the *tar* archiving utility (the `tar` command available in `*$PATH*`) and the command line interpreter (the `/bin/sh` command); this allows your image to use the fastest possible build path. If the `tar` or `/bin/sh` command is not available, the -`sti build` script is forced to automatically perform an additional Docker build +`sti build` process is forced to automatically perform an additional Docker build to put both the sources and the scripts inside the image, and only then run the usual `sti build` procedure. @@ -49,12 +50,12 @@ See the following diagram for the basic S2I build workflow: .Build Workflow image::sti-flow.png[S2I workflow] -//// -* Run build's responsibility is to untar the sources, scripts and artifacts (if such exist) and invoke `assemble` script. If this is second run (after catching `tar`/`/bin/sh` error) it's responsible only for invoking `assemble` script, since both scripts and sources are already there. -//// +* Run build's responsibility is to untar the sources, scripts and artifacts (if such exist) and invoke the `assemble` script. If this is the second run (after catching `tar`/`/bin/sh` not found error) it is responsible only for invoking `assemble` script, since both scripts and sources are already there. + +[#s2i-scripts] == S2I Scripts -You can write S2I scripts in any programming language as long as the scripts are +You can write S2I scripts in any programming language, as long as the scripts are executable inside the builder image. S2I supports multiple options providing `assemble`/`run`/`save-artifacts` scripts. All of these locations are checked on each build in the following order: @@ -63,7 +64,7 @@ each build in the following order: 2. A script found in the application source `.sti/bin` directory 3. A script found at the default image URL (`io.s2i.scripts-url` label) -Both `io.s2i.scripts-url` label specified in the image and `--scripts-url` flag +Both the `io.s2i.scripts-url` label specified in the image and the `--scripts-url` flag can take one of the following form: - `image://path_to_scripts_dir` - absolute path inside the image to a directory where the S2I scripts are located @@ -86,7 +87,7 @@ or `io.s2i.destination` label applies only to sources and artifacts. and places them into appropriate directories inside the image. The workflow for this script is: -. Restore build artifacts. If you want to support incremental builds, make sure to define *_save-artifacts_* as well. +. Restore build artifacts. If you want to support incremental builds, make sure to define *_save-artifacts_* as well (optional). . Place the application source in the desired location. . Build the application artifacts. . Install the artifacts into locations appropriate for them to run. @@ -100,8 +101,8 @@ this script is: |The *_save-artifacts_* script gathers all dependencies that can speed up the build processes that follow. For example: -- For Ruby, *gems* is installed by Bundler. -- For Java, *.m2* contents are installed. +- For Ruby, *gems* installed by Bundler. +- For Java, *.m2* contents. These dependencies are gathered into a tar file and streamed to the standard output. @@ -119,16 +120,15 @@ image is working correctly. The proposed flow of that process is: . Build the image. . Run the image to verify the *_usage_* script. . Run `sti build` to verify the *_assemble_* script. -. Run `sti build` again to verify the *_save-artifacts_* script and the *_usage_* script's restore artifacts functionality. (optional) +. Run `sti build` again to verify the *_save-artifacts_* and *_assemble_* scripts save and restore artifacts functionality. (optional) . Run the image to verify the test application is working. See the link:sti_testing.html[Testing S2I Images] topic for more information. NOTE: The suggested location to put the test application built by your *_test/run_* script is the *_test/test-app_* directory in your image repository. -See the -https://github.com/openshift/source-to-image/blob/master/docs/cli.md#sti-create[S2I -documentation] for more information. +See the https://github.com/openshift/source-to-image/blob/master/docs/cli.md#sti-create[S2I documentation] +for more information. |=== *Example S2I Scripts* @@ -213,19 +213,21 @@ example: See the https://docs.docker.com/reference/builder/#onbuild[Docker documentation] for more information on `ONBUILD`. -S2I has a different strategy when a Docker image with `ONBUILD` instructions is -used as a builder image for the application source code. During the S2I build, +Upon start S2I detects whether the builder image uses `ONBUILD` instructions. +If there are none, the regular S2I link:#build-process[build] is performed, +otherwise a different strategy is chosen. During such a S2I build, all `ONBUILD` instructions are executed in the order they were defined in the -builder image Dockerfile. The S2I scripts are not required for this strategy, -but they can be used as supplementary scripts to existing `ONBUILD` -instructions. +builder image's Dockerfile. The S2I scripts are not required for this strategy, +but they can be used as a supplement to existing `ONBUILD` instructions. Many official Docker images that use `ONBUILD` do not declare the image `CMD` or `ENTRYPOINT`, and for that, S2I must know how to run your application. There are two methods for defining the `ENTRYPOINT`: -- Include the *_run_* script in your application root folder. S2I then recognizes it and sets it as the application image `ENTRYPOINT`. +- Include the *_run_* script in your application root folder. S2I recognizes it +and sets it as the application image `ENTRYPOINT`. -- Use the S2I scripts. If you provide the URL from where the S2I can fetch the scripts, the S2I *_run_* script is then -set as an image `ENTRYPOINT`. If the S2I scripts location also includes the *_assemble_* script, the script is then -executed as the last instruction of the Docker build. +- Use the link:#s2i-scripts[S2I scripts]. If you provide them, the *_run_* script +is set as an image `ENTRYPOINT`. If the S2I scripts location also includes the +*_assemble_* script, that script is executed as the last instruction of the underlying +Docker build. diff --git a/creating_images/sti_testing.adoc b/creating_images/sti_testing.adoc index 1d0324c64ad7..e664b981c8c8 100644 --- a/creating_images/sti_testing.adoc +++ b/creating_images/sti_testing.adoc @@ -36,19 +36,19 @@ invoked by the OpenShift S2I image builder and it could be a simple Bash script or a static Go binary. The *_test/run_* script performs the S2I build, so you must have the S2I binary -available on your system. If required, follow the installation instructions in -the S2I -https://github.com/openshift/source-to-image/blob/master/README.md#installation[README]. +available in your `*$PATH*`. If required, follow the installation instructions in +the https://github.com/openshift/source-to-image/blob/master/README.md#installation[S2I README]. -S2I combines the application source code and Docker image, so in order to test +S2I combines the application source code and builder image, so in order to test it you need a sample application source to verify that the source successfully -converts into a Docker image. The sample application should be simple, but it -should also exercise the `assemble` command. +transforms into a runnable Docker image. The sample application should be simple, +but it should exercise the crucial steps of `assemble` and `run` scripts. == Generating Scripts and Tools The S2I tooling comes with powerful generation tools to speed up the process of -creating a new S2I image. The `sti create` command produces all necessary S2I +creating a new S2I image. The `sti create` command produces all the necessary S2I scripts and testing tools along with the *_Makefile_*: + **** `$ sti create __ __` **** @@ -84,7 +84,7 @@ test: ==== == Basic Testing Workflow -The *_test_* script assumes you have already built the image that you want to +The *_test_* script assumes you have already built the image you want to test. If required, first build the S2I image using: **** @@ -98,7 +98,7 @@ The following steps describe the default workflow to test S2I image builders: ==== **** -`$ docker run __` +`$ docker run __ .` **** ==== @@ -112,7 +112,8 @@ The following steps describe the default workflow to test S2I image builders: **** ==== -. If you support *_save-artifacts_*, then execute step 2 again to verify that restoring artifacts works properly. +. Optionally, if you support *_save-artifacts_*, execute step 2 once again to +verify that saving and restoring artifacts works properly. . Run the container: + @@ -125,16 +126,17 @@ The following steps describe the default workflow to test S2I image builders: . Verify the container is running and the application is responding. -Executing these steps is generally enough to tell if the S2I scripts are -operating properly. +Executing these steps is generally enough to tell if the builder image is +working as expected. == Using OpenShift Build for Automated Testing Another way you can execute the S2I image tests is to use the OpenShift platform itself as a continuous integration system. The OpenShift platform is capable of building Docker images and is highly customizable. -To set up an S2I image builder continuous integration system, define a special `*CustomBuild*` and use the -*openshift/sti-image-builder* image. This image executes all the steps mentioned +To set up an S2I image builder continuous integration system, define a +link:../architecture/core_objects/builds.html#custom-build[Custom build] and use +the *openshift/sti-image-builder* image. This image executes all the steps mentioned in the link:#basic-testing-workflow[Basic Testing Workflow] section and creates a new S2I builder image. @@ -193,6 +195,7 @@ a new S2I builder image. } ---- + ==== You can use the `oc create` command to create this `*BuildConfig*`. After you create the `*BuildConfig*`, you can start the build using the following command: @@ -204,10 +207,10 @@ You can use the `oc create` command to create this `*BuildConfig*`. After you cr **** ==== -If your OpenShift instance is hosted on a public IP address, then the build is -triggered each time you push into your S2I builder image GitHub repository. +If your OpenShift instance is hosted on a public IP address, the build can be +triggered each time you push into your S2I builder image GitHub repository. See +link:../dev_guide/builds.html#webhook-triggers[webhook triggers] for more information. -You can also use the `*CustomBuild*` to trigger a rebuild for your applications -based on the S2I image you updated. To do this you must specify the `Output` -field in the `parameters` section and define to which Docker registry the image -should be pushed after a successful build. +You can also use the `*CustomBuild*` to trigger a rebuild of your application +based on the S2I image you updated. See link:../dev_guide/builds.html#image-change-triggers[image change triggers] +for more information. diff --git a/dev_guide/builds.adoc b/dev_guide/builds.adoc index 875f86b356eb..81dd60e55d5d 100644 --- a/dev_guide/builds.adoc +++ b/dev_guide/builds.adoc @@ -81,17 +81,17 @@ every time a Docker image tag or the source code changes: <1> This specification will create a new buildConfig named ruby-sample-build. <2> You can specify a list of triggers, which causes a new build to be created. -The `*ImageChange*` trigger creates a new build each time the Docker image specified in the `*strategy.from*` field changes. The `*github*` and `*generic*` triggers starts the build when a callback is made from the GitHub or other source code repository hosting service, indicating change in the source code specified in `*source*` section. +The `*ImageChange*` trigger creates a new build each time the Docker image specified in the `*strategy.from*` field changes. The `*Github*` and `*Generic*` triggers starts the build when a callback is made from the GitHub or other source code repository hosting service, indicating change in the source code specified in `*source*` section. <3> The `*source*` section defines the source code repository location. You can provide additional options, such as `*sourceSecret*` or `*contextDir*` here. <4> The `*strategy*` section describes the build strategy used to execute the build. You can specify `*Source*`, `*Docker*` and `*Custom*` strategies here. This specification uses the `*ruby-20-centos7*` Docker image that Source-To-Image will use for the application build. <5> After the Docker image is successfully built, it will be pushed into the repository described in the `*output*` section. ==== == Defining an incremental buildConfig -Source-To-Image (S2I) builds provide an option to perform incremental build, which means they will -reuse artifacts from previously built image. To create an incremental build one -needs to create a BuildConfig as previously explained, but with a slight modification -in strategy definition as presented below: +The Source-To-Image (S2I) strategy provides an option to perform incremental builds, +which means it will reuse artifacts from previously built image. To create an incremental +build one needs to create a BuildConfig as previously explained, but with a slight +modification in strategy definition as presented below: ==== @@ -111,43 +111,49 @@ in strategy definition as presented below: } ---- -<1> You need to specify here an image that supports incremental builds. Note: the S2I images provided by OpenShift do not implement artifact reuse, so setting incremental to true will have no effect on BuildConfigs using those builder images. -<2> This flag controls whether an incremental build is attempted. If the builder image does not support incremental builds, the build will still succeed but you will get a log message that the incremental build was not successful because of a missing `save-artifacts` script. +<1> You need to specify an image that supports incremental builds. Note that the S2I +images provided by the OpenShift do not implement artifact reuse, so setting `incremental` + to `true` will have no effect on Builds using those builder images. + +<2> This flag controls whether an incremental build is attempted. If the builder +image does not support incremental builds, the build will still succeed but you +will get a log message stating the incremental build was not successful because +of a missing `save-artifacts` script. ==== -NOTE: For information on how to create a builder image which supports incremental build see link:../creating_images/sti.html[creating images guide]. +NOTE: For information on how to create a builder image supporting incremental builds see link:../creating_images/sti.html[creating images guide]. == Starting a Build You can manually invoke a build using the following command: **** -`$ oc start-build __` +`$ oc start-build __` **** A build can be re-run using the `--from-build` flag: **** -`$ oc start-build --from-build=__` +`$ oc start-build --from-build=__` **** Specify the `--follow` flag to stream the build's logs in stdout: **** -`$ oc start-build __ --follow` +`$ oc start-build __ --follow` **** == Canceling a Build You can manually cancel a build using the following command: **** -`$ oc cancel-build __` +`$ oc cancel-build __` **** == Accessing Build Logs To allow access to build logs, use the following command: **** -`$ oc build-logs __` +`$ oc build-logs __` **** *Source Build Logs* @@ -187,6 +193,7 @@ Level 0:: Produces output from containers running the *_assemble_* script and al Level 1:: Produces basic information about the executed process. Level 2:: Produces very detailed information about the executed process. Level 3:: Produces very detailed information about the executed process, and a listing of the archive contents. +Level 5:: Produces everything mentioned on previous levels and additionally provides docker push messages. == Source Code The source code location is one of the required parameters for the @@ -208,7 +215,7 @@ is later built. The source code location definition is part of the } ---- -<1> The `*type*` field describes what SCM is used to fetch your source code. +<1> The `*type*` field describes which SCM is used to fetch your source code. <2> In this example, the `*git*` field contains the URI to the remote Git repository where your source code lives. It might optionally specify the `*ref*` field if you want to check out a specific Git reference. A valid `*ref*` can be @@ -222,15 +229,17 @@ default location (the root folder) using this field. [[configuring-the-source-environment]] == Source Environment -There are two ways to make environment variables available to the link:../architecture/core_objects/builds.html#source-build[Source] -build process and resulting image, Environment files and BuildConfig environment values. +There are two ways to make environment variables available to the +link:../architecture/core_objects/builds.html#source-build[Source build] +process and resulting \image: link:#environment-files[environment files] and +link:#buildconfig-environment[BuildConfig environment] values. === Environment files -Source enables you to set environment values -in your application by specifying them in a *_.sti/environment_* file in the -source repository. The environment variables are then present during the build -process and in the final docker image. The complete list of supported -environment variables are available in the documentation for each image. +Source build enables you to set environment values (one per line) +inside your application, by specifying them in a *_.sti/environment_* file in the +source repository. The environment variables specified in this file are present during +the build process and in the final docker image. The complete list of supported +environment variables is available in the documentation for each image. If you provide a *_.sti/environment_* file in your source repository, S2I reads this file during the build. This allows customization of the build behavior as @@ -248,22 +257,41 @@ application to be started in `development` mode instead of `production`. === BuildConfig Environment You can add environment variables to the SourceStrategy definition of the BuildConfig. -Environment variables defined here will be visible during the *_assemble_* script +Environment variables defined there will be visible during the *_assemble_* script execution and will be defined in the output image, making them also available to the *_run_* script and application code. +For example disabling assets compilation for your Rails application: + +==== + +---- +{ + "sourceStrategy": { + ... + "env": [ + { + "Name": "DISABLE_ASSET_COMPILATION", + "Value": "true" + } + ] + } +} +---- +==== + == Build Triggers When defining a `*BuildConfig*`, you can define triggers to control the circumstances in which a build should be run for the `*BuildConfig*`. There are two types of triggers available: -* Webhook -* Image change +* link:#webhook-triggers[Webhook] +* link:#image-change-triggers[Image change] === Webhook Triggers Webhook triggers allow you to trigger a new build by sending a request to the OpenShift API endpoint. You can define these triggers using -https://developer.github.com/webhooks/[GitHub webhooks] or generic webhooks. +https://developer.github.com/webhooks/[GitHub webhooks] or Generic webhooks. *GitHub Webhooks* @@ -462,12 +490,13 @@ are performed using consistent image tags for ease of reproduction. [#using-docker-credentials-for-pushing-and-pulling-images] == Using Docker Credentials for Pushing and Pulling Images -Supply the `.dockercfg` file with valid Docker Registry credentials in order to push the output image into a private Docker Registry or pull the +You must supply the `.dockercfg` file with valid Docker Registry credentials in +order to push the output image into a private Docker Registry or pull the builder image from the private Docker Registry that requires authentication. For the OpenShift Docker Registry, you don't have to do this because the Secrets are generated automatically for you by OpenShift. -The *_.dockercfg_* JSON file exists in your home directory by default and has +The *_.dockercfg_* JSON file exists in your home directory by default and has the following format: ==== @@ -528,7 +557,7 @@ that you created, which in the above example is `*dockerhub*`: "name": "private-image" }, "pushSecret":{ - "name":"dockerhub" + "name": "dockerhub" } } } @@ -536,7 +565,7 @@ that you created, which in the above example is `*dockerhub*`: ---- ==== -Pull the builder Docker image from a private Docker registry by specifying the +To pull the builder Docker image from a private Docker registry specify the `PullSecret` field, which is part of the build strategy definition: ==== @@ -562,7 +591,7 @@ Pull the builder Docker image from a private Docker registry by specifying the [#using-private-repositories-for-builds] == Using Private Repositories for Builds -Supply valid credentials to build an application from a private repository. +You must supply valid credentials to build an application from a private repository. Currently, only SSH key based authentication is supported. The repository keys are located in the `$HOME/.ssh/` directory, and are named `id_dsa.pub`, `id_ecdsa.pub`, `id_ed25519.pub` or `id_rsa.pub` by default. Generate SSH key @@ -575,7 +604,7 @@ $ ssh-keygen -t rsa -C "your_email@example.com" ---- ==== -Two files will be created: the public key (as explained above) and a +As a result, two files will be created: the public key (as explained above) and a corresponding private key (one of `id_dsa`, `id_ecdsa`, `id_ed25519` or `id_rsa`). With both of these in place you should consult your source control management (SCM) system's manual on how to upload the public key. The private