Templates Includes

Templates and Includes

Templates allow you to define your logic once and reuse it several times. Using templates, you can combine the contents of multiple YAML files into a single pipeline.

Templates can be defined locally or included from remote git repositories.

Step re-use

One or more steps can be reused across several pipelines. We can include steps from multiple files, and include additional steps explicitly in POET pipeline before or after the inclusion.

As mentioned, both local and remote inclusion will be possible. The following examples focus on local as an example, remote inclusion is discussion in its own section.

Let's define two local templates:

# File: templates/docker-compose-publish.yml
# Note, split into two steps as an example of a template
# that expands to multiple steps
steps:
  - name: build-containers
    image: ${plugins}/docker
    commands:
      - docker-compose build
  - name: publish-containers
    image: ${plugins}/docker
    secrets:
      - source: jenkinsdeployer
        target:
          - DOCKER_USER
          - DOCKER_PASS
    commands:
      - echo ${DOCKER_PASS} | docker login -u ${DOCKER_USER} --password-stdin ${ARTIFACTORY}
      - docker-compose push
      - docker-compose down -v --rmi=all
      - docker logout ${ARTIFACTORY}

# File: templates/slack.yml
steps:
  - name: slack-notification
    image: ${plugins}/slack
    secrets:
      - source: pipeline_notifier_slack_hook
        target:
          - SLACK_WEBHOOK
    commands:
      - python /app/slack.py
    when:
      status: [ success, failure ]
      branch: master

Now we'll incorporate them into POET pipeline:

# File: pipeline.yml
pipeline:
  steps:
    - name: build-jar
      image: mvn:3-jdk-11
      commands: [ mvn package ]
    # the following include will expand to 2 steps
    - include: templates/docker-compose-publish.yml
    - include: templates/slack.yml # template include (1 step)
      environment: # note: we're adding to the environment
        ROOM: "#my_room"
    - name: say-hello
      image: ubuntu:14.04
      commands: [ echo "hello" ]

Note that the resulting pipeline would contain 5 steps, since we have 1 step before and after the included steps, and one of the includes will expand to 2 steps.

Configuration Re-use

There may be other global pipeline information we wish to share. For example, in a microservice type architecture, we may want to share a base version or other standard configuration information.

Like at the step level, you can include templates at the pipeline level that can import global and environment information from a list of templates. In the future, this may include other information such as secrets, services/sidecar containers, etc.

# File: templates/microservice-env.yml
pipeline:
  appVersion:
    master: 1.0.0
  environment:
    LOG_LEVEL: "info"

# File: pipeline.yml
pipeline:
  include:
    - templates/microservice-env.yml

include at the pipeline level is only valid for configuration information. Templates included at this level must not include pipeline steps.

Evaluation order

• Included files will be processed in order, with later files overriding previous ones.
• Definitions from the host pipeline.yml file will be evaluated last and override any previous definitions.

Passing data to templates

Like explicit steps, templates will have access to the Standard Environment Variables and the pipeline global Environment.

Also like explicit steps, these variables will be for expansion in any step.name or step.image properties in a template. Likewise, they will be available as environment variables in the container for the commands at runtime.

Passing additional environment variables

At the use site of the template, the containing pipeline can add additional variables for use in the template by adding an environment section when referencing the template.

pipeline:
  steps:
    - include: templates/slack.yml
      environment:
        ROOM: "#my_room"

This will add the environment variable ROOM to the environment for all steps contained in the template. If any of the steps previously had a value for ROOM in its environment, it will be overwritten.

Remote includes (using other repositories)

Templates can be defined in arbitrary repositories.

For example, we could define a "standard java" template in an sre-pipeline-templates repository:

# Repo: ssh://[email protected]/...pipeline-templates.git
# File: standard-java.yml
pipeline:
  steps:
    - name: build-jar
      image: maven:${maven_version:-3-jdk-9}
      commands:
        - mvn clean package
    - name: run-sonarqube
      image: ...
      ...
    - name: publish-to-artifactory
      image: ...
      ...

Now we can reuse this template in multiple repositories.

At the top-level in our pipeline.yml, we can provide repository details under a resources element.

To specify that the file is coming from a repo, we use @ followed by the name we gave the repository.

We can include files from multiple remote repositories, as well as mix with local templates.

# File: pipeline.yml
resources:
  repositories:
    - name: templates
      uri: ssh://[email protected]/...pipeline-templates.git
      # label is optional (defaults to master), and can reference branch, tag, commit
      label: master      
      # Note that larger design for secrets is still pending, credentials details may change
      credentials:
        id: buildmaster

pipeline:
  steps:
    - include: standard-java.yml@templates
    - include: slack.yml@templates
    - include: templates/my_local_template.yml

• Repositories are resolved only once, at pipeline start-up
  • the same resource is used for the duration of the pipeline
• Once expanded, the final pipeline runs as if it were defined entirely in the source repo
  • you can't use scripts or other files from the template repo

Templates in Templates

We allow templates to include other templates up to a maximum depth of 5.

Resources and scope

Like the host pipeline.yml file, both step and config templates can include their own resources section where they may specify other repositories to use for template includes.

The parent repositories are not consulted, each template is treated independently.

A template may refer to other templates within its repository in a local/relative fashion.

Recursive Include Example

# File: pipeline.yml
resources:
  repositories:
    - name: my-templates
      uri: ssh://[email protected]/...pipeline-templates.git
      credentials:
        id: buildmaster

pipeline:
  steps:
    - include: java-build.yml@my-templates

# Repository: ssh://[email protected]/...pipeline-templates.git
# File: java-build.yml

resources:
  repositories:
    - name: templates
      uri: ...
      credentials:
        id: ...

steps:
  - include: mvn-package.yml # note this will be in the same repo as this template
  - include: slack.yml@templates # remote repo
  - name: a-directly-defined-step
    image: ...
    ...

# Repository: ssh://[email protected]/...pipeline-templates.git
# File: mvn-package.yml
steps:
  - name: mvn-package
    image: mvn:3
    commands:
      - mvn clean package

Minimal pipeline.yml file

As mentioned, we have explicitly separated out importing configuration information and importing steps.

This makes the minimal possible pipeline.yml that only re-uses existing definitions something like:

# File: pipeline.yml
pipeline:
  include:
    - templates/shared-env.yml
  steps:
    - include: templates/shared-steps.yml