Skip to content

Commit

Permalink
TEP-0118: Implement Matrix Fanning Out logic to support Matrix Includ…
Browse files Browse the repository at this point in the history
…e Parameters in a Task Run

This commit adds implementation logic and testing for fanning out the Matrix Include Parameters in a Task Run to allow users to generate explicit combinations and support adding a specific combination of input values for Matrix Parameters.
  • Loading branch information
EmmaMunley committed Mar 27, 2023
1 parent 8638170 commit f6e1499
Show file tree
Hide file tree
Showing 13 changed files with 2,214 additions and 51 deletions.
211 changes: 200 additions & 11 deletions docs/matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,10 @@ weight: 406
- [Results from fanned out PipelineTasks](#results-from-fanned-out-pipelinetasks)
- [Retries](#retries)
- [Examples](#examples)
- [`PipelineTasks` with `Tasks`](#pipelinetasks-with-tasks)
- [`PipelineTasks` with `Custom Tasks`](#pipelinetasks-with-custom-tasks)
- [`Matrix` Combinations with `Matrix.Params` only](#-matrix--combinations-with--matrixparams--only)
- [`Matrix` Combinations with `Matrix.Params` and `Matrix.Include`](#-matrix--combinations-with--matrixparams--and--matrixinclude-)
- [`PipelineTasks` with `Tasks`](#-pipelinetasks--with--tasks-)
- [`PipelineTasks` with `Custom Tasks`](#-pipelinetasks--with--custom-tasks-)

## Overview

Expand Down Expand Up @@ -62,25 +64,95 @@ The `Matrix.Params` is used to generate combinations to fan out a `PipelineTask`
...
```

### Explicit Combinations
Combinations generated

> :warning: This feature is in a preview mode.
> It is still in a very early stage of development and is not yet fully functional.
```json!
{ "platform": "linux", "browser": "safari" }
{ "platform": "linux", "browser": "chrome"}
{ "platform": "mac", "browser": "safari" }
{ "platform": "mac", "browser": "chrome"}
```
[See another example](#-matrix--combinations-with--matrixparams--only)

The `Matrix.Include` is used to add explicit combinations to fan out a `PipelineTask`, but is not yet functional.
### Explicit Combinations

The `Matrix.Include` is used to add explicit combinations to fan out a `PipelineTask`.

```yaml
matrix:
params:
- name: platform
value:
- linux
- mac
- name: browser
value:
- safari
- chrome
include:
- name: s390x-no-race
- name: linux-url
params:
- name: GOARCH
value: "linux/s390x"
- name: flags
value: "-cover -v"
- name: platform
value: linux
- name: url
value: some-url
- name: non-existent-browser
params:
- name: browser
value: "i-do-not-exist"
...
```

The first `Matrix.Include` clause adds `"url": "some-url"` only to the original `matrix` combinations that include `"platform": "linux"` and the second `Matrix.Include` clause cannot be added to any original `matrix` combination without overwriting any `params` of the original combinations, so it is added as an additional `matrix` combination:

Combinations generated
```json!
{ "platform": "linux", "browser": "safari", "url": "some-url" }
{ "platform": "linux", "browser": "chrome", "url": "some-url"}
{ "platform": "mac", "browser": "safari" }
{ "platform": "mac", "browser": "chrome"}
{ "browser": "i-do-not-exist"}
```

[See another example](#-matrix--combinations-with--matrixparams--and--matrixinclude-)

The `Matrix.Include` can also be used without `Matrix.Params` to generate explicit combinations to fan out a `PipelineTask`.

```yaml
matrix:
include:
- name: build-1
params:
- name: IMAGE
value: "image-1"
- name: DOCKERFILE
value: "path/to/Dockerfile1"
- name: build-2
params:
- name: IMAGE
value: "image-2"
- name: DOCKERFILE
value: "path/to/Dockerfile2"
- name: build-3
params:
- name: IMAGE
value: "image-3"
- name: DOCKERFILE
value: "path/to/Dockerfile3"
...
```

This configuration allows users to take advantage of `Matrix` to fan out without having an auto-populated `Matrix`. `Matrix` with include section without `Params` section creates the number of `TaskRuns` specified in the `Include` section with the specified `Parameters`.


Combinations generated

```json!
{ "IMAGE": "image-1", "DOCKERFILE": "path/to/Dockerfile1" }
{ "IMAGE": "image-2", "DOCKERFILE": "path/to/Dockerfile2"}
{ "IMAGE": "image-3", "DOCKERFILE": "path/to/Dockerfile3}
```

## Concurrency Control

The default maximum count of `TaskRuns` or `Runs` from a given `Matrix` is **256**. To customize the maximum count of
Expand Down Expand Up @@ -321,6 +393,123 @@ spec:

## Examples

### `Matrix` Combinations with `Matrix.Params` only

```yaml
matrix:
params:
- name: GOARCH
value:
- "linux/amd64"
- "linux/ppc64le"
- "linux/s390x"
- name: version
value:
- "go1.17"
- "go1.18.1"
```

This `matrix` specification will result in six `taskRuns` with the following `matrix` combinations:

```json!
{ "GOARCH": "linux/amd64", "version": "go1.17" }
{ "GOARCH": "linux/amd64", "version": "go1.18.1" }
{ "GOARCH": "linux/ppc64le", "version": "go1.17" }
{ "GOARCH": "linux/ppc64le", "version": "go1.18.1" }
{ "GOARCH": "linux/s390x", "version": "go1.17" }
{ "GOARCH": "linux/s390x", "version": "go1.18.1" }
```

Let's expand this use case to showcase a little more complex combinations in the next example.

### `Matrix` Combinations with `Matrix.Params` and `Matrix.Include`

Now, let's introduce `include` with a couple of `Parameters`: `"package"`, `"flags"` and `"context"`:

```yaml
matrix:
params:
- name: GOARCH
value:
- "linux/amd64"
- "linux/ppc64le"
- "linux/s390x"
- name: version
value:
- "go1.17"
- "go1.18.1"
include:
- name: common-package
params:
- name: package
value: "path/to/common/package/"
- name: s390x-no-race
params:
- name: GOARCH
value: "linux/s390x"
- name: flags
value: "-cover -v"
- name: go117-context
params:
- name: version
value: "go1.17"
- name: context
value: "path/to/go117/context"
- name: non-existent-arch
params:
- name: GOARCH
value: "I-do-not-exist"
```

The first `include` clause is added to all the original `matrix` combintations without overwriting any `parameters` of
the original combinations:

```json!
{ "GOARCH": "linux/amd64", "version": "go1.17", **"package": "path/to/common/package/"** }
{ "GOARCH": "linux/amd64", "version": "go1.18.1", **"package": "path/to/common/package/"** }
{ "GOARCH": "linux/ppc64le", "version": "go1.17", **"package": "path/to/common/package/"** }
{ "GOARCH": "linux/ppc64le", "version": "go1.18.1", **"package": "path/to/common/package/"** }
{ "GOARCH": "linux/s390x", "version": "go1.17", **"package": "path/to/common/package/"** }
{ "GOARCH": "linux/s390x", "version": "go1.18.1", **"package": "path/to/common/package/"** }
```

The second `include` clause adds `"flags": "-cover -v"` only to the original `matrix` combinations that include
`"GOARCH": "linux/s390x"`:
```json!
{ "GOARCH": "linux/s390x", "version": "go1.17", "package": "path/to/common/package/", **"flags": "-cover -v"** }
{ "GOARCH": "linux/s390x", "version": "go1.18.1", "package": "path/to/common/package/", **"flags": "-cover -v"** }
```

The third `include` clause adds `"context": "path/to/go117/context"` only to the original `matrix` combinations
that include `"version": "go1.17"`:

```json!
{ "GOARCH": "linux/amd64", "version": "go1.17", "package": "path/to/common/package/", **"context": "path/to/go117/context"** }
{ "GOARCH": "linux/ppc64le", "version": "go1.17", "package": "path/to/common/package/", **"context": "path/to/go117/context"** }
{ "GOARCH": "linux/s390x", "version": "go1.17", "package": "path/to/common/package/", "flags": "-cover -v", **"context": "path/to/go117/context"** }
```

The fourth `include` clause cannot be added to any original `matrix` combination without overwriting any `params` of the
original combinations, so it is added as an additional `matrix` combination:

```json!
* { **"GOARCH": "I-do-not-exist"** }
```

The above specification will result in seven `taskRuns` with the following matrix combinations:

```json!
{ "GOARCH": "linux/amd64", "version": "go1.17", "package": "path/to/common/package/", "context": "path/to/go117/context" }
{ "GOARCH": "linux/amd64", "version": "go1.18.1", "package": "path/to/common/package/" }
{ "GOARCH": "linux/ppc64le", "version": "go1.17", "package": "path/to/common/package/", "context": "path/to/go117/context" }
{ "GOARCH": "linux/ppc64le", "version": "go1.18.1", "package": "path/to/common/package/" }
{ "GOARCH": "linux/s390x", "version": "go1.17", "package": "path/to/common/package/", "flags": "-cover -v", "context": "path/to/go117/context" }
{ "GOARCH": "linux/s390x", "version": "go1.18.1", "package": "path/to/common/package/", "flags": "-cover -v" }
{ "GOARCH": "I-do-not-exist" }
```

### `PipelineTasks` with `Tasks`

When a `PipelineTask` has a `Task` and a `Matrix`, the `Task` will be executed in parallel `TaskRuns` with
Expand Down
12 changes: 4 additions & 8 deletions docs/pipeline-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1481,8 +1481,7 @@ TaskSpec
<h3 id="tekton.dev/v1.IncludeParams">IncludeParams
</h3>
<div>
<p>IncludeParams allows passing in a specific combinations of Parameters into the Matrix.
Note this struct is in preview mode and not yet supported</p>
<p>IncludeParams allows passing in a specific combinations of Parameters into the Matrix.</p>
</div>
<table>
<thead>
Expand Down Expand Up @@ -1562,8 +1561,7 @@ IncludeParamsList
</td>
<td>
<em>(Optional)</em>
<p>Include is a list of IncludeParams which allows passing in specific combinations of Parameters into the Matrix.
Note that Include is in preview mode and not yet supported.</p>
<p>Include is a list of IncludeParams which allows passing in specific combinations of Parameters into the Matrix.</p>
</td>
</tr>
</tbody>
Expand Down Expand Up @@ -8614,8 +8612,7 @@ TaskSpec
<h3 id="tekton.dev/v1beta1.IncludeParams">IncludeParams
</h3>
<div>
<p>IncludeParams allows passing in a specific combinations of Parameters into the Matrix.
Note this struct is in preview mode and not yet supported</p>
<p>IncludeParams allows passing in a specific combinations of Parameters into the Matrix.</p>
</div>
<table>
<thead>
Expand Down Expand Up @@ -8695,8 +8692,7 @@ IncludeParamsList
</td>
<td>
<em>(Optional)</em>
<p>Include is a list of IncludeParams which allows passing in specific combinations of Parameters into the Matrix.
Note that Include is in preview mode and not yet supported.</p>
<p>Include is a list of IncludeParams which allows passing in specific combinations of Parameters into the Matrix.</p>
</td>
</tr>
</tbody>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: mytask
annotations:
description: |
A task that does something cool with platforms and browsers
spec:
params:
- name: IMAGE
- name: DOCKERFILE
steps:
- name: echo
image: alpine
script: |
echo "$(params.IMAGE) and $(params.DOCKERFILE)"
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: explicit-combos
spec:
serviceAccountName: 'default'
pipelineSpec:
tasks:
- name: matrix-include
matrix:
include:
- name: build-1
params:
- name: IMAGE
value: image-1
- name: DOCKERFILE
value: path/to/Dockerfile1
- name: build-2
params:
- name: IMAGE
value: image-2
- name: DOCKERFILE
value: path/to/Dockerfile2
- name: build-3
params:
- name: IMAGE
value: image-3
- name: DOCKERFILE
value: path/to/Dockerfile3
taskRef:
name: mytask
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: mytask
annotations:
description: |
A task that does something cool with GOARCH and version
spec:
params:
- name: GOARCH
default: ''
- name: version
default: ''
- name: flags
default: ''
- name: context
default: ''
- name: package
default: ''
steps:
- name: echo
image: alpine
script: |
echo $(params.GOARCH) and $(params.version) flags? $(params.flags) context? $(params.context) package? $(params.package)
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: matrixed-include-pr
spec:
serviceAccountName: default
pipelineSpec:
tasks:
- name: matrix-include
matrix:
params:
- name: GOARCH
value:
- linux/amd64
- linux/ppc64le
- linux/s390x
- name: version
value:
- go1.17
- go1.18.1
include:
- name: common-package
params:
- name: package
value: path/to/common/package/
- name: s390x-no-race
params:
- name: GOARCH
value: linux/s390x
- name: flags
value: '-cover -v'
- name: go117-context
params:
- name: version
value: go1.17
- name: context
value: path/to/go117/context
- name: non-existent-arch
params:
- name: GOARCH
value: I-do-not-exist
taskRef:
name: mytask
Loading

0 comments on commit f6e1499

Please sign in to comment.