TEP-0118: Implement Matrix FanOut() logic for Matrix Include Parameters

[TEP-0090: Matrix] introduced `Matrix` to the `PipelineTask` specification such that the `PipelineTask` executes a list of `TaskRuns` or `Runs` in parallel with the specified list of inputs for a `Parameter` or with different combinations of the inputs for a set of `Parameters`.

To build on this, Tep-0018 introduced Matrix.Include, which allows passing in a specific combinations of `Parameters` into the `Matrix`.

**This commit adds implementation logic and testing for fanning out the Matrix Include Parameters to allow users to generate explicit combinations and support adding a specific combination of input values for Matrix Parameters**

docs/matrix.md

@@ -12,7 +12,9 @@ weight: 406
   - [Concurrency Control](#concurrency-control)
   - [Parameters](#parameters)
     - [Specifying both `params` and `matrix` in a `PipelineTask`](#specifying-both-params-and-matrix-in-a-pipelinetask)
-    - [Include](#Include)
+  - [Include](#Include)
+    - [Include specific combinations in the Matrix](#include-specific-combinations-in-the-matrix)
+    - [Define explicit combinations in the Matrix](#define-explicit-combinations-in-the-matrix)
   - [Context Variables](#context-variables)
   - [Results](#results)
     - [Specifying Results in a Matrix](#specifying-results-in-a-matrix)
@@ -158,6 +160,8 @@ spec:
 > It is still in a very early stage of development and is not yet fully functional.
 The `Include` section in the `Matrix` field exists, but is not yet functional.
 
+The `Include` section in the `Matrix` can be specified with or without `Params` section adding a specific combination of input values for `Matrix Parameters` or defining explicit combinations in the `Matrix` without `Params`.
+
 The `Matrix.Include` will take `Parameters` of type `"string"` only.
 
 ```yaml
@@ -172,6 +176,98 @@ The `Matrix.Include` will take `Parameters` of type `"string"` only.
   ...
 ```
 
+#### Include specific combinations in the Matrix
+
+In the example below, `Pipeline` will need to define a `Matrix` that will execute golang-test in three different linux flavors.
+
+While running on amd64 and ppc64le, golang-test should execute `go test github.com/tektoncd/pipeline/pkg/... -race -cover -v`.
+While running on linux/s390x, golang-test should execute `go test github.com/tektoncd/pipeline/pkg/... -cover -v`.
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline
+metadata:
+  name: pipeline-to-build-and-test-go-project
+spec:
+  workspaces:
+    - name: shared-workspace
+  tasks:
+    - ...
+    - name: golang-test
+      taskRef:
+        name: golang-test
+      workspaces:
+        - name: source
+          workspace: shared-workspace
+      params:
+        - name: package
+          value: "github.com/tektoncd/pipeline"
+        - name: packages
+          value: "./pkg/..."
+      matrix:
+        params:
+          - name: GOARCH
+            value:
+              - "linux/amd64"
+              - "linux/ppc64le"
+              - "linux/s390x"
+        include:
+          - name: s390x-no-race
+            params:
+              - name: GOARCH
+                value: "linux/s390x"
+              - name: flags
+                value: "-cover -v"
+    - ...
+```
+
+The above `Matrix` specification will create three `TaskRuns`, one for each architecture in GOARCH ("linux/amd64", "linux/ppc64le", and "linux/s390x") and with default flags. Now, when a `TaskRun` is created for the GOARCH value of linux/s390x, an additional `Parameter` called flags with the value of `-cover -v` will be included in the taskRun.
+
+The `Matrix.Include` section can list params which does not exist in the matrix.params section. In this example specification, flags was not listed in `Matrix.Params`. At the same time, `Matrix.Include` section can list params which does exist in the `Matrix.Params` section.
+
+### Define explicit combinations in the Matrix
+
+In the example below, the user needs to specify explicit mapping between IMAGE and DOCKERFILE, such as:
+
+```yaml
+apiVersion: tekton.dev/v1beta1
+kind: Pipeline
+metadata:
+  name: pipeline-to-build-images-from-a-single-repo
+spec:
+  workspaces:
+    - name: shared-workspace
+  tasks:
+    - ...
+    - name: kaniko-build
+      taskRef:
+        name: kaniko
+      workspaces:
+        - name: source
+          workspace: shared-workspace
+      matrix:
+        include:
+          - name: build-1
+            params:
+              - name: IMAGE
+                value: "image-1"
+              - name: DOCKERFILE
+                value: "path/to/Dockerfile1"
+          - name: build-2
+            params:
+              - name: IMAGE
+                value: "image-2"
+              - name: DOCKERFILE
+                value: "path/to/Dockerfile2"
+          - name: build-3
+            params:
+              - name: IMAGE
+                value: "image-3"
+              - name: DOCKERFILE
+                value: "path/to/Dockerfile3"
+```
+This configuration allows users to take advantage of `Matrix` to fan out without having an auto-populated `Matrix`. `Matrix` with include section without `Params` section creates the number of `TaskRuns` specified in the `Include` section with the specified `Parameters`.
+
 ### Context Variables
 
 Similarly to the `Parameters` in the `Params` field, the `Parameters` in the `Matrix` field will accept

go.mod

@@ -60,6 +60,7 @@ require (
 	go.opentelemetry.io/otel/exporters/jaeger v1.14.0
 	go.opentelemetry.io/otel/sdk v1.14.0
 	go.opentelemetry.io/otel/trace v1.14.0
+	golang.org/x/exp v0.0.0-20230307190834-24139beb5833
 	k8s.io/utils v0.0.0-20221108210102-8e77b1f39fe2
 )
 

pkg/apis/pipeline/v1/matrix_types.go

@@ -56,9 +56,27 @@ type MatrixInclude struct {
 // Combination is a map, mainly defined to hold a single combination from a Matrix with key as param.Name and value as param.Value
 type Combination map[string]string
 
-// Combinations is a list of combinations
+// Combinations is a Combination list
 type Combinations []Combination
 
+// FanOut generates Combinations, which is a list of mapped Combination by fanning out matrix
+// Parameters including Matrix Include Parameters
+func (m *Matrix) FanOut() Combinations {
+	var combinations Combinations
+	// If only Matrix Include Parameters exists, generate and return explicit Combinations
+	if m.hasInclude() && !m.hasParams() {
+		return m.fanOutExplictCombinations()
+	}
+	// Fan out Matrix Parameters
+	for _, parameter := range m.Params {
+		combinations = combinations.fanOut(parameter)
+	}
+	// Replace initial Combinations generated with Matrix Include Parameters
+	mappedMatrixIncludeParamsSlice := m.extractIncludeParams()
+	combinations = combinations.replaceCombinations(mappedMatrixIncludeParamsSlice)
+	return combinations
+}
+
 // ToParams transforms Combinations from a slice of map[string]string to a slice of Params
 // such that, these combinations can be directly consumed in creating taskRun/run object
 func (cs Combinations) ToParams() []Params {
@@ -86,7 +104,22 @@ func (cs Combinations) fanOut(param Param) Combinations {
 	return cs.distribute(param)
 }
 
-// distribute generates a new combination of Parameters by adding a new Parameter to an existing list of Combinations.
+// fanOutExplictCombinations handles the use case when there are only Matrix Include Parameters
+// and no Matrix Parameters to generate explicit combinations
+func (m *Matrix) fanOutExplictCombinations() Combinations {
+	var combinations Combinations
+	for i := 0; i < len(m.Include); i++ {
+		includeParams := m.Include[i].Params
+		newCombination := make(Combination)
+		for _, param := range includeParams {
+			newCombination[param.Name] = param.Value.StringVal
+		}
+		combinations = append(combinations, newCombination)
+	}
+	return combinations
+}
+
+// distribute generates a new Combination of Parameters by adding a new Parameter to an existing list of Combinations.
 func (cs Combinations) distribute(param Param) Combinations {
 	var expandedCombinations Combinations
 	for _, value := range param.Value.ArrayVal {
@@ -101,7 +134,94 @@ func (cs Combinations) distribute(param Param) Combinations {
 	return expandedCombinations
 }
 
-// initializeCombinations generates a new combination based on the first Parameter in the Matrix.
+// replaceCombinations checks if any of the include Parameters are missing from the initial fanned out Combinations.
+// It will add any missing combinations to either the existing combination or generate a new combination if the
+// parameter value does not exist.
+func (cs Combinations) replaceCombinations(mappedMatrixIncludeParamsSlice []map[string]string) Combinations {
+	for _, matrixIncludeParamMap := range mappedMatrixIncludeParamsSlice {
+		hasMissingParamName := cs.hasMissingParamName(matrixIncludeParamMap)
+		hasMissingParamVal := cs.hasMissingParamVal(matrixIncludeParamMap)
+		for _, c := range cs {
+			hasAtLeastOneMatch := c.hasAtLeastOneMatch(matrixIncludeParamMap)
+			containsSubset := c.containsSubset(matrixIncludeParamMap)
+			if hasAtLeastOneMatch && containsSubset || hasMissingParamName {
+				// add the Matrix Include Parameters to the existing Combination
+				maps.Copy(c, matrixIncludeParamMap)
+			}
+		}
+
+		if hasMissingParamVal && !hasMissingParamName {
+			// generate a new Combination for the missing parameter value
+			if len(matrixIncludeParamMap) == 1 {
+				for name, val := range matrixIncludeParamMap {
+					cs = append(cs, map[string]string{name: val})
+				}
+			}
+		}
+	}
+	return cs
+}
+
+// hasAtLeastOneMatch returns true if at least one Matrix Include Parameter
+//
+//	name and value exist in a given Combination
+func (c Combination) hasAtLeastOneMatch(paramNamesMap map[string]string) bool {
+	for name, val := range c {
+		if paramVal, exist := paramNamesMap[name]; exist {
+			if val == paramVal {
+				return true
+			}
+		}
+	}
+	return false
+}
+
+// containsSubset returns true if all parameter names and values that exist in Matrix
+// Include Parameters also exist in a given Combination
+func (c Combination) containsSubset(matrixIncludeParamMap map[string]string) bool {
+	matchedParamsCount := 0
+	missingParamsCount := 0
+	for name, val := range matrixIncludeParamMap {
+		if combinationVal, ok := c[name]; ok {
+			if combinationVal == val {
+				matchedParamsCount++
+			}
+		} else {
+			missingParamsCount++
+		}
+	}
+	return matchedParamsCount+missingParamsCount == len(matrixIncludeParamMap)
+}
+
+// hasMissingParamName returns true if a Matrix Include Parameter name is missing from
+// all Combinations
+func (cs Combinations) hasMissingParamName(matrixIncludeParamMap map[string]string) bool {
+	for _, c := range cs {
+		for name := range matrixIncludeParamMap {
+			if _, exist := c[name]; exist {
+				return false
+			}
+		}
+	}
+	return true
+}
+
+// hasMissingParamVal returns true if a Matrix Include Parameter value is missing from
+// all combinations
+func (cs Combinations) hasMissingParamVal(matrixIncludeParamMap map[string]string) bool {
+	for _, c := range cs {
+		for name, val := range matrixIncludeParamMap {
+			if cVal, exist := c[name]; exist {
+				if val == cVal {
+					return false
+				}
+			}
+		}
+	}
+	return true
+}
+
+// initializeCombinations generates a new Combination based on the first Parameter in the Matrix.
 func initializeCombinations(param Param) Combinations {
 	var combinations Combinations
 	for _, value := range param.Value.ArrayVal {
@@ -110,7 +230,7 @@ func initializeCombinations(param Param) Combinations {
 	return combinations
 }
 
-// sortCombination sorts the given Combination based on the param names to produce a deterministic ordering
+// sortCombination sorts the given Combination based on the Parameter names to produce a deterministic ordering
 func (c Combination) sortCombination() ([]string, Combination) {
 	sortedCombination := make(Combination, len(c))
 	order := make([]string, 0, len(c))
@@ -126,28 +246,19 @@ func (c Combination) sortCombination() ([]string, Combination) {
 	return order, sortedCombination
 }
 
-// FanOut produces combinations of Parameters of type String from a slice of Parameters of type Array.
-func (m *Matrix) FanOut() Combinations {
-	var combinations Combinations
-	for _, parameter := range m.Params {
-		combinations = combinations.fanOut(parameter)
-	}
-	return combinations
-}
-
-// CountCombinations returns the count of combinations of Parameters generated from the Matrix in PipelineTask.
+// CountCombinations returns the count of Combinations of Parameters generated from the Matrix in PipelineTask.
 func (m *Matrix) CountCombinations() int {
-	// Iterate over matrix.params and compute count of all generated combinations
+	// Iterate over Matrix Parameters and compute count of all generated Combinations
 	count := m.countGeneratedCombinationsFromParams()
 
-	// Add any additional combinations generated from matrix include params
+	// Add any additional Combinations generated from Matrix Include Parameters
 	count += m.countNewCombinationsFromInclude()
 
 	return count
 }
 
-// countGeneratedCombinationsFromParams returns the count of combinations of Parameters generated from the matrix
-// parameters
+// countGeneratedCombinationsFromParams returns the count of Combinations of Parameters generated from the Matrix
+// Parameters
 func (m *Matrix) countGeneratedCombinationsFromParams() int {
 	if !m.hasParams() {
 		return 0
@@ -159,8 +270,8 @@ func (m *Matrix) countGeneratedCombinationsFromParams() int {
 	return count
 }
 
-// countNewCombinationsFromInclude returns the count of combinations of Parameters generated from the matrix
-// include parameters
+// countNewCombinationsFromInclude returns the count of Combinations of Parameters generated from the Matrix
+// Include Parameters
 func (m *Matrix) countNewCombinationsFromInclude() int {
 	if !m.hasInclude() {
 		return 0
@@ -173,7 +284,7 @@ func (m *Matrix) countNewCombinationsFromInclude() int {
 	for _, include := range m.Include {
 		for _, param := range include.Params {
 			if val, exist := matrixParamMap[param.Name]; exist {
-				// If the matrix include param values does not exist, a new combination will be generated
+				// If the Matrix Include param values does not exist, a new Combination will be generated
 				if !slices.Contains(val, param.Value.StringVal) {
 					count++
 				} else {
@@ -193,6 +304,16 @@ func (m *Matrix) hasParams() bool {
 	return m != nil && m.Params != nil && len(m.Params) > 0
 }
 
+// extractIncludeParams returns mapped include params with the key name: param.Name and
+// val: param.value.stringVal
+func (m *Matrix) extractIncludeParams() []map[string]string {
+	var includeParamsMapped []map[string]string
+	for _, include := range m.Include {
+		includeParamsMapped = append(includeParamsMapped, include.Params.extractParamMapStrVals())
+	}
+	return includeParamsMapped
+}
+
 func (m *Matrix) validateCombinationsCount(ctx context.Context) (errs *apis.FieldError) {
 	matrixCombinationsCount := m.CountCombinations()
 	maxMatrixCombinationsCount := config.FromContextOrDefaults(ctx).Defaults.DefaultMaxMatrixCombinationsCount
@@ -202,8 +323,7 @@ func (m *Matrix) validateCombinationsCount(ctx context.Context) (errs *apis.Fiel
 	return errs
 }
 
-// validateParams validates the type of parameter
-// for Matrix.Params and Matrix.Include.Params
+// validateParams validates the type of Parameter for Matrix.Params and Matrix.Include.Params
 // Matrix.Params must be of type array. Matrix.Include.Params must be of type string.
 // validateParams also validates Matrix.Params for a unique list of params
 // and a unique list of params in each Matrix.Include.Params specification
@@ -231,7 +351,7 @@ func (m *Matrix) validateParams() (errs *apis.FieldError) {
 	return errs
 }
 
-// validatePipelineParametersVariablesInMatrixParameters validates all pipeline paramater variables including Matrix.Params and Matrix.Include.Params
+// validatePipelineParametersVariablesInMatrixParameters validates all pipeline parameter variables including Matrix.Params and Matrix.Include.Params
 // that may contain the reference(s) to other params to make sure those references are used appropriately.
 func (m *Matrix) validatePipelineParametersVariablesInMatrixParameters(prefix string, paramNames sets.String, arrayParamNames sets.String, objectParamNameKeys map[string][]string) (errs *apis.FieldError) {
 	if m.hasInclude() {