Skip to content

Convert the test config map to a config list #1900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
supakeen merged 6 commits into from
Oct 10, 2025
Merged

Convert the test config map to a config list #1900

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b898ae5
test: convert the config-map to a config-list
achilleas-k Sep 24, 2025
5ba66ed
cmd/gen-manifests: read config list instead of map
achilleas-k Sep 24, 2025
03f91f8
test/scripts: adapt scripts and their tests
achilleas-k Sep 24, 2025
103bfe3
docs,test: update test and developer docs
achilleas-k Sep 24, 2025
7b7aaa3
test: rm test/config-map.json
achilleas-k Sep 24, 2025
4199368
test: drop usage of config variants for distros
achilleas-k Sep 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .github/workflows/gitlab-helper.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,9 @@ jobs:
exit "0"
fi

- name: Check that the config-map is valid
- name: Check that the config-list is valid
run: |
./test/scripts/validate-config-map
./test/scripts/validate-config-list


gitlab-ci-helper:
Expand Down
39 changes: 24 additions & 15 deletions cmd/gen-manifests/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// Standalone executable for generating all test manifests in parallel.
// Collects list of image types from the distro list. Must be run from the
// root of the repository and reads test/data/repositories for repositories
// test/config-map.json to match image types with configuration files.
// test/config-list.json to match image types with configuration files.
// Collects errors and failures and prints them after all jobs are finished.
package main

Expand Down Expand Up @@ -142,24 +142,31 @@ func (bc BuildConfigs) Get(distro, arch, imageType string) []*buildconfig.BuildC
return configs
}

func loadConfigMap(configPath string, opts *buildconfig.Options) *BuildConfigs {
func loadConfigList(configPath string, opts *buildconfig.Options) *BuildConfigs {

type configFilters struct {
ImageTypes []string `json:"image-types"`
Distros []string `json:"distros"`
SkipDistros []skipDistro `json:"skip-distros"`
Arches []string `json:"arches"`
}
type configMap map[string]configFilters

type configItem struct {
Path string `json:"path"`
Filters configFilters `json:"filters"`
}

type configList []configItem

fp, err := os.Open(configPath)
if err != nil {
panic(fmt.Sprintf("failed to open config map %q: %s", configPath, err.Error()))
panic(fmt.Sprintf("failed to open config list %q: %s", configPath, err.Error()))
}
defer fp.Close()

var cfgMap configMap
if err := json.NewDecoder(fp).Decode(&cfgMap); err != nil {
panic(fmt.Sprintf("failed to unmarshal config map %q: %s", configPath, err.Error()))
var cfgList configList
if err := json.NewDecoder(fp).Decode(&cfgList); err != nil {
panic(fmt.Sprintf("failed to unmarshal config list %q: %s", configPath, err.Error()))
}

emptyFallback := func(list []string) []string {
Expand All @@ -173,8 +180,10 @@ func loadConfigMap(configPath string, opts *buildconfig.Options) *BuildConfigs {

// load each config from its path
cm := newBuildConfigs()
for path, filters := range cfgMap {
// config paths can be relative to the location of the config map
for _, cfgItem := range cfgList {
// config paths can be relative to the location of the config list
path := cfgItem.Path
filters := cfgItem.Filters
if !filepath.IsAbs(path) {
cfgDir := filepath.Dir(configPath)
path = filepath.Join(cfgDir, path)
Expand All @@ -199,8 +208,8 @@ func loadConfigMap(configPath string, opts *buildconfig.Options) *BuildConfigs {
}

// loadImgConfig loads a single image config from a file and returns a
// a BuildConfigs map with the config mapped to all distros, arches, and
// image types.
// BuildConfigs map with the config mapped to all distros, arches, and image
// types.
func loadImgConfig(configPath string, opts *buildconfig.Options) *BuildConfigs {
cm := newBuildConfigs()
config, err := buildconfig.New(configPath, opts)
Expand Down Expand Up @@ -408,8 +417,8 @@ func main() {
flag.IntVar(&nWorkers, "workers", 16, "number of workers to run concurrently")
flag.StringVar(&cacheRoot, "cache", "/tmp/rpmmd", "rpm metadata cache directory")
flag.BoolVar(&metadata, "metadata", true, "store metadata in the file")
flag.StringVar(&configPath, "config", "", "image config file to use for all images (overrides -config-map)")
flag.StringVar(&configMapPath, "config-map", "test/config-map.json", "configuration file mapping image types to configs")
flag.StringVar(&configPath, "config", "", "image config file to use for all images (overrides -config-list)")
flag.StringVar(&configMapPath, "config-list", "test/config-list.json", "configuration file mapping image types to configs")
flag.BoolVar(&skipNoconfig, "skip-noconfig", false, "skip distro-arch-image configurations that have no config (otherwise fail)")
flag.BoolVar(&skipNorepos, "skip-norepos", false, "skip distro-arch-image configurations that have no repositories (otherwise fail)")
flag.BoolVar(&buildconfigAllowUnknown, "buildconfig-allow-unknown", false, "allow unknown keys in buildconfig")
Expand Down Expand Up @@ -447,10 +456,10 @@ func main() {
var configs *BuildConfigs
opts := &buildconfig.Options{AllowUnknownFields: buildconfigAllowUnknown}
if configPath != "" {
fmt.Println("'-config' was provided, thus ignoring '-config-map' option")
fmt.Println("'-config' was provided, thus ignoring '-config-list' option")
configs = loadImgConfig(configPath, opts)
} else {
configs = loadConfigMap(configMapPath, opts)
configs = loadConfigList(configMapPath, opts)
}

if err := os.MkdirAll(outputDir, 0770); err != nil {
Expand Down
2 changes: 1 addition & 1 deletion docs/developer/cmds.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ a new image type, might be:
2. Make changes in an existing image definition or add a new image type.
3. Add appropriate configuration changes:
- If a new image type is added, add it to the [config
map](test/config-map.json) under an appropriate configuration file or
list](test/config-list.json) under an appropriate configuration file or
write a new one.
- If an existing image type is being modified, and the change depends on an
image customization, make sure the modification is covered by an existing
Expand Down
12 changes: 6 additions & 6 deletions test/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ sudo ./bin/build --output ./buildtest --rpmmd /tmp/rpmmd --distro fedora-41 --ty
```
will build a Fedora 38 qcow2 image using the configuration specified in the file `embed-containers.json`

- [./cmd/gen-manifests](../cmd/gen-manifests) generates manifests based on the configs specified in [./test/config-map.json](./config-map.json). The config map maps configuration files to image types, distributions, and architectures. An empty list means it applies to all values. Globs are supported.
- [./cmd/gen-manifests](../cmd/gen-manifests) generates manifests based on the configs specified in [./test/config-list.json](./config-list.json). The config list maps configuration files to image types, distributions, and architectures. An empty list means it applies to all values. Globs are supported.

The config map is also used in CI to dynamically generate test builds using the [./test/scripts/generate-build-config](./scripts/generate-build-config) and [./test/scripts/generate-ostree-build-config](./scripts/generate-ostree-build-config) scripts.
The config list is also used in CI to dynamically generate test builds using the [./test/scripts/generate-build-config](./scripts/generate-build-config) and [./test/scripts/generate-ostree-build-config](./scripts/generate-ostree-build-config) scripts.

- [./test/data/repositories/](./data/repositories/) contains repository configurations for manifest generation ([./cmd/gen-manifests](../cmd/gen-manifests)) and image building ([./cmd/build](../cmd/build)).

Expand Down Expand Up @@ -123,10 +123,10 @@ The config generator:
- For example [iot-ostree-pull-empty](./configs/iot-ostree-pull-empty.json)) will cause a manifest to be generated for `iot-container` with the `empty` config for all distros.
- Determines the container name and tag from the build name and manifest ID and pulls each container from the registry.
- Runs each container with a unique port mapped to the internal web service port.
- For each build config that defines a dependency and for each image that config applies to, creates build configs and a config map that defines the URL, port, and ref for the ostree commit source.
- For example, the config [iot-ostree-pull-empty](./configs/iot-ostree-pull-empty.json)) is mapped in the [config-map](config-map.json) to the image types `iot-ami`, `iot-installer`, `iot-raw-image`, and `iot-vsphere`. This will create four configs for each distro, one for each image type, that will all have ostree options to pull an ostree commit from an `iot-container` of the same distro.
- Generates all the manifests defined in the config map that was generated in the previous step.
- Note that this manifest generation step uses the `-skip-noconfig` flag, which means that any image type not defined in the map is skipped.
- For each build config that defines a dependency and for each image that config applies to, creates build configs and a config list that defines the URL, port, and ref for the ostree commit source.
- For example, the config [iot-ostree-pull-empty](./configs/iot-ostree-pull-empty.json)) is mapped in the [config-list](config-list.json) to the image types `iot-ami`, `iot-installer`, `iot-raw-image`, and `iot-vsphere`. This will create four configs for each distro, one for each image type, that will all have ostree options to pull an ostree commit from an `iot-container` of the same distro.
- Generates all the manifests defined in the config list that was generated in the previous step.
- Note that this manifest generation step uses the `-skip-noconfig` flag, which means that any image type not defined in the list is skipped.
- Downloads the test build cache.
- Filters out any manifest with an ID that exists in the build cache.
- For each remaining manifest, creates a job which builds, boots (if applicable), and uploads the results to the build cache for a given distro, image type, and config file.
Expand Down
Loading
Loading