Add GO_TEST_EXEC variable to use gotestsum on CI#2107
Merged
Conversation
0334e91 to
63a6f0b
Compare
Introduce a Makefile variable GO_TEST_EXEC (defaulting to "go test") that is set to "gotestsum --" on CI via the GitHub Actions workflow env. Also install gotestsum via mise in the test environment. Closes #2106 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous approach set GO_TEST_EXEC in the workflow env and relied on Make's ?= to pick it up. This worked for the sharded path (make test) but not the non-sharded path, which called go test directly in the workflow YAML. This change moves all test invocations through make by: - Adding TESTTAGS ?= all to control build tags (replaces hardcoded -tags=all) - Replacing direct `go test` calls in the non-sharded workflow steps with `make TESTTAGS=<language> GOTESTARGS=<flags> test` GO_TEST_EXEC ?= go test in the Makefile now correctly controls the test executor for both sharded and non-sharded paths, and the workflow env GO_TEST_EXEC: "gotestsum --" is picked up via ?= in both cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fc349db to
f500900
Compare
This was referenced Mar 16, 2026
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Mar 16, 2026
…ailures (#2113) Part of #2106 ## Problem After #2107 was merged, several providers started failing CI with: ``` FAIL github.com/pulumi/pulumi-postgresql/provider/v3/cmd/pulumi-resource-postgresql [setup failed] ``` Affected providers: - pulumi/pulumi-postgresql#886 - pulumi/pulumi-newrelic#1312 - pulumi/pulumi-auth0#1098 - pulumi/pulumi-scm#474 ## Root Cause The `cmd/pulumi-resource-*` packages embed a generated `schema.json` file (via `//go:embed`) that only exists after `make tfgen` runs. At test time in CI, this file does not exist. The old workflow ran `go test .` (root package of `provider/` only), which never compiled the `cmd/` packages. After #2107, the `Run provider tests` step was changed to call `make test_provider`, which uses `./...` — and this is the first time the `cmd/` packages were compiled in CI, exposing the pre-existing build failure. ## Fix Change `test_provider_cmd` from `./...` to `.` in the base Makefile template, restoring the original scope (root package of `provider/` only). The `cmd/` packages are binary entrypoints with no tests; they do not belong in the test scope. ## Test plan - [ ] CI on this PR passes - [ ] Re-run CI on one of the affected providers after this merges to confirm the fix ## Verification PRs PRs created from branch `pose/fix/test-provider-scope` for the 21 providers with open sub-tickets: - 🔄 in-progress pulumi/pulumi-eks#2193 - ✅ passed pulumi/pulumi-sdwan#460 - ✅ passed pulumi/pulumi-ise#492 - ✅ passed pulumi/pulumi-snowflake#1136 - 🔄 in-progress pulumi/pulumi-cloudflare#1521 - ❌ failed pulumi/pulumi-tls#990 (unrelated: certificate issue pulumi/pulumi-tls#962) - ✅ passed pulumi/pulumi-dbtcloud#514 - ✅ passed pulumi/pulumi-datadog#1115 - ✅ passed pulumi/pulumi-scm#481 - ✅ passed pulumi/pulumi-digitalocean#1295 - ✅ passed pulumi/pulumi-harness#634 - ✅ passed pulumi/pulumi-gitlab#1146 - ✅ passed pulumi/pulumi-random#2027 - ✅ passed pulumi/pulumi-newrelic#1321 - ✅ passed pulumi/pulumi-postgresql#893 - ✅ passed pulumi/pulumi-auth0#1105 - ✅ passed pulumi/pulumi-meraki#508 - ✅ passed pulumi/pulumi-artifactory#1298 - ✅ passed pulumi/pulumi-junipermist#631 - ✅ passed pulumi/pulumi-okta#1120 - ✅ passed pulumi/pulumi-pagerduty#1048 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2106.
Context
We previously used
gotestfmtto improve test output readability on CI, but it was removed because it was mis-attributing logs from one test to another and not reporting logs when a panic was encountered — making failures harder to diagnose rather than easier.gotestsumis a more robust alternative. This PR introduces two Makefile variables:GO_TEST_EXEC ?= go test— set to"gotestsum --format github-actions --"on CI via the workflowenv:blockTESTTAGS ?= all— overridden per-language in the non-shorted matrix pathWhy the workflow had to change too
An earlier attempt set
GO_TEST_EXECin the workflow env and relied on Make's?=to pick it up. This worked for the sharded path (which callsmake test), but not the non-sharded path, which calledgo testdirectly in the workflow YAML — bypassing the Makefile entirely. awsx uses the non-sharded path, soGO_TEST_EXECwas never consulted.This change moves all test invocations through
makesoGO_TEST_EXECis always honoured, and also aligns with the strategic goal of keeping workflows "dumb" — test logic lives in the Makefile, not scattered across workflow YAML.Changes
GO_TEST_EXEC ?= go testandTESTTAGS ?= allto bothbaseandparameterized-goMakefile templates$(GO_TEST_EXEC)and$(TESTTAGS)intestandtest_providermake targetsGO_TEST_EXEC: "gotestsum --format github-actions --"in the test workflow's globalenv:go testworkflow steps withmake TESTTAGS=<language> GOTESTARGS=<flags> testgotestsumvia aqua tomise.test.tomlso it is installed in test workflow environmentsTest plan
make test-go test-providerspasses — all 15 test providers regenerate and pass actionlint🤖 Generated with Claude Code