Skip to content

Add GO_TEST_EXEC variable to use gotestsum on CI#2107

Merged
pose merged 3 commits into
masterfrom
pose/chores/make-all-providers-use-gotestsum-2106
Mar 12, 2026
Merged

Add GO_TEST_EXEC variable to use gotestsum on CI#2107
pose merged 3 commits into
masterfrom
pose/chores/make-all-providers-use-gotestsum-2106

Conversation

@pose

@pose pose commented Mar 11, 2026

Copy link
Copy Markdown
Member

Fixes #2106.

Context

We previously used gotestfmt to improve test output readability on CI, but it was removed because it was mis-attributing logs from one test to another and not reporting logs when a panic was encountered — making failures harder to diagnose rather than easier.

gotestsum is a more robust alternative. This PR introduces two Makefile variables:

  • GO_TEST_EXEC ?= go test — set to "gotestsum --format github-actions --" on CI via the workflow env: block
  • TESTTAGS ?= all — overridden per-language in the non-shorted matrix path

Why the workflow had to change too

An earlier attempt set GO_TEST_EXEC in the workflow env and relied on Make's ?= to pick it up. This worked for the sharded path (which calls make test), but not the non-sharded path, which called go test directly in the workflow YAML — bypassing the Makefile entirely. awsx uses the non-sharded path, so GO_TEST_EXEC was never consulted.

This change moves all test invocations through make so GO_TEST_EXEC is always honoured, and also aligns with the strategic goal of keeping workflows "dumb" — test logic lives in the Makefile, not scattered across workflow YAML.

Changes

  • Add GO_TEST_EXEC ?= go test and TESTTAGS ?= all to both base and parameterized-go Makefile templates
  • Use $(GO_TEST_EXEC) and $(TESTTAGS) in test and test_provider make targets
  • Set GO_TEST_EXEC: "gotestsum --format github-actions --" in the test workflow's global env:
  • Replace non-sharded go test workflow steps with make TESTTAGS=<language> GOTESTARGS=<flags> test
  • Add gotestsum via aqua to mise.test.toml so it is installed in test workflow environments

Test plan

🤖 Generated with Claude Code

@corymhall corymhall left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀🚀🚀🚀🚀🚀

@pose pose self-assigned this Mar 11, 2026
@pose pose added this to the 0.134 milestone Mar 11, 2026
@pose pose force-pushed the pose/chores/make-all-providers-use-gotestsum-2106 branch from 0334e91 to 63a6f0b Compare March 12, 2026 10:19
pose and others added 3 commits March 12, 2026 16:01
Introduce a Makefile variable GO_TEST_EXEC (defaulting to "go test")
that is set to "gotestsum --" on CI via the GitHub Actions workflow env.
Also install gotestsum via mise in the test environment.

Closes #2106

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous approach set GO_TEST_EXEC in the workflow env and relied on
Make's ?= to pick it up. This worked for the sharded path (make test) but
not the non-sharded path, which called go test directly in the workflow YAML.

This change moves all test invocations through make by:
- Adding TESTTAGS ?= all to control build tags (replaces hardcoded -tags=all)
- Replacing direct `go test` calls in the non-sharded workflow steps with
  `make TESTTAGS=<language> GOTESTARGS=<flags> test`

GO_TEST_EXEC ?= go test in the Makefile now correctly controls the test
executor for both sharded and non-sharded paths, and the workflow env
GO_TEST_EXEC: "gotestsum --" is picked up via ?= in both cases.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@pose pose force-pushed the pose/chores/make-all-providers-use-gotestsum-2106 branch from fc349db to f500900 Compare March 12, 2026 16:01
@pose pose added this pull request to the merge queue Mar 12, 2026
Merged via the queue into master with commit ea5e03e Mar 12, 2026
11 checks passed
@pose pose deleted the pose/chores/make-all-providers-use-gotestsum-2106 branch March 12, 2026 16:24
github-merge-queue Bot pushed a commit that referenced this pull request Mar 16, 2026
…ailures (#2113)

Part of #2106

## Problem

After #2107 was merged, several providers started failing CI with:

```
FAIL github.com/pulumi/pulumi-postgresql/provider/v3/cmd/pulumi-resource-postgresql [setup failed]
```

Affected providers:
- pulumi/pulumi-postgresql#886
- pulumi/pulumi-newrelic#1312
- pulumi/pulumi-auth0#1098
- pulumi/pulumi-scm#474

## Root Cause

The `cmd/pulumi-resource-*` packages embed a generated `schema.json`
file (via `//go:embed`) that only exists after `make tfgen` runs. At
test time in CI, this file does not exist.

The old workflow ran `go test .` (root package of `provider/` only),
which never compiled the `cmd/` packages. After #2107, the `Run provider
tests` step was changed to call `make test_provider`, which uses `./...`
— and this is the first time the `cmd/` packages were compiled in CI,
exposing the pre-existing build failure.

## Fix

Change `test_provider_cmd` from `./...` to `.` in the base Makefile
template, restoring the original scope (root package of `provider/`
only). The `cmd/` packages are binary entrypoints with no tests; they do
not belong in the test scope.

## Test plan

- [ ] CI on this PR passes
- [ ] Re-run CI on one of the affected providers after this merges to
confirm the fix

## Verification PRs

PRs created from branch `pose/fix/test-provider-scope` for the 21
providers with open sub-tickets:

- 🔄 in-progress pulumi/pulumi-eks#2193
- ✅ passed pulumi/pulumi-sdwan#460
- ✅ passed pulumi/pulumi-ise#492
- ✅ passed pulumi/pulumi-snowflake#1136
- 🔄 in-progress pulumi/pulumi-cloudflare#1521
- ❌ failed pulumi/pulumi-tls#990 (unrelated:
certificate issue pulumi/pulumi-tls#962)
- ✅ passed pulumi/pulumi-dbtcloud#514
- ✅ passed pulumi/pulumi-datadog#1115
- ✅ passed pulumi/pulumi-scm#481
- ✅ passed pulumi/pulumi-digitalocean#1295
- ✅ passed pulumi/pulumi-harness#634
- ✅ passed pulumi/pulumi-gitlab#1146
- ✅ passed pulumi/pulumi-random#2027
- ✅ passed pulumi/pulumi-newrelic#1321
- ✅ passed pulumi/pulumi-postgresql#893
- ✅ passed pulumi/pulumi-auth0#1105
- ✅ passed pulumi/pulumi-meraki#508
- ✅ passed pulumi/pulumi-artifactory#1298
- ✅ passed pulumi/pulumi-junipermist#631
- ✅ passed pulumi/pulumi-okta#1120
- ✅ passed pulumi/pulumi-pagerduty#1048

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make all providers use gotestsum

2 participants