Add release checklist docs and speed up functional release tests - #319
Add release checklist docs and speed up functional release tests#319dbmikus wants to merge 2 commits into
Conversation
The SSH helper retried transport failures (exit 255) with uncapped exponential backoff over 6 attempts (~155s), and waitForSnapshot polled for up to 5 minutes. Both made a failing staging run drag on far longer than necessary. - sshRun: 5 attempts, 2s base, capped at 10s per attempt (~24s total) - waitForSnapshot: reduce timeout from 5 minutes to 2 minutes
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 95bfe6759e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ```bash | ||
| AMIKA_API_URL=https://app.staging-amika.dev/ \ | ||
| AMIKA_API_TOKEN=<your-api-key> \ | ||
| pnpm --dir sdk/typescript test:functional -- --reporter=verbose |
There was a problem hiding this comment.
For every documented TypeScript invocation, pnpm forwards this separator: pnpm help run gives Usage: pnpm run <command> [<args>...], and the checked command expands to vitest run --config vitest.functional.config.ts -- --reporter=verbose. Vitest treats tokens after -- as positional filters rather than CLI options, so the release command can search for a test named --reporter=verbose instead of enabling the reporter and may run no tests. Drop the extra --.
AGENTS.md reference: AGENTS.md:L129-L129
Useful? React with 👍 / 👎.
| slug: string, | ||
| targetState = "active", | ||
| timeoutMs = 5 * 60 * 1000, | ||
| timeoutMs = 2 * 60 * 1000, |
There was a problem hiding this comment.
When a valid Daytona snapshot remains capturing for between two and five minutes, this now fails the release test even though the enclosing test deliberately allows up to 15 minutes. The stated goal of surfacing server-side failures faster can be achieved without rejecting slow successful snapshots by retaining the previous timeout and immediately throwing when snap.state === "failed".
Useful? React with 👍 / 👎.
| AMIKA_RUN_E2E=1 \ | ||
| AMIKA_API_URL=https://app.amika.dev/ \ | ||
| AMIKA_API_KEY=<your-api-key> \ | ||
| make test-e2e |
There was a problem hiding this comment.
With this invocation, AMIKA_RUN_E2E_API is unset, so go/test/e2e/e2e_test.go:103-106 skips every api-* case and baseEnvFor at lines 51-64 strips the provided production URL and key from all remaining cases. Consequently, this command does not exercise production at all and can give a release operator a false production-validation signal; describe it as an offline CLI sanity check or provide an explicit way to select safe API cases.
AGENTS.md reference: AGENTS.md:L129-L129
Useful? React with 👍 / 👎.
| AMIKA_RUN_E2E=1 AMIKA_RUN_E2E_API=1 \ | ||
| AMIKA_API_URL="$STAGING" AMIKA_API_KEY="$API_KEY" \ | ||
| go -C go test -v -timeout 20m ./test/e2e/... \ | ||
| 2>&1 | tee scratch/test-results/e2e-go.txt |
There was a problem hiding this comment.
On a fresh checkout, scratch/ is gitignored and absent, and neither saving-results command creates scratch/test-results/. In that scenario tee exits with No such file or directory, no result file is saved, and the pipeline reports failure even if the test command succeeded. Add mkdir -p scratch/test-results before these pipelines.
AGENTS.md reference: AGENTS.md:L129-L129
Useful? React with 👍 / 👎.
1 participant
Summary
docs/release-checklist.mddocumenting how to run the Go e2e suite and the TypeScript SDK functional suite against staging before a release, including credentials, env vars, the--reporter=verboseflag for legible per-test output, and how to save results.release.functional.test.tsso a failing staging run surfaces failures far faster.Details
The SSH helper (
sshRun) retried transport failures (exit 255) with uncapped exponential backoff over 6 attempts (~155s per SSH call). It now uses 5 attempts with a 2s base capped at 10s per attempt (~24s total).waitForSnapshotpolled for up to 5 minutes for a snapshot to reachactive. It now waits up to 2 minutes. This matters when a snapshot fails server-side (e.g. the/etc/environmentscrub abort tracked in KAPRO-711): the helper doesn't short-circuit on thefailedstate, so a shorter cap keeps a doomed poll from dragging on.