Use GitHub Actions matrix for integration tests#238
Conversation
Run Westend parachain and Polkadot solochain tests in parallel instead of sequentially, eliminating ~75 lines of duplication and cutting CI wall time roughly in half. Each runtime gets its own isolated runner, avoiding flaky cross-runtime state issues (#237).
Instead of hardcoding the runtime list in the workflow, read from scripts/runtimes-matrix.json and filter by a new `integration_tests` flag. This keeps the single source of truth for runtime definitions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@x3c41a why is this hanging? |
The matrix job reports per-runtime check names (e.g. "Integration Tests (bulletin-polkadot)") which don't match the branch protection rule expecting a single "Integration Tests" status. Add a summary job that aggregates matrix results under that exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
I guess rate limiting. I had rate limits before, they looked similar |
The Setup job uses POLKADOT_SDK_VERSION and ZOMBIENET_VERSION in cache keys, but these are defined in .github/env. Without loading that file, the cache keys resolve to empty strings causing cache misses every run. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
looks like the github Configuration protect branch with required jobs name changed :) |
| - name: Test authorize-and-store ws | ||
| working-directory: examples | ||
| run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "bulletin-westend-runtime" "ws" | ||
| run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "ws" |
There was a problem hiding this comment.
| - name: Test authorize-and-store ws | |
| working-directory: examples | |
| run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "bulletin-westend-runtime" "ws" | |
| run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "ws" | |
| - name: Test authorize-and-store ws | |
| env: | |
| RUNTIME_PACKAGE: ${{ matrix.runtime.package }} | |
| working-directory: examples | |
| run: just run-test-authorize-and-store "$TEST_DIR" "$RUNTIME_PACKAGE" "ws" |
There was a problem hiding this comment.
@karolk91 why is this better approach? Then we need to adjust also other steps. Why not then add RUNTIME_NAME to echo "TEST_DIR=$TEST_DIR" >> "$GITHUB_ENV" or something like that, so we don't need to add env: RUNTIME_PACKAGE: ${{ matrix.runtime.package }} to every step?
but the actual version, looks also easy:
run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "ws"
any security concerns like we discussed yesterday?
There was a problem hiding this comment.
the general idea is to avoid using ${{ xxx }} substitutions especially in cases where these can be provided by external user as these may lead to command injections. This is because how github action will substitute values for these variables before "shell" can handle them.
There are some other levels of protections so in our specific case we are probably safe without these changes - but to be future proof and follow best practices, lets use ${{ }} via env (as this path will make sure that these are substituted by "shell" itself instead of special handling from github actions where some special characters etc that may lead to command injections)
There was a problem hiding this comment.
using something like
echo "RUNTIME_PACKAGE=${{ matrix.runtime.package }}" >> "$GITHUB_ENV"
doesn't solve the issue (in general) because we just moved ${{ }} to a different place
| - name: Test authorize-and-store smoldot | ||
| working-directory: examples | ||
| run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "bulletin-westend-runtime" "smoldot" | ||
| run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "smoldot" |
There was a problem hiding this comment.
| - name: Test authorize-and-store smoldot | |
| working-directory: examples | |
| run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "bulletin-westend-runtime" "smoldot" | |
| run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "smoldot" | |
| - name: Test authorize-and-store smoldot | |
| env: | |
| RUNTIME_PACKAGE: ${{ matrix.runtime.package }} | |
| working-directory: examples | |
| run: just run-test-authorize-and-store "$TEST_DIR" "$RUNTIME_PACKAGE" "smoldot" |
Set RUNTIME_PACKAGE once in $GITHUB_ENV alongside TEST_DIR, then reference both as plain env vars in all subsequent steps. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@x3c41a , lets merge it. I will address security concerns in a separate PR Best practices for the record: Also, please lets avoid jumping to implementations in the future until we are all aligned in a discussion regarding some topic |
Great doc! I think we should add it to our Claude Happy to review your PR! |
* Use GitHub Actions matrix for integration tests Run Westend parachain and Polkadot solochain tests in parallel instead of sequentially, eliminating ~75 lines of duplication and cutting CI wall time roughly in half. Each runtime gets its own isolated runner, avoiding flaky cross-runtime state issues (#237). * Reuse runtimes-matrix.json for integration test matrix Instead of hardcoding the runtime list in the workflow, read from scripts/runtimes-matrix.json and filter by a new `integration_tests` flag. This keeps the single source of truth for runtime definitions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add summary job for Integration Tests status check The matrix job reports per-runtime check names (e.g. "Integration Tests (bulletin-polkadot)") which don't match the branch protection rule expecting a single "Integration Tests" status. Add a summary job that aggregates matrix results under that exact name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Load env vars in Setup job to fix cache key resolution The Setup job uses POLKADOT_SDK_VERSION and ZOMBIENET_VERSION in cache keys, but these are defined in .github/env. Without loading that file, the cache keys resolve to empty strings causing cache misses every run. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use GITHUB_ENV for RUNTIME_PACKAGE instead of per-step matrix refs Set RUNTIME_PACKAGE once in $GITHUB_ENV alongside TEST_DIR, then reference both as plain env vars in all subsequent steps. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Branislav Kontur <bkontur@gmail.com>


Summary
Test plan
Westend parachain,Polkadot solochain) appear in the Actions tab