Skip to content

Use GitHub Actions matrix for integration tests#238

Merged
bkontur merged 8 commits into
mainfrom
matrix_again
Feb 13, 2026
Merged

Use GitHub Actions matrix for integration tests#238
bkontur merged 8 commits into
mainfrom
matrix_again

Conversation

@x3c41a
Copy link
Copy Markdown
Contributor

@x3c41a x3c41a commented Feb 12, 2026

Summary

Test plan

  • Verify both matrix jobs (Westend parachain, Polkadot solochain) appear in the Actions tab
  • Verify they run in parallel and all tests pass
  • Verify artifact names are unique per runtime on failure

Run Westend parachain and Polkadot solochain tests in parallel
instead of sequentially, eliminating ~75 lines of duplication and
cutting CI wall time roughly in half. Each runtime gets its own
isolated runner, avoiding flaky cross-runtime state issues (#237).
@x3c41a x3c41a requested a review from bkontur February 12, 2026 10:24
Comment thread .github/workflows/integration-test.yml Outdated
Instead of hardcoding the runtime list in the workflow, read from
scripts/runtimes-matrix.json and filter by a new `integration_tests`
flag. This keeps the single source of truth for runtime definitions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bkontur bkontur requested a review from karolk91 February 12, 2026 12:44
@bkontur
Copy link
Copy Markdown
Collaborator

bkontur commented Feb 12, 2026

@x3c41a why is this hanging?
image

The matrix job reports per-runtime check names (e.g. "Integration Tests
(bulletin-polkadot)") which don't match the branch protection rule
expecting a single "Integration Tests" status. Add a summary job that
aggregates matrix results under that exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@x3c41a
Copy link
Copy Markdown
Contributor Author

x3c41a commented Feb 12, 2026

@x3c41a why is this hanging? image

I guess rate limiting. I had rate limits before, they looked similar

The Setup job uses POLKADOT_SDK_VERSION and ZOMBIENET_VERSION in cache
keys, but these are defined in .github/env. Without loading that file,
the cache keys resolve to empty strings causing cache misses every run.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bkontur
Copy link
Copy Markdown
Collaborator

bkontur commented Feb 12, 2026

I guess rate limiting. I had rate limits before, they looked similar

looks like the github Configuration protect branch with required jobs name changed :)

Comment thread .github/workflows/integration-test.yml Outdated
Comment on lines +175 to +177
- name: Test authorize-and-store ws
working-directory: examples
run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "bulletin-westend-runtime" "ws"
run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "ws"
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Test authorize-and-store ws
working-directory: examples
run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "bulletin-westend-runtime" "ws"
run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "ws"
- name: Test authorize-and-store ws
env:
RUNTIME_PACKAGE: ${{ matrix.runtime.package }}
working-directory: examples
run: just run-test-authorize-and-store "$TEST_DIR" "$RUNTIME_PACKAGE" "ws"

Copy link
Copy Markdown
Collaborator

@bkontur bkontur Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@karolk91 why is this better approach? Then we need to adjust also other steps. Why not then add RUNTIME_NAME to echo "TEST_DIR=$TEST_DIR" >> "$GITHUB_ENV" or something like that, so we don't need to add env: RUNTIME_PACKAGE: ${{ matrix.runtime.package }} to every step?

but the actual version, looks also easy:

run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "ws"

any security concerns like we discussed yesterday?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@karolk91 thx for the suggestion, it opened the right discussion I believe!
I used @bkontur approach to avid adding env: ... at every matrix step

any security concerns?

+1 to previous question

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the general idea is to avoid using ${{ xxx }} substitutions especially in cases where these can be provided by external user as these may lead to command injections. This is because how github action will substitute values for these variables before "shell" can handle them.

There are some other levels of protections so in our specific case we are probably safe without these changes - but to be future proof and follow best practices, lets use ${{ }} via env (as this path will make sure that these are substituted by "shell" itself instead of special handling from github actions where some special characters etc that may lead to command injections)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using something like

echo "RUNTIME_PACKAGE=${{ matrix.runtime.package }}" >> "$GITHUB_ENV"

doesn't solve the issue (in general) because we just moved ${{ }} to a different place

Comment thread .github/workflows/integration-test.yml Outdated
Comment on lines +179 to +181
- name: Test authorize-and-store smoldot
working-directory: examples
run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "bulletin-westend-runtime" "smoldot"
run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "smoldot"
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Test authorize-and-store smoldot
working-directory: examples
run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "bulletin-westend-runtime" "smoldot"
run: just run-test-authorize-and-store "${{ env.TEST_DIR }}" "${{ matrix.runtime.package }}" "smoldot"
- name: Test authorize-and-store smoldot
env:
RUNTIME_PACKAGE: ${{ matrix.runtime.package }}
working-directory: examples
run: just run-test-authorize-and-store "$TEST_DIR" "$RUNTIME_PACKAGE" "smoldot"

bkontur and others added 2 commits February 12, 2026 23:34
Set RUNTIME_PACKAGE once in $GITHUB_ENV alongside TEST_DIR, then
reference both as plain env vars in all subsequent steps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@karolk91
Copy link
Copy Markdown
Collaborator

@x3c41a , lets merge it. I will address security concerns in a separate PR

Best practices for the record:
https://docs.github.com/en/actions/reference/security/secure-use#use-an-intermediate-environment-variable

Also, please lets avoid jumping to implementations in the future until we are all aligned in a discussion regarding some topic

@bkontur bkontur merged commit b1ecf3d into main Feb 13, 2026
21 of 23 checks passed
@bkontur bkontur deleted the matrix_again branch February 13, 2026 09:38
@x3c41a
Copy link
Copy Markdown
Contributor Author

x3c41a commented Feb 13, 2026

@x3c41a , lets merge it. I will address security concerns in a separate PR

Best practices for the record: https://docs.github.com/en/actions/reference/security/secure-use#use-an-intermediate-environment-variable

Also, please lets avoid jumping to implementations in the future until we are all aligned in a discussion regarding some topic

Great doc! I think we should add it to our Claude /review skill for reference.

Happy to review your PR!

antkve pushed a commit that referenced this pull request Feb 16, 2026
* Use GitHub Actions matrix for integration tests

Run Westend parachain and Polkadot solochain tests in parallel
instead of sequentially, eliminating ~75 lines of duplication and
cutting CI wall time roughly in half. Each runtime gets its own
isolated runner, avoiding flaky cross-runtime state issues (#237).

* Reuse runtimes-matrix.json for integration test matrix

Instead of hardcoding the runtime list in the workflow, read from
scripts/runtimes-matrix.json and filter by a new `integration_tests`
flag. This keeps the single source of truth for runtime definitions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add summary job for Integration Tests status check

The matrix job reports per-runtime check names (e.g. "Integration Tests
(bulletin-polkadot)") which don't match the branch protection rule
expecting a single "Integration Tests" status. Add a summary job that
aggregates matrix results under that exact name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Load env vars in Setup job to fix cache key resolution

The Setup job uses POLKADOT_SDK_VERSION and ZOMBIENET_VERSION in cache
keys, but these are defined in .github/env. Without loading that file,
the cache keys resolve to empty strings causing cache misses every run.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Use GITHUB_ENV for RUNTIME_PACKAGE instead of per-step matrix refs

Set RUNTIME_PACKAGE once in $GITHUB_ENV alongside TEST_DIR, then
reference both as plain env vars in all subsequent steps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants