Add E2E test suite CI job with code coverage#129
Merged
Conversation
Add a dedicated `e2e-integration-tests` CI job that runs all 10 E2E workflows against an inmemory provider, replacing the individual demo script invocations that the suite supersedes. JS coverage uses c8 (V8 native) to measure api.js and common.js coverage. Rust coverage uses LLVM instrument-coverage on the provider binary built into a separate target-cov/ directory; profraw data is merged after graceful SIGTERM shutdown. Both text summaries print to CI logs and artifacts are uploaded for downstream consumption. Remove the standalone demo scripts (bucket-membership, bucket-with- storage, checkpoint-rewards, drive-lifecycle, s3-lifecycle) and their justfile recipes, as the E2E suite covers these flows. Closes #126
Add the 10 E2E workflow files, runner, and helpers that the e2e-integration-tests CI job executes. Also include the new extrinsic wrappers in api.js that the workflows depend on.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
bkontur
reviewed
Jun 3, 2026
…tions E2E test fixes: - Fix TDZ crash in waitForNextBlock/waitForBlock (ReplaySubject fires synchronously before const sub is assigned) - Add .catch/.finally to main() in all 10 workflows to ensure clean process exit and prevent 10-minute timeout hangs - Fix wrong field names in createBucketWithStorage callers: max_capacity → max_bytes, max_payment → max_price_per_byte - Fix test 6.4: use self-demotion (pallet enforces ensure!(member==who)) - Fix extend_agreement callers: duration → additional_duration - Fix setExtensionsBlocked: remove extra provider param, use provider signer instead of client signer - Fix test 8.4: manually accept agreement instead of waiting for auto-accept (Charlie has no provider node running) CI workflow DRY-up: - Extract start-zombienet composite action (3 copies → 1) - Extract wait-for-parachain composite action (3 copies → 1) - Extract wait-for-provider-health composite action (4 copies → 1)
Remote had added wait-for-parachain inline; keep our start-zombienet composite action and deduplicate the wait-for-parachain step.
- 02: remove unused UNIT, use balBefore in reject assertion - 04: remove unused putChunk, providerFetch imports - 05: remove unused sameAddress import - 07: remove unused waitForBlock, getFree imports - 08: remove unused UNIT, completeDeregister, registerProvider - 09: remove unused fmtRole import
…5x speedup - submitTx now delegates to waitForTransaction(TX_MODE_IN_BLOCK) instead of signAndSubmit (which waits for finalization ~36s per tx) - submitTxExpectFailure uses signSubmitAndWatch with txBestBlocksState - waitForNextBlock/waitForBlock use bestBlocks$ instead of finalizedBlock$ - Fix test 4.7 S3 list: data.objects -> data.contents matching ListResult
Best-block confirmation is too fast — subsequent api.query calls read from finalized state and don't see the tx's state changes yet. Switch submitTx back to TX_MODE_FINALIZED_BLOCK (still using the cleaner observable-based waitForTransaction). Keep submitTxExpectFailure and waitForNextBlock/waitForBlock at best-block for speed since they don't depend on post-tx state reads.
Skip the relay chain entirely for E2E tests — run polkadot-omni-node directly with --dev-block-time 2000. Blocks finalize instantly, cutting per-tx wait from ~18-36s (relay finalization) to ~2-4s. - Add start-e2e-chain composite action (generates spec + runs omni-node) - Add chain_spec_script field to runtimes-matrix.json - Add just start-e2e-chain for local dev - Remove zombienet dependency from e2e-integration-tests job
omni-node refuses to start without a stable network key. Since E2E uses --tmp (ephemeral), pass --unsafe-force-node-key-generation to skip.
- 7.1: increase extDuration to 200 (> original 100) since extend_agreement resets expires_at = current_block + additional_duration, not additive - 8.1-8.3: use Ferdie (fresh provider, no prior agreements) instead of Charlie who accumulates committed_bytes from earlier workflows - 10.2: increase duration to 10,000 so reserved payment dwarfs tx fees - 10.5: add checkpoint before freeze (pallet requires snapshot) and fix frozen field assertion to use frozen_start_seq
- 2.3: compare balance after reject vs after request (not vs before),
since client paid tx fees for create_bucket + request_primary_agreement
- 2.6: pass burn_percent to Enum("Burn", { burn_percent: 100 }) since
the pallet's EndAction::Burn variant requires this field
- api.js: add optional actionValue param to endAgreement for enum data
- Run E2E tests against web3-storage-paseo only (storage pallet config is identical across runtimes — running both just doubles CI time) - Add dedicated "Print JS coverage" step so metrics are visible in CI logs instead of buried in 500+ lines of test output - Drop runtime-matrix dependency since E2E job no longer needs it
- Export LLVM_PROFILE_FILE explicitly in shell script instead of relying on step-level env (may not propagate to nohup'd background process in container environments) - Add diagnostics: log provider PID at start, list coverage files and process state before generating report - Add else branches with clear warnings when profraw/llvm-tools missing - Send SIGKILL as fallback if provider doesn't exit after SIGTERM
Test 2.6: payment_locked (price_per_byte × max_bytes × duration) was 10,485,760 with price=1, 1MiB, 10 blocks — below the existential deposit (1,000,000,000). The treasury transfer fails with Token::BelowMinimum when the treasury account doesn't exist yet. Fix: use 1 GiB max_bytes so payment_locked = 10,737,418,240 > ED. Test 10: remove unused sameAddress import, charlie variable, and balBefore declaration flagged by CodeQL.
- Use inline env var (LLVM_PROFILE_FILE=... nohup) instead of export to guarantee the env var reaches the backgrounded process - Save provider PID to GITHUB_ENV and use kill -TERM $PID directly instead of pkill -f which is unreliable in containers - Remove %m from profraw filename (unnecessary for single binary) - Add binary instrumentation verification after build (nm check) - Verify LLVM_PROFILE_FILE in /proc/PID/environ after start - Wait up to 15s for graceful exit before SIGKILL - Search /tmp and cwd for stray profraw files - Better diagnostics on failure (provider log tail)
…allel
The integration-tests and sc-integration-tests jobs each rebuilt the runtime
and provider before running their demos. Factor the shared compilation into a
single 'build' job and have both consume its artifact:
build Compile the COMMON stuff once: every runtime wasm
(looping the runtimes matrix file, for zombienet's
chain_spec_command) and the provider node, uploaded as
one 'build' artifact.
integration-tests Download the artifact, spawn its own chain + inmemory
and disk providers, run the L0 / fs / s3 / papi demos.
fs/s3 still 'cargo run' their examples (incremental off
the restored build cache).
sc-integration-tests Download the artifact, install solc/resolc + build the
example contracts, spawn its own chain + inmemory
provider, run the smart-contract demos.
The two test jobs run in parallel and skip the runtime/provider build. Demos
stay sequential within each job (drain-tx-pool-then preserved).
Reflect the build-once + sharded-matrix CI refactor onto feat/e2e-coverage: - New 'build' job compiles every matrix runtime wasm + the provider once and publishes them as a single 'build' artifact. - integration-tests and sc-integration-tests drop their build steps and consume the artifact, running in parallel (timeout 30m). - e2e-integration-tests consumes the artifact's runtime wasm but still builds its own coverage-instrumented provider. - Existing composite actions (start-zombienet, start-e2e-chain, wait-for-parachain, wait-for-provider-health) and the e2e coverage job are preserved.
Eliminate the local/CI divergence and make the matrix's chain_spec_script field load-bearing instead of metadata-only: - justfile start-e2e-chain takes a RUNTIME arg (default web3-storage-paseo, matching CI) and reads build_command + chain_spec_script from scripts/runtimes-matrix.json, so a local run exercises the same runtime and spec as CI instead of hardcoding build-runtime + build-chain-spec.sh. - The e2e CI job resolves chain_spec_script from the matrix (by runtime name) into E2E_CHAIN_SPEC_SCRIPT and passes it to start-e2e-chain, instead of hardcoding scripts/build-paseo-chain-spec.sh.
The build-once artifact uploads both target/release/storage-provider-node and the runtime wasm under target/release/wbuild, so upload-artifact roots the archive at their least common ancestor (target/release) and strips that prefix. The test jobs downloaded into '.', landing the binary at ./storage-provider-node, so 'chmod +x target/release/storage-provider-node' failed with 'No such file or directory'. Download into target/release so the stripped prefix is restored where chmod, the chain_spec_command scripts, and the demos expect it. Also upload only the final *.compact.compressed.wasm per runtime instead of the entire wbuild Cargo scratch tree, cutting the artifact from ~927 MiB / 3453 files to a handful of files and making the download/extract near-instant. LCA stays target/release, so the download fix covers both.
The e2e-integration-tests job never consumed the shared build artifact: its provider is built here with coverage instrumentation (target-cov), and its runtime wasm is rebuilt by the chain-spec script at chain start. The download step pulled the artifact into '.' (where nothing read it) and 'needs: build' serialized the job behind the build for no benefit. Drop both, and correct the misleading comment. The job now runs in parallel with build. The build artifact is still consumed by integration-tests and sc-integration-tests, and build remains in the completion gate's needs.
The header comment claimed the build artifact carries the fs/s3 example binaries and the Solidity contracts and that the test jobs compile nothing. Neither is true: the artifact holds only the provider binary and the compressed runtime wasm, the fs/s3 demos still 'cargo run' their example crates, and the sc job still installs solc/resolc and builds contracts. Rewrite both comments to describe what actually happens and note the test jobs lean on the shared Rust cache.
…at/e2e-coverage # Conflicts: # .github/workflows/integration-tests.yml # examples/papi/common.js
submitTx now resolves at in-block inclusion rather than finalization, so a
demo that reads state right after a write was racing the ~6-block finality
lag: storage reads default to the finalized head, so the just-included write
wasn't visible yet (e.g. full-flow's fetchChallengeProof failing with
"No challenges at deadline").
Point every post-write read in the PAPI demos at the best (non-finalized)
block via a single shared READ_OPTS = { at: "best" } exported from common.js
-- one switch for the whole suite (flip to "finalized" there to revert).
Covers the CI demos (full-flow/api, s3-lifecycle, drive-lifecycle, sc-flow,
sc-coverage, sc-token-gated, common helpers) plus the non-CI demos
(checkpoint-rewards, checkpoint-missed, bucket-membership, provider-discovery)
for consistency.
…at/e2e-coverage # Conflicts: # examples/papi/bucket-membership.js # examples/papi/checkpoint-rewards.js # examples/papi/drive-lifecycle.js # examples/papi/s3-lifecycle.js
The suite submits txs at in-block (best-block) inclusion, not finalization.
State assertions that read right after submitTx via api.query.*.getValue/
getEntries default to finalized state, which lags inclusion and races the
just-applied change. Append READ_OPTS ({ at: "best" }) to all 54 such reads
across the 10 e2e workflow files + helpers.js, matching what 391f8bf did for
the standalone demos. No assertion, submitTx, or HTTP logic changed.
full-flow's verification used api.event...ChallengeDefended.watch(), which only observes FINALIZED blocks. With in-block tx submission the two defenses land in best (not-yet-finalized) blocks, so a count taken right after (even behind a 3s sleep) read 0 -> "Expected 2 ChallengeDefended events, got 0". The respond_to_challenge dispatch emits ChallengeDefended in the same block the tx lands in, so extract it from each response's in-block events via requireOneEvent instead. Deterministic and race-free; drops the sleep.
ilchu
approved these changes
Jun 4, 2026
…engeNotFound
A challenge's id embeds the block height it was created at, and
respond_to_challenge references that exact id. With in-block submission the
creating tx sits in a best block that — even with a single collator — can be
reorged before it's relay-backed/finalized, rolling the challenge out from
under the response (seen as 'respond_to_challenge dispatch failed:
ChallengeNotFound' when the response lands a few blocks later, as with the
slower disk provider).
Add submitTxFinalized (waits for a finalized block) and use it only for the
challenge creators: challengeOffchain + challengeCheckpoint (api.js), and the
challengeCheckpoint precompile call in sc-coverage via a new callContract
{ finalized: true } opt. Everything else stays in-block. Two extra finality
waits per affected demo, still far under the 30-min cap.
Smaller requested duration shortens the 'wait for agreement expiry' step (expires_at = accept_block + duration). 15 stays >= the provider's min_duration (10), and max_payment scales automatically off duration.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Provider coverage: - Use %c (continuous mode) in LLVM_PROFILE_FILE — profraw is mmap'd and flushed in real-time, so coverage data survives SIGKILL. The previous approach relied on atexit handlers which never ran because the provider didn't exit on SIGTERM within 15s. - Clean up stray build-step profraw files (default_*.profraw from proc-macros/build-scripts) that were polluting the coverage report with 0% data. Pallet coverage: - Add pallet unit test coverage step. The pallet runs as WASM on-chain so E2E tests cannot instrument it. Instead, run `cargo test -p storage-provider-pallet --lib` with native -C instrument-coverage to get pallet/src/ coverage. - Generate separate pallet coverage report (pallet-lcov.info).
This was referenced Jun 4, 2026
…erage gates Split the 3,729-line monolithic tests.rs into 12 focused sub-files under pallet/src/tests/ for maintainability. Move shared test helpers into mock.rs. Add coverage regression checks to CI: - check.yml: pallet-coverage job compares PR vs base branch pallet coverage - integration-tests.yml: extract coverage summary (pallet, provider, JS), cache baseline on main/dev pushes, and coverage-gate job fails PRs if any metric decreases
…or provider Pallet tests run without --release, so the test binary is in target-cov/debug/deps/, not release/deps/. Also send SIGTERM before SIGKILL when stopping the provider so the LLVM atexit handler can flush profraw data when continuous mode (%c) is not supported.
ui-e2e.yml filtered at the WORKFLOW level (on.*.paths), so on PRs that touch none of those paths the whole workflow was skipped and the required status check 'UI E2E Tests (chain + provider + UIs)' never reported — leaving such PRs permanently BLOCKED in branch protection (e.g. CI-only / PAPI-demo changes). Adopt the same shape ui-checks.yml already uses: drop the top-level paths filter, detect relevant changes in a 'changes' job, gate the heavy e2e job (now 'UI E2E run') on it, and add an always-running aggregate job that carries the required context name and is green on skip, red on real failure/cancellation.
Condense the comment blocks across the integration-tests/ui-e2e workflows and the PAPI demos (READ_OPTS, submitTx/submitTxFinalized, challenge-finalize and event-read notes) to one or two lines each, keeping the 'why' and dropping the restated mechanics. Comments only; no behaviour change.
…at/e2e-coverage # Conflicts: # .github/workflows/integration-tests.yml
bkontur
reviewed
Jun 4, 2026
bkontur
approved these changes
Jun 4, 2026
…path - Add git safe.directory in pallet-coverage so git stash/checkout works inside the CI container (different user owns the workspace) - Fix pallet test binary path from release/deps to debug/deps (cargo test runs without --release) - Use SIGTERM before SIGKILL for provider shutdown so LLVM atexit handler can flush profraw data
# Conflicts: # .github/workflows/integration-tests.yml # examples/papi/bucket-membership.js # examples/papi/checkpoint-rewards.js # examples/papi/drive-lifecycle.js # examples/papi/s3-lifecycle.js
…in E2E Move pallet + provider-node coverage to Basic Checks using cargo-llvm-cov unit tests (reliable, fast). Strip all Rust coverage instrumentation from the E2E job — no more target-cov, RUSTFLAGS, profraw, or SIGTERM dance. E2E job now only collects JS c8 coverage. Coverage-gate compares JS only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a comprehensive E2E test suite (10 workflows, 95 tests) that exercises the full storage pallet lifecycle against a real chain + provider node, with dual JS/Rust code coverage.
Chain infrastructure: dev mode instead of zombienet
The E2E job runs
polkadot-omni-nodein standalone dev mode (--dev-block-time 2000) instead of spawning a full zombienet relay+parachain network. This is a strategic change:The
start-e2e-chaincomposite action generates a chain spec viachain-spec-builderand launchespolkadot-omni-node --alice --tmp --dev-block-time 2000.Test execution model
e2e/runner.js): discoversNN-*.jsfiles, runs each as a child process with 10-minute timeoutapi.jsandcommon.jscoverage-C instrument-coverage, profraw merged after graceful SIGTERMTest workflows
Bug fixes discovered during E2E development
Fixed 8 bugs in test logic and shared utilities:
waitForNextBlock/waitForBlock(common.js): ReplaySubject fired callback synchronously beforeconst subwas assignedpapi.destroy(), open WebSocket kept Node alivecreateBucketWithStoragecallers: passedmax_capacity/max_paymentbut pallet expectsmax_bytes/max_price_per_bytesetExtensionsBlockedAPI: passed extraproviderparam pallet doesn't accept, and signed as client instead of provideracceptAgreementextend_agreementresetsexpires_at = current_block + additional_duration(not additive) —extDurationmust exceed original durationEnum("Burn")missing requiredburn_percentfield forEndAction::Burn { burn_percent: u8 }Other changes
bucket-membership,bucket-with-storage,checkpoint-rewards,drive-lifecycle,s3-lifecycle)start-e2e-chain,start-zombienet,wait-for-parachain,wait-for-provider-healthjust e2eandjust e2e-singlerecipesTest plan
js-coverage-*andrust-coverage-*artifacts downloadable from Actions runintegration-testsandsc-integration-testsjobs still pass unaffectedCloses #126