test: add automated testing runner and matrix#3440
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0fe0cc1230
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| pass_count = sum(1 for a in attempts if a.status == "PASS") | ||
| if pass_count >= 2: | ||
| return True |
There was a problem hiding this comment.
This early-stop condition is incorrect for critical tests that run with max_attempts=5: returning after only two PASS attempts can produce a final PASS before a true majority is possible, and returning after two ERROR attempts can similarly short-circuit recoverable runs. Because run_test_with_retries applies this to all critical-tagged tests, the majority-vote reliability guarantee is broken for a large part of the matrix.
Useful? React with 👍 / 👎.
| python -m runner.runner --matrix "$MATRIX" --tag smoke --single | ||
| SMOKE_EXIT=$? |
There was a problem hiding this comment.
With set -euo pipefail enabled, this command exits the script immediately on any non-zero status, so the later $SMOKE_EXIT/$FULL_EXIT branching and artifact collection logic does not run on failures. In practice, smoke/full failures bypass the script’s documented exit-code handling and can skip report copying, reducing CI diagnosability.
Useful? React with 👍 / 👎.
CharlVS
force-pushed
the
qa/automated-testing
branch
from
0fe0cc1 to
faed95c
Compare
|
Visit the preview URL for this PR (updated for commit faed95c): https://walletrc--pull-3440-merge-d2q3l3tm.web.app (expires Thu, 19 Mar 2026 14:54:35 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: f66a4ff03faa546f12f0ae5a841bd9eff2714dcc |
1 participant
Summary
automated_testing/automated_testing/as part of the testing packageNote
This PR targets
polish/05-docs-and-release-notesbecauseqa/automated-testingwas branched from it. Afterpolish/05-docs-and-release-notesis merged, please switch this PR base todev.