Repository navigation
feat: tolerate known upstream test failures in checks - #61
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Upstream added a MessagesTimeline browser test (pingdotgg/t3code#29166) that fails on the plain nightly 0.1.7-nightly.20261103.4138 without any patch (
expected 577 to be close to 473, 3 of 3 runs on devbox1). Becausechecks-tt.jsonlists that file, everychecks,server,portand 04:30 unattended run failed, so the unattended-port grant never applied.What
known-upstream-failures.json(file, full test name, upstream reference, date added), seeded with that test.run_test_stepnow passes a failing vitest step only when every failed test in its JSON report matches an entry exactly, the file has no load error, and the test case's source is identical at the nightly commit and at HEAD. Any other failure, an unhandled error or an unparsable report still fails the step. Each tolerated test printspass (known upstream failure: <name>).checksrecords whether each entry failed or passed.statuslists the entries and marks one stale ("remove it") once it has passed.recipe_hash()now includes the list, so the checks stamp and the grant reflect changes to it.docs/machines.md(new "Known upstream failures" section),docs/porting.md, a row indocs/fork-changes.md, and the ownership table inAGENTS.md.Checks
python3 -m py_compile t3code-customizepython3 -m unittest discover -s tests(210 tests, OK)./t3code-customize statuschecks --source <port checkout> --target 115739c1e02e --iosrun tolerates the test in both stages.