Repository navigation
test: pin slow-test ratchet JSONL bridge #2830
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
e4afbdd
test: pin slow test ratchet jsonl bridge
briansrls 60b8c2b
test: cover non-warn timeout manifest rows
briansrls 636524c
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 3414165
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls e20ec35
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 9d23c59
ci: run test-timeout ratchet self-test
briansrls 4362a53
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls c21365b
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 77199f3
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls afc8142
WIP: T-WAD Slice 6: slow_test_exemptions_dissolved (test cost-dim cuts)
briansrls c0d4de6
Merge origin/main into session/sharp-deer-576
briansrls b7d74d2
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 142b06d
build: mark timeout ratchet self-test phony
briansrls c903a59
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 062b416
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls d108ce7
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 82867a0
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls d7cb2c6
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls c5cc237
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 3d154c8
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 5c12be3
test: tighten timeout ratchet wording
briansrls 972f1b3
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 8f3aec5
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 1a4e2fa
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 911b381
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 023b728
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 12fd064
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls 58f6a09
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls ee72364
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls ad802b3
Merge remote-tracking branch 'origin/main' into session/sharp-deer-576
briansrls File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| #!/usr/bin/env bash | ||
| # Self-test for scripts/check-test-timeout.sh. | ||
| # | ||
| # Pins the T-WAD Slice 6 bridge behavior: slow-test policy comes from the | ||
| # structured JSONL manifest, and over-budget tests not warn-listed in the | ||
| # manifest fail closed. The retired slow-test-exemptions.txt count floor must not be | ||
| # required for the consumer to run. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| ROOT="$(cd "$(dirname "$0")/.." && pwd)" | ||
| cd "$ROOT" | ||
|
|
||
| CONSUMER="$ROOT/scripts/check-test-timeout.sh" | ||
| TMPDIR="$(mktemp -d)" | ||
| cleanup() { | ||
| rm -rf "$TMPDIR" | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| MANIFEST="$TMPDIR/test-node-wall-clock-ratchet.jsonl" | ||
| LOG="$TMPDIR/libtest.log" | ||
|
|
||
| cat > "$MANIFEST" <<'EOF' | ||
| {"test":"slow_warned_test","policy":"warn"} | ||
| {"test":"not_warn_policy","policy":"fail"} | ||
| EOF | ||
|
|
||
| write_log() { | ||
| cat > "$LOG" <<'EOF' | ||
| test fast_test ... ok <0.001s> | ||
| test slow_warned_test ... ok <2.500s> | ||
| EOF | ||
| } | ||
|
|
||
| test_warn_manifest_allows_known_slow_test() { | ||
| write_log | ||
| local output | ||
| if ! output=$(TEST_TIMEOUT_MANIFEST="$MANIFEST" bash "$CONSUMER" "$LOG" 2000 2>&1); then | ||
| echo "FAIL [known-warn]: consumer rejected a manifest warn-policy test" | ||
| printf '%s\n' "$output" | sed 's/^/ | /' | ||
| return 1 | ||
| fi | ||
| if [[ "$output" != *"warn-policy tests exceeded budget"* ]]; then | ||
| echo "FAIL [known-warn]: consumer passed but did not report warn-policy backlog" | ||
| printf '%s\n' "$output" | sed 's/^/ | /' | ||
| return 1 | ||
| fi | ||
| } | ||
|
|
||
| test_unknown_slow_test_fails_closed() { | ||
| write_log | ||
| cat >> "$LOG" <<'EOF' | ||
| test slow_unknown_test ... ok <2.100s> | ||
| EOF | ||
|
|
||
| local output exit_code | ||
| if output=$(TEST_TIMEOUT_MANIFEST="$MANIFEST" bash "$CONSUMER" "$LOG" 2000 2>&1); then | ||
| exit_code=0 | ||
| else | ||
| exit_code=$? | ||
| fi | ||
|
|
||
| if [ "$exit_code" -eq 0 ]; then | ||
| echo "FAIL [unknown-slow]: consumer passed with an over-budget test absent from JSONL" | ||
| printf '%s\n' "$output" | sed 's/^/ | /' | ||
| return 1 | ||
| fi | ||
| if [[ "$output" != *"slow_unknown_test"* ]] || [[ "$output" != *"not warn-listed in manifest"* ]]; then | ||
| echo "FAIL [unknown-slow]: consumer failed but did not name the missing slow test" | ||
| printf '%s\n' "$output" | sed 's/^/ | /' | ||
| return 1 | ||
| fi | ||
| } | ||
|
|
||
| test_non_warn_manifest_policy_fails_closed() { | ||
| write_log | ||
| cat >> "$LOG" <<'EOF' | ||
| test not_warn_policy ... ok <2.100s> | ||
| EOF | ||
|
|
||
| local output exit_code | ||
| if output=$(TEST_TIMEOUT_MANIFEST="$MANIFEST" bash "$CONSUMER" "$LOG" 2000 2>&1); then | ||
| exit_code=0 | ||
| else | ||
| exit_code=$? | ||
| fi | ||
|
|
||
| if [ "$exit_code" -eq 0 ]; then | ||
| echo "FAIL [non-warn-policy]: consumer treated a non-warn manifest row as warn-listed" | ||
| printf '%s\n' "$output" | sed 's/^/ | /' | ||
| return 1 | ||
| fi | ||
| if [[ "$output" != *"not_warn_policy"* ]] || [[ "$output" != *"not warn-listed in manifest"* ]]; then | ||
| echo "FAIL [non-warn-policy]: consumer failed but did not name the non-warn slow test" | ||
| printf '%s\n' "$output" | sed 's/^/ | /' | ||
| return 1 | ||
| fi | ||
| } | ||
|
|
||
| test_zero_parsed_lines_fails_closed() { | ||
| printf 'not libtest output\n' > "$LOG" | ||
|
|
||
| local output exit_code | ||
| if output=$(TEST_TIMEOUT_MANIFEST="$MANIFEST" bash "$CONSUMER" "$LOG" 2000 2>&1); then | ||
| exit_code=0 | ||
| else | ||
| exit_code=$? | ||
| fi | ||
|
|
||
| if [ "$exit_code" -eq 0 ]; then | ||
| echo "FAIL [zero-parsed]: consumer passed on a log with no libtest timing lines" | ||
| return 1 | ||
| fi | ||
| if [[ "$output" != *"zero test-result lines parsed"* ]]; then | ||
| echo "FAIL [zero-parsed]: consumer failed but did not report parser drift guard" | ||
| printf '%s\n' "$output" | sed 's/^/ | /' | ||
| return 1 | ||
| fi | ||
| } | ||
|
|
||
| failures=0 | ||
|
|
||
| for test_fn in \ | ||
| test_warn_manifest_allows_known_slow_test \ | ||
| test_unknown_slow_test_fails_closed \ | ||
| test_non_warn_manifest_policy_fails_closed \ | ||
| test_zero_parsed_lines_fails_closed; do | ||
| echo "Test: $test_fn..." | ||
| if "$test_fn"; then | ||
| echo " PASS" | ||
| else | ||
| failures=$((failures + 1)) | ||
| fi | ||
| done | ||
|
|
||
| if [ "$failures" -ne 0 ]; then | ||
| echo "FAIL: $failures check-test-timeout self-test(s) failed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "PASS: check-test-timeout JSONL manifest behavior verified" | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This comment was marked as resolved.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.