Skip to content

test_runner: add message assertions for toContainEqual/toBeArrayOfSize labels - #34320

Open
robobun wants to merge 1 commit into
mainfrom
claude/b685de8e/matcher-msg-labels
Open

robobun wants to merge 1 commit into
mainfrom
claude/b685de8e/matcher-msg-labels

Conversation

@robobun

@robobun robobun commented Jul 16, 2026 •

Copy link
Copy Markdown
Collaborator

The "Expected to contain:" / "Received:" labels were being dropped from these matchers' failure messages (the format literal went to a discarded _fmt parameter instead of being baked into format_args!):

1.3.14: "expect(received).toContainEqual(expected)\n\nExpected to contain: {\n  a: 2,\n}\nReceived: [\n  {\n    a: 1,\n  }\n]\n"
1.4:    "expect(received).toContainEqual(expected){\n  a: 2,\n}[\n  {\n    a: 1,\n  }\n]"

The underlying bug was fixed by #34343 as part of migrating every matcher to the throw! macro (which takes the format literal as a first-class argument). This PR adds message-content assertions for all four affected paths (toContainEqual, not.toContainEqual, toBeArrayOfSize, not.toBeArrayOfSize) so the label text can't silently drift again.

Both tests fail on the 1.4 canary that predates #34343 and pass on current main.


[stamp-90s] gate passed · iteration 2 · 2 files touched

passes on PR (with fix)
Test-only change.

Debug/ASAN (expected pass):
$ bun bd test 'test/js/bun/test/expect.test.js' 'test/js/bun/test/jest-extended.test.js'
$ BUN_DEBUG_QUIET_LOGS=1 bun scripts/build.ts --profile=debug --quiet test test/js/bun/test/expect.test.js test/js/bun/test/jest-extended.test.js
info: syncing channel updates for nightly-2026-05-06-x86_64-unknown-linux-gnu
info: latest update on 2026-05-06 for version 1.97.0-nightly (e95e73209 2026-05-05)
info: component rust-src is up to date
info: checking for self-update (current version: 1.29.0)
bun test v1.4.0 (49dc17712)

test/js/bun/test/jest-extended.test.js:
(pass) jest-extended > pass() [15.44ms]
(pass) jest-extended > fail() [11.49ms]
(pass) jest-extended > toBeEmpty() > "" [3.38ms]
(pass) jest-extended > toBeEmpty() > [] [0.41ms]
(pass) jest-extended > toBeEmpty() > {} [0.30ms]
(pass) jest-extended > toBeEmpty() > Set {} [0.22ms]
(pass) jest-extended > toBeEmpty() > Map {} [0.23ms]
(pass) jest-extended > toBeEmpty() > "" [0.22ms]
(pass) jest-extended > toBeEmpty() > [] [0.21ms]
(pass) jest-extended > toBeEmpty() > Uint8Array(0) [] [0.20ms]
(pass) jest-extended > toBeEmpty() > {} [0.30ms]
(pass) jest-extended > toBeEmpty() > <Buffer > [0.22ms]
(pass) jest-extended > toBeEmpty() > Headers {} [0.21ms]
(pass) jest-extended > toBeEmpty() > URLSearchParams {} [0.22ms]
(pass) jest-extended > toBeEmpty() > {} [0.26ms]
(pass) jest-extended > toBeEmpty() > {} [7.39ms]
(pass) jest-extended > toBeEmpty() > FileRef ("/tmp/jest-extended-7vwzvz/empty.txt") {  type: "text/plain;charset=utf-8"} [0.39ms]
(pass) jest-extended > not.toBeEmpty() > " " [1.31ms]
(pass) jest-extended > not.toBeEmpty() > [ "" ] [0.35ms]
(pass) jest-extended > not.toBeEmpty() > [ undefined ] [0.27ms]
(pass) jest-extended > not.toBeEmpty() > {  "": "",} [0.69ms]
(pass) jest-extended > not.toBeEmpty() > Set(1) {  "",} [0.34ms]
(pass) jest-extended > not.toBeEmpty() > Map(1) {  "": "",} [0.24ms]
(pass) jest-extended > not.toBeEmpty() > " " [0.24ms]
(pass) jest-extended > not.toBeEmpty() > [ empty item ] [0.31ms]
(pass) jest-extended > not.toBeEmpty() > Uint8Array(1) [ 0 ] [0.24ms]
(
... (truncated)
Exit: 0
diff hotspot
test/js/bun/test/expect.test.js        | 19 +++++++++++++++++++
 test/js/bun/test/jest-extended.test.js | 17 +++++++++++++++++
 2 files changed, 36 insertions(+)

gate history · 2 passed · 0 rejected · iteration 2

evidence per changed file
file                                    reads  edits  tests
test/js/bun/test/expect.test.js             1      1      0
test/js/bun/test/jest-extended.test.js      1      1      0

@coderabbitai

coderabbitai Bot commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 26 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: af6861a5-41b3-4c82-88ac-3a7059d74aca

📥 Commits

Reviewing files that changed from the base of the PR and between 3ff6fc5 and 49dc177.

📒 Files selected for processing (2)
  • test/js/bun/test/expect.test.js
  • test/js/bun/test/jest-extended.test.js

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Jul 16, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 12:05 AM PT - Jul 18th, 2026

❌ @robobun, your commit 49dc177 has 1 failures in Build #74851 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 34320

That installs a local version of the PR into your bun-34320 executable, so you can run:

bun-34320 --bun

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — straightforward fix baking the label literals into format_args! at four call sites, matching the pattern already used by toContain.rs and the other matchers. The inline nit about comment length is non-blocking.

Extended reasoning...

Overview

Restores the "Expected/Received" label text in toContainEqual and toBeArrayOfSize failure messages. Four call sites were passing the label literal to throw_fmt's ignored _fmt parameter instead of baking it into format_args!; they now call throw() directly with the full format string, matching toContain.rs. The throw_fmt debug assertion is tightened to probe args for a leading '\n' so this class of bug trips in debug builds. Tests added to the existing expect.test.js and jest-extended.test.js files.

Security risks

None. This only affects the text of test-runner assertion failure messages.

Level of scrutiny

Low. Mechanical fix following an established pattern; the debug-assertion change is #[cfg(debug_assertions)]-gated and compiles out of release builds. I audited every remaining throw_fmt caller (.pass(), .fail(), toBeCloseTo, snapshot matchers) — all either pass _fmt = "" (skipped by the guard) or already lead args with "\n\n", so the tightened assert won't false-positive on any of them. fmt::Arguments is Copy, so passing it to the probe and then to self.throw is fine.

Other factors

Tests are placed in the correct existing files per repo convention, cover both the .not and positive variants, and assert on message content (labels + values) rather than exact formatting. The one finding is a CLAUDE.md comment-length nit on the debug-assert block — purely stylistic and non-blocking.

Comment thread src/runtime/test_runner/expect.rs Outdated
@robobun

robobun commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator Author

Opened #34328 for the same bug before the dedupe bot flagged this one; closing mine in favor of this.

One difference: this PR keeps throw_fmt and tightens its debug_assert, whereas #34328 deletes throw_fmt entirely and migrates the remaining 19 callers to throw() (once _fmt is gone the wrapper is identical to throw). Branch is claude/farm/6c0302ea/fix-throw-fmt-labels if that variant is preferred.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the comment-length nit is addressed in 493a0e1 and no new issues found.

Checked: the four call-site fixes match the toContain.rs pattern exactly (throw() + format_args!(concat!(...))). Grepped all remaining throw_fmt callers (toBeCloseTo, snapshot matchers, .pass/.fail, inline-snapshot paths in expect.rs) — every one passes format_args! beginning with "\\n\\n", so the tightened debug probe won't false-positive. The probe is #[cfg(debug_assertions)]-gated so release builds are unaffected.

The alternative in #34328 (delete throw_fmt entirely) is a reasonable follow-up but doesn't block this fix.

Extended reasoning...

Overview

Restores the Expected:/Received: label text in toContainEqual and toBeArrayOfSize failure messages, which regressed when these matchers were ported to the throw_fmt calling convention. Four call sites in toContainEqual.rs and toBeArrayOfSize.rs are rewritten to bake the literal label text into format_args! and call throw() directly — the exact pattern already used by toContain.rs:117-140. The throw_fmt debug assertion in expect.rs is tightened from args.as_str() != Some("") (which format_args!("{}", v) trivially satisfies since it returns None) to a short-circuiting fmt::Write probe that checks the first non-empty write starts with '\n'. New message-content tests in expect.test.js and jest-extended.test.js cover both the not and non-not variants of each matcher.

Security risks

None. This touches only test-runner assertion-failure message formatting; no user input parsing, no I/O, no auth/crypto surface.

Level of scrutiny

Low-to-moderate. The call-site changes are mechanical pattern alignment with an existing sibling (toContain.rs). The debug-assert probe is slightly novel but is #[cfg(debug_assertions)]-gated, so it cannot affect release builds; I verified via grep that every remaining throw_fmt caller (in toBeCloseTo.rs, toMatchSnapshot.rs, toThrowErrorMatchingSnapshot.rs, and expect.rs itself for .pass/.fail/inline-snapshot) passes a format_args! literal beginning with \"\\n\\n\", so the new assert holds for all of them. The probe returning Err(fmt::Error) after the first write correctly short-circuits so dynamic args aren't rendered twice.

Other factors

  • My prior nit (6-line comment vs. CLAUDE.md rule 13) was addressed in 493a0e1; the diff now shows the 3-line version and the thread is resolved.
  • robobun noted an alternative approach in #34328 that deletes throw_fmt entirely and migrates the remaining ~19 callers to throw(). That's a valid but larger refactor; this PR's more conservative fix-the-callers-plus-tighten-the-guard approach is correct on its own and doesn't preclude the follow-up. Not a blocker.
  • Tests were added to the existing test files (expect.test.js, jest-extended.test.js) rather than new files, per repo convention, and the PR description confirms they fail on main and pass with the fix.
  • Bug hunter found no issues this run.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the comment-length nit from the previous pass is addressed in 493a0e1.

Checked: the four call-site rewrites match the throw() pattern already used by toContain and neighbors; the tightened throw_fmt debug-assert is #[cfg(debug_assertions)]-only and I verified every remaining throw_fmt caller (toBeCloseTo, pass/fail, snapshot matchers) either passes _fmt="" or leads args with "\n\n", so no new false positives. New tests cover all four fixed paths (both .not variants).

Extended reasoning...

Overview

Restores the "Expected to contain:" / "Received:" label text in toContainEqual and toBeArrayOfSize failure messages. Four call sites in toContainEqual.rs / toBeArrayOfSize.rs were passing label literals to throw_fmt's ignored _fmt param instead of baking them into format_args!; they now call throw() directly with the literal inlined, matching the pattern every other matcher uses. Also tightens the debug-only guard in Expect::throw_fmt to probe the first non-empty write of args for a leading '\n', and adds message-content tests in expect.test.js and jest-extended.test.js.

Security risks

None. This is pure test-runner error-message formatting on the failure path; no untrusted input parsing, no auth/crypto/permissions surface.

Level of scrutiny

Low-to-moderate. The four call-site changes are mechanical throw_fmt(g, sig, LIT, format_args!("{}", v)) → throw(g, sig, format_args!(LIT, v)) rewrites. The only non-trivial piece is the new Probe writer in throw_fmt, which is gated behind #[cfg(debug_assertions)] so it has zero release-build impact. I traced its behavior: empty writes are skipped, the first non-empty write sets the flag and returns Err(fmt::Error) to short-circuit, and if args is entirely empty the flag stays false and the assert fires (correct — that's the original bug shape). I grepped every remaining throw_fmt caller and confirmed each either passes _fmt="" (check skipped) or leads format_args! with a "\n\n..." literal, so the tightened assert introduces no false positives.

Other factors

  • My prior review's only finding (6-line comment vs. CLAUDE.md rule 13) was addressed in 493a0e1; the comment is now 3 lines and the thread is resolved.
  • New tests use the same try/catch + expect.unreachable() pattern as the existing toBeOneOf test in the same file; if the matcher failed to throw, the positive toContain("\n\nExpected to contain: ") assertion would fail on the unreachable-error message, so the tests are sound.
  • No CODEOWNERS match these paths (.github/CODEOWNERS only covers /test/expectations.txt for anything expect-related).
  • The thread notes an alternative (#34328) that deletes throw_fmt entirely. That's a valid follow-up but orthogonal — this PR's more conservative keep-and-tighten approach is correct on its own and has the smaller blast radius.

@robobun

robobun commented Jul 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

Rebased onto main. The src/ fix is no longer needed here: #34343 migrated every matcher to the throw! macro and fixed the dropped-label bug as a side effect. This PR now adds only the regression tests, which fail on the 1.4 canary predating #34343 and pass on current main.

Note: with no src/ diff the fail-before gate will see both runs pass identically; a maintainer merge is needed.

…e labels

The "Expected to contain:" / "Received:" labels were being dropped
from these matchers' failure messages (the format literal was passed
to a discarded parameter). #34343 fixed the underlying bug as part of
migrating all matchers to the throw! macro; these tests lock the
message shape so it can't regress again.
@robobun
robobun force-pushed the claude/b685de8e/matcher-msg-labels branch from fb6d4bd to 49dc177 Compare July 18, 2026 00:27
@robobun robobun changed the title test_runner: restore Expected/Received labels in toContainEqual/toBeArrayOfSize failure messages test_runner: add message assertions for toContainEqual/toBeArrayOfSize labels Jul 18, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — test-only regression coverage for matcher failure-message labels.

What was reviewed:

  • Verified the asserted label strings (\n\nExpected to contain: , \n\nExpected to not contain: , \n\nReceived: ) against the current throw! format literals in toContainEqual.rs and toBeArrayOfSize.rs.
  • Checked the try/expect.unreachable()/catch pattern is non-vacuous (matches the existing toBeOneOf message test in the same file) and that the catch-block assertions would fail on the pre-#34343 output.
  • Confirmed tests are added alongside existing matcher tests in the correct files, no new files created.
Extended reasoning...

Overview

This PR adds two small regression tests (~36 lines total) to test/js/bun/test/expect.test.js and test/js/bun/test/jest-extended.test.js, asserting that toContainEqual / not.toContainEqual and toBeArrayOfSize / not.toBeArrayOfSize failure messages include their "Expected to contain:" / "Received:" label text. The underlying bug (labels dropped because a format literal went to a discarded _fmt parameter) was already fixed by #34343; this PR only adds coverage so it can't silently regress.

The PR originally also touched src/runtime/test_runner/expect.rs (my earlier inline nit about comment length applied there, and was addressed), but after rebasing onto #34343 the src/ change was dropped and the diff is now test-only.

Security risks

None. Pure test additions to the test-runner suite; no runtime, network, filesystem, or auth code touched.

Level of scrutiny

Low. Test-only, ~36 lines, follows the exact try/expect.unreachable()/catch pattern already used a few lines away for toBeOneOf in jest-extended.test.js. I cross-checked each asserted substring against the current concat! format literals in toContainEqual.rs:120,132-133 and toBeArrayOfSize.rs:56,66 — they match. The catch-block assertions are non-vacuous: if the matcher failed to throw, expect.unreachable() would throw an error whose message does not contain the asserted label, so the .toContain checks would fail.

Other factors

  • Tests are placed in the existing files for these matchers (per REVIEW.md "add your test to the existing test file"), not new files.
  • Both positive and .not variants are covered.
  • The PR description states both tests fail on the pre-#34343 1.4 canary and pass on current main; the author noted the fail-before CI gate can't distinguish because there's no src/ diff, which is expected for a test-only follow-up to an already-merged fix.
  • No outstanding reviewer comments; my earlier nit was resolved and is now moot since the file it targeted is no longer in the diff.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant