Repository navigation
fix(backend): eval runner: failed calls no longer score as passed + dotAll regex - #13413
Merged
diegosouzapw merged 3 commits intoSep 15, 2026
Conversation
The 'does not duplicate custom Jina specialty models' test asserted
catalog IDs prefixed with 'jina-ai/' (the connection/provider ID), but
the catalog builder uses the provider alias 'jina' — which was
introduced in v3.8.36. The test above it in the same file ('does not
duplicate imported Jina specialty models') already asserts 'jina/' and
passes; this second test was simply never updated.
Fix the two assert.equal strings from 'jina-ai/' to 'jina/' to match
the current product behavior.
Fixes cheaper-inference#13313 (Jina portion — pt-BR compression integration tests need
separate investigation)
…heaper-inference#13308) The VACUUM INTO path in createManagedDbBackup() — reached from the health-check-repair flow — never called the retention helper, so db_backups/ grew without bound (observed: 10+ GB across 18 snapshots in 8 days). The sibling path (backupDbFile → backup.ts) already calls cleanupDbBackups() after each snapshot; the health-check path simply forgot to do the same. Fix: after a successful VACUUM INTO in createManagedDbBackup(), call pruneBackupDirectory() from backupRetention.ts using the same env-var precedence (DB_BACKUP_MAX_FILES / DB_BACKUP_RETENTION_DAYS) as the manual/scheduled backup path. Pruning is wrapped in a try/catch so a best-effort failure never obscures the backup result. Regression test: db-backup-healthcheck-prune-13308.test.ts seeds a backup directory with MAX_DB_BACKUPS + 5 families and asserts that pruneBackupDirectory removes exactly the overflow.
…otAll regex (cheaper-inference#13137, cheaper-inference#13138) Two eval runner bugs fixed: cheaper-inference#13137 — runSuite() never forced passed=false when caseMetrics had an error. An eval case whose upstream call failed could still be scored as passed if the error text happened to match the expected pattern. Now: if metrics.error is set, result.passed is forced to false. cheaper-inference#13138 — evaluateCase() compiled regex patterns from strings without the dotAll (s) flag, so "." did not match newlines. LLM answers are routinely multi-line, causing correct answers to be marked wrong. Now: string patterns are compiled with 's' flag; RegExp objects get the 's' flag added if not already present. Added 9 regression tests covering multi-line regex matching (SQL, numbered lists, generic), RegExp flag preservation, and error-forced failures. All existing eval tests continue to pass. Fixes cheaper-inference#13137 Fixes cheaper-inference#13138
This was referenced Sep 15, 2026
diegosouzapw
merged commit Sep 15, 2026
fdfa921
into
cheaper-inference:release/v3.8.51
8 of 16 checks passed
diegosouzapw
pushed a commit
that referenced
this pull request
Sep 15, 2026
…bility (fixes #13467) (#13539) Adds a `DISABLE_IOREG_STRATEGY=1` escape hatch to the macOS `ioreg` strategy in `getMachineIdRaw()`, so `machineId.test.ts` can reach the fallback strategies on darwin instead of always resolving the real hardware UUID (#13467). Maintainer note: this branch also carried the "force passed=false when upstream call failed" eval commit, which already landed in #13413. The branch was reset to the current release tip plus only the ioreg commit (authorship preserved) so the squash carries just this change. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
diegosouzapw
added a commit
to dajiaohuang/OmniRoute
that referenced
this pull request
Sep 18, 2026
Resolve src/lib/evals/runtime.ts: base (cheaper-inference#13413) landed the exact same two opt-out headers with a different comment. Keep one copy, with this PR's comment pointing at the buildEvalCaseRequest docblock it also introduces. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…otAll regex (cheaper-inference#13413) Eval runner: a case whose upstream call failed can no longer score as passed when the grading regex happens to match the error text (cheaper-inference#13137), and regex grading compiles with dotAll so `.` spans newlines in multi-line answers, for both string and `RegExp` patterns (cheaper-inference#13138). Chosen over cheaper-inference#13542 / cheaper-inference#13530, which each covered half. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…bility (fixes cheaper-inference#13467) (cheaper-inference#13539) Adds a `DISABLE_IOREG_STRATEGY=1` escape hatch to the macOS `ioreg` strategy in `getMachineIdRaw()`, so `machineId.test.ts` can reach the fallback strategies on darwin instead of always resolving the real hardware UUID (cheaper-inference#13467). Maintainer note: this branch also carried the "force passed=false when upstream call failed" eval commit, which already landed in cheaper-inference#13413. The branch was reset to the current release tip plus only the ioreg commit (authorship preserved) so the squash carries just this change. Validated in one consolidated batch of this series (37 PRs boarded together on `release/v3.8.51`): `typecheck:core`, `check:open-sse-typecheck` and `check:dashboard-typecheck` clean; ESLint clean on every changed file; file-size, complexity, cognitive-complexity, changelog-integrity, docs-counts, docs-sync and migration-numbering gates green (only the pre-existing `open-sse/utils/stream.ts` file-size red remains, inherited from the base); 3,743 focused `node:test` cases plus 34 vitest cases green. Thanks @KooshaPari!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes two eval runner bugs that inflated reported pass rates and marked correct multi-line answers as wrong.
Bug 1: Failed upstream calls scored as passed (#13137)
runSuite()never forcedpassed=falsewhencaseMetricscontained an error. An eval case whose upstream API call failed returned[ERROR] ...as output, and if the grading regex happened to match that error text (e.g./error/i), the case scored as passed.Fix: When
metrics.erroris set,result.passedis now forced tofalse.Bug 2: Regex grading ignores newlines (#13138)
evaluateCase()compiled string regex patterns without thedotAll(s) flag, so.did not match newlines. Since LLM answers are routinely multi-line, correct answers were being marked wrong.Three built-in cases were affected:
gs-09Counting:1.*2.*3.*4.*5vs1\n2\n3\n4\n5code-03SQL:SELECT.*FROM.*WHEREvs multiline SQLinstr-02Numbered list:1\..*2\..*3\..*4\..*5\.vs numbered itemsFix: String patterns are now compiled with
sflag. RegExp objects getsadded if not already present. Thegandyflags continue to be stripped (inappropriate fortest()calls).Changes
evalRunner.tsline ~156: Addedsflag to regex compilation (both string and RegExp paths)evalRunner.tsline ~256: Addedif (metrics?.error) result.passed = falseafter error attachmentTest results
Plus all existing eval tests continue to pass (4/4).
Fixes #13137
Fixes #13138