Repository navigation
fix: improve Jev resolver timing and cost diagnostics - #16
Merged
Merged
Conversation
MrGTV-love
force-pushed
the
fm/fm-jev-resolver-observability
branch
from
October 5, 2026 16:59
24eb33c to
fbc86a8
Compare
…me_total for API timing
…g-lib clock helpers
…RESULT jq in bin/fm-dispatch-resolve.sh printed the raw response .model in the legacy `model:` line, so a crafted model string could print the key or brief text. The telemetry returned_model check did not cover that line. Fix: the numeric Jev model-ID check is now defined once, as JEV_MODEL_ID_JQ (def jev_model_id). Both jq programs use it: the telemetry usage line (returned_model) and the resolution result (model: ($r.model | jev_model_id)). Valid ids such as jev-1.13.0 still print. Unsafe ids print as `-` (unknown). Routing, usage, cost and exit codes do not change. Test: tests/fm-dispatch-resolve.test.sh has a new successful-quota case that reuses the sensitive-model fixture. I ran it before the fix and it failed: the `model:` line contained the key and private-brief-marker-4417. After the fix the full resolver suite passes, both with default bash and with RESOLVER_BASH=/bin/bash (Bash 3.2). The header comment, docs/configuration.md and docs/verification/dispatch-resolve.md now say the same thing. shellcheck reports only the old SC1091 info notes. tests/fm-lint.test.sh was still running when this result was due. Its finished cases all passed, and I saw no `not ok` lines, but I did not see the full run finish
…uthoritative ownership
… privacy verification
MrGTV-love
force-pushed
the
fm/fm-jev-resolver-observability
branch
from
October 5, 2026 17:26
fbc86a8 to
e247ff7
Compare
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
do all of hte proposed fork fixes (2026-09-29, answering the Jev value report data/fm-jev-value-scout/report.md section 5, 'Fork follow-up shortlist'). Goal stated with it: "The user is looking to maximize performance while minimizing cost. Jev seems like a massive lever." Jev is typesafe.ai's System One model; firstmate calls it only from bin/fm-dispatch-resolve.sh (typed dispatch resolution). Work lands on the fork MrGTV-love/firstmate only; never push to, open, or comment on anything in kunchenguid/firstmate.
This task covers shortlist items 1 and 6:
Report evidence: on macOS system Bash 3.2 (no EPOCHREALTIME) the resolver printed latency_ms: 0 for a ~211 ms API call and 1,000 ms elsewhere; the quota snapshot (~3.4-3.7 s), not Jev (~0.2 s), dominates the ~4 s resolver wall time.
I am concerned that prior evaluations of jev were not constructed properly (other models), so I would instruct the workstreams not to anchor on them
the Jev workstream is also important ... if we implement Jev, it could really helpful to our current work (improve speed, quality and reduce costs)
What Changed
Risk Assessment
✅ Low: The changes are bounded to diagnostic timing and cost evidence, preserve routing and shared epoch-clock compatibility, and apply the approved model-output privacy fix consistently without introducing a substantiated material defect.
Testing
The focused resolver behavior suite and live CLI scenarios passed using real curl, real quota-axi, Bash 3.2, and disposable Bash 5.3. Evidence shows separate API/quota/total timing, correct input-token cost, secret-safe model and error output, and preserved routing guards. The initial transport-adapter setup was corrected before successful validation. API responses were disposable fixtures, not a Jev-quality evaluation; all temporary setup was removed.
Evidence: Live resolver CLI transcripts and transport observations
Source: Live resolver CLI transcripts and transport observations
Evidence: Real Bash clock paths, shared epoch, and no-Perl evidence
Source: Real Bash clock paths, shared epoch, and no-Perl evidence
Evidence: Real quota-axi output from disposable snapshot data
Source: Real quota-axi output from disposable snapshot data
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
/bin/bash tests/fm-dispatch-resolve.test.shwith inherited resolver/home overrides cleared and TMPDIR inside the disposable worktree setup; passed. Its Bash fast-path check skipped because PATH supplied Bash 3.2; the separate live Bash 5 check covered that path.Executed/bin/bash bin/fm-dispatch-resolve.sh <disposable brief> --project disposable-resolver-validationacross 20 isolated invocations using real curl, a disposable loopback HTTPS service, and real quota-axi withQUOTA_AXI_SNAPSHOT.Compared emitted api_ms with real curl%{time_total}; checked latency alias, stage accounting, selected profile, input-only cost, returned model, request boundaries, and child-environment secret exclusion.Drove missing, zero, negative, and fractional usage; failed and unavailable quota; hostile response models; HTTP 429/500; malformed responses; real timeout and connection refusal; unavailable curl; absent opt-in; and never-send suppression.Executed disposable copies of the base-commit resolver and timing library against the same local services; reproduced sensitive model text in successful-quota output and latency_ms=0.Downloaded official GNU Bash 5.2.37 into the worktree, built with./configure --without-bash-malloc --disable-nlsandmake -j4, then exercised real timing helpers under Bash 3.2 and Bash 5.2.Executed timing start/record/render with subsecond delays, shared cross-shell epoch origin, blocked Perl on the builtin path, unavailable Perl fallback, and disabled recording.Saved CLI transcripts and structured runtime evidence; stopped the local HTTPS server and removed all disposable fixtures, downloaded source, and build outputs.✅ No issues found.
/bin/bash tests/fm-dispatch-resolve.test.shwith isolated HOME and worktree-local TMPDIR — passed; its Bash-5-only check was covered separately using a disposable source-built Bash 5.3../configure --without-bash-malloc --disable-nls && make -j4— built Bash 5.3 entirely inside the disposable worktree lab./bin/bash bin/fm-dispatch-resolve.sh <disposable brief> --project isolated-live-validation— drove the unchanged resolver through real curl against a local trusted TLS endpoint and real quota-axi against its supported QUOTA_AXI_SNAPSHOT fixture.Exercised successful dispatch, missing and invalid usage, five adversarial response-model values, quota snapshot failure, HTTP 503 with sensitive response text, malformed HTTP-200 response, and an actual curl timeout.Exercised low-confidence ambiguity, captain-approval escalation, absent-key silence, never-send rejection, and minimal no-rules output.Executed the timing helpers under real Bash 3.2 and Bash 5.3; verified decimal conversion, shared epoch propagation, and no Perl execution across the complete Bash-5 resolver.Published CLI transcripts, clock evidence, and actual quota-axi output; stopped the TLS fixture and removed the disposable lab, downloaded sources, build outputs, keys, configuration, and caches.✅ **Document** - passed
✅ No issues found.
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.