Repository navigation
feat: add vendored hyper-jev skill and integration starter - #41
Merged
Merged
Conversation
… and runner documentation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
This is an excellent skill that disler created for Jev: https://github.com/disler/ten-levels-of-jev/blob/main/.claude/skills/hyper-jev/SKILL.md
Jev work is best guided by a skill. We can defer if Kun has added one upstream to firstmate
Context: Main checked upstream kunchenguid/firstmate main: it has no Jev skill (only bin/fm-jev-mem-guard.* and docs/jev-guards.md), so there is nothing to defer to. Standing captain rules: use the reference he provides as-is and change it only where our codebase forces it; Jev calls go direct to TypeSafe (lower latency), never via OpenRouter; the TypeSafe key lives only in the primary home .env as TYPESAFE_API_KEY.
What Changed
TYPESAFE_API_KEYonly from the primary home.env, overriding conflicting upstream guidance.SKILL.mdsuite mapping.Risk Assessment
🚨 High: The vendored skill contradicts the authoritative direct-TypeSafe and primary-home-only credential requirements, so it should not merge without resolving those conflicts; the remaining findings are bounded offline correctness and test-quality issues.
Testing
Live Claude skill discovery and overlay interpretation, isolated credential handling, bundled CLI execution, and adversarial changed-file selection passed; the audience check and upstream byte-contract comparison also passed. Initial socket-length and disabled-discovery setup issues were corrected before re-driving the scenarios. CLI transcripts and comparison evidence were retained, and disposable state was removed. No authenticated Jev API request was made; the offline demo is not evidence of live model accuracy.
Evidence: Live project skill discovery and complete Claude transcript
Source: Live project skill discovery and complete Claude transcript
Evidence: Claude response applying firstmate provider and credential rules
Source: Claude response applying firstmate provider and credential rules
Evidence: Named skill discovery and isolated credential boundary
Source: Named skill discovery and isolated credential boundary
Evidence: Bundled offline support-routing demonstration
Source: Bundled offline support-routing demonstration
Evidence: Bundled CLI refusal without credentials
Source: Bundled CLI refusal without credentials
Evidence: Changed-file selector acceptance and adversarial refusal
Source: Changed-file selector acceptance and adversarial refusal
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
.agents/skills/hyper-jev/SKILL.md:21- The authoritative criterion says “Jev calls go direct to TypeSafe (lower latency), never via OpenRouter,” but the added instruction says “Prefer TYPESAFE_API_KEY, otherwise OPENROUTER_API_KEY.” In production, with only an OpenRouter key, the documented new JevClient() path selects OpenRouter and sends the complete decision state there; explicit provider and JEV_BACKEND overrides also select it even when TypeSafe credentials exist. Related sites under .agents/skills/hyper-jev/: templates/starter/src/core/client.ts:16 (provider union), :19 (endpoint), :24 (model), :29 (key), :113 (explicit provider acceptance), :132 (custom endpoint override), :240 (selection precedence), :242 (OpenRouter override), :246 (automatic selection), :194 (transport); cookbook/01-structure-and-setup.md:60 (fallback), :63 (override), :65 (guidance); cookbook/02-input-output.md:10 (endpoint), :40 (request adaptation); cookbook/04-client-operations.md:3 (interface), :13 (selection), :18 (fallback policy), :23 (example), :41 (option); cookbook/05-production.md:9 (provider guidance); cookbook/README.md:16 (transport); cookbook/use-cases/README.md:11 (live setup); templates/starter/README.md:16 and :18 (live setup); templates/starter/.env.example:3 (credential); templates/starter/tests/client.test.ts:58, :59, :89, :119, :131 and :148 (selection and transport expectations). This alternate live-provider component is forbidden, not required by the intent. Remove OpenRouter selection/transport and narrow live endpoint configuration to direct TypeSafe, updating its instructions and tests together; preserve the remaining upstream material..agents/skills/hyper-jev/templates/starter/README.md:16- The criterion says “the TypeSafe key lives only in the primary home .env as TYPESAFE_API_KEY,” but the added setup instructs a destination application to use a “private local .env” with node --env-file=.env src/example.ts. The setup cookbook explicitly creates a separate destination service, then gives the same service-local credential command. Following that path provisions a second credential location rather than using the primary home's existing key. Related sites under .agents/skills/hyper-jev/: cookbook/01-structure-and-setup.md:44–48 (destination creation), :55 (secret-manager provisioning), :67–73 (local environment-file setup); templates/starter/.env.example:2 (local key slot); SKILL.md:34 (initialization guidance lacks the required primary-home source); cookbook/use-cases/README.md:13 (credential loading guidance). Adapt these instructions to reference the primary home's existing .env without copying or provisioning the key in the destination service. The required primary-home credential boundary is currently absent from the skill..agents/skills/hyper-jev/templates/starter/src/core/mock.ts:147- The offline backend silently loses valid proto keys because it constructs ordinary objects and assigns dynamic keys through bracket notation. For example, a question map parsed from {"proto":{"type":"noul","instructions":"Is this a greeting?"}} passes validation, but assignment changes the answers object's prototype; serialization and result cloning return answers: {} without rejecting the call. The same invariant fails at mock.ts:110–111 for Choice probabilities: an own proto criterion loses its probability. All question-type writes at mock.ts:149–151 must preserve own keys. Downstream affected sites are client.ts:177–182 (returns the mock result without response validation), levels/level09/wikiracing.ts:26 and :37 (dynamic titles and returned probabilities), and levels/level09/hierarchical-classify.ts:50 and :69–70 (dynamic category names and probability multiplication, which can yield NaN from the inherited value). Paths are relative to .agents/skills/hyper-jev/templates/starter/src/. Construct both dynamic maps with own-key-safe operations, such as Object.fromEntries or null-prototype objects, at the shared mock boundary..agents/skills/hyper-jev/templates/starter/tests/starter-regressions.test.ts:42- Both assertions at lines 42–43 merely require the phrasediff_shapein instruction strings returned by buildReviewRubric. They still pass if judgePullRequest sends different state fields, stops sending these questions, or otherwise breaks the claimed alignment. This is the prohibited instruction-content-only test, not a check of the final emitted interface. Remove this wording-only case, or exercise judgePullRequest and inspect its actual outgoing request as an explicitly owned generated prompt/state contract. Update the corresponding claim in .agents/skills/hyper-jev/cookbook/use-cases/10-dynamic-questions.md:47.🔧 Fix applied.
2 warnings still open:
.agents/skills/hyper-jev/templates/starter/src/core/mock.ts:147- The offline backend silently loses valid proto keys because it constructs ordinary objects and assigns dynamic keys through bracket notation. For example, a question map parsed from {"proto":{"type":"noul","instructions":"Is this a greeting?"}} passes validation, but assignment changes the answers object's prototype; serialization and result cloning return answers: {} without rejecting the call. The same invariant fails at mock.ts:110–111 for Choice probabilities: an own proto criterion loses its probability. All question-type writes at mock.ts:149–151 must preserve own keys. Downstream affected sites are client.ts:177–182 (returns the mock result without response validation), levels/level09/wikiracing.ts:26 and :37 (dynamic titles and returned probabilities), and levels/level09/hierarchical-classify.ts:50 and :69–70 (dynamic category names and probability multiplication, which can yield NaN from the inherited value). Paths are relative to .agents/skills/hyper-jev/templates/starter/src/. Construct both dynamic maps with own-key-safe operations, such as Object.fromEntries or null-prototype objects, at the shared mock boundary..agents/skills/hyper-jev/templates/starter/tests/starter-regressions.test.ts:42- Both assertions at lines 42–43 merely require the phrasediff_shapein instruction strings returned by buildReviewRubric. They still pass if judgePullRequest sends different state fields, stops sending these questions, or otherwise breaks the claimed alignment. This is the prohibited instruction-content-only test, not a check of the final emitted interface. Remove this wording-only case, or exercise judgePullRequest and inspect its actual outgoing request as an explicitly owned generated prompt/state contract. Update the corresponding claim in .agents/skills/hyper-jev/cookbook/use-cases/10-dynamic-questions.md:47.✅ **Test** - passed
✅ No issues found.
node --version,tmux -V,claude --help, andomp --helpconfirmed available runtime and harness interfaces.Compared all 89 vendored files, including LICENSE, byte-for-byte against the GitHub archive for upstream commit777adaf47d37ae0553220d35b2f15b3a3a063305.bin/fm-doc-audience-check.shvalidated the documentation inventory and local links.bin/fm-skill-suggest.sh --task-file <disposable-task> --format briefexercised named skill discovery with missing TypeSafe credentials and an OpenRouter-only environment.Executedfm_typesafe_key "$FM_HOME"with a synthetic credential stored only in the disposable primary home's.env; verified resolution, absence from child-process environment, and no additional files.Ran two read-only Claude primary sessions on a private, workspace-local tmux socket: one provenance-guided evaluation and one/hyper-jevinvocation using normal project skill discovery. Captured the actual responses and stopped both servers.JEV_BACKEND=mock node src/example.tsexercised the bundled CLI's documented offline demonstration.node src/example.tswith both provider credentials and backend overrides unset verified that startup refuses missing credentials rather than inventing a decision.bin/fm-test-run.sh --list --changed --base HEADin a disposable repository exercised vendor-only changes, SKILL.md selection, and refusal of unrelated unmapped source.bin/fm-lab-home.sh teardown <disposable-home>completed, followed by removal of all transient worktree fixtures.🔧 **Document** - 3 issues found → auto-fixed ✅
.agents/skills/hyper-jev/SOURCE.md:14- The provenance note says SOURCE.md is the only firstmate-added file besides LICENSE, but FIRSTMATE.md is now also present. Correcting this authoritative statement requires permission because the recorded decision permits only adding the top-of-file pointer and explicitly forbids other changes.README.md:207- The skill-layout owner claims every .agents/skills skill requires a live firstmate home and carries metadata.internal: true. The newly vendored hyper-jev skill is portable and lacks that metadata. CONTRIBUTING.md:57 repeats the stale claim. Qualify the README owner for this exception and reduce the contributor duplicate to a pointer, without modifying upstream files. These edits exceed the recorded exact-changes-only decision and require permission.bin/fm-test-run.sh:139- The runner header describes an unmapped bin/ path as the sole deliberately narrow changed-file selection case, but the added hyper-jev arm selects no suite for non-SKILL.md paths. The authoritative header should acknowledge this exception while retaining the preceding SKILL.md family-selection rule. A documentation-only correction exceeds the recorded exact-changes-only decision and requires permission.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.