docs: update external harness token capture guide

[ananthsub]

Explains how to collect trainable rollouts from an external agent harness that drives its own model calls and returns text without token ids.

How it works

flowchart LR
    H[External agent harness] -->|model calls| M[Gym model server]
    M -->|exact token ids and log probabilities| S[(Capture storage)]
    H -->|finished rollout| C[Rollout collector]
    S --> C
    C -->|rebuild verified call chain| R[Token-bearing response]
    R --> T[Trainer]

Loading

Gym records token ids while they are still available in the model server. It links each call to its rollout, rebuilds the verified chain that produced the scored response, and masks incomplete or ambiguous rollouts instead of guessing.

Summary

• Shows when external-harness token capture is needed and when native response token ids should be used instead.
• Walks through model-server, capture-store, agent opt-in, sampling, tokenizer, and tool-parser configuration.
• Explains the rebuilt response, first-run health metrics, terminal attribution, masking, and capture cleanup.
• Documents optional verified prefix supply and its backend requirements, including the stock vLLM limitation.
• Covers both Gym's node-local file store and framework-owned capture transports without exposing internal storage or matching implementation details.
• Provides the consumer lifecycle and rollout-id requirements needed by custom training loops.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-ananthsub-tokidcap-docs-extras.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/tutorials/training-tutorials/external-agent-harnesses

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ananthsub]

/claude review

[claude]

SHIP — LGTM, no reliability concerns.

Docs-and-tests-only change to the token-id-capture external-harness guide plus two new tests. No runtime library code touched; no verifier, scorer, async, or public-API surface affected.

Verified against the implementation:

• Doc protocol signatures (TokenSink.put/mark_incomplete/close, TokenSource.seal/drop/close, LineageStore.resolve/record/close) match nemo_gym/token_id_capture/protocols.py exactly, including the async mark_incomplete and the seal_id/version conditional-drop contract.
• The documented finalize_rollout_token_capture(result, source) / retire_rollout_token_capture(rollout_id, source, built) call shape matches delivery.py.
• The referenced overlay responses_api_models/vllm_model/configs/vllm_model_supply_prefix.yaml exists.
• New tests assert real behavior: test_file_lineage_concurrent_idempotent_publication_stays_unique exercises 4 concurrent FileLineageStore writers and asserts a unique resolve; test_an_external_endpoint_round_trips_the_sink_and_source_protocols drives a real request through the server and checks seal → entries → conditional drop → empty re-seal. Test API usage (build_source, FileLineageStore.record/resolve, TokenCaptureSnapshot) matches current signatures.

No BLOCKER/RISK/NOTE findings.

[ananthsub]

/claude review

[claude]

SHIP — LGTM, no reliability concerns.

Docs-plus-test change to token_id_capture. Reviewed both changed files against the diff and the trusted base implementation.

• Docs (external-agent-harnesses.mdx): the new protocol descriptions match source — TokenSink.mark_incomplete/close are async in protocols.py, TokenSource.freeze/drop(*, snapshot_id, version) -> bool and LineageStore.resolve/record signatures are accurate, and the rebuild_response/paired-source startup requirement matches config.py validation. No factual drift.
• Tests: both new tests assert real behavior, not pass-throughs.
  • test_file_lineage_concurrent_idempotent_publication_stays_unique exercises FileLineageStore's lock + idempotent record (byte-identical payload is a no-op) under concurrent writers, then confirms resolve returns the single unique parent — matches _record/_resolve.
  • test_an_external_endpoint_round_trips_the_sink_and_source_protocols drives a configured sink/source through the server, freezes via build_source(), and verifies conditional drop retires the exact snapshot — matches TokenCaptureSnapshot and build_source.
  • Remaining changes are comment rewrites only.

No async, API-compat, or config-convention concerns. (pytest execution was gated by sandbox permissions; verification was static against the base implementation.)

[ananthsub]

/claude review

[ananthsub]

/claude review

[claude]

SHIP — LGTM, no reliability concerns.

Docs-only rewrite of external-agent-harnesses.mdx. No runtime, verifier, scorer, or async code touched — zero blast radius on training/eval correctness.

Verified every technical claim against the trusted base:

• Config keys (delta_records, max_mask_fraction, mask_fraction_min_samples, all_agents, rebuild_response, return_token_id_information, sampling_overrides) all exist in nemo_gym/token_id_capture/config.py and responses_api_models/vllm_model/app.py.
• File paths (vllm_model_for_training.yaml, vllm_model_supply_prefix.yaml) exist.
• API symbols (finalize_rollout_token_capture, retire_rollout_token_capture, run_conformance) exist in delivery.py/conformance.py.
• Metric field names (n_calls, chains, delivered_fraction, quarantined_calls, empty_generation_calls, unresolved_parent_calls, mask_sample, terminal_attribution.chain = delivered) match consumer.py/builder.py.
• Prefix-supply incompatibility with use_completions_api and is_responses_native matches the guards in app.py:228-231.
• YAML nesting for the per-agent opt-in and run-wide capture block matches the schema.

No factual drift found.

[ananthsub]

/ok to test 654b421