feat(reborn): observability seams — trajectory observer + LLM provider injection

[pranavraja99]

Two small reborn observability seams that let an external host (nearai-bench) drive and observe a reborn agent run with parity to the legacy agent path.

① Trajectory observer hook

pub trait RebornTrajectoryObserver + RebornRuntimeInput::with_trajectory_observer(...). LocalDevCapabilityIo forwards per-step capability input (on_capability_input: call_id, tool, args) and result (on_capability_result) at the points it already records display previews — no new collection, no-op when unset, best-effort. Lets a consumer reconstruct what the agent did (tool calls + results), which build_reborn_runtime otherwise keeps in internal stores.

② Host LLM-provider injection

ResolvedRebornLlm::with_provider(Arc<dyn LlmProvider>) + build_llm_gateway honors it. Normally the runtime builds its provider from config with no interception point; this lets a host pass in a pre-wrapped provider (e.g. an instrumented wrapper that counts tokens/cost and captures reasoning). This is the only viable capture path for reborn — its model calls run in spawned worker tasks a per-task tracing subscriber can't reach, so wrapping the provider is required.

Consumer

nearai-bench uses ① to render the trajectory and ② to capture reasoning/tokens/cost when running its suite against reborn.

Related

• feat(reborn): extra-capabilities seam — register host-supplied tools as reborn capabilities #4671 (extra-capabilities seam — registering host-supplied tools) is stacked on this PR.
• Needs a rebase onto main (currently conflicting) — flagging rather than resolving blind.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[Copilot]

Pull request overview

This PR adds three “host seams” to the Reborn composition/runtime so an external harness (e.g., nearai-bench) can (a) observe tool-call trajectories in real time, (b) inject a wrapped LLM provider implementation, and (c) register extra externally-dispatched tools as loop capabilities.

Changes:

• Introduces a trajectory observer hook plumbed through loop-support ports and local-dev capability IO.
• Adds ResolvedRebornLlm::with_provider(...) and updates gateway construction to honor a caller-supplied provider.
• Adds a bench-oriented “extra tools” decorator/port that exposes external tools as bench.<name> capabilities.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
crates/ironclaw_reborn_composition/src/trajectory_observer.rs	Re-exports the loop-support observer trait under a Reborn-facing name.
crates/ironclaw_reborn_composition/src/runtime/local_dev.rs	Threads the observer through local-dev capability wiring and forwards result events.
crates/ironclaw_reborn_composition/src/runtime/local_dev/tests.rs	Updates local-dev wiring call sites for the new observer parameter.
crates/ironclaw_reborn_composition/src/runtime.rs	Plumbs trajectory_observer/extra_tools into runtime build; honors LLM provider override.
crates/ironclaw_reborn_composition/src/runtime_input.rs	Adds runtime input fields + builder methods; adds LLM provider override field/method.
crates/ironclaw_reborn_composition/src/lib.rs	Exposes new public API items (observer + extra-tools types).
crates/ironclaw_reborn_composition/src/legacy_tools.rs	Implements the extra-tools decorator/port and dispatch surface.
crates/ironclaw_loop_support/src/lib.rs	Re-exports the new observer trait.
crates/ironclaw_loop_support/src/capability_port.rs	Defines CapabilityTrajectoryObserver and forwards input events from the host port.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[serrrfirat]

Review Summary

Verdict: REQUEST_CHANGES

Critical Issues: 2 High-severity test coverage gaps

Stats by Category

• Security: 0 findings (✅ clean)
• Bugs: 0 findings (✅ clean)
• Performance: 0 findings (✅ clean)
• Tests: 2 findings (❌ blocking)
• Conventions: 1 finding (Low severity, non-blocking)
• Maintainability: 0 findings (✅ clean)

---

Blocking Issues

1. Missing trajectory observer integration tests

Severity: High | Confidence: 85% | Category: missing-integration-test

The PR adds CapabilityTrajectoryObserver trait with on_capability_input and on_capability_result callbacks, wired into multiple call sites (HostRuntimeLoopCapabilityPort, LocalDevCapabilityIo). All test changes pass None for the observer — no tests verify the callbacks are invoked.

AGENTS.md violation:

"Test through the caller, not just the helper. When a predicate/classifier/transform helper gates a side effect...add a test that drives the actual call site."

The trajectory observer is a side effect hook that downstream consumers (nearai-bench) rely on for trajectory capture. Without integration tests, callback invocation regressions would go undetected.

Required fix: Add integration test(s) that:

1. Instantiate a mock CapabilityTrajectoryObserver (counting calls, capturing args)
2. Wire it via HostRuntimeLoopCapabilityPortFactory::with_trajectory_observer(Some(mock))
3. Execute a capability through the port
4. Assert on_capability_input called with correct call_id, capability_id, arguments
5. Assert on_capability_result called with matching data

2. Missing LLM provider override test

Severity: High | Confidence: 80% | Category: missing-integration-test

The PR adds ResolvedRebornLlm::with_provider(...) and build_llm_gateway honors provider_override (lines 2338-2344 in runtime.rs). No test verifies this bypass works.

If the override logic regresses, instrumented providers (token counting, reasoning capture) would silently fail to be used by nearai-bench.

Required fix: Add test that:

1. Creates a mock LlmProvider
2. Builds ResolvedRebornLlm::with_provider(mock_provider)
3. Builds runtime/gateway
4. Verifies the mock provider is invoked (not a fresh provider from config)

---

Non-Blocking Observations

Conventions (Low)

New public trait without usage example: CapabilityTrajectoryObserver has good doc comments but no concrete implementation example in tests. Similar observer traits in the repo have test mocks showing the pattern. Consider adding a minimal example or pointing to tests/support/.

---

Approval Bar

Per strict reviewer contract, APPROVE only when ALL of:

• ✅ No security issue (confidence≥50)
• ✅ No correctness bug (confidence≥60)
• ✅ No file over 1000 lines
• ✅ No hardcoded strings needing constants
• ✅ No silent wildcard drops
• ✅ No breaking API without migration
• ❌ FAIL: No reachable failure path missing test coverage (confidence≥65)

The trajectory observer callbacks and provider override are reachable production paths that downstream code depends on. Integration tests are required.

[pranavraja99]

Addressed both blocking findings in 662600a — thanks @serrrfirat.

1. Trajectory observer integration tests — added tests that drive the real call sites with a recording CapabilityTrajectoryObserver:

• invoke_capability_forwards_resolved_input_to_trajectory_observer (capability_port.rs): wires the observer via HostRuntimeLoopCapabilityPortFactory::with_trajectory_observer, invokes a capability, and asserts on_capability_input fires with the resolved capability id + tool-call arguments.
• capability_io_forwards_input_and_result_to_trajectory_observer (local_dev/tests.rs): drives register_provider_tool_call_input + write_capability_result and asserts both on_capability_input and on_capability_result fire and correlate by input ref.

2. LLM provider override test — build_llm_gateway_drives_provider_override_not_config injects a counting mock LlmProvider via ResolvedRebornLlm::with_provider, points config at a dead endpoint, and asserts the gateway returns the mock's sentinel response (proving the override is driven, not a config-built chain).

Also fixed pre-existing breakage this surfaced: 5 LocalDevLoopCapabilityPortFactory test initializers were missing the trajectory_observer field this PR added, so the composition crate's tests didn't compile under --features root-llm-provider. That's why the override seam had no coverage — the test target couldn't build. Fixed.

Results: loop_support 301 passed; composition (--features root-llm-provider) 520 passed; 0 failures.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.