feat(harness-bench): bind any registered harness by name (ACP + community plugins)

[PattaraS]

Related issue

N/A

Summary

Tests the bench's "plugs in with no bench edit" promise against two new harnesses
and closes the gap it exposed:

• acp — the in-repo generic ACP harness (feat(acp): generic ACP harness + Omnigent-tool MCP bridge for all ACP harnesses #2152), ACP_SUBPROCESS.
• rovo / rovo-cli — an external community plugin (omnigent-rovo),
  discovered via the omnigent.community.harness entry point; ACP-based, and it
  declares no capabilities entry (only a harness module + install spec).

Both are registered in the omnigent registry, but resolve_profile("acp") and
resolve_profile("rovo") KeyError'd — the bench only knew official profiles
(4 SDK harnesses + auto-derived NATIVE_TUI) or a dotted module:attr reference.
So --harness acp / --harness rovo couldn't run.

Fix: a registry fallback in resolve_profile. After the official + reference
checks, _registry_profile derives a BenchProfile for any harness in the
omnigent registry:

• resolves aliases (rovo -> rovo-cli),
• keys off harness_modules() so it covers plugins with no capabilities entry,
• maps integration_mode -> transport family (SDK / CLI / ACP subprocess ->
  the sdk-inproc family = existing drivers; NATIVE_TUI -> native-tui),
• skip-gates on the harness's install-spec binary when present (rovo -> acli).

No new transport driver. An ACP harness registers as an omnigent agent
(config.harness=acp:<slug>) and runs on the existing SDK-wrap drivers
(sdk-inproc / full-server) — the same server+runner+session path as
claude-sdk/codex/pi. Both harnesses are OWN_AUTH, so they run only where their
vendor binary is installed + authenticated, and skip cleanly otherwise.
tool_calling / policy_deny stay · for ACP (the agent runs its own tools and
gates via session/request_permission, not a server-dispatched builtin) — the
same documented observation gap as native harnesses.

Test Plan

• Offline: pytest tests/harness_bench/ -> 71 passed / 18 skipped; ruff clean.
  New tests: resolve_profile binds acp (sdk-inproc) and rovo/rovo-cli
  (alias resolves, cli_binary=acli); an unknown name still KeyErrors; the
  plugin cases pytest.skip when omnigent-rovo isn't installed (so CI without
  the plugin passes).
• Bind check (offline render, no creds): --harness acp --harness rovo --no-live
  renders both rows, no KeyError. acp declares streaming/interrupt (✓); rovo has
  no caps entry so those are ? (honest UNKNOWN).
• Live skip-gate (this host has neither an ACP binary nor acli):
  --harness rovo --profile oss -> rovo-cli skipped: 'acli' CLI is not on PATH
  — the bind + install-spec skip-gate proven end-to-end.

Demo

$ python -m tests.harness_bench --harness rovo --profile oss
[rovo-cli] skipped: 'acli' CLI is not on PATH
rovo-cli [full-server]   ·   ·   ·   ·   ·   ·
Skipped harnesses:
- `rovo-cli`: 'acli' CLI is not on PATH

The bench bound to an external entry-point plugin by name, resolved its vendor
binary, and skipped honestly — zero per-harness bench edits.

Type of change

• Bug fix
• Feature
• UI / frontend change
• Refactor / chore
• Docs
• Test / CI
• Breaking change

Test coverage

• Unit tests added / updated
• Integration tests added / updated
• E2E tests added / updated
• Manual verification completed
• Existing tests cover this change
• Not applicable

Coverage notes

Profile resolution for acp + the rovo plugin (name + alias + skip-gate) is
covered offline; the plugin tests skip when omnigent-rovo isn't installed. The
live turn path for an ACP harness needs a vendor-authed ACP binary on the host
(not available in CI), so it was verified by hand to the skip-gate boundary; a
full live turn is host-dependent.

Changelog

The harness bench can now probe any registered harness by name — including the
generic ACP harness and community entry-point plugins — with no bench edits.

[omnigent-ci]

Polly AI Review

Blocking issues

NATIVE_SERVER harnesses silently mis-bind to the SDK driver family. The new _INTEGRATION_MODE_TRANSPORT map covers SDK_IN_PROCESS, CLI_SUBPROCESS, ACP_SUBPROCESS, and NATIVE_TUI, but omits IntegrationMode.NATIVE_SERVER. Since _registry_profile does _INTEGRATION_MODE_TRANSPORT.get(mode, "sdk-inproc"), any registered native-server harness falls through to the "sdk-inproc" default. This is not hypothetical: opencode-native is registered as NATIVE_SERVER, and with this PR resolve_profile("opencode-native") now returns a profile with transport="sdk-inproc" and cli_binary=None — i.e. it binds a runner-owned vendor-server harness to the wrong driver family and drops the native binary skip-gate (opencode). Verified locally against the applied diff:

opencode-native -> sdk-inproc  cli: None

Before this PR that name KeyError'd (unrunnable, which is at least honest). Now the bench's "bind any registered harness by name" path returns a profile that would attempt to run a native-server harness on the SDK-wrap drivers. The fix mirrors the NATIVE_TUI case: map NATIVE_SERVER explicitly (either to a native transport or, if the bench can't run it yet, to a sentinel that skip-gates rather than silently degrading to sdk-inproc). The bare-else "default to SDK family" is only safe for genuinely unmodeled modes (plugins with no caps) — it should not swallow a modeled mode the map simply forgot.

Security vulnerabilities

None. No new deps/lockfile/extras changes; no new network, deserialization, or auth-boundary surface. Both new harnesses remain OWN_AUTH and skip when their vendor binary/login is absent.

Non-blocking notes

• _registry_cli_binary uses getattr(spec, "binary", None), but HarnessInstallSpec.binary is a required field — the defensive getattr is harmless but slightly obscures that spec is either a real spec or None (already guarded). Minor.
• The two plugin/acp tests pytest.skip when the harness/plugin isn't present, so in a build without omnigent-rovo (and without acp registered) the positive registry-fallback path is entirely unexercised in CI. The existing test_resolve...unknown KeyError test still runs, but consider adding a fixture/monkeypatched fake registered harness so the fallback's happy path (name + alias + _registry_cli_binary) has non-skipped coverage independent of optional plugins.
• Consider a test asserting that a NATIVE_SERVER-mode harness resolves to the intended transport (this would have caught the blocking issue above).

Summary

Clean, well-scoped change that closes a real gap — the registry fallback correctly makes acp and entry-point plugins like rovo/rovo-cli resolvable by name/alias with no bench edits, and the diff applies and existing tests pass. The one substantive concern is that the integration_mode → transport map omits NATIVE_SERVER, so a registered native-server harness (opencode-native) silently binds to the sdk-inproc driver via the "sdk-inproc" default instead of failing loudly or routing correctly; that should be handled explicitly before merge. Test coverage of the new positive path is skip-gated on optional plugins and would benefit from a plugin-independent case.

---

_{Automated review by Polly · workflow run}

[PattaraS]

Thanks - the blocking issue was a real bug; fixed in acef57f.

Blocking: NATIVE_SERVER silent mis-bind. Fixed. _registry_profile now distinguishes an unmodeled mode (no capabilities entry, e.g. a plain plugin like rovo) from a modeled mode with no bench driver. A modeled mode not in the transport map (NATIVE_SERVER / opencode-native) returns None, so resolve_profile raises a clean KeyError instead of degrading to sdk-inproc. The bare "default to SDK family" now only applies when there are no caps at all. Verified resolve_profile("opencode-native") KeyErrors again. Added test_registry_refuses_native_server_mode.

Non-blocking: skip-gated positive-path coverage. Added test_registry_profile_happy_path_no_plugin — a monkeypatched fake registered CLI harness exercising name+alias resolution, the integration_mode->sdk-inproc mapping, and the install-spec skip-gate, independent of whether omnigent-rovo is installed.

Non-blocking: getattr(spec, "binary"). Kept as-is; spec is already null-guarded and getattr is harmless, but I left a note that it's defensive.

Also fixed two things a live rovo run surfaced (no acli present):

• The ACP-session / vendor-login failure now classifies as an infra SKIP, not a false !!✓>✗ DRIFT (an own-auth harness with no vendor login is an environment gap).
• Registry profiles no longer stamp a databricks-* placeholder model on own-auth harnesses (the runner drops the gateway model for them); own-auth/capless -> empty model.

[PattaraS]

Live-verified end to end: the registry fallback now drives a real ACP agent turn.

Configured a qwen ACP agent (qwen --acp) as acp:qwenacp and ran the bench with no bench edits — it bound, registered, provisioned the full server, spawned the ACP subprocess, and drove real turns:

acp:qwenacp [full-server]   Basic turn ✓   Streaming ✓   Tool calling ·   Policy DENY ·   Model override ✓   Interrupt ✓

• Basic turn ✓ (marker echoed), Streaming ✓ (39 token-level deltas), Interrupt ✓ (cancelled after 261 chars streamed), Model override ✓.
• Tool calling / Policy DENY · — the documented ACP gap (the agent runs its own tools and gates via session/request_permission, not a server-dispatched builtin), reported as a clean SKIP.

Also validated the honest-skip path against two other ACP harnesses whose vendor-side setup wasn't satisfied: a community plugin blocked on an interactive vendor onboarding prompt, and qwen before its own auth was configured — both reported specific, diagnosed SKIPs rather than false verdicts or hangs. That's the seamlessness guarantee working: bind by name, run on the existing drivers, and never lie about a capability the environment simply couldn't exercise.

[PattaraS]

/review

[omnigent-ci]

Polly AI Review

Summary

Solid, well-scoped change. It adds a registry fallback to resolve_profile so any harness in the omnigent registry (in-repo acp, acp:<slug> ids, and entry-point community plugins like rovo-cli) resolves to a BenchProfile with no per-harness bench edit, plus a name-sanitization fix for colon-bearing harness ids in the full-server agent name and two new infra-skip markers for own-auth ACP harnesses. The design is clean: it keys off harness_modules() (so caps-less plugins are covered), maps integration_mode → transport family, refuses MODELED modes the bench can't drive (NATIVE_SERVER → None → clean KeyError) rather than mis-binding, and reuses _native_profile for the native-tui path so the two derivations agree. I verified resolve_profile("acp"), acp:qwen, the acp: empty-slug refusal, the unknown-name KeyError, and the family-gate acceptance of the placeholder model all behave as claimed; the full bench suite and ruff pass. No blocking or security issues found. Changes are entirely within tests/harness_bench/ with no production or user-facing surface, so no visual demonstration is warranted.

Blocking issues

None.

Security vulnerabilities

None. No new dependencies, no lockfile or extras changes, no auth-boundary changes. Own-auth harnesses continue to run only where their vendor binary is present and skip-gate otherwise.

Non-blocking notes

• Broad infra markers. The new _INFRA_ERROR_MARKERS entry "are logged in" (and to a lesser degree "ACP session") is a fairly generic substring. Because these only downgrade a failed turn to SKIPPED for own-auth harnesses, the blast radius is small, but a genuine capability failure whose error text happens to contain "are logged in" would be misclassified as an environment skip and hidden as drift. Consider anchoring on the ACP-specific markers (AcpProcessExited, ACP subprocess) and a more specific login phrase (e.g. the full "Ensure `acli` is installed and you are logged in" shape) to reduce false positives.

• model_override on own-auth ACP. For an ACP harness the profile stamps the inert databricks-* placeholder that the runner drops before spawn; if a live turn ever runs, the model_override probe's live half could complete on the agent's own model and read as SUPPORTED even though the override was ignored. In practice acp/rovo aren't in model_env_keys() and is_native_harness("acp") is False, so declared["model_override"] stays UNKNOWN and won't raise drift — but the live SUPPORTED verdict is slightly optimistic. Worth a one-line note in the probe or profile if a future ACP harness ever routes gateway models.

• getattr(spec, "binary", None) in _registry_cli_binary is defensive against a required dataclass field (HarnessInstallSpec.binary); harmless, just slightly looser than necessary.

• Test coverage is good: the monkeypatched test_registry_profile_happy_path_no_plugin and test_registry_refuses_native_server_mode cover the fallback's positive path and the NATIVE_SERVER refusal without depending on the optional plugin, so CI without omnigent-rovo still exercises the core logic.

---

_{Automated review by Polly · workflow run}

[github-actions]

🏷️ Doc impact: no-doc-update

Changes are confined to the internal harness_bench test tooling (profile resolution, skip-gating, tests) with no user-facing surface, integration, or built-in policy change.

_{Auto-classified on merge. Set the label manually before merging to override. · run}