feat: evidence-grade NIM discovery + all-modality cost-quality benchmark

[seonghobae]

Purpose

Implement issue #86 as a provider-neutral, evidence-grade NVIDIA NIM discovery and benchmark harness while preserving standalone operation and the modular CWL MSA boundary.

Exact integration identity

• Current exact contributor head: 26f8d8dc5634f0371fad0801056e9a3450c78bff
• Stacked base branch: PR fix(security): pin provider egress and repair the Atheris lock #96 fix/atheris-interpreter-lock
• Current live base-branch tip: 3703d0da9823b8258a0be94f1801aa5d61bfad9f
• GitHub PR metadata still records historical base snapshot 2df63e080737d5aa29d40871af99aa13411c3d76; it is not treated as the current stacked-base tip.
• GitHub reports this PR as Draft / not mergeable against the moved stack.

The base branch is independently maintained and may move. Before every mutation and acceptance decision, live GitHub PR/ref metadata is authoritative. Any check, status, review, approval, mergeability result, or synthetic merge associated with another head/base pair is historical only.

Current-head verification state

For exact contributor head 26f8d8dc5634f0371fad0801056e9a3450c78bff:

• Tests run 31182517889: success;
• Security run 31182517906: success;
• Fuzz run 31182518012: success;
• CodeRabbit commit status: success;
• Security Scan: absent on this exact head;
• SAST Semgrep: absent on this exact head;
• trusted central 100% statement/branch/public-docstring/package evidence: absent pending protected-main central prerequisites;
• fresh exact-head OpenCode, Noema, Strix, and qualifying independent non-author approval: absent.

Absent evidence is not success. The current GitHub review-thread inventory reports zero unresolved inline threads. The addressed transactional-publication finding still records test-first staging, complete-set validation, same-filesystem publication, rollback/recovery, and cleanup; its resolved state is not approval and does not substitute for post-reconciliation exact-head evidence.

The stacked base has advanced after this branch's earlier reconciliation, so no historical mergeability/check evidence is reused. After PR #96 integrates, this PR must be reconciled or retargeted from the exact integrated protected head and every required gate must be regenerated.

Implemented benchmark scope

• dynamically discover the OpenAI-compatible GET /v1/models catalog without a hard-coded authoritative inventory;
• deduplicate and provenance-hash catalog evidence;
• compute the complete discovery, all-model capability-probe, and equal-budget evaluation plan before capability egress;
• fail closed before the first capability probe when the hard request cap cannot cover the complete plan;
• probe every discovered model under deterministic bounded concurrency;
• cover chat, completions, Responses, embeddings, image, valid deterministic one-frame H.264 video, audio understanding, transcription, and speech contracts;
• classify supported, unsupported, throttled, timeout, unavailable, failed, and malformed outcomes with machine-readable reasons;
• compare every eligible direct single-model arm, route_once, bounded conduct, and an optional reviewed cheapest-worker scenario under common total token and call envelopes;
• separate reviewed actual free-to-caller access evidence from hypothetical paid pricing scenarios, never infer prices, and keep unsupported cost evidence unknown;
• record provenance, configured and observed budgets, task completion, uncertainty, paired comparisons, evidence sufficiency, and Pareto frontiers in secret-redacted JSON/CSV/Markdown artifacts;
• preserve every policy/task cell's exact step, role, agent, and model assignments in deterministic CSV evidence with JSON/CSV identity parity;
• transactionally stage, validate, enrich, and publish the complete artifact set with ordinary-failure rollback, residue cleanup, bounded crash recovery, and final-path-only success output.

Strict scoring validity boundary

A test-first buyer-visible slice repairs construct-irrelevant score inflation in the locked task set.

• RED contract head 337356117ede1b69ba1b47282038bfa0fcc9c360 required complete-answer scorers that did not yet exist.
• Numeric scorer exact_number_match version 2 accepts only one complete finite decimal response; prose, negation, units, multiple values, NaN, and infinities do not earn credit.
• Text scorer exact_text_match version 1 compares the complete NFC-normalized, whitespace-normalized, case-folded response against explicit declared alternatives; substrings and undeclared aliases do not match.
• malformed answer keys, ambiguous locked scorer contracts, and scorer identity collisions fail before provider egress;
• the supported CLI explicitly derives a private versioned strict manifest, while historical authoring-manifest and exploratory scorer behavior remain isolated from headline comparisons;
• the derived manifest version and SHA-256 are written into existing provenance;
• ordinary import contextual_orchestrator does not import the benchmark, CSV adapter, or strict-scoring adapter;
• the strict scorer and supported publication path are included in permanent statement/branch/docstring, package build/install/import, and end-to-end dry-run contracts.

An isolated networkless harness exercised all 142 strict-scoring production statements and 54 branches and passed; that diagnostic validates the bounded module logic but is not repository CI, trusted central coverage, or merge evidence. Authoritative doctoring remains in docs/doctoring/nim-benchmark-strict-answer-scoring.md.

Provider and credential security boundary

• provider sockets dial only validation-time globally routable addresses;
• original hostname authority, TLS SNI, and certificate verification are preserved;
• ambient proxies and redirects are disabled;
• credentials are never forwarded outside the validated origin or serialized;
• provider responses are bounded to 8 MiB;
• dry-run jobs receive no NVIDIA_NIM_API_KEY; only the bounded live benchmark step receives the GitHub Secret;
• no COPILOT_GITHUB_TOKEN path is introduced;
• optional benchmark adapters remain lazy and do not mutate the normal package at import time;
• no one-shot, self-modifying, encoded-patch, branch-local repair, or temporary write-capable workflow exists in the current PR tree.

Historical central dependencies — read-only

Protected central .github main remains exact tip 6eb06cdd08c79a06f7b390069d4ffa49e2eb7dba.

The live central continuation is .github#937 at exact head 8726df151e64eecb89d91a4c029e809a785ee126 and .github#939 at exact head ac5665148bb113f92e97d2fc49a729bca2f050b5. #937 has ten terminal-success workflows, zero unresolved threads, and COMMENTED-only reviews; #939 has nine terminal-success workflows, zero threads, and no formal review. Both are Ready but neither is protected-main authority or qualifying approval. Closed or superseded #888/#906/#929 evidence remains historical and does not transfer.

PR #96 remains Draft at exact head 3703d0da9823b8258a0be94f1801aa5d61bfad9f on protected repository main@6841b71935e0b7cb98fb52bcb4709cc5100c8d87. This benchmark PR must reconcile only after the accepted protected #96 result exists and must regenerate every exact-head gate.

This repository does not write central branches or dispatch repair agents there.

Evidence contracts

Permanent regressions cover complete-plan admission, one-request-short zero-partial egress, deterministic bounded concurrency, valid media fixtures, DNS rebinding and non-global egress, redirects, proxy and credential isolation, bounded provider bodies, equal token/call budgets, reviewed pricing expiry, strict scoring, complete assignment evidence, Pareto exclusion labels, schema completeness, transactional publication, rollback/crash recovery, package import isolation, exact contributor-head workflow checkout, fuzz instrumentation, and buyer-readable evidence-status semantics.

Stack order and merge acceptance

1. the live central remediation path currently represented by fix(scheduler): require approved aggregate review state .github#937 and #939 must satisfy its own protection, exact-head evidence, and qualifying independent-review gates and reach protected central main; their current Ready state and green workflows are not protected-main authority;
2. PR fix(security): pin provider egress and repair the Atheris lock #96 must regenerate trusted exact-head central evidence, obtain qualifying independent non-author approval, satisfy branch protection/repository policy, and merge without bypass;
3. this PR must reconcile or retarget to the exact integrated protected result, preserving valid changes without relying on predecessor base metadata, then rerun every exact-head gate;
4. resolve the publication thread only after its stated exact-head evidence requirement is actually satisfied;
5. merge only when Tests, Security, Fuzz, Security Scan, SAST Semgrep, trusted central 100% production statement/branch/public-docstring/package evidence, required automated reviews, zero valid unresolved findings, qualifying independent non-author approval, security gates, repository rules, and branch protection all apply to one unchanged exact integrated head.

Queued, pending, skipped-required, cancelled, absent, stale-head, predecessor-head, failed, status-only, author-only, rate-limited, or synthetic-merge evidence is never success.

Refs #86.
Depends on #96 and the live protected-main central remediation path. Refetch its current successor before any acceptance decision; no historical PR number, head, or workflow evidence transfers.

Live dependency refresh (2026-08-12 05:07 UTC)

Read-only refetch superseding the historical dependency section above:

• This Draft PR remains exact head 26f8d8dc5634f0371fad0801056e9a3450c78bff on branch claude/nim-all-models-support-fecb0b.
• Its stacked base fix/atheris-interpreter-lock remains exact fix(security): pin provider egress and repair the Atheris lock #96 head 3703d0da9823b8258a0be94f1801aa5d61bfad9f; protected repository main remains 6841b71935e0b7cb98fb52bcb4709cc5100c8d87.
• Protected central .github/main remains 6eb06cdd08c79a06f7b390069d4ffa49e2eb7dba.
• Ready central #937 remains exact head 67d834f510fe044dd9d53cd4f4b9783353e303bd, with 11 terminal-success exact-head workflows, zero unresolved threads, and no qualifying approval.
• Ready central #939 remains exact head ac5665148bb113f92e97d2fc49a729bca2f050b5, with nine terminal-success exact-head workflows, zero unresolved threads, and no formal review.
• Ready central #943 advanced to exact head 601b254f3a8ea4cc593e7089d6baeadd9d8d3ee4, with nine terminal-success exact-head workflows, zero unresolved threads, and no formal review. It is the bounded trusted-download User-Agent repair and does not replace #939's separate cross-repository review-status semantics.
• This PR still has 16 formal review submissions and one intentionally unresolved publication-evidence thread. The repair is implemented, but the thread's own exact-head trusted evidence condition has not been satisfied.

Every dependency remains planning evidence only. No predecessor, status-only, synthetic-merge, queued, or unintegrated central evidence transfers to this contributor head.

Live central review-authority correction (2026-08-12 06:29 UTC)

Read-only dependency refetch: protected central main remains 6eb06cdd08c79a06f7b390069d4ffa49e2eb7dba. Central #937 remains open at 67d834f510fe044dd9d53cd4f4b9783353e303bd with eleven terminal-success workflows, zero unresolved threads, and one OpenCode APPROVED formal model review. That model review is not a qualifying independent human approval and does not make the central branch protected authority. Central #939 remains open at ac5665148bb113f92e97d2fc49a729bca2f050b5 with nine terminal-success workflows, zero threads, and no formal review; #943 remains open at 601b254f3a8ea4cc593e7089d6baeadd9d8d3ee4 with the same nine-success/zero-thread/no-review classification.

This repository loop did not mutate the central repository. This PR's unchanged source head and local workflow evidence remain active_pr; protected central integration and qualifying independent non-author approval remain absent. Keep the PR Draft.

[coderabbitai]

📝 Walkthrough

Walkthrough

NVIDIA NIM 비용·품질 벤치마크 하네스를 추가했습니다. 카탈로그 탐색, modality 검증, 정책 평가, 비용·증거 검증, 결정적 dry-run, live CI 실행, artifact 생성 및 품질 테스트를 포함합니다.

Changes

NIM 벤치마크 기능과 검증

Layer / File(s)	Summary
벤치마크 엔진과 실행 입력 contextual_orchestrator/nim_benchmark.py, contextual_orchestrator/__main__.py, examples/nim_*.json	NIM 모델 카탈로그 탐색, 전 modality probe, 요청 예산, 가격 검증, 정책 비교, 통계, provenance, artifact 생성 및 nim-benchmark CLI를 추가했습니다.
CI 실행과 카탈로그 퍼징 .github/workflows/*, fuzz/*, tests/fuzz/*, .gitignore	dry-run 및 live benchmark 작업을 추가했습니다. pull request 대상 제한을 제거했습니다. NIM 카탈로그 퍼징과 관련 Hypothesis 테스트를 추가했습니다.
벤치마크 계약 및 릴리스 검증 tests/test_nim_benchmark*.py, tests/test_pr_workflow_oidc_boundary.py, tests/test_repository_security_metadata.py	보안 경계, 요청 계획, 예산, capability 결과, 가격 증거, token usage, MP4 fixture, provenance, secret 격리 및 workflow 계약을 검증합니다.
증거 정책과 운영 문서 README.md, docs/*, CHANGELOG.md, docs/papers/README.md, conductor/tracks.md	실행 방법, 보안 제약, 증거 등급, 비용 처리, 운영 승인 규칙, 평가 방법론 및 변경 기록을 문서화했습니다.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• ContextualWisdomLab/contextual-orchestrator#96: provider-egress 보안 작업과 Atheris 퍼징 변경이 연결됩니다.

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant GitHubActions
  participant NIMBenchmark
  participant NIMProvider
  participant ArtifactStore

  Operator->>GitHubActions: dry-run 또는 live 실행 요청
  GitHubActions->>NIMBenchmark: 고정된 예산·manifest·provenance 전달
  NIMBenchmark->>NIMProvider: 카탈로그 조회 및 capability probe
  NIMProvider-->>NIMBenchmark: 모델·probe 결과 반환
  NIMBenchmark->>NIMProvider: 평가 task 요청 전송
  NIMProvider-->>NIMBenchmark: 응답·usage 반환
  NIMBenchmark->>ArtifactStore: JSON·CSV·Markdown artifact 업로드

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 55.94% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	제목은 NIM 모델 탐색과 전체 모달리티 비용·품질 벤치마크라는 변경의 핵심을 정확하고 간결하게 요약합니다.

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

⚔️ Resolve merge conflicts 💡

• Resolve merge conflict in branch claude/nim-all-models-support-fecb0b

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/nim-all-models-support-fecb0b

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Exact-head maintainer audit of 064973ea3ac3dfd8975342f8453483fa16a09aa6 found three release-blocking gaps. This is not approval evidence.

1. Provider egress is not DNS-pinned and redirects/proxies are not closed. build_default_transport() validates DNS once, then calls urllib.request.urlopen() against the hostname. That creates a second DNS resolution, follows redirects by default, and honors environment proxy configuration. A changed DNS answer or redirect can therefore forward the NVIDIA bearer credential outside the validated address set. The live policy-evaluation path also uses this branch's pre-fix(security): pin provider egress to validated public addresses #76 ModelClient transport. Stack this work on fix(security): pin provider egress to validated public addresses #76 and reuse its validation-time-address pinning, original-host TLS verification, direct no-proxy connection, and redirect rejection for both catalog/probe calls and model evaluation. Add adversarial tests for DNS rebinding, redirect credential forwarding, proxy bypass, non-global addresses including RFC 6598, and pinned-IP/original-host SNI behavior.

2. The compared policies do not have an equal adjusted token budget. Each request receives the same max_output_tokens, while conduct_bounded may issue up to five calls and route_once/direct baselines issue one. Conduct can therefore consume roughly five times the output-token allowance, contrary to issue [Product Gap] Evidence-grade NVIDIA NIM model discovery and cost-quality benchmark #86. Introduce a per-policy/task total token allowance and the same declared maximum-call envelope, decrement it across every step, stop or fail cells when exhausted, and record configured versus observed budgets in every cell and the provenance parameters. Tests must prove identical total allowances across direct, route-once, and conduct arms.

3. Actual-free cost is asserted without versioned evidence provenance. The report hard-codes actual_cost_usd = 0.0 and a static honesty label. NVIDIA currently documents Developer Program hosted endpoints as free for prototyping/research/testing, but evidence-grade artifacts must record the reviewed source identity, observation/review date, access/program context, and uncertainty or require a versioned run input. Do not silently carry the zero-cost assertion forward if access terms change.

Do not mark this PR ready or resolve these findings from stale checks. Re-run all required workflows and independent review on the exact repaired head.

[seonghobae]

Blocking security finding

CRITICAL — benchmark HTTPS transport reintroduces DNS-rebinding SSRF

require_public_https_endpoint() validates one DNS answer, but build_default_transport() then calls urllib.request.urlopen(), which resolves the hostname again. The validated_hosts cache widens the gap by skipping validation on later calls. A provider hostname can therefore pass public-address validation and later connect to loopback, private, RFC 6598, link-local, or otherwise non-global infrastructure. Redirects are also delegated to urllib's default handler, so credentials may be forwarded to an unvalidated destination.

This is the same TOCTOU class repaired by PR #76. Keep this PR Draft until #76 is in main, then reuse or generalize its DNS-pinned transport so the socket dials only validation-time public IPs while preserving the original hostname for HTTP authority, TLS SNI, and certificate verification. Reject redirects, bypass environment proxies, require is_global, clear stale pins before each validation, and deterministically close failed sockets/responses.

Required regression evidence:

• no transport-time DNS re-resolution;
• public IPv4/IPv6 pinning and hostname/SNI preservation;
• RFC 6598/private/loopback/link-local/multicast/reserved/unspecified rejection;
• redirect rejection with no authorization propagation;
• approved-IP fallback and all-address failure;
• exact-head statement/branch coverage and full repository checks.

Do not merge based on the current local 100% claim; it does not cover this security invariant.

[seonghobae]

Exact-head review — still blocked

The latest head addresses the three audit themes, but the implementation is not yet mergeable.

1. HIGH — production security behavior is coupled to a test monkeypatch

nim_benchmark_hardening._build_secure_transport() deliberately falls back to urllib.request.urlopen() whenever that global has been replaced. This keeps the old offline tests green by changing production behavior instead of replacing the obsolete tests. It reintroduces the hostname-resolving, proxy-aware, redirect-capable path that this repair is supposed to eliminate, and requires a nosemgrep waiver on the exact sink.

Remove the compatibility branch entirely. Rewrite the transport tests to inject the pinned connection/resolver seam and prove the real direct transport. Do not let test instrumentation select a less-secure production path.

2. HIGH — the new 531-line hardening module has no direct regression suite

The current tests/test_nim_benchmark.py still asserts the old urlopen behavior and contains no evidence for EqualBudgetModelClient, expiry enforcement, actual-cost evidence validation, configured/observed budget fields, or the installed wrappers. A 100% repository claim cannot be accepted while the newly shipped module and its branches are not directly exercised.

Add behavior tests that fail without each contract, then prove exact-head statement and branch coverage at 100% for both nim_benchmark.py and nim_benchmark_hardening.py, plus 100% public docstrings.

3. HIGH — cited cost source does not support the recorded claim

ACTUAL_COST_EVIDENCE names https://docs.nvidia.com/nim/large-language-models/latest/faq.html, but the current NVIDIA FAQ location is different and the available FAQ content does not establish that the API Catalog hosted endpoint used by this run is free to the caller. Current build.nvidia.com model pages do label prototype endpoints as free, while NIM offering/licensing documentation distinguishes free exploratory offerings from NVIDIA AI Enterprise production support. Record the exact reviewed page/version or immutable evidence artifact that supports the hosted endpoint claim; do not cite a generic or moved FAQ as proof.

4. MEDIUM — optional adapter boundary is lost

contextual_orchestrator.__init__ now imports the entire benchmark and monkeypatch installer for every package import. Keep the NIM evaluator optional: integrate the fix directly into its module, or install it only from the benchmark entrypoint. Runtime gateway users should not import benchmark-only code or mutate benchmark globals as a side effect of import contextual_orchestrator.

5. MEDIUM — temporary privileged workflow remains in the PR tree

.github/workflows/temporary-nim-transport-fix.yml has top-level contents: write, persists checkout credentials, and triggers on every branch push. It must be removed from the final tree. Any one-shot repair must use read-only defaults, isolate credentials to the publication step, validate exact changed paths, and disappear before review.

Keep the PR Draft. Replace the compatibility installer with direct, test-first implementation; remove the temporary workflow; update the PR body to the actual head; then rerun all exact-head checks and independent review.

[seonghobae]

Exact-head maintainer follow-up for the new hardening commits:

1. Keep the existing GHAS thread unresolved until contextual_orchestrator/nim_benchmark.py itself no longer contains the hostname-re-resolving urllib.request.urlopen implementation. Installing a replacement through package import does not remove the vulnerable/static-scanned implementation and forces the optional benchmark adapter into every package import.
2. Correct the cost-evidence source to the current official NVIDIA NIM General FAQ: https://docs.api.nvidia.com/nim/docs/product. That page explicitly states free Developer Program access to NIM API endpoints for prototyping and distinguishes production use, which requires NVIDIA AI Enterprise. The current docs.nvidia.com/nim/large-language-models/latest/faq.html identity and the docs.nvidia.com-only validator do not match the reviewed current source.
3. Preserve lazy/optional adapter loading: install benchmark-specific behavior from the benchmark module or integrate it directly, rather than eagerly importing the full benchmark and compatibility installer from package __init__.

Do not resolve the security thread or mark the PR ready until these are implemented and verified on the exact repaired head.

[seonghobae]

Still the primary vehicle for issue #86 (NIM discovery + cost-quality benchmark) but CONFLICTING. Autonomous loop will rebase or re-implement on main after #111 merges rather than waiting on conflict resolution as a blocker.

[seonghobae]

Closed by autonomous commercial loop: PR targets non-main base fix/atheris-interpreter-lock (stack intermediate). Re-open against current main after #111–#114 land if the product gap remains. (feat: evidence-grade NIM discovery + all-modality cost-quality benchmark)