Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,22 @@ this file. The format follows Keep a Changelog, and versioned releases follow
Semantic Versioning where the repository publishes a release.

## [Unreleased]
- Bump the vendored `ORCHESTRATOR_PIN_SHA` from `30c6d71680e659f25a0a433d4726ad0d437f9757`
to `0adca4703df67f8f31d3ea5b04a1e07ed775dd6c` (`contextual-orchestrator`
`main` tip as of 2026-08-31) in the sidecar script, its contract test, and
ADR-0003 -- vendoring `contextual-orchestrator`'s request-time failover
classification fix (#922) plus PR #941 (stop collapsing the independent
`nvidia_nim`/`nvidia_nim_sub` credentials into one outage-domain family
for pool-diversity purposes -- an assumption ADR-0015 already contradicted)
and #945 (its matching stale-test fix), and everything else that landed on
that repo's `main` since the last pin. `requirements.lock` is unchanged
and every CLI change is additive, so the sidecar's dependency surface is
unaffected; see `docs/product-technical-gap-baseline.md` for the full
review and the still-open live-verification follow-up. This is a
prerequisite for the separate `PROVIDER_FAMILIES` fix in
`scripts/ci/contextual_orchestrator_review_policy.py` (its own,
independent copy of the same collapsing bug) to reflect reality in the
free/ZDR diversity evidence served to central review.
- Web verification now runs backend, frontend, and E2E commands inside an
isolated Linux bubblewrap workspace by default (`--isolation required`),
mounting a read-only runtime root with a single writable `/workspace`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ all five, and auto-optimize routing by cost.

1. **Vendoring, pinned**: `scripts/ci/contextual_orchestrator_review_sidecar.sh`
clones `ContextualWisdomLab/contextual-orchestrator` at an exact SHA
(`30c6d71680e659f25a0a433d4726ad0d437f9757` today) into `RUNNER_TEMP`. The
(`0adca4703df67f8f31d3ea5b04a1e07ed775dd6c` today) into `RUNNER_TEMP`. The
source's `requirements.lock` is installed with `--require-hashes` and
`--no-deps`, so dependency resolution cannot silently move the reviewed
runtime.
Expand Down
84 changes: 84 additions & 0 deletions docs/product-technical-gap-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -1715,6 +1715,90 @@ string, a bare number) confirmed to fail against the pre-fix script (`KeyError:
signature as the original round-4 bug) before passing after the fix. 1930 tests pass; 100% coverage and
100% docstring coverage on `scripts/ci/`.

## 2026-08-31 sidecar pin bumped past #922's failover fix, #941's NVIDIA-account-independence fix, and current `main`

- Bumps `ORCHESTRATOR_PIN_SHA` from `30c6d71680e659f25a0a433d4726ad0d437f9757`
(the #1430 pin) to `0adca4703df67f8f31d3ea5b04a1e07ed775dd6c`
(`contextual-orchestrator` `main` tip as of 2026-08-31) in the same three
places #1430 established as the contract: the sidecar script default
(`scripts/ci/contextual_orchestrator_review_sidecar.sh`), the contract
test's `ORCH_PIN_SHA` (`tests/test_contextual_orchestrator_review_sidecar_contract.py`),
and `docs/adr/0003-contextual-orchestrator-vendored-free-zdr.md`'s "today"
reference. An earlier revision of this branch (PR #1465) targeted
`79c6841b9b3645d1f14b943985825d2302071f5b`; that intermediate target has
since fallen behind two more merged `contextual-orchestrator` PRs found
during this session's own investigation (below), so the bump was extended
the rest of the way to current `main` instead of landing a pin that would
already be stale on merge.
- Vendors `contextual-orchestrator` PR #922 (`fix(routing): classify primary
provider transport failures explicitly` — the classification fix behind
PR #1437's own request-time failover acceptance contract) plus everything
else that landed on that repo's `main` since the #1430 pin, most notably:
- **#941** (`fix(discovery): keep credential accounts independent`) and its
follow-up test fix **#945**: removes the wrong assumption that the two
independent `nvidia_nim`/`nvidia_nim_sub` KV credentials share one model
catalog/outage domain — an assumption `docs/planning/adrs/0015-durable-
provider-catalog.md` (accepted 2026-08-22, in `contextual-orchestrator`)
already contradicted. `model_discovery._provider_family` (the collapsing
helper) is deleted outright; `select_bootstrap_discovered_agents`,
`select_provider_diverse_models`, and `evaluate_provider_credential_
inventory` all now diversify/tolerate per credential-backed provider
account, not per collapsed family. This is the exact fix this session's
investigation was chasing: without this bump, every hosted review run
(`noema-review`/`opencode-review`/`strix`) kept using the old, buggy
collapsing logic regardless of what was already merged upstream. It is a
**prerequisite** for the separate, independent fix to
`scripts/ci/contextual_orchestrator_review_policy.py`'s own
`PROVIDER_FAMILIES` hardcoded map (this repo's own copy of the identical
wrong assumption, never touched by #941 since that PR only fixed
`contextual-orchestrator` itself) to have any live effect on the
`free_family_diversity` evidence PR #1433 added and PR #1437 (open) gates
Strix's `orchestrator/free` eligibility on.
Comment on lines +1750 to +1756

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 NVIDIA diversity remains partially disabled

Upstream now separates both NVIDIA accounts, while local PROVIDER_FAMILIES still collapses them. Catalog caps and diversity evidence remain unchanged pending follow-up.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

- a new, unrelated free-pool correctness fix also on this span: models that
declare a non-text input modality (e.g. NVIDIA NIM's vision-input
`meta/llama-3.2-90b-vision-instruct`, the root cause of
`ContextualWisdomLab/.github#1198`'s incident) are now excluded from the
blind general-chat `orchestrator/free` pool via a new
`general_free_serving_candidates`/`_is_general_free_agent` predicate,
while remaining fully counted in price-only free inventory and reachable
through capability-scoped free routes. Additive only (new function, new
`discover-models --verbose`/`general_free_serving_count` fields); no
removed CLI surface.
- #912 (video job resource normalization), #868 (gateway default chat
model + configured-gateway discovery + OpenRouter ZDR/privacy metadata),
#920/#921 (cleanup/docs), #928 (provider-catalog-sync outage tolerance),
#925 (`stream_options.include_usage` for tools passthrough), and #929
(Bytez raw-token `Authorization` header fix) — already reviewed by
PR #1465's own prior pass, not re-reviewed in depth here.
- Reviewed for anything that would break the sidecar's actual dependency
surface before bumping, since ADR-0003 requires this — this session's own
incremental review covers only the span from the #1465 pin
(`79c6841b9b3645d1f14b943985825d2302071f5b`) to the new target
(`0adca4703df67f8f31d3ea5b04a1e07ed775dd6c`), i.e. #941/#945 plus the
vision-input exclusion above; the earlier span was already reviewed by
PR #1465 and is not repeated: `requirements.lock` is byte-for-byte
unchanged across the whole span (no `--require-hashes` drift, confirmed by
diffing `requirements.lock`/`pyproject.toml` directly between the two
SHAs), and every change to `contextual_orchestrator/__main__.py` (the CLI
the launcher invokes) is strictly additive -- new optional `--verbose`
flags and a new discovery report field, zero removed arguments or
subcommands. The `register-credential` and `--serve --agents ... --port
... --auth-token ...` invocations the sidecar/launcher depend on are
untouched (confirmed directly in `contextual_orchestrator/__main__.py`).
- Not independently verified live: this sandbox has no provider credentials
(`BYTEZ_API_KEY`/`NVIDIA_NIM_API_KEY`/`NVIDIA_NIM_API_KEY_SUB`/
`OPENROUTER_API_KEY`/`OPENAI_API_KEY`), so an actually-executed end-to-end
sidecar run against real providers isn't possible here the way ADR-0003's
own PoC bar asks for. Local verification is `.github`'s own full suite
(100% coverage/docstrings) plus the static sidecar/pin contract tests; the
real live proof is the next hosted `noema-review`/`opencode-review`/`strix`
run against this new pin, once the ongoing org-wide `opencode-review`
outage (tracked on PR #1437 and elsewhere in this baseline) clears enough
to observe one. Left as this entry's own concrete follow-up rather than a
bypass-merge, since (unlike the scheduler-script hotfixes bypass-merged
earlier this session) this change's actual correctness depends on live
provider behavior this sandbox cannot exercise.

## 5. 실행 루프와 고객의 다음 행동

각 hourly pass는 아래 순서를 유지한다.
Expand Down
2 changes: 1 addition & 1 deletion scripts/ci/contextual_orchestrator_review_sidecar.sh
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# (fail-closed zero-cost) pool.
set -euo pipefail

ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-30c6d71680e659f25a0a433d4726ad0d437f9757}"
ORCHESTRATOR_PIN_SHA="${ORCHESTRATOR_PIN_SHA:-0adca4703df67f8f31d3ea5b04a1e07ed775dd6c}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Vision routes remain in general reviews

The pinned release cannot exclude multimodal routes because the sidecar drops their input tags. They can pass preflight and reject real review requests.

Prompt for agents
Update the sidecar integration so its catalog uses the pinned orchestrator's general-purpose free-serving eligibility, or preserve discovered input-modality metadata as input:<modality> agent tags. The current launcher calls free_discovered_models and _report_rows omits input_modalities, while contextual_orchestrator_review_policy emits only review/cost/ZDR tags. Consequently both general_free_serving_candidates and TaskOrchestrator._is_general_free_agent lack the evidence needed to exclude vision-input routes. Add an integration regression using a free text+image model that can pass a plain chat probe but must not enter orchestrator/free.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 OpenRouter slowdown aborts all reviews

The pinned release can spend the entire 180-second startup budget fetching optional OpenRouter metadata. A slowdown kills every review before provider preflight.

Prompt for agents
Bound the newly pinned OpenRouter privacy-metadata fanout within the sidecar's total startup deadline. At this revision, discover_provider_models calls _openrouter_free_model_endpoints for every token-free OpenRouter model using eight workers and a per-request 15-second timeout; the current catalog contains dozens of candidates, so stalled requests can consume or exceed the shell's 180-second readiness watchdog before other discovery and route preflight complete. Add one shared discovery deadline or a much smaller bounded metadata budget, and verify that optional OpenRouter metadata failure degrades only that evidence source rather than terminating the entire review sidecar.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

ORCHESTRATOR_GIT_URL="${ORCHESTRATOR_GIT_URL:-https://github.com/ContextualWisdomLab/contextual-orchestrator.git}"
# The Strix gate and Noema SSRF guard accept this one process-local origin.
# Keep it fixed so an environment override cannot create an unvalidated sidecar.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
)

GATEWAY_MODEL = "contextual-orchestrator/orchestrator/free"
ORCH_PIN_SHA = "30c6d71680e659f25a0a433d4726ad0d437f9757"
ORCH_PIN_SHA = "0adca4703df67f8f31d3ea5b04a1e07ed775dd6c"


def _read(path: Path) -> str:
Expand Down
Loading