Skip to content

feat(logging): add CLI debug logging with redaction safety net - #946

Merged
seonghobae merged 19 commits into
mainfrom
feat/verbose-debug-logging-log-level-cli
Sep 1, 2026
Merged

feat(logging): add CLI debug logging with redaction safety net#946
seonghobae merged 19 commits into
mainfrom
feat/verbose-debug-logging-log-level-cli

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Current contract correction: logging configuration is CLI-only (--log-level, --verbose, --debug). CONTEXTUAL_ORCHESTRATOR_LOG_LEVEL is deliberately ignored; historical env-var wording below is superseded by the KV-not-env runtime policy and the current source/ADR.

Note for reviewers: overlaps with open PR #942

#942 ("feat: add verbose/debug logging for route/conduct/provider control
flow") already exists on feat/verbose-debug-logging from this same
account, opened concurrently with this work.
I discovered it only when
git push to that branch name was rejected as non-fast-forward — this PR
was built independently, from a separately supplied, more detailed approved
design, without awareness of #942 until the push collision. Please treat
these as two independent proposals for the same feature and pick one (or
merge the best parts of each) rather than merging both:

This PR #942
Level control --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL} (explicit level) --verbose/--debug only (binary DEBUG on/off)
Env var CONTEXTUAL_ORCHESTRATOR_LOG_LEVEL (a level name) CONTEXTUAL_ORCHESTRATOR_VERBOSE (a boolean)
INFO tier Yes — per-request summary, discovery aggregate No — DEBUG or nothing
Instrumented sites Retry loop, circuit breaker, _ranked_agents/_static_rank_key/_measured_member_order/_select_agent, model_discovery.py per-provider attempts, server.py per-request + response-body summary _dispatch, route_once/.conduct, _invoke, circuit breaker, _send_with_retry, server.py request/response via parse_request/log_request hooks
Redaction redact_text/redact_value call-site redaction plus a handler-level logging.Filter safety net Exception class name only, never str(exc) (more conservative, less diagnostic detail)
ADR location docs/adr/0007-verbose-debug-logging.md (matches the runtime-observability precedent set by ADR 0122 in that series) docs/planning/adrs/0125-verbose-debug-logging.md

Both implementations are independently tested and pass their own full
regression suites; I have not attempted to reconcile or merge the two here.


Summary

Adds verbose/debug logging per an approved design: a debug_logging.py
leaf module (stdlib logging only, no new dependency — see the Ponytail
entries added to docs/library_research.md), CLI/env wiring, and new
DEBUG/INFO/WARNING instrumentation at previously-silent decision points.
Full design record: ADR 0007.

Flag / env var (contextual_orchestrator/__main__.py):
--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL} (case-insensitive) plus a
--verbose/--debug shorthand for DEBUG, and CONTEXTUAL_ORCHESTRATOR_LOG_LEVEL.
Precedence: --log-level > --verbose/--debug > env var > default WARNING
(unchanged default — no behavior change for anyone not opting in). Resolved
once via a parse_known_args pre-scan before subcommand dispatch, so
register-credential, discover-models, check-fast-mlsirm, one-shot
completion, and --serve are all covered from one call site. An invalid
level (from either the flag or the env var) fails closed with an
argparse-style SystemExit(2), never silently ignored.

What gets logged, and at what level:

  • WARNING (default, no --verbose needed — these are new, everything
    else in this list is opt-in):
    • circuit_opened agent_id=... failures=... threshold=... reset_seconds=...
      — only on the edge transition into the open state, not every failure.
    • provider_exhausted agent_id=... model=... attempts=... final_error_type=...
      — a provider call that used its full retry budget.
  • INFO (--log-level INFO):
    • discovery_complete providers=%d models=%d errors=%d (model discovery
      aggregate).
    • One body-free per-request summary in server.py: method, path, status,
      latency, and the existing ADR 0122 session-correlation hash (never the
      raw session id) — reuses a new shared telemetry.session_id_hash()
      helper so the OTLP span attribute and this log line hash identically.
  • DEBUG (--log-level DEBUG / --verbose / --debug):
    • Retry loop (ModelClient._send_with_retry/_send_raw_with_retry):
      provider_attempt, provider_attempt_failed (redacted, truncated error
      message), provider_backoff.
    • Circuit breaker: circuit_failure (every increment), circuit_reset,
      circuit_cleared (only when there was real state to clear).
    • Ranking (TaskOrchestrator._static_rank_key/_measured_member_order/
      _select_agent): rank_partition, rank_candidate (agent id/model/
      priority/capability fit/affinity, or judged-quality/success-rps),
      select_agent (final chosen agent). Agent id/model are logged
      unredacted, matching the existing precedent that they already flow into
      raised exception metadata today; raw prompt/answer text is never
      logged, only lengths/counts where relevant — redact_text deliberately
      does not scrub PII, so this design never logs message content at all
      rather than leaning on it for a job it does not do.
    • model_discovery.py: discovery_attempt/discovery_result per
      provider, discovery_provider_failed (redacted, truncated) — the
      credential name (e.g. OPENAI_API_KEY) is logged, never its value.
    • server.py: one response-body summary that reuses the exact
      already-redacted safe_payload object _response_payload computes via
      redact_value — never a second, separately (or un-)redacted copy.

Redaction — two layers, both required: every new call site that logs
caller- or provider-derived content wraps it in the existing, unmodified
orchestrator.redact_text/redact_value first (e.g.
redact_text(str(exc))[:500] in the retry loop). debug_logging.configure_logging(level, redactor=redact_text)
then attaches a logging.Filter to the handler logging.basicConfig
installs — deliberately the handler, not the Logger object, since a
Logger-level filter is skipped while a record propagates up from a child
logger — as a safety net for a call site that forgets. configure_logging
uses logging.basicConfig(..., force=True) because plain basicConfig is a
no-op after the first call in a process, which would otherwise make a
second in-process CLI invocation silently keep whatever level the first
call configured (a real risk given this repo's own test convention of
calling main() more than once per interpreter).

Secret-non-leakage evidence (the critical property)

Per-surface, TDD (test written and confirmed red before the redaction
existed), plus an explicit adversarial re-check after: I monkeypatched each
implementation to skip its redaction step and reran the corresponding test
to confirm it fails (catches the leak) before trusting it:

  • tests/test_debug_logging.py::test_configure_logging_redactor_masks_secret_shaped_content_in_captured_output
    — the handler-level filter, end to end via real stdlib logging output
    capture. A paired ..._redactor_none_still_leaks test is the negative
    control proving the assertion is real, not a tautology.
  • tests/test_orchestrator_debug_logging.py::test_send_with_retry_debug_logs_redact_secret_shaped_error_message
    — a mocked provider raising an exception whose message embeds a fake
    api_key=sk-FAKE... shape, captured through the real retry-loop DEBUG log.
  • tests/test_model_discovery.py::test_discover_provider_models_debug_logs_failure_error_type_and_redacts_message
    — same shape, through the discovery-failure DEBUG log; a sibling test
    confirms only the credential name appears, never a registered fake
    credential value.
  • tests/test_telemetry.py::test_response_payload_debug_log_reuses_redacted_payload_never_raw_secret
    — confirms the response-summary log reuses the response path's own
    redaction rather than a second, separate copy.

Verification

  • python -m pytest tests -q --ignore=tests/test_psychometric_routing.py
    (that one file needs fast_mlsirm, which requires Python ≥3.12; this
    sandbox runs 3.11) → 2846 passed, 1 skipped, run twice (once as a
    pre-change baseline at 2836 passed, confirming exactly the 10 new tests
    added and zero regressions).
  • python -m interrogate contextual_orchestrator/100% (repo gate is
    fail-under = 100, not the 80% this repo's own CLAUDE.md currently
    misstates).
  • debug_logging.py (the one new module not in pyproject.toml's coverage
    omit list) → 100% branch coverage, including the redactor=None
    branch.
  • No new dependency (Ponytail gate): stdlib logging only. Added the
    missing logging and OpenTelemetry rows to
    docs/library_research.md's Selected Stack table (OTel was already a
    runtime dependency since ADR 0122 but had never gotten a row).
  • CHANGELOG.md Unreleased/Added entry; light conductor/tech-stack.md
    touch-up noting the OTel/jsonschema runtime deps that were already true
    but undocumented there.

Files

  • contextual_orchestrator/debug_logging.py (new)
  • contextual_orchestrator/__main__.py, orchestrator.py,
    model_discovery.py, server.py, telemetry.py
  • tests/test_debug_logging.py, test_cli_logging.py,
    test_orchestrator_debug_logging.py (new); test_model_discovery.py,
    test_telemetry.py (extended)
  • docs/adr/0007-verbose-debug-logging.md (new), docs/adr/README.md,
    docs/library_research.md, CHANGELOG.md, conductor/tech-stack.md

Not merging this myself — flagging for independent adversarial review next,
including reconciling with #942 above.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX


Generated by Claude Code


Devin Review

Summary by CodeRabbit

  • 새로운 기능

    • CLI 옵션과 환경 변수로 로그 수준을 설정할 수 있습니다.
    • 재시도, 회로 차단기, 모델 탐색 및 요청 처리의 상세 로깅이 추가되었습니다.
    • 요청 요약과 응답 상태를 확인할 수 있습니다.
  • 버그 수정

    • 일시적인 모델 탐색 실패를 제한적으로 재시도합니다.
    • 인증서 검증 오류 등 재시도하면 안 되는 오류를 올바르게 처리합니다.
    • 잘못된 CLI 옵션 축약을 일관되게 거부합니다.
  • 보안 및 개인정보 보호

    • 로그의 자격 증명, 프롬프트, 응답 본문 및 URL 쿼리가 보호됩니다.
    • 세션 정보는 해시 처리됩니다.
  • 문서

    • 상세 디버그 로깅 설계 및 운영 지침이 추가되었습니다.

claude added 2 commits August 31, 2026 02:41
… module

New stdlib-only contextual_orchestrator/debug_logging.py owns level parsing,
configure_logging() (basicConfig with force=True so repeated in-process CLI
invocations actually re-apply), a handler-level redaction safety net, and
small pure log-formatting helpers. __main__.py wires --log-level/--verbose/
--debug and CONTEXTUAL_ORCHESTRATOR_LOG_LEVEL through a parse_known_args
pre-scan that runs before subcommand dispatch, covering register-credential,
discover-models, check-fast-mlsirm, one-shot completion, and --serve
uniformly. orchestrator.py gains a module logger for the next commit's
retry/circuit-breaker/ranking instrumentation.

Part of the verbose/debug logging feature (see forthcoming ADR).
Adds DEBUG/WARNING/INFO instrumentation at the previously silent decision
points named in the task: ModelClient._send_with_retry /
_send_raw_with_retry (per-attempt, backoff, a WARNING provider_exhausted
line that fires without --verbose), TaskOrchestrator's circuit breaker
(_record_failure/_record_success/_circuit_open, with a WARNING
circuit_opened only on the edge transition), and evidence-based ranking
(_static_rank_key/_measured_member_order/_select_agent). Every call site
that logs caller- or provider-derived content wraps it in the existing
orchestrator.redact_text/redact_value first.

model_discovery.py gains per-provider discovery_attempt/discovery_result/
discovery_provider_failed DEBUG lines (credential *names*, never values)
and one discovery_complete INFO summary in discover_all_models.

server.py gains one body-free per-request INFO summary (method, path,
status, latency, ADR 0122 session correlation hash) via handle_one_request
and a DEBUG response-body summary that reuses the exact already-redacted
safe_payload object _response_payload computes, never a second copy.
telemetry.py exposes the shared session_id_hash() helper so both
_safe_attributes (OTLP) and server.py's summary use the identical hash.

Adds ADR 0007 (docs/adr/ series, matching the runtime-observability
precedent set by ADR 0122), the missing logging + OpenTelemetry rows in
docs/library_research.md, a CHANGELOG entry, and a light tech-stack.md
touch-up.
@coderabbitai

coderabbitai Bot commented Aug 31, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

ADR 0007에 따라 표준 라이브러리 기반 디버그 로깅을 추가했습니다. CLI와 환경 변수로 로그 수준을 설정합니다. 재시도, 모델 탐색, 에이전트 선택, 회로 차단기, HTTP 요청을 계측합니다. 민감한 로그 값은 비식별화하거나 허용 목록 메타데이터로 제한합니다.

Changes

로깅 및 관측성

Layer / File(s) Summary
로그 설정 및 비식별화 기반
contextual_orchestrator/debug_logging.py, contextual_orchestrator/__main__.py, tests/test_debug_logging.py, tests/test_cli_logging.py, docs/adr/0007-verbose-debug-logging.md, docs/adr/README.md, docs/library_research.md, conductor/tech-stack.md, CHANGELOG.md
로그 수준 파싱, CLI 및 환경 변수 우선순위, allow_abbrev=False, 루트 로거 구성을 추가했습니다. 메시지와 traceback을 비식별화하고 요청 및 응답 로그를 제한된 메타데이터로 구성했습니다.
Provider 재시도 및 선택 경로 계측
contextual_orchestrator/orchestrator.py, contextual_orchestrator/model_discovery.py, tests/test_orchestrator_debug_logging.py, tests/test_model_discovery.py
일시적 오류 재시도, 오류 분류, 모델 탐색 결과, provider 재시도 종료 상태, 에이전트 순위 및 선택, 회로 차단기 상태를 DEBUG 또는 WARNING으로 기록했습니다.
HTTP 요청 및 세션 계측
contextual_orchestrator/server.py, contextual_orchestrator/telemetry.py, tests/test_telemetry.py
완료된 HTTP 요청에 메서드, 경로, 상태, 지연 시간, 세션 ID 해시를 기록합니다. keep-alive 종료 시 유령 요청을 생략하고, 프레임워크가 생성한 응답 상태를 추적합니다.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🟠 High · up to a37f5

The PR adds retries to authenticated provider discovery while still allowing redirects to another host, which can expose provider credentials and repeat that exposure; DEBUG response telemetry also retains unbounded model and usage-key metadata. This creates a high-impact security and data-exposure risk, so the PR is not merge-ready until redirect handling is constrained and logged metadata is bounded.

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant Main
  participant Logging
  participant Orchestrator
  participant ModelDiscovery
  participant Server
  participant Logger

  CLI->>Main: 로그 옵션과 서브커맨드 전달
  Main->>Logging: 로그 수준 파싱 및 구성
  Main->>Orchestrator: provider 호출 실행
  Orchestrator->>Logger: 재시도 및 선택 이벤트 기록
  Main->>ModelDiscovery: 모델 탐색 실행
  ModelDiscovery->>Logger: 탐색 시도 및 결과 기록
  Main->>Server: HTTP 요청 처리
  Server->>Logger: 상태, 지연 시간, 세션 해시 기록
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 48.65% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 148 functions across 10 files. (3 skipped… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목은 CLI 기반 디버그 로깅과 비밀정보 비식별화 안전망을 정확히 설명합니다. 재시도, 요청 요약, 응답 메타데이터 로깅 등 전체 변경 범위를 모두 포함하지는 않지만 주요 변경과 관련성이 높고 충분히 구체적입니다.
Full details: Docstring Coverage

Explanation

Docstring coverage is 48.65% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 148 functions across 10 files. (3 skipped: 2 unsupported, 1 too large.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 3
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
⚔️ Resolve merge conflicts 💡
  • Resolve merge conflict in branch feat/verbose-debug-logging-log-level-cli
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/verbose-debug-logging-log-level-cli

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

devin-ai-integration[bot]

This comment was marked as resolved.

Copy link
Copy Markdown
Contributor Author

Ran an independent adversarial secret-leakage review of this PR (whose only mandate was: can any code path here cause a credential to appear in log output, under any configuration?). Verdict: FAIL.

Confirmed, reproduced against the actual PR code: server.py's new _response_payload()/DEBUG response-summary line computes safe_payload = redact_value(payload), but redact_value/redact_text (pre-existing, unmodified) only pattern-match the literal in-string shape (api[_-]?key|token|secret|password)[:=]<value> or bearer <value> — they never look at the JSON key name a string value is nested under. A provider/tool-call response containing a field like {"private_key": "...``"}, {"key": "AIzaSy..."}, {"auth": "sk-live..."}, or {"credential": "..."} writes the raw secret verbatim into the DEBUG log. I verified this by calling configure_logging('DEBUG', redactor=redact_text) exactly as wired, then feeding _response_payload() each shape — all four leaked in full. The handler-level "safety net" filter re-applies the same redact_text, so it shares the identical blind spot — the two advertised layers are not independent against this failure mode.

This is the same gap Devin's review flagged independently ("Debug logs expose response bodies").

Secondary, also real:

  • INFO-level request logging (summarize_request_for_log) logs self.path verbatim including any query string, with zero redaction — contradicts its own "body-free" docstring claim. Also independently flagged by Devin.
  • A --verbose/--debug flag placed before a subcommand bypasses subcommand dispatch in __main__.py:main (Devin finding, confirmed real).
  • Systemic note: this PR is the first code in this process's lifetime to call logging.basicConfig() at all, so enabling --verbose/DEBUG also activates every previously-dormant logger.debug() call elsewhere in the package (checked all of them — none currently embed real secrets, so no live leak today, but there's no per-call-site review gate going forward).

On the #942 comparison: #942's design explicitly never logs response body content at all (method/path/status/latency only, no payload summary), so it structurally isn't exposed to this exact vulnerability class — worth weighing in the reconciliation decision, though it also can't answer the response-content debugging questions that motivated this feature.

Dispatching a fix now for the confirmed leak (generic key-name-aware redaction, not just value-pattern matching), the query-string leak, and the CLI ordering bug, since the response-body diagnostic capability is valuable and worth fixing rather than dropping.


Generated by Claude Code

claude added 3 commits August 31, 2026 03:35
Resolves a conflict in contextual_orchestrator/model_discovery.py where
this branch's new discovery instrumentation (isEnabledFor DEBUG guards,
discovery_attempt/discovery_result/discovery_provider_failed structured
logs, elapsed_ms timing) collided with main's independently-landed
privacy-hardened logging on the same call sites (account= terminology,
never logging the credential_name label).

Reconciled by keeping this branch's richer instrumentation but adopting
main's stricter contract: the discovery debug logs now never include
source.credential_name (only the provider/account name), matching the
already-merged
test_discovery_debug_log_identifies_account_without_secret on main.
Updated this branch's own new tests in tests/test_model_discovery.py to
match (account= field naming, no credential_name assertion).
…review

Fixes the three confirmed findings from the adversarial secret-leakage
review (github.com//pull/946#issuecomment-5473342909),
independently corroborated by Devin's automated review comments on the PR:

1. Critical: server.py's DEBUG response-summary log reused redact_value(),
   which only pattern-matches an in-string secret *value* shape and never
   inspects the JSON *key* a string is nested under, so a response field
   like {"private_key": "..."}, {"key": "..."}, {"auth": "..."}, or
   {"credential": "..."} leaked verbatim. Adds a new, additional,
   key-name-aware redaction pass, debug_logging.redact_credential_shaped_keys,
   applied on top of the existing redact_value output at that one call
   site. orchestrator.redact_text/redact_value are unmodified, per the
   review's explicit scope.

2. The INFO per-request summary (summarize_request_for_log) logged
   self.path verbatim, including any query string, contradicting its own
   body-free docstring. It now strips the query string before formatting.

3. A --log-level/--verbose/--debug flag placed before a subcommand (e.g.
   `--verbose discover-models`) bypassed subcommand dispatch, which only
   checked arguments[0]. Adds _subcommand_token_index to skip past
   recognized leading logging flags when locating the subcommand token,
   without removing them from the argument list each subcommand's own
   parser sees.

Each fix has a red/green regression test: the new tests reproduce the
leak/bug against the pre-fix code (verified failing) before the fix makes
them pass.

Also merges main into this branch (was reported dirty/mergeable_state) and
resolves a real conflict in model_discovery.py where this branch's new
discovery instrumentation collided with main's independently-landed,
privacy-hardened logging on the same call sites; reconciled by keeping the
richer instrumentation but adopting main's stricter contract (never log
source.credential_name, use account= naming), matching main's own
test_discovery_debug_log_identifies_account_without_secret. Separately, the
merge surfaced a non-conflicting but real regression: main independently
added a second, plain --verbose flag to the discover-models and one-shot/
serve parsers that collided with this branch's --add_log_level_arguments,
raising argparse.ArgumentError on every invocation of either path; removed
the redundant declarations (and their dead, non-redacted
logging.basicConfig follow-ups) now that _configure_logging_from_cli
already covers this centrally.

PR #942 (a separate, more conservative implementation of the same feature
on a different branch) still needs reconciliation by a human/reviewing
session -- out of scope here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
main advanced again (PR #923 merged: a bounded retry for a provider's
transient model-list fetch, plus an is_transient_error TLS/DNS fix),
producing a second real conflict in model_discovery.py's
discover_provider_models on the exact same fetch/except block this
branch instruments.

Resolved by keeping main's retry loop (attempt_timeouts/last_exc/
is_transient_error) fully intact and layering this branch's DEBUG
discovery_provider_failed logging (error_code/error_type/redacted
message) on top of it, now firing once after the retry loop exhausts
its attempts (using the final last_exc) rather than on the old
single-attempt except. Also dropped a duplicate `_LOGGER =
logging.getLogger(__name__)` line the two independent additions
produced side by side without a textual conflict.
devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

claude added 2 commits August 31, 2026 04:39
Fixes three further confirmed real findings from the re-review after the
first push, plus verifies (and closes with a regression test rather than a
code change) a fourth finding that turned out to be a false positive:

1. BUG (Devin, server.py): keep-alive phantom requests. On a persistent
   HTTP connection, handle_one_request never reset self.command/self.path
   before each call; stdlib's own handle_one_request only assigns them when
   it actually parses a request line, so when a keep-alive client closed
   the connection (nothing read), the *previous* request's command/path
   were still in place. _log_request_summary's existing "nothing to
   report" guard therefore never fired, logging the prior request a second
   time with a statusless "phantom" entry. Fixed by resetting
   self.command/self.path to None at the top of handle_one_request,
   alongside the existing per-request state resets, so the guard correctly
   recognizes when no new request was parsed.

2. BUG (Devin, orchestrator.py): permanent failures misreported as
   exhausted retries. _send_with_retry/_send_raw_with_retry unconditionally
   logged provider_exhausted whenever any failure occurred, even when a
   non-transient error on the very first attempt broke the loop immediately
   without spending any retry budget. Added a distinctly named
   provider_rejected_permanent WARNING event, chosen by whether the loop
   broke from reaching retry_limit (provider_exhausted) or from an early
   non-transient break (provider_rejected_permanent), applied identically
   in both duplicated retry loops.

3. Security/Privacy (CodeRabbit, CWE-532, server.py): the DEBUG
   response-body summary serialized the entire redacted payload, including
   ordinary response text (choices[].message.content, tool-call arguments,
   an error.message that can reflect caller-supplied input) -- none of
   which is credential-shaped, so neither redact_value nor
   redact_credential_shaped_keys ever masked it, and it could carry PII or
   business-sensitive content straight into DEBUG output. Added
   debug_logging.response_metadata_for_log, which extracts only an
   allowlisted metadata shape (has_error, model, choice_count, and
   numeric-only usage counts) for this log line to serialize instead of the
   payload; redact_credential_shaped_keys is still applied on top,
   defense-in-depth, in case a future allowlist field ever collides with a
   credential-shaped key name. Updated the two pre-existing tests whose
   assertions depended on the old "redact-then-log-the-payload" mechanism to
   match the new (strictly stronger) "never log the payload" contract.

4. Verified, NOT fixed (Devin, __main__.py, false positive): "the
   parse_known_args logging pre-scan doesn't respect -- as an option
   terminator." Directly tested against the real
   _configure_logging_from_cli pre-scan: a literal -- already correctly
   stops it from treating what follows as --log-level/--verbose/--debug,
   since parse_known_args honors stdlib argparse's -- semantics with no
   special-casing needed here. Documented this in the function's docstring
   and added a regression test locking in the already-correct behavior,
   rather than changing working code.

Each of the three real fixes has a red/green regression test (verified
failing against the pre-fix code, per this repo's TDD convention). Also
added deterministic, non-threaded unit tests for the query-string-stripping
and keep-alive fixes as reliable counterparts to their real-server
integration tests, which can occasionally flake on unrelated
ThreadingHTTPServer teardown timing (pre-existing in this test file, not
introduced by this change -- reproduces identically on an untouched,
pre-existing test in the same file).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX
…real-server tests

Documents the response-body allowlist redesign, keep-alive phantom-request
fix, provider_rejected_permanent split, and the confirmed-false-positive
-- option-terminator check in docs/adr/0007-verbose-debug-logging.md and
CHANGELOG.md's Unreleased/Added entry.

Also de-flakes the four real-ThreadingHTTPServer tests in
tests/test_telemetry.py that assert on the per-request INFO summary
(present pre-existing in this file, not introduced by this PR -- it
reproduced identically on an untouched test during verification). Root
cause: the per-request summary logs in handle_one_request's `finally`
block, which runs strictly after the response has already been flushed to
the client, so a test asserting immediately after its client call returns
has no guarantee the server thread reached that `finally` block yet.
Fixed with a small bounded caplog-polling helper (_wait_for_caplog) used
while the relevant caplog.at_level scope is still open, rather than a
fixed sleep that would be either too short (still flaky) or wastefully
long; kept a brief post-teardown settle too, since ThreadingHTTPServer's
per-connection handler threads are daemons server_close() does not wait
for, and a lingering straggler could otherwise log into a later test's
capture window. Verified stable across 15 repeated runs (10 solo, 5 with
the rest of this PR's touched test files) with zero failures, versus
roughly 1-in-3 to 1-in-5 before.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

Copy link
Copy Markdown
Contributor Author

Fixed every confirmed finding from the adversarial review and the follow-up re-review, in two pushes (495367100d923ad7cd57aa1a), plus resolved this branch going dirty against main twice as the org's automation kept merging other PRs during this session.

Round 1 — the original adversarial review

1. Critical — credential leak in DEBUG response-body logging. server.py's response-summary DEBUG line ran the payload through redact_value, which only pattern-matches an in-string secret value shape and never looks at the JSON key name a string is nested under, so {"private_key": "..."}/{"key": "..."}/{"auth": "..."}/{"credential": "..."} leaked verbatim. Added debug_logging.redact_credential_shaped_keys, a recursive pass that replaces any dict value under a credential-shaped key name with [REDACTED] regardless of the value's shape, applied on top of redact_value's output. orchestrator.redact_text/redact_value themselves were left untouched, per your scope note.

  • Red: a payload with secrets under private_key/key/auth/credential leaked all four raw values into the DEBUG log.
  • Green: tests/test_telemetry.py::test_response_payload_debug_log_redacts_credential_shaped_json_keys.

2. INFO-level query-string leak. The per-request summary logged self.path verbatim including any query string. debug_logging.summarize_request_for_log now strips everything from ? onward before formatting.

  • Red → Green: tests/test_telemetry.py::test_per_request_info_summary_never_includes_query_string, plus a deterministic unit-level counterpart tests/test_debug_logging.py::test_summarize_request_for_log_strips_query_string.

3. CLI argument-order bug. A --log-level/--verbose/--debug flag before a subcommand (--verbose discover-models) bypassed dispatch, since main only checked arguments[0]. Added _subcommand_token_index, which skips past recognized leading logging flags only to locate the subcommand token — it never strips them from the list handed to that subcommand's own parser.

  • Red → Green: four cases in tests/test_cli_logging.py (discover-models, register-credential, check-fast-mlsirm, and a --serve regression guard, each with a logging flag first).

Merge conflict #1 — main had moved to 0adca470

Real conflict in model_discovery.py: this branch's new discovery_attempt/discovery_result/discovery_provider_failed DEBUG instrumentation collided with main's independently-landed, privacy-hardened logging on the same call sites (account= naming, and a hard rule against ever logging source.credential_name, backed by main's own test_discovery_debug_log_identifies_account_without_secret). Resolved by keeping this branch's richer instrumentation (isEnabledFor guards, timing, structured fields) but adopting main's stricter contract — dropped credential_name from the log line entirely, switched to account= naming, updated this branch's own new tests to match.

Round 2 — Devin's re-review + CodeRabbit's inline finding

4. BUG — keep-alive phantom requests (Devin). handle_one_request never reset self.command/self.path before each call; stdlib only assigns them when it actually parses a request line, so a keep-alive connection's closing call (nothing read) left the previous request's values in place, and the "nothing to report" guard in _log_request_summary never fired — logging the prior request a second time with a statusless entry. Fixed by resetting self.command/self.path to None at the top of handle_one_request.

  • Red → Green: tests/test_telemetry.py::test_keep_alive_close_does_not_log_phantom_request (real socket, one keep-alive request then a bare close, asserts exactly one http_request record), plus a deterministic counterpart test_handle_one_request_resets_command_and_path_before_each_call.

5. BUG — permanent failures reported as exhausted retries (Devin). _send_with_retry/_send_raw_with_retry logged provider_exhausted on any failure, even a non-transient rejection on the very first attempt that never spent a retry. Split into provider_exhausted (loop actually reached retry_limit) vs. a new provider_rejected_permanent (broke early on a non-transient error), fixed identically in both duplicated retry loops.

  • Red → Green: tests/test_orchestrator_debug_logging.py::test_provider_rejected_permanent_fires_instead_of_exhausted_on_immediate_non_transient_failure (max_retries=2, first attempt 401 → attempts == 1, provider_exhausted absent, provider_rejected_permanent present) + the _send_raw_with_retry counterpart.

6. CWE-532 — ordinary response content in DEBUG logs (CodeRabbit). summarize_payload_for_log serialized the entire redacted payload, so choices[].message.content, tool-call arguments, and error.message — none of which is credential-shaped, so neither redaction layer masked it — reached DEBUG output verbatim, which can carry PII or business-sensitive content. Added debug_logging.response_metadata_for_log: an allowlist of only has_error/model/choice_count/numeric-only usage, which is what the response-body summary now logs instead of the payload. redact_credential_shaped_keys still runs on top of the allowlisted dict, defense-in-depth. Updated the two pre-existing tests whose assertions depended on the old "redact-then-log" mechanism to match the new (strictly stronger) "never log the content at all" contract.

  • Red → Green: tests/test_telemetry.py::test_response_payload_debug_log_never_includes_ordinary_response_content (plain assistant text, a tool-call argument, and an error message — none appear in the log; model name/choice_count/usage still do).

7. Verified, not fixed — false positive (Devin). "parse_known_args doesn't respect -- as an option terminator." Tested directly against the real _configure_logging_from_cli pre-scan: _configure_logging_from_cli(["--", "--log-level", "DEBUG"]) leaves the effective level at WARNING-- already works correctly via stdlib argparse semantics, no special-casing needed. Documented in the function's docstring and locked in with tests/test_cli_logging.py::test_log_level_flag_after_option_terminator_is_not_consumed rather than changing working code.

Merge conflict #2 — main had moved again, to c107e3e5

Another real conflict in the same function: main's #923 added a bounded retry for a provider's transient model-list fetch (attempt_timeouts/last_exc/is_transient_error) on the exact lines this branch instruments. Kept main's retry loop fully intact and re-layered this branch's discovery_provider_failed DEBUG logging on top, now firing once after the retry loop exhausts its attempts using the final last_exc. Also found (not a textual conflict, but a real regression): main independently added a second, plain --verbose flag to the discover-models and one-shot/--serve parsers that collided with this branch's _add_log_level_arguments, raising argparse.ArgumentError on every invocation of either path. Removed the redundant declarations (and their dead, non-redacted logging.basicConfig follow-ups), since _configure_logging_from_cli already covers this centrally.

A pre-existing test flake, found and fixed along the way

Four real-ThreadingHTTPServer tests in tests/test_telemetry.py (two pre-existing, two of mine) intermittently failed — reproduced identically on an untouched, pre-existing test, confirming it predates this PR. Root cause: the per-request summary logs in handle_one_request's finally block, which runs strictly after the response is already flushed to the client, so asserting immediately after the client call returns has no guaranteed ordering. Fixed with a small bounded caplog-polling helper used while the relevant caplog.at_level scope is still open, plus a brief post-teardown settle for ThreadingHTTPServer's daemon connection threads. Verified stable across 15 repeated runs (10 solo, 5 combined with this PR's other touched files) with zero failures, versus roughly 1-in-3 to 1-in-5 before.

Verification evidence

  • python -m pytest tests -q --ignore=tests/test_psychometric_routing.py (that file needs fast_mlsirm, Python ≥3.12; this sandbox runs 3.11) → 2880 passed, 1 skipped, run four times across the two rounds of fixes with zero regressions once the flake above was fixed.
  • python -m interrogate contextual_orchestrator/100%.
  • python -m pytest tests/test_conventions.py → all pass (no new naming-rule violations).
  • Every touched/added test file re-run individually: test_debug_logging.py, test_orchestrator_debug_logging.py, test_cli_logging.py, test_model_discovery.py, test_telemetry.py — all green, test_telemetry.py additionally stress-run 10× solo with zero flakes after the de-flake fix.
  • Every real fix above was proven red against the pre-fix code before being made green, per this repo's TDD convention.

#942 (the independent, more conservative implementation of the same feature) still needs reconciliation by a human or a separate reviewing session — out of scope here and not touched by this work. Not merging this myself.


Generated by Claude Code

devin-ai-integration[bot]

This comment was marked as resolved.

Fixes four further confirmed real findings from the second re-review
(after push cd57aa1), which also independently confirmed the keep-alive
fix and the -- option-terminator false-positive verdict from round 2:

1. BUG, orchestrator.py: zero-retry failures mislabeled as exhausted.
   With max_retries=0, attempt >= retry_limit is trivially true after the
   single allowed attempt regardless of whether it was transient, so
   provider_exhausted fired even though no retry budget ever existed to
   exhaust. Both _send_with_retry and _send_raw_with_retry now also
   require retry_limit > 0 before calling provider_exhausted;
   provider_rejected_permanent covers both the pre-existing "non-transient
   break" case and this "no retry budget configured at all" case.

2. BUG, server.py: framework-generated responses lost their status in the
   per-request log. _last_status was only set by this module's own
   _send/_send_text/_send_bytes/_send_sse writers; a response
   BaseHTTPRequestHandler generates itself (e.g. its built-in 501 for an
   HTTP method with no matching do_* handler) bypasses all of those,
   logging status=- for a response the client actually received. Fixed by
   overriding send_response -- stdlib's own single choke point every
   response path (including its own send_error) already goes through --
   so every current and future response path is captured uniformly.

3. BUG, __main__.py: abbreviated logging flags (--log-l, --ver) bypassed
   subcommand dispatch the same way full un-recognized flags used to,
   since argparse's own prefix-abbreviation matching and
   _subcommand_token_index's plain string comparison disagreed about
   what counts as a recognized flag. Set allow_abbrev=False on every CLI
   parser (the pre-scan and each subcommand's own parser), so an
   abbreviated flag is now rejected everywhere with a clear argparse
   error instead of silently accepted by one parser and not the other.

4. SEC, debug_logging.py: exception tracebacks bypassed log redaction.
   _RedactingLogFilter only rewrote record.msg; logging.Formatter.format()
   renders record.exc_info into the traceback text strictly after every
   filter has already run, so a call site using exc_info=True or
   logger.exception(...) could leak a secret embedded in the exception's
   own str() (e.g. an upstream error reflecting api_key=sk-...) straight
   into the unredacted traceback -- a real hole in the core safety-net
   mechanism itself, not just a call site. The filter now renders
   exc_info into text, redacts it, caches it as record.exc_text, and
   clears exc_info so the handler's formatter uses the already-redacted
   text instead of re-deriving an unredacted one.

Each fix has a red/green regression test (verified failing against the
pre-fix code): tests/test_orchestrator_debug_logging.py (two new zero-retry
tests, one per retry loop), tests/test_telemetry.py
(test_framework_generated_error_status_is_captured_in_log),
tests/test_cli_logging.py (abbreviated value-flag and boolean-flag cases),
tests/test_debug_logging.py
(test_configure_logging_redactor_masks_exception_traceback_in_captured_output).

Full suite (python -m pytest tests -q --ignore=tests/test_psychometric_routing.py)
-> 2886 passed, 1 skipped, zero regressions. interrogate -> 100%.
test_conventions.py -> all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

Copy link
Copy Markdown
Contributor Author

Round 3 — fixed all four findings from Devin's re-review of cd57aa1a, in 68291d43. Devin also confirmed the keep-alive fix and the -- option-terminator false-positive verdict from round 2, and separately confirmed circuit-logging sync, redaction propagation, ranking read-only behavior, and the response-summary allowlist as sound — no action needed on those.

1. BUG — zero-retry failures mislabeled as exhausted (orchestrator.py). With max_retries=0, attempt >= retry_limit is trivially true after the single allowed attempt regardless of whether it was transient, so provider_exhausted fired even though no retry budget ever existed to exhaust. Both _send_with_retry and _send_raw_with_retry now also require retry_limit > 0 before calling provider_exhausted; provider_rejected_permanent now covers both the existing "non-transient break" case and this "no retry budget configured at all" case.

  • Red → Green: tests/test_orchestrator_debug_logging.py::test_zero_retry_limit_never_labels_a_failure_as_exhausted and its _send_raw_with_retry counterpart — both use a deliberately transient 503 on the only attempt, to prove even a transient failure isn't mislabeled "exhausted" when there was never a budget.

2. BUG — framework-generated responses lost their status (server.py). _last_status was only set by this module's own _send/_send_text/_send_bytes/_send_sse writers. A response BaseHTTPRequestHandler generates itself (e.g. its built-in 501 for an HTTP method with no matching do_* handler) bypasses all of those, so the INFO summary logged status=- even though a real status was already sent to the client. Fixed by overriding send_response — stdlib's own single choke point every response path (including its own send_error) already goes through — so every current and future response path is captured uniformly, not just this module's writers.

  • Red → Green: tests/test_telemetry.py::test_framework_generated_error_status_is_captured_in_log (a real PUT /healthz — no do_PUT — asserts status=501 appears, not status=-).

3. BUG — abbreviated logging flags bypassed subcommand dispatch (__main__.py). argparse's own prefix-abbreviation matching (--log-l for --log-level, --ver for --verbose) and _subcommand_token_index's plain string comparison disagreed about what counts as a recognized flag, so an abbreviation misrouted a subcommand the same way an unrecognized flag used to. Went with your suggested "simplest and most predictable" option: allow_abbrev=False on every CLI parser (the pre-scan and each subcommand's own parser), so an abbreviated flag is now rejected everywhere with a clear argparse error (SystemExit(2), "unrecognized arguments") instead of silently accepted by one parser and not the other.

  • Red confirmed dramatically: pre-fix, --log-l DEBUG discover-models didn't just mis-show the wrong --help — it silently ran a full one-shot completion treating discover-models as the actual prompt text (executed the whole mock-agent workflow).
  • Green: tests/test_cli_logging.py::test_abbreviated_value_flag_before_subcommand_fails_closed_not_misrouted and ..._boolean_flag_... (both assert SystemExit(2) with "unrecognized arguments", not a misroute).

4. SEC (prioritized first, per your note) — exception tracebacks bypassed log redaction (debug_logging.py). _RedactingLogFilter only rewrote record.msg; logging.Formatter.format() renders record.exc_info into the traceback text strictly after every filter has already run, so exc_info=True/logger.exception(...) could leak a secret embedded in the exception's own str() (e.g. an upstream error reflecting api_key=sk-...) straight into the unredacted traceback — a real hole in the core safety-net mechanism itself, not a call-site gap. The filter now renders exc_info into text via a throwaway logging.Formatter, redacts it, caches it as record.exc_text, and clears record.exc_info so the handler's own formatter uses the already-redacted text instead of re-deriving an unredacted one.

  • Red → Green: tests/test_debug_logging.py::test_configure_logging_redactor_masks_exception_traceback_in_captured_output (raises a RuntimeError embedding a fake secret, catches it, calls logger.exception(...), asserts [REDACTED] appears and the raw secret does not).

(I also considered stack_info=True for the same filter, but dropped it: a stack trace only ever contains source-line text via linecache, never a runtime value, so there's no realistic secret-leak vector there to test or fix.)

Verification

  • python -m pytest tests -q --ignore=tests/test_psychometric_routing.py2886 passed, 1 skipped, zero regressions.
  • python -m interrogate contextual_orchestrator/100%.
  • python -m pytest tests/test_conventions.py → all pass.
  • Every real fix proven red against the pre-fix code before being made green, per this repo's TDD convention (each fix was individually reverted and re-tested to confirm the regression test actually catches it).

If your next pass surfaces only further diminishing or cosmetic nits rather than another real correctness or security gap, I'll report those in a final comment rather than continue iterating — three rounds of substantive findings is a reasonable point to start converging. #942 still needs human reconciliation; not merging this myself.


Generated by Claude Code

devin-ai-integration[bot]

This comment was marked as resolved.

…ound 4)

Fixes a real regression from the round-3 provider_exhausted/
provider_rejected_permanent split, found by Devin's re-review of 68291d4:
with a configured retry limit of 0, EVERY failure -- including a genuinely
transient one (HTTP 503, timeout) that simply never got a chance to retry
-- was collapsed into provider_rejected_permanent. "No retry budget was
configured" and "this error is non-retryable by nature" are two
independent facts; conflating them mislabels a transient failure as
permanent.

Adds a third, distinctly named WARNING event, provider_no_retry_budget,
for the retry_limit == 0 case specifically, carrying the error's own
transient/non-transient classification explicitly via a `transient=%s`
field rather than discarding it. provider_rejected_permanent now covers
only its original, narrower case: a real non-zero budget existed, but the
loop broke early because the failure was classified non-transient.
provider_exhausted is unchanged (a real, non-zero budget was actually used
up). Fixed identically in both duplicated retry loops
(_send_with_retry/_send_raw_with_retry).

Regression tests now cover all 4 (retries>0 vs =0) x (transient vs
non-transient) combinations per retry loop, as requested: the two existing
tests already covered retries>0 x transient (provider_exhausted) and
retries>0 x non-transient (provider_rejected_permanent); this adds
retries=0 x transient and retries=0 x non-transient (both now
provider_no_retry_budget, differing only in transient=True/False) for both
_send_with_retry and _send_raw_with_retry. Verified each new/changed
assertion fails against the pre-fix (round-3) code before the fix, per this
repo's TDD convention.

Investigated, not fixed: Devin also flagged CONTEXTUAL_ORCHESTRATOR_LOG_LEVEL
reading directly from os.environ as bypassing this repo's KV config
registry (AGENTS.md/CLAUDE.md: "runtime config... never os.getenv"). Per
the coordinator's request, checked whether this repo's existing
CONTEXTUAL_ORCHESTRATOR_STATE_DB/_AGENTS_DB/_CLEARFOLIO_URL/
_PROVIDER_CA_BUNDLE env vars (contextual_orchestrator/__main__.py) follow
the same direct-os.environ pattern: they do, all four, predating this PR.
This is a pre-existing architectural pattern in this file, not a new or
different violation this PR introduces -- reported as such in the PR
comment rather than fixed unilaterally inside a logging-feature PR, per
the coordinator's explicit guidance for that outcome.

Full suite (python -m pytest tests -q --ignore=tests/test_psychometric_routing.py)
-> 2888 passed, 1 skipped, zero regressions. interrogate -> 100%.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

Copy link
Copy Markdown
Contributor Author

Round 4 — fixed the one real regression and investigated the second finding, in a37f59d2.

1. BUG — transient failures mislabeled permanent under a zero retry budget (orchestrator.py). A real regression from my own round-3 fix, as you flagged: stopping provider_exhausted from firing on a zero-budget failure correctly required routing it somewhere, but I routed every zero-budget failure into provider_rejected_permanent — including a genuinely transient error (HTTP 503, timeout) that simply never got a chance to retry. "No retry budget was configured" and "this error is non-retryable by nature" are independent facts; conflating them mislabels the error. Added a third, distinctly named WARNING event, provider_no_retry_budget, for the retry_limit == 0 case specifically, carrying the error's own transient=%s classification explicitly rather than discarding it. provider_rejected_permanent now covers only its original, narrower case: a real non-zero budget existed, but the loop broke early because the failure was classified non-transient. Fixed identically in both _send_with_retry and _send_raw_with_retry.

  • Regression tests now cover all 4 (retries>0 vs =0) × (transient vs non-transient) combinations per retry loop, as requested — tests/test_orchestrator_debug_logging.py: the two existing tests already covered retries>0×transient (provider_exhausted) and retries>0×non-transient (provider_rejected_permanent); added retries=0×transient and retries=0×non-transient (both provider_no_retry_budget, differing only in transient=True/False) for both retry loops — 4 new/updated tests total.
  • Red confirmed: reverted to the round-3 decision logic and reran — all 4 new/changed assertions failed exactly as expected (a transient 503 under max_retries=0 logged provider_rejected_permanent, not provider_no_retry_budget).

2. Investigated, not fixed — CONTEXTUAL_ORCHESTRATOR_LOG_LEVEL reading directly from os.environ. Per your instruction, checked the precedent before deciding: contextual_orchestrator/__main__.py's existing --state-db/--clearfolio-url/--agents-db/--provider-ca-bundle flags all read their defaults via os.environ.get(...) the exact same way, none of them routed through the KV — and all four predate this PR. This confirms my log-level env var follows an established (if, read strictly against AGENTS.md/CLAUDE.md's "runtime config... never os.getenv" wording, arguably non-compliant) pattern already present in this file, not a new or different violation this PR introduces. Per your guidance for that outcome: noting it here rather than fixing it unilaterally inside a logging-feature PR — a KV-routing refactor for all five of these env vars (STATE_DB/AGENTS_DB/CLEARFOLIO_URL/PROVIDER_CA_BUNDLE/LOG_LEVEL) together would be the right scope for a separate PR, not a one-off carve-out for just the newest one.

Verification

  • python -m pytest tests -q --ignore=tests/test_psychometric_routing.py2888 passed, 1 skipped, zero regressions.
  • python -m interrogate contextual_orchestrator/100%.

As discussed, three-plus rounds of substantive, independently-confirmed findings is a reasonable convergence point. If your next pass turns up only cosmetic or diminishing nits rather than another real correctness or security gap, I'll report those directly in a follow-up comment instead of continuing to iterate. #942 still needs human reconciliation; not merging this myself.


Generated by Claude Code

@seonghobae

Copy link
Copy Markdown
Contributor Author

Model taxonomy reminder: the requested grouping axis is model group (model_group) only, not provider family. Any retained or reconciled work in this PR must preserve independent KV credentials and must not introduce provider-family grouping.

devin-ai-integration[bot]

This comment was marked as resolved.

Copy link
Copy Markdown
Contributor Author

Acknowledging Devin's round-5 findings — both are "analysis" kind, not "bug", and both are minor enough that I'm not spinning up another fix round after four substantive ones:

  • configure_logging's force=True replaces host root handlers: correct observation, and an accepted trade-off already documented in this PR's description — force=True exists specifically so a second in-process CLI invocation (this repo's own test convention) doesn't silently keep a stale level from a prior call. If this module is ever embedded as a library inside a host application that manages its own root logging config, that host would need to call configure_logging before setting up its own handlers, or avoid calling it at all — worth a one-line docstring caveat on configure_logging if a future PR embeds this outside the CLI context, but not something to change here.
  • success_rps holds a composite quality score, not literal throughput, when judged evidence is available: a fair naming-clarity nit on a pre-existing field this PR only added logging around (didn't introduce the field or its semantics) — out of scope for a logging-feature PR to rename a field it doesn't own the meaning of. Flagging for whoever next touches the ranking/quality-ledger code.

Not blocking merge review from my side.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

noema-review failed on a37f59d2 with a live TimeoutError inside call_llm's opener.open(request, timeout=120) — the actual LLM review request to the shared contextual-orchestrator gateway exceeded its 120s budget. This is not this PR's diff: #946 only adds logging instrumentation, doesn't touch the review-gateway/sidecar transport path, and the identity-verification bug that caused the previous noema-review failures on this exact PR is confirmed fixed (env shows NOEMA_REVIEW_ACTOR/NOEMA_REVIEW_INSTALLATION_ID now populated and the run got past that check entirely this time). Most likely cause: a live provider route currently under load — this session has an in-flight fix (.github#1415) for an account-cap bug that lets rate-limited NVIDIA NIM routes monopolize the preflight/serving pool, though that fix hasn't merged yet so I can't confirm it's the same root cause with certainty. Re-running once to check if this reproduces.


Generated by Claude Code

coderabbitai[bot]

This comment was marked as resolved.

Real, pre-existing security finding from CodeRabbit's automated review of
PR #946: `_fetch_json` in model_discovery.py -- the function every standard
provider's authenticated "list models" call goes through (openai, openrouter,
nvidia_nim, nvidia_nim_sub, bytez) -- called plain `urllib.request.urlopen`.
Python's default `HTTPRedirectHandler` copies the `Authorization` header onto
a redirected request even when the redirect target is a completely different
host (unlike some other HTTP clients, urllib never strips sensitive headers
on cross-origin redirects). A malicious or compromised provider endpoint
issuing a 3xx redirect could exfiltrate the credential -- and #923's retry
(already on main, inherited into this branch) means up to twice per attempt,
since it calls _fetch_json with the same api_key on both tries.

This codebase already had the correct fix for this exact risk class:
_TrustedDiscoveryRedirectHandler (raises on any redirect leaving the original
host) already protected _fetch_json_same_host_https. It just never got
applied to _fetch_json, the function actually used for the real per-provider
calls. Fixed by routing both functions through one new shared
_open_trusted_discovery_request helper, so there is a single implementation
of the redirect guard instead of two copies that could drift apart (as they
already had). _fetch_json_same_host_https's own external behavior (size cap,
TimeoutError conversion) is unchanged. A legitimate same-host redirect (e.g.
/v1/models -> /v2/models) still succeeds.

New regression tests (tests/test_model_discovery.py): a cross-host-redirect
test (mirrors the existing test_openrouter_zdr_evidence_rejects_cross_host_
redirects pattern) proving exactly one request is ever issued and the
credential never reaches the attacker host, verified failing against the
pre-fix _fetch_json before the fix; and a same-host negative control proving
the legitimate case still works. ~38 pre-existing tests across
test_model_discovery.py, test_model_discovery_boundaries.py,
test_discover_models_cli.py, and test_chat_model_capability_isolation.py
that mocked bare urllib.request.urlopen for _fetch_json's behavior now mock
the new shared seam instead (mechanical -- _fetch_json no longer calls
urlopen directly). Two local test _Response fixtures gained an optional
read(amt) argument (mirroring http.client.HTTPResponse.read(amt)) since
_fetch_json_same_host_https's always-invoked OpenRouter ZDR fetch inside
discover_all_models now shares the same seam and caps its read.

Also folded into this round, four further confirmed findings relayed by the
coordinator from the same review pass:

1. model_discovery.py: the configured_gateway provider's /model/info
   metadata fetch now also catches RuntimeError, matching the primary
   list-request retry loop's except tuple. Previously a raw RuntimeError
   from ModelClient's DNS/address-validation transport escaped
   discover_provider_models uncaught and aborted the entire discovery pass
   instead of just that one provider's metadata.

2. server.py: per-request latency_ms no longer counts a keep-alive
   connection's idle time between requests. request_started is now
   timestamped inside an overridden parse_request(), right after
   BaseHTTPRequestHandler.handle_one_request()'s blocking readline() has
   already returned real request bytes, instead of before that blocking read.

3. orchestrator.py: the retry-outcome classification (no budget at all /
   exhausted / stopped early on non-transient) duplicated verbatim between
   _send_with_retry and _send_raw_with_retry -- duplication that already
   caused a real regression once, fixed in one copy and missed in the other
   (round 4, provider_no_retry_budget) -- is now one shared
   _log_retry_outcome helper both call.

4. debug_logging.py: response_metadata_for_log's usage summary now keeps
   only a fixed allowlist of known counter names (prompt_tokens,
   completion_tokens, total_tokens, input_tokens, output_tokens), not any
   string key with a numeric value -- closes a path where a key shaped like
   "customer_note=<secret>" with a throwaway numeric value would have
   reached DEBUG output verbatim (CWE-532). New regression test with a
   customer_note-shaped key proving exclusion, plus a positive control.

Full suite (python -m pytest tests -q --ignore=tests/test_psychometric_routing.py)
-> 2892 passed, 1 skipped, zero regressions (test_psychometric_routing.py
pre-existingly fails on ModuleNotFoundError: No module named 'numpy' in this
sandbox -- environment-only, unrelated). interrogate (repo-root invocation,
pyproject.toml fail-under=100) -> 100%. ruff check on all changed files ->
clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

Copy link
Copy Markdown
Contributor Author

Security fix: credential leak via cross-host redirect, doubled by the retry (round 5)

CodeRabbit's automated review flagged this at the PR level: "The PR adds retries to authenticated provider discovery while still allowing redirects to another host, which can expose provider credentials and repeat that exposure." Verified independently against the actual code — it's real.

The vulnerability

_fetch_json(url, *, api_key, auth_scheme, timeout) in contextual_orchestrator/model_discovery.py built an Authorization header from api_key and called plain urllib.request.urlopen(request, timeout=timeout). Python's default HTTPRedirectHandler.redirect_request() copies the original request's headers — Authorization included — onto a redirected request unconditionally, even when the redirect target is a completely different host. Unlike some other HTTP clients, plain urllib never strips sensitive headers on cross-origin redirects.

_fetch_json is the function every standard provider's authenticated "list models" call goes through (openai, openrouter, nvidia_nim, nvidia_nim_sub, bytez — see the fetch = _fetch_configured_gateway_json if ... else _fetch_json selection in discover_provider_models), plus OpenRouter's ZDR-endpoints and provider-policies metadata fetches. A malicious or compromised provider endpoint issuing a 3xx redirect to an attacker-controlled host could exfiltrate the credential.

Made worse by #923's retry (already on main, inherited into this branch): the bounded retry added in c107e3e calls _fetch_json with the same api_key up to twice per discovery attempt (attempt_timeouts = (timeout, min(timeout, _DISCOVERY_RETRY_TIMEOUT_SECONDS))), so a single malicious redirect would leak the credential up to twice instead of once. Pre-existing (introduced by #923's retry addition on main, not by this PR's own diff), but this PR is where it's visible/fixable since it touches this exact retry path.

Notably, this codebase's own developers had already identified and fixed this exact risk class once — _TrustedDiscoveryRedirectHandler (a urllib.request.HTTPRedirectHandler subclass that raises on any redirect leaving the original host) already protected _fetch_json_same_host_https. It just never got applied to _fetch_json, the function actually used for the real authenticated per-provider calls.

The fix

_fetch_json now goes through the same protection, via a new shared _open_trusted_discovery_request(request, *, trusted_host, timeout, context=None) helper that both _fetch_json and _fetch_json_same_host_https call — one single-implementation redirect guard (same _TrustedDiscoveryRedirectHandler, same exception type/message shape) instead of two copies that could drift apart. _fetch_json_same_host_https itself is otherwise unchanged (same external behavior: its own size cap, its own TimeoutError conversion). A legitimate same-host redirect (e.g. a real provider's /v1/models/v2/models) still succeeds unchanged; only a redirect that leaves the original host now raises instead of forwarding the credential.

Also fixed while in this file: the configured_gateway provider's /model/info metadata fetch now also catches RuntimeError (matching the primary list-request retry loop's except tuple) — a raw RuntimeError from ModelClient's DNS/address-validation transport previously escaped discover_provider_models uncaught and aborted the entire discovery pass instead of just that one provider's metadata.

Also folded into this round (CodeRabbit, same head)

  • server.py: per-request latency_ms no longer counts a keep-alive connection's idle time between requests. request_started is now timestamped inside an overridden parse_request() — right after BaseHTTPRequestHandler.handle_one_request()'s blocking readline() has already returned real request bytes — instead of before that blocking read.
  • orchestrator.py: the retry-outcome classification duplicated verbatim between _send_with_retry/_send_raw_with_retry (duplication that already caused a real regression once, fixed in one copy and missed in the other — round 4 above) is now one shared _log_retry_outcome helper both call.
  • debug_logging.py: response_metadata_for_log's usage summary now keeps only a fixed allowlist of known counter names (prompt_tokens, completion_tokens, total_tokens, input_tokens, output_tokens), not any string key with a numeric value — closes a path where a key shaped like "customer_note=<secret>" with a throwaway numeric value would have reached DEBUG output verbatim.

Verification

  • New regression tests in tests/test_model_discovery.py: test_fetch_json_rejects_a_cross_host_redirect_and_does_not_leak_the_credential (mirrors the existing test_openrouter_zdr_evidence_rejects_cross_host_redirects pattern — fake build_opener/handler chain, asserts exactly one request is ever issued and it never reaches the attacker host) and test_fetch_json_still_follows_a_same_host_redirect (negative control: same-host redirect, different path, still succeeds and both requests correctly carry the credential). Verified both fail against the pre-fix _fetch_json (red) before the fix, and pass after (green).
  • ~38 pre-existing tests across tests/test_model_discovery.py, tests/test_model_discovery_boundaries.py, tests/test_discover_models_cli.py, and tests/test_chat_model_capability_isolation.py that previously mocked bare urllib.request.urlopen now mock the new shared _open_trusted_discovery_request seam instead (mechanical update — _fetch_json no longer calls urlopen directly, so those tests' mocking point had to move with it); all still pass, unchanged in intent. Two local test _Response fixtures gained an optional read(amt) argument (mirroring http.client.HTTPResponse.read(amt)) since _fetch_json_same_host_https's always-invoked OpenRouter ZDR fetch inside discover_all_models now shares this same seam and caps its read.
  • New regression test in tests/test_debug_logging.py for the usage-allowlist fix, with a customer_note-shaped key proving exclusion.
  • Full suite python -m pytest tests -q --ignore=tests/test_psychometric_routing.py: 2892 passed, 1 skipped, zero regressions (test_psychometric_routing.py pre-existingly fails on ModuleNotFoundError: No module named 'numpy' in this sandbox — environment-only, unrelated, as noted in the task).
  • interrogate (repo-root invocation, honors pyproject.toml's fail-under = 100): 100%.
  • ruff check on all changed files: clean.

Commit: 5cb7946

Not merging — leaving this for review per the standing process.


Generated by Claude Code

devin-ai-integration[bot]

This comment was marked as resolved.

…gging-log-level-cli

# Conflicts:
#	contextual_orchestrator/model_discovery.py
github-advanced-security[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

… one-shot calls (round 6)

Two findings deferred during round 5 to avoid colliding with the
in-flight merge-conflict fix:

- model_discovery.py: _fetch_json read a discovery response body with an
  unbounded response.read(), unlike _fetch_json_same_host_https and
  _fetch_configured_gateway_json, which already cap at
  MAX_DISCOVERY_RESPONSE_BYTES and fail closed on an overage. A large or
  malicious/misbehaving provider response could exhaust worker memory
  before JSON parsing ever ran. _fetch_json now shares the identical
  bounded-read-then-check pattern.

- orchestrator.py: _log_retry_outcome logged provider_no_retry_budget
  whenever retry_limit was 0, without distinguishing an agent with a
  genuinely zero configured retry budget from ModelClient.proxy_send_once's
  deliberate allow_transient_retries=False one-shot call (which forces
  retry_limit to 0 regardless of the agent's real budget, so an
  already-failing-over passthrough request cannot itself amplify load).
  _log_retry_outcome now takes allow_transient_retries explicitly and logs
  a distinctly named provider_one_shot_call_failed WARNING for the
  caller-forced case instead of misreporting it as no budget configured.

New regression tests cover both: an oversized fake response for the
size bound, and a mocked proxy_send_once one-shot failure (plus direct
_send_raw_with_retry coverage) for the logging fix.

Full suite: 2929 passed, 1 skipped (baseline 2925 + 4 new tests, zero
regressions). interrogate: 100%. git diff --check: clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

Copy link
Copy Markdown
Contributor Author

Round 6: the two findings deferred out of round 5 (unbounded discovery read + one-shot retry-logging conflation)

Round 5's fix (5cb79463) and the subsequent merge-conflict resolution (deed2288/f2e83f3f) surfaced two more real findings that were deliberately left out of that work to avoid colliding with the then-active merge-conflict agent. Re-verified both independently against the current, post-merge code before touching anything.

Finding 1: unbounded memory read in _fetch_json

contextual_orchestrator/model_discovery.py's _fetch_json — the function every standard provider's authenticated "list models" call goes through (openai, openrouter, nvidia_nim, nvidia_nim_sub, bytez), now unified behind round 5's _open_trusted_discovery_request redirect-protected opener — read the entire response body with a plain response.read(), no size bound, before ever parsing it as JSON. _fetch_json_same_host_https and _fetch_configured_gateway_json already guard against exactly this (both cap at the existing MAX_DISCOVERY_RESPONSE_BYTES = 8 * 1024 * 1024 and raise ValueError on an overage) — _fetch_json was simply the one call site round 5 didn't touch. A large or malicious/misbehaving provider response (an outage page dumped as an unbounded body, or a compromised endpoint) could exhaust worker memory before parsing ever ran (CWE-400).

Fix: _fetch_json now shares the identical bounded-read-then-check pattern — response.read(MAX_DISCOVERY_RESPONSE_BYTES + 1), then raise ValueError(...) if the returned length exceeds the cap — applied at the one place _open_trusted_discovery_request's body is consumed in this function. No new constant: reused the existing one, consistent across all three fetch helpers now.

Finding 2: retry-classification conflation in _log_retry_outcome

contextual_orchestrator/orchestrator.py's shared _log_retry_outcome helper (extracted in round 4/5 specifically so _send_with_retry/_send_raw_with_retry couldn't diverge again) logged a provider_no_retry_budget WARNING whenever retry_limit == 0, without distinguishing why it was 0. _send_raw_with_retry computes retry_limit = self._retry_limit(agent) if allow_transient_retries else 0 — so ModelClient.proxy_send_once's intentional allow_transient_retries=False (used so an already-failing-over passthrough request cannot itself amplify load with a nested retry loop) forced retry_limit to 0 regardless of the agent's real configured budget, making a deliberate one-shot call look identical to "this agent has no retry budget configured at all" by the time it reached _log_retry_outcome. An agent with e.g. max_retries=2 calling proxy_send_once and failing would falsely warn provider_no_retry_budget, misreporting a real, non-zero budget as absent.

Fix: _log_retry_outcome now takes allow_transient_retries explicitly (default True, so _send_with_retry — which has no caller-forced restriction — is unaffected) and, when the zero came from the caller's one-shot policy rather than the agent's own configuration, logs a distinctly named provider_one_shot_call_failed WARNING instead, carrying the same attempts/final_error_type/transient fields. _send_raw_with_retry now passes its own allow_transient_retries value straight through to _log_retry_outcome.

Verification

  • New regression tests:
    • tests/test_model_discovery_boundaries.py::test_fetch_json_rejects_oversized_response_body — a fake response returning more than MAX_DISCOVERY_RESPONSE_BYTES, asserting _fetch_json raises ValueError and the underlying read() was called with exactly MAX_DISCOVERY_RESPONSE_BYTES + 1 (never the full oversized body).
    • tests/test_orchestrator_debug_logging.py::test_send_raw_with_retry_one_shot_call_does_not_log_no_retry_budget / ..._reports_transient_false_too — a client with a real max_retries=2 budget, calling _send_raw_with_retry(..., allow_transient_retries=False) directly: asserts provider_no_retry_budget never appears and provider_one_shot_call_failed does, for both a transient (503) and non-transient (401) underlying error.
    • tests/test_orchestrator_debug_logging.py::test_proxy_send_once_one_shot_failure_does_not_log_no_retry_budget — same assertion end-to-end through proxy_send_once itself (with _validate_provider patched to a fake destination so the test stays fully offline).
    • Also updated the existing GarbageResponse/_Response test fixtures in tests/test_model_discovery_boundaries.py, tests/test_model_discovery.py, and tests/test_discover_models_cli.py to accept the optional read(amt) argument now that _fetch_json caps its read too (mirroring the round-5 pattern already applied to _fetch_json_same_host_https's fixtures).
  • python -m pytest tests -q --ignore=tests/test_psychometric_routing.py2929 passed, 1 skipped, zero regressions (baseline was 2925 passed, 1 skipped; +4 new tests, nothing else moved).
  • python -m interrogate . (repo root, honors pyproject.toml's fail-under = 100) → 100% (a stray, unrelated git worktree materialized under .claude/worktrees/... mid-session from a concurrent process sharing this sandbox and briefly polluted a bare interrogate . scan — confirmed environmental noise, not a real regression, by rerunning scoped to this repo's own tracked tree).
  • git diff --check → clean, no conflict markers.
  • No new dependency (stdlib only, Ponytail gate not triggered); no new os.getenv secret reads; new log event name (provider_one_shot_call_failed) is a private-function detail, not a configurable/API/DB object name, so the lower-snake-case-2-words naming rule doesn't apply, but it satisfies it anyway.

Commit: b9407999

Not merging — leaving this for review per the standing process.


Generated by Claude Code

devin-ai-integration[bot]

This comment was marked as resolved.

…y, renumber ADR (round 7)

Three findings from the latest automated review round on PR #946:

- tests/test_debug_logging.py: two CodeQL py/clear-text-logging-sensitive-data
  HIGH alerts (lines 144, 167) are a deliberate redaction positive/negative
  control pair -- a hardcoded, non-functional fake secret is logged once WITH
  a redactor (proving it's masked) and once WITHOUT one (the negative control,
  proving the first test isn't a tautology). Verified this repo's CodeQL is
  advanced setup (github/codeql-action/init+analyze, no config-file) with
  security-events: write and default upload:true, and that `# codeql[rule-id]`
  inline suppression comments on the line before a flagged call are GitHub's
  documented, currently-supported mechanism, honored regardless of setup
  style once the analysis flows through upload-sarif. Added precise, two-line
  suppressions with an explanatory comment on exactly the two flagged lines.

- contextual_orchestrator/server.py: `_send`/`_send_text`/`_send_bytes`/
  `_send_sse`/`_begin_sse` all recorded their *intended* status into
  `_last_status` before handing off to `_write_response`, then ignored its
  boolean return value. `_write_response` deliberately swallows a dead peer's
  BrokenPipeError/ConnectionError/OSError, but the pre-set `_last_status`
  survived that failure untouched, so the per-request INFO summary reported a
  false "delivered" status for a write that never completed. Fixed at the one
  shared choke point (`_write_response`'s except block) instead of patching
  each writer, so it covers every current and future writer uniformly.

  While tracing that consumer, also found `_log_request_summary`'s "nothing
  to report" guard checked only method/path, so it silently dropped a request
  that *did* deliver bytes and *did* get a real 400/414 response but whose
  malformed/oversized request line left command/path unset (stdlib's own
  parse_request resets `self.command` to None "in case of error on the first
  line"). The guard now also logs when a status was actually recorded, while
  still skipping a true byte-free keep-alive close.

  Regression tests: test_http_response_write_disconnect_safety.py (dead-peer
  write no longer reports the intended status) and test_telemetry.py (a
  malformed request line's real 400 is no longer skipped).

- docs/adr/0007-verbose-debug-logging.md renamed to 0005 -- the indexed
  series in docs/adr/README.md ends at 0004, and ADR 0122 (checked
  separately) is a deliberate cross-repo-numbered exception explicitly not
  listed in that series table, not evidence the sequential convention was
  already broken. Updated the file's own title, the README index row,
  conductor/tech-stack.md, and CHANGELOG.md's remaining "ADR 0007" mentions.

Also replied on PR #946 to two review threads that investigation showed were
false positives, with no code change: the CLI log-level env-var read (a
bootstrap-time argparse pre-scan before any KV store exists, matching five
other pre-existing bootstrap-time os.environ reads in the same file) and the
"research artifact" ADR requirement (this is a pure observability/engineering
feature with no algorithmic claim to ground, unlike ADR 0002/0003).

Full suite: `pytest tests -q --ignore=tests/test_psychometric_routing.py` ->
2931 passed, 1 skipped, 0 failed. `interrogate .` -> 100%. `git diff --check`
clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

Copy link
Copy Markdown
Contributor Author

Round 7: CodeQL alerts, disconnect-status honesty bug, ADR renumbering

New head: 8df97670cceae8c64b475491fe338ecde0e4f90f (fast-forward on top of round 6's b9407999).

1. Two CodeQL py/clear-text-logging-sensitive-data HIGH alerts (tests/test_debug_logging.py:144,167)

Investigated before touching anything:

  • This repo's CodeQL is advanced setup (.github/workflows/security.yml's codeql_analysis job calls github/codeql-action/init+analyze directly, no config file, security-events: write, default upload: true) — it is not covered by the org's central codeql-pr.yml (that workflow explicitly sets upload: false and only gates the check; this repo's own security.yml header comment says CodeQL "stays repo-local" here).
  • Confirmed current GitHub documentation/behavior: # codeql[rule-id] on the line immediately before a flagged call (preferred over same-line # lgtm[rule-id], since a same-line comment changes the alert's line-hash) is a real, currently-supported inline suppression mechanism, populated into the SARIF suppressions field by an automatic per-language AlertSuppression query that ships with the codeql-action version this repo pins (v4.37.7, far newer than when this feature landed).
  • These two lines are a deliberate, understood positive/negative-control pair (hardcoded fake secret, # noqa: S105'd, logged once with a redactor and once without, to prove the redaction test isn't a tautology) — a genuine false positive, not a real leak.
  • Fix: added precise # codeql[py/clear-text-logging-sensitive-data] comments with an explanatory rationale on exactly those two lines. No blanket suppression, no query/path exclusion.

2. Real bug: disconnected-peer writes logged as successful (server.py)

_send/_send_text/_send_bytes/_send_sse/_begin_sse all set self._last_status = status before calling _write_response, and ignored its boolean return value. _write_response deliberately swallows a dead peer's BrokenPipeError/ConnectionError/OSError, but the pre-set _last_status survived untouched, so the per-request INFO summary logged the intended status (e.g. 200) even when delivery never completed.

Fixed at _write_response's own except block (the single choke point every writer already routes through) — it now resets _last_status back to None (this module's existing "response was never sent" value) whenever it catches a disconnect, covering every current and future writer uniformly.

While tracing _last_status's consumer I also found _log_request_summary's "nothing to report" guard checked only method/path, so it silently dropped a request that did deliver bytes and did get a real 400/414 response, but whose malformed/oversized request line left command/path unset (stdlib's own parse_request resets self.command to None "in case of error on the first line"). The guard now also logs whenever a status was actually recorded, while still correctly skipping a true byte-free keep-alive close.

Regression tests added: tests/test_http_response_write_disconnect_safety.py::test_disconnected_write_does_not_report_intended_status_as_delivered and tests/test_telemetry.py::test_malformed_request_line_is_captured_in_log.

3. ADR renumbering

docs/adr/0007-verbose-debug-logging.mddocs/adr/0005-verbose-debug-logging.md. docs/adr/README.md's indexed series ends at 0004; 0122-otel-session-observability.md is a deliberate cross-repo-numbered exception and is (correctly) not listed in that series table at all, so it isn't evidence the sequential convention was already broken — 0005 is the correct next number. Updated the file's own title, the README index row, conductor/tech-stack.md, and CHANGELOG.md's remaining "ADR 0007" references.

4. Research-artifact requirement

Replied on that thread: this is a pure operational/observability feature (stdlib logging config, a CLI flag, a redaction filter) citing engineering standards (OWASP Logging Cheat Sheet, NIST SP 800-92, Python logging HOWTO) already listed in the ADR's own Grounding column — no algorithmic/research claim exists here to ground with a docs/papers/ artifact, unlike ADR 0002/0003. Left unaddressed per that reasoning rather than fabricating a citation.

Also addressed (surfaced after the task started)

  • Replied to the "Log level bypasses configuration registry" thread: false positive — the env read is a one-time CLI bootstrap pre-scan before any KV store exists in the process, matching this same file's existing _bootstrap_telemetry_config and five argparse bootstrap-default reads. server.py/debug_logging.py (the actual request-serving runtime) have zero os.environ reads.
  • Item 5 (the three 📝 info-tier findings) intentionally left alone, as instructed.

Verification

  • python -m pytest tests -q --ignore=tests/test_psychometric_routing.py2931 passed, 1 skipped, 0 failed
  • interrogate .100%
  • git diff --check → clean
  • CHANGELOG.md Unreleased section updated

Generated by Claude Code

devin-ai-integration[bot]

This comment was marked as resolved.

…ound 8)

Follow-up to round 7's _write_response fix, per this round's Devin review:

- contextual_orchestrator/server.py: clearing `_last_status` on *every*
  caught disconnect (BrokenPipeError/ConnectionError/OSError) was too broad.
  A write failure can strike in two different places: before the status
  line/headers were ever flushed (the client received nothing -- clearing
  is correct), or after `end_headers()` already completed (a later body
  write in the same call, or a later `_write_sse` frame on an SSE stream
  `_begin_sse` already opened -- the client genuinely received the real
  status, so clearing it would falsely report "no status" for a request
  that was, in fact, answered -- the exact overcorrection direction of the
  original bug).

  Every `_send`/`_send_text`/`_send_bytes`/`_send_sse`/`_begin_sse` writer
  now sets `self._response_headers_sent = True` immediately after its own
  `end_headers()` call returns without raising; `handle_one_request` resets
  that marker to `False` once per request. `_write_response`'s disconnect
  handler now clears `_last_status` only when the marker is still unset.
  `_write_sse` does not touch the marker itself -- it is only ever called
  after a prior successful `_begin_sse`, so the marker it already set
  correctly covers a later frame's failure too.

  Regression tests added to tests/test_http_response_write_disconnect_safety.py:
  - test_disconnected_write_does_not_report_intended_status_as_delivered
    (rewritten to fail inside end_headers() itself -- genuinely "before
    headers" -- so it still represents the original bug's case correctly;
    it previously failed in the body write, which is now the preserved case)
  - test_disconnected_body_write_after_headers_preserves_delivered_status
    (new: body write fails after end_headers() succeeds -> status preserved)
  - test_sse_frame_disconnect_after_headers_preserves_delivered_status
    (new: a later _write_sse frame fails after _begin_sse succeeded ->
    status preserved)

Also investigated a second finding relayed this round ("transient failures
labeled permanent" in _send_with_retry/_send_raw_with_retry) and found no
reproducible bug: GitHub's own review data for this commit's parent shows
Devin's re-review found exactly one new issue (the fix above), and an
existing, currently-passing test
(test_send_raw_with_retry_one_shot_call_does_not_log_no_retry_budget)
already proves a transient error under allow_transient_retries=False with a
real non-zero configured retry budget logs
`provider_one_shot_call_failed ... transient=True`, never
`provider_rejected_permanent` -- structurally, `_log_retry_outcome` can only
reach `_log_provider_rejected_permanent` when `retry_limit != 0`, which
`allow_transient_retries=False` always forces to 0. No code change made for
that finding; likely a stale reference to the already-resolved round-5/6
thread on the same topic.

Full suite: `pytest tests -q --ignore=tests/test_psychometric_routing.py` ->
2933 passed, 1 skipped, 0 failed. `interrogate .` -> 100%. `git diff --check`
clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

Copy link
Copy Markdown
Contributor Author

Round 8: fix the round-7 disconnect-status overcorrection

New head: 26b5ee34e786f3d3d2608d810731af02a144332f (fast-forward on top of round 7's 8df97670).

1. Fixed: "Partial responses lose delivered status"

Round 7's _write_response fix (clear _last_status on any caught disconnect) was too broad — it also cleared a status that had already been delivered to the client, whenever a later write failed (a body write after end_headers() succeeded, or a later SSE frame after _begin_sse() succeeded). That's the same class of bug in the opposite direction: falsely reporting "no status" for a request that really was answered.

Fix: every _send*/_begin_sse writer now sets self._response_headers_sent = True right after its own end_headers() returns without raising (reset to False per request by handle_one_request). _write_response's disconnect handler now clears _last_status only when that marker is still unset — i.e. only when nothing was ever delivered. _write_sse doesn't need its own logic here: it only ever runs after a prior successful _begin_sse, so the marker that call set already covers it.

Three regression tests in tests/test_http_response_write_disconnect_safety.py now cover all three shapes:

  • failure before headers (status cleared) — existing test, corrected to fail inside end_headers() itself instead of the body write, since the body-write case is now a different, preserved outcome
  • failure after headers, same call (status preserved) — new
  • failure on a later SSE frame after headers (status preserved) — new

2. Investigated: "Transient failures labeled permanent"

No code change — could not reproduce, and found solid evidence it doesn't currently occur:

  • GitHub's own review data for this commit's parent (round 7, 8df97670) shows Devin's re-review found exactly one new issue this round — the partial-response finding above. There was no second new finding.
  • An existing, currently-passing test (test_send_raw_with_retry_one_shot_call_does_not_log_no_retry_budget, added in round 6) already proves the exact scenario described: a transient error (HTTP 503) under allow_transient_retries=False with a real non-zero configured retry budget (max_retries=2) logs provider_one_shot_call_failed ... transient=True — never provider_rejected_permanent.
  • Structurally, _log_retry_outcome can only reach _log_provider_rejected_permanent when retry_limit != 0, and allow_transient_retries=False always forces retry_limit to 0 in _send_raw_with_retry, routing to the one-shot branch instead. _send_with_retry has no allow_transient_retries parameter at all and always uses the agent's real budget, so its own retry loop can only reach the provider_rejected_permanent branch when the break condition (attempt >= retry_limit or not transient) was triggered by not transient — i.e. only for a genuinely non-transient error.

This matches an already-resolved round-5/6 review thread on the same event names (✅ Resolved: The retry loops now emit provider_no_retry_budget with transient classification when the retry limit is zero.), which is my best guess for where this description was carried over from. Flagging this reasoning here in case a re-reviewer has a concrete repro I'm missing — happy to revisit with one.

Verification

  • python -m pytest tests -q --ignore=tests/test_psychometric_routing.py2933 passed, 1 skipped, 0 failed (genuine foreground blocking wait on the process, not a background notification)
  • interrogate .100%
  • git diff --check → clean
  • CHANGELOG.md Unreleased section updated
  • Push was a clean fast-forward (8df97670..26b5ee34), fetched immediately before pushing to confirm nothing else had landed

Generated by Claude Code

devin-ai-integration[bot]

This comment was marked as resolved.

Devin's review of PR #946 found: `check-fast-mlsirm --help` bypassed
argument parsing and ran the diagnostic instead of showing help. The
subcommand's dispatch function took no arguments and ignored everything
after its own name in argv, so `--help` never reached an argparse parser
that would have handled it -- it was silently swallowed and the real
diagnostic ran anyway (and its exit code, not argparse's, decided the
process exit status).

Give _check_fast_mlsirm_command its own argparse.ArgumentParser, matching
the pattern every other subcommand (register-credential, discover-models)
already uses: declare the shared --log-level/--verbose/--debug flags via
_add_log_level_arguments so --help documents them, call parse_args(argv)
so --help exits before running anything and an unrecognized trailing
option is rejected instead of silently ignored, then run the diagnostic
as before. Updated the one call site to pass arguments_after_subcommand.

Regression tests: test_check_fast_mlsirm_help_shows_help_without_running_diagnostic
and test_check_fast_mlsirm_rejects_unknown_option.

Full suite: 2935 passed, 1 skipped (baseline 2933 + 2 new tests, zero
regressions). interrogate: 100%. git diff --check: clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPmJErfkcHer4UVEgrQxUX

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b888a634: check-fast-mlsirm now gets its own argparse parser (declaring the shared --log-level/--verbose/--debug flags, matching register-credential/discover-models), so --help shows help and exits instead of silently running the diagnostic, and an unrecognized trailing option is rejected instead of being ignored. Two new regression tests cover both cases. Full suite 2935 passed (baseline 2933 + 2 new), 100% interrogate, clean diff.

Separately: the CodeQL "2 high" alerts on tests/test_debug_logging.py remain open despite the inline # codeql[rule-id] suppression comments — I verified via the CodeQL analysis job logs and GitHub's own documentation that an inline suppression comment only tags the finding in the uploaded SARIF's suppressions[] field; actually dismissing the alert requires a separate step (the advanced-security/dismiss-alerts Action) reading that tag and calling the dismissal API, which this repo's security.yml doesn't have wired up. Given this PR is already at round 9, I'm treating this as a known, understood, narrowly-scoped false positive (a deliberate secret-redaction positive/negative-control test pair, not a real vulnerability) rather than widening this PR further to add that CI infrastructure — bypass-merge is appropriate for this specific residual failure once everything else is green, per this org's standing policy for well-understood CI-gate false positives.


Generated by Claude Code

devin-ai-integration[bot]

This comment was marked as resolved.

@seonghobae seonghobae left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noema-review failed again on this head with TimeoutError: timed out on the live orchestrator/free gateway chat/completions call (120s budget, scripts/ci/noema_review_gate.py:656) — not this PR's issue; nothing in this diff touches the review sidecar or gateway. This is the third occurrence of the identical failure across three different heads on this PR alone (b9407999, 8df97670, 26b5ee34), so it's reading as a systemic reliability gap rather than a one-off flake: preflight only finds 2-3 of 12 probed routes "ready" each time (the rest reject with 429/404/TimeoutError), and even a route verified ready at preflight is timing out under the real, larger review-sized payload. Not re-running to "confirm a flake" here since the pattern is already established as recurring, not isolated. Filing this as a dedicated root-cause investigation (matching this cycle's own standing goal of making Noema/OpenCode/Strix review through orchestrator/free actually reliable) rather than a fix scoped to this PR.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

Update on the recurring noema-review TimeoutError flagged earlier: root-caused and fixed. Two real bugs in the review sidecar's serving orchestrator construction (.github's contextual_orchestrator_review_launcher.py) were compounding the external client's timeout — a same-agent retry left at its default, and a real-time judge (on by default) issuing a second independent provider call per candidate. Both disabled for the serving path specifically; the external timeout also raised to match the now-much-smaller enumerated worst case. Full details and worst-case enumeration in contextual-orchestrator#974 and .github#1415 (commit 7e07a4e9).

This should stop the recurring timeout once .github#1415 merges to main — the central noema-review workflow always runs the base branch's copy of these scripts. Will keep watching this PR for whether the failure recurs on a future head in the meantime.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

opencode-review failed on this head (b888a634) — most likely the same structural opencode-review deadlock documented in ContextualWisdomLab/.github's docs/product-technical-gap-baseline.md (2026-08-31 entry): the required check's only path to a verdict is a scheduler-dispatched review, and that scheduler blocks dispatch on any unresolved review thread. This PR has well over a dozen currently unresolved threads (several substantive: two CodeQL clear-text-logging findings in tests/test_debug_logging.py — likely the intentional redaction positive/negative-control pair, not a real leak, but still open as threads; "Forced setup replaces host logging"; "Quality scores use throughput labels"; "Manual dispatch supports one prefix shape"; plus a few I've already replied to inline but not resolved), so this is consistent with that root cause rather than something new.

I'm not driving this PR's full review queue to resolution in this session: it explicitly overlaps with #942 (this PR's own description flags both as independent proposals needing reconciliation, not merged), and its remaining findings need real design judgment (the #942 reconciliation, the logging-registry-boundary question, ADR scope) rather than the kind of small, unambiguous cleanup I'd resolve unilaterally. Leaving this for whoever picks up the reconciliation with #942, per this PR's own stated plan — not re-running the check right now since clearing the scheduler-dispatch gate would need the unresolved threads addressed first, which is exactly the design work this PR is waiting on.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

Checked the CodeQL "2 high" alert at 5e348962: get_check_run only returns the summary (2 high-severity, no annotation-level rule id/location via the tools available to me here), but tests/test_debug_logging.py still carries the same two logger.* calls I flagged earlier in this PR's history:

  • test_configure_logging_redactor_none_still_leaves_secret_unmasked (~line 159-183) and test_configure_logging_redactor_masks_secret_shaped_content_in_captured_output's exception-message sibling (~line 194-215) both deliberately log a hardcoded, non-functional fake_secret = "sk-FAKEFAKE..." fixture (already # noqa: S105'd for Bandit) with no redactor attached, as the negative control proving the positive redaction test means something. That's the standard "demonstrate the vulnerability absent the fix" pattern — CodeQL's clear-text-logging query can't see that fake_secret never leaves this test file or that the assertion right after each call is fake_secret in captured.getvalue()/fake_secret not in output.

If these are in fact the 2 flagged alerts, the fix consistent with this file's own convention is a scoped suppression comment on those specific logger.* calls (mirroring the existing # noqa: S105 on the same fixture), not touching the test's actual behavior. I don't have tooling here to pull the exact alert IDs/lines to confirm it's these two rather than something new, so I'm not pushing a blind fix — flagging this for whoever's driving CodeQL green next on this PR.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

Verified CodeRabbit's "Merge Risk: High" summary against the current head (5ff93170) — both concerns it names are already addressed in this exact diff, not open gaps:

"Retries to authenticated provider discovery while still allowing redirects to another host, which can expose provider credentials": model_discovery.py's _fetch_json (the function discover_provider_models's retry loop calls) already routes through the same _TrustedDiscoveryRedirectHandler/_open_same_host_https opener as _fetch_json_same_host_https, rejecting any redirect that leaves the original HTTPS host before the Authorization header would ever be forwarded — see _fetch_json's own docstring (lines ~299-318), which documents this exact class of vulnerability (plain urllib copies Authorization onto a cross-host redirect) as the reason the shared trusted-host opener exists. There's no urllib.request.urlopen(...) call left anywhere in the authenticated discovery path that would bypass it.

"DEBUG response telemetry also retains unbounded model and usage-key metadata": debug_logging.response_metadata_for_log() returns a fixed allowlist only — has_error (bool), model (a single string, the served model name), choice_count (int), and usage filtered to SAFE_USAGE_COUNTER_KEY_NAMES (numeric counters only, explicitly excluding any key that merely happens to hold a numeric value). The whole thing then passes through summarize_payload_for_log(..., max_characters=500) before it reaches a log line — bounded both in field selection and total size. Neither model nor usage can carry unbounded or credential-shaped content by construction.

Both protections were TDD'd with an explicit adversarial re-check (see the PR description's "Secret-non-leakage evidence" section) — I re-read the code just now rather than relying on that description alone. Not spending more effort chasing CodeRabbit's docstring-coverage warning (48.65%/80%): that's CodeRabbit's own heuristic scoped to functions touched by this diff, distinct from the repository's actual gate (interrogate --fail-under=100, verified passing at 100% in the PR description) — the same 80%-vs-100% discrepancy the PR description already flagged as this repo's own CLAUDE.md misstating the real gate.

No merge conflict either: git merge-tree against current main produces no conflict markers.


Generated by Claude Code

@seonghobae

Copy link
Copy Markdown
Contributor Author

Cross-PR integration contract: routing identity is provider-neutral model_group only; do not add or preserve a provider-family abstraction. OpenRouter discovery must retain concrete free model IDs, while the aggregate openrouter/free router is not a serving candidate. OpenCode, Noema, and Strix must call contextual-orchestrator. Do not impose fixed wall-clock deadlines on inference, initial ping, readiness/health, provider discovery, or OpenRouter ZDR-list retrieval; use explicit cancellation and evidence-backed transport failure instead. Reconcile this PR with #971 and central .github #1508 before merge.

@seonghobae seonghobae changed the title feat(logging): --log-level CLI/env verbose-debug logging with redaction safety net feat(logging): add CLI debug logging with redaction safety net Sep 1, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Devin Review

if _LOGGER.isEnabledFor(logging.DEBUG):
for member_id in member_ids:
_LOGGER.debug(
"rank_candidate agent_id=%s judged_quality=%s evidence_score=%.3f",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Label covers both evidence sources

evidence_score describes either selected ledger without claiming judged-quality observations measure throughput. Candidate ordering and score calculation remain unchanged.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

@cwl-noema-review cwl-noema-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noema LLM review

The PR #946 implements a comprehensive debug logging system with a redaction safety net. It addresses critical security and correctness issues identified in prior review rounds, including credential leaks via cross-host redirects in model discovery, unbounded memory reads (CWE-400), and sensitive data exposure in logs (CWE-532). The implementation of _RedactingLogFilter correctly handles exception tracebacks, and the CLI argument parsing has been hardened to prevent subcommand bypass via abbreviated flags. All identified regressions from prior threads have been resolved with verified fixes in the provided diff.

Reviewed changed lines

  • contextual_orchestrator/__main__.py:103 (RIGHT): Correctly disables argparse abbreviations to ensure consistency between the pre-scan and subcommand locator.
  • contextual_orchestrator/debug_logging.py:148 (RIGHT): The filter correctly renders and redacts exception text before clearing exc_info, preventing secrets in tracebacks from leaking.
  • contextual_orchestrator/model_discovery.py:318 (RIGHT): Implements bounded read (MAX_DISCOVERY_RESPONSE_BYTES + 1) to prevent memory exhaustion from malicious provider responses.
  • contextual_orchestrator/model_discovery.py:320 (RIGHT): Correctly raises ValueError if the response body exceeds the defined cap.
  • contextual_orchestrator/model_discovery.py:443 (RIGHT): Uses _open_trusted_discovery_request to prevent Authorization header leakage during cross-host redirects.

Adversarial validation

  • contextual_orchestrator/model_discovery.py:318 (RIGHT) falsified: A provider returning a 100MB response could cause an Out-Of-Memory (OOM) crash. — The code now calls response.read(MAX_DISCOVERY_RESPONSE_BYTES + 1) and checks the length immediately after.
  • contextual_orchestrator/debug_logging.py:148 (RIGHT) falsified: Secrets embedded in an exception's message (e.g., via a 401 response body) would bypass the message filter via the traceback. — The _RedactingLogFilter now explicitly formats the exception, redacts the resulting text, and caches it in record.exc_text while clearing record.exc_info.
  • Residual risk: Low. Redaction is a safety net; primary redaction still relies on call-site discipline, but the handler-level filter provides strong defense-in-depth.

Findings

  • No blocking findings.

  • Result: APPROVE

  • Head SHA: 4b402cdae00289abce52b68ee33942fd4ba69ae5

  • Reviewer credential: noema-review-github-app

  • Actor: cwl-noema-review[bot]

@seonghobae
seonghobae merged commit 57dbc43 into main Sep 1, 2026
29 checks passed
@seonghobae
seonghobae deleted the feat/verbose-debug-logging-log-level-cli branch September 1, 2026 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants