feat(context-engine): select_context() + on_turn_complete() ABC verbs — salvage of #51226 (@chaos-xxl, RFC #36765)

[teknium1]

Salvage of PR #51226 by @chaos-xxl — implements RFC #36765: two additive, no-op-default, fail-open verbs on the ContextEngine ABC, cherry-picked onto current main (848 commits ahead of the PR's base) with authorship preserved, plus two maintainer follow-up commits (a default-path zero-cost short-circuit and the scoping docs).

What lands

• select_context() — pre-request selection. An engine may replace which messages enter THIS provider request (retrieval, topic routing, role/branch switching). Request-only: persisted history is never mutated. This is the headline verb — nothing else on the surface can express replace (pre_llm_call is inject-only by documented cache design).
• on_turn_complete() — post-turn observation from the turn-finalization seam, with a shallow-copied finalized transcript and the turn's canonical usage buckets (the same dict shape fed to update_from_response; None on turns that never reached a provider response). Accepted as the coupled observation mirror of MemoryProvider.sync_turn() for engines that already select.
• Host call sites in conversation_loop.py (_apply_context_engine_selection, before cache-control and all sanitizers) and turn_finalizer.py (_notify_context_engine_turn_complete), both fail-open.
• Docs: context-engine-plugin.md gains the two hooks in the optional-methods table, the full contract, and a new maintainer scoping section (see below).

Consolidates the 4-PR cluster: #41918 (@johnnykor82 — on_turn_complete design; Co-authored-by preserved on the cherry-picked commits), #24949, #47109, and the observation half of #50053.

Credit

• @chaos-xxl — primary author: RFC [RFC] Treat "context selection/routing" as a first-class ContextEngine concern, distinct from compression #36765, the full implementation, and a responsive multi-round review cycle (fixed the usage=None forwarding, the all([]) empty-list fail-open hole with a regression test, and the shallow-copy of live inputs with a mutation regression test — all before this salvage).
• @johnnykor82 — on_turn_complete design + two caught defects during review; Co-authored-by preserved on commits 1–2.
• @arminanton — scope correction on the original cluster.

Verification checklist (run against the composed diff on current main)

#	Check	Result
1	select_context() call-site ordering: before Anthropic cache-control and before _sanitize_api_messages / _drop_thinking_only_and_merge_users	✅ OK — selection at conversation_loop.py ~L1349, cache-control ~L1363, sanitizers ~L1374/L1384 post-train
2	Default-path byte-identity + no-op short-circuit actually skips work	⚠️ Found gap, fixed — ContextCompressor implements neither verb (pinned), but the call site's hasattr guard couldn't skip the inherited ABC default: the built-in path paid per-request history copies + a hook call. Maintainer commit adds the base-method identity short-circuit (mirroring the existing one in _notify_context_engine_turn_complete) + 2 pins
3	Fail-open contract: non-list / list-of-non-dicts / empty list / exception → original request	✅ OK — the and selected and guard survived composition; all four branches pinned by tests
4	_last_turn_usage: reset at turn start, stashed from the canonical bucket dict fed to update_from_response, forwarded from finalize_turn, None when no provider response	✅ OK — reset at current turn-start position (~L979, before the codex_app_server early return), stash is dict(usage_dict) at the single canonical site (~L2762), finalizer reads via getattr(..., None)
5	on_turn_complete fires once per turn from the finalization seam; shallow copy; return ignored	✅ OK with documented scope — exactly one finalize_turn call site (run_conversation's single return). Author's coverage claim is honest: abnormal early returns (content-policy block, terminal provider failures) bypass finalization and are explicitly documented as NOT covered (docstring + docs + a pinning test), not claimed covered
6	Guard quartet audit (test-double density)	✅ OK — all engine access via getattr(agent, "context_compressor", None) + hasattr/callable; no bare truthiness on new flags (_last_turn_usage read type-agnostically and only forwarded); no new agent.<method>() calls; sidecar/finalizer double suites green
7	Composition with this week's compression train (#69865 preflight arm, #70254 prune wiring, #70285 lock-defer)	✅ OK — one conflict total (both-added block in context_engine.py, composed keeping prune_tool_results_only + select_context); test_engine_preflight_wire.py, test_proactive_prune_loop_wiring.py, test_413_compression.py all green on the composed branch

Maintainer additions (separate commits)

1. fix(context-engine): short-circuit the inherited no-op select_context — zero per-request work on the default path (checklist item 2), + pins that the base no-op is never invoked and that ContextCompressor.__dict__ contains neither verb. Also adds the contributors/emails/ mapping for @chaos-xxl.
2. docs(context-engine): scoping guidance — select_context() is for engines that must replace per-request context; observation-only plugins should implement a MemoryProvider (sync_turn) instead of a context engine; and an explicit note that a non-no-op select_context() naturally changes the prompt-cache prefix on turns where the selection changes — engines should return stable selections when nothing changed.

Validation

• scripts/run_tests.sh targeted: test_context_engine_select_context.py (25 tests incl. 2 new pins), test_context_engine_on_turn_complete_usage.py, test_context_engine.py, test_turn_context.py — all green.
• Sibling train suites: test_engine_preflight_wire.py, test_proactive_prune_loop_wiring.py, test_413_compression.py — all green (83 tests across the 5-file core set).
• Finalizer + test-double sensitive suites: test_turn_finalizer_cleanup_guard.py, test_turn_finalizer_iteration_limit_exit.py, test_turn_finalizer_interrupt_alternation.py, test_turn_finalizer_final_response_persistence.py, test_api_content_sidecar.py — all green.
• ruff check clean on all touched files; ast.parse clean after every conflict resolution; branch rebased to 0-behind origin/main pre-push, diff contains only this PR's 7 files.

Closes #51226 on merge (with credit). Rebase-merge to preserve @chaos-xxl's per-commit authorship.

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 85e2736}

all good!

[chaos-xxl]

Thanks for the salvage and for preserving authorship — really appreciate it. The two maintainer additions look right to me: the inherited-no-op short-circuit is the correct zero-cost default path, and the scoping guidance (select vs. MemoryProvider, cache-prefix stability) is a good clarification I'd have wanted in the docs anyway.

Happy to pick up the early-return on_turn_complete coverage or anything else as a follow-up if it's useful.