feat(OMN-20926): the content-capture producer stores detected secrets and emits references

contracts/OMN-20926.yaml

@@ -1,50 +1,72 @@
-schema_version: "1.0.0"
+schema_version: 1.0.0
 ticket_id: OMN-20926
-title: "The Skip Count Ratchet no longer refuses the OMN-20818 onboarding pin tests"
-summary: >
-  The OMN-20818 onboarding pin tests skip on a depth-1 checkout because the
-  pinned script commit is absent there, and the Skip Count Ratchet (OMN-18776)
-  refuses a test that is collected and then skipped. This change makes the
-  pin tests fetch the pinned commit by sha, and unshallow once when an
-  ancestry answer from a shallow checkout is not an answer, so they run on a
-  depth-1 checkout instead of skipping. The change covers only this test
-  module; the captured-secret-reference work the ticket also carries is
-  bound by its own pull requests.
+title: The content-capture producer stores detected secrets and emits references, and the Skip Count Ratchet no longer refuses the onboarding pin tests
+summary: A secret the content scrubber finds is offered to a secret store sink and the captured record carries a secret reference in its place. References stand through every scrub and are never cut at a chunk boundary or the size cap. A store that cannot take the secret falls back to the existing marker and journals the reason and count, never a value. Capture is never blocked. The OMN-20818 onboarding pin tests skip on a depth-1 checkout because the pinned script commit is absent there, and the Skip Count Ratchet (OMN-18776) refuses a test that is collected and then skipped. This change makes the pin tests fetch the pinned commit by sha, and unshallow once when an ancestry answer from a shallow checkout is not an answer, so they run on a depth-1 checkout instead of skipping.
 is_seam_ticket: false
 interface_change: false
 interfaces_touched: []
 requirements:
-  - id: req-pin-tests-run-on-depth-1
-    statement: "The OMN-20818 onboarding pin tests run, and do not skip, in a checkout that holds only the pull request's head commit."
-    acceptance:
-      - id: "AC1"
-        statement: |-
-          AC1: when the pinned script commit is absent from the checkout, the pin tests fetch it by sha before they skip, so a depth-1 checkout runs them -- falsifier: uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly
-      - id: "AC2"
-        statement: |-
-          AC2: the ancestry check between the multi-model script commit and the pinned commit unshallows a shallow checkout once before it answers, and still fails when the guide pins a commit older than the multi-model script -- falsifier: uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly -k not_older
+- id: req-capture-secret-refs
+  statement: The content-capture producer replaces a detected secret's value with a stored secret reference, or with the marker when the store cannot take it.
+  acceptance:
+  - id: AC3
+    statement: 'AC3: a detected secret is offered to the secret store sink, only its value is replaced by the returned reference, a labelled secret keeps its label, and the same secret is one stored value and one reference -- falsifier: uv run pytest tests/unit/hooks/test_hook_content_capture_secret_refs_omn20926.py -q -k "stored or labelled or same_secret"'
+  - id: AC4
+    statement: 'AC4: a reference already in the text stands through every scrub and is never cut at a chunk boundary or the size cap -- falsifier: uv run pytest tests/unit/hooks/test_hook_content_capture_secret_refs_omn20926.py -q -k "survives or chunking or reference_spans or sub_keeping"'
+  - id: AC5
+    statement: 'AC5: a store failure falls back to the marker, journals the reason and count and never a value, a machine that does not take part falls back quietly, and capture is never blocked -- falsifier: uv run pytest tests/unit/hooks/test_hook_content_capture_secret_refs_omn20926.py -q -k "falls_back or does_not_take_part or alarm"'
+  - id: AC6
+    statement: 'AC6: the mirrored capture_redaction.yaml matches the contract omnimarket owns, secret_reference block included -- falsifier: uv run python scripts/validation/generate_event_registry.py --daemon-registry "$OMNI_HOME/omnimarket/src/omnimarket/nodes/node_emit_daemon/registries/topics.yaml" --check'
+- id: req-pin-tests-run-on-depth-1
+  statement: The OMN-20818 onboarding pin tests run, and do not skip, in a checkout that holds only the pull request's head commit.
+  acceptance:
+  - id: AC1
+    statement: 'AC1: when the pinned script commit is absent from the checkout, the pin tests fetch it by sha before they skip, so a depth-1 checkout runs them -- falsifier: uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly'
+  - id: AC2
+    statement: 'AC2: the ancestry check between the multi-model script commit and the pinned commit unshallows a shallow checkout once before it answers, and still fails when the guide pins a commit older than the multi-model script -- falsifier: uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly -k not_older'
 evidence_requirements:
-  - kind: tests
-    description: "The OMN-20818 onboarding pin tests pass at this head with the fetch-by-sha and one-time unshallow in place"
-    command: "uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly"
+- kind: tests
+  description: Secret store sink, reference survival, chunking and fallback tests pass
+  command: uv run pytest tests/unit/hooks/test_hook_content_capture_secret_refs_omn20926.py tests/unit/hooks/test_hook_content_capture_omn19551.py tests/hooks/test_capture_redaction_omn17959.py tests/hooks/test_capture_redaction_sync_omn18357.py -q
+- kind: tests
+  description: The OMN-20818 onboarding pin tests pass at this head with the fetch-by-sha and one-time unshallow in place
+  command: uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly
 emergency_bypass:
   enabled: false
-  justification: ""
-  follow_up_ticket_id: ""
+  justification: ''
+  follow_up_ticket_id: ''
 dod_evidence:
-  - id: dod-omn20926-ac1-pin-tests-fetch-by-sha
-    description: "The three OMN-20818 pin tests pass at this head; the pinned commit is fetched by sha when absent, so a depth-1 checkout runs them instead of skipping them."
-    source: manual
-    checks:
-      - check_type: test_passes
-        check_value: "uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly"
-    binds_ac:
-      - AC1
-  - id: dod-omn20926-ac2-pin-ancestry-unshallow
-    description: "The pin-is-not-older test answers from the ancestry check after one unshallow of a shallow checkout; the same module's digest test still compares the guide's sha256 with the script at the pinned commit."
-    source: manual
-    checks:
-      - check_type: test_passes
-        check_value: "uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly"
-    binds_ac:
-      - AC2
+- id: dod-omn20926-capture-secret-refs
+  description: The producer's secret store sink, reference survival through scrubs and chunking, and the quiet and loud fallbacks are covered by the focused suites.
+  source: manual
+  checks:
+  - check_type: test_passes
+    check_value: uv run pytest tests/unit/hooks/test_hook_content_capture_secret_refs_omn20926.py tests/unit/hooks/test_hook_content_capture_omn19551.py tests/hooks/test_capture_redaction_omn17959.py tests/hooks/test_capture_redaction_sync_omn18357.py -q
+  binds_ac:
+  - AC3
+  - AC4
+  - AC5
+- id: dod-omn20926-mirror-parity
+  description: The mirrored contract matches omnimarket's owning copy.
+  source: manual
+  checks:
+  - check_type: test_passes
+    check_value: uv run pytest tests/unit/hooks/test_hook_content_capture_secret_refs_omn20926.py -q -k drifted_secret_reference
+  binds_ac:
+  - AC6
+- id: dod-omn20926-ac1-pin-tests-fetch-by-sha
+  description: The three OMN-20818 pin tests pass at this head; the pinned commit is fetched by sha when absent, so a depth-1 checkout runs them instead of skipping them.
+  source: manual
+  checks:
+  - check_type: test_passes
+    check_value: uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly
+  binds_ac:
+  - AC1
+- id: dod-omn20926-ac2-pin-ancestry-unshallow
+  description: The pin-is-not-older test answers from the ancestry check after one unshallow of a shallow checkout; the same module's digest test still compares the guide's sha256 with the script at the pinned commit.
+  source: manual
+  checks:
+  - check_type: test_passes
+    check_value: uv run pytest tests/skills/test_onboarding_script_pin_omn20818.py -q -p no:randomly
+  binds_ac:
+  - AC2

plugins/onex/hooks/lib/hook_content_capture.py

@@ -79,6 +79,7 @@
 import os
 import sys
 import time
+import uuid
 from datetime import UTC, datetime, timedelta
 from pathlib import Path
 from types import ModuleType
@@ -118,6 +119,25 @@
 OPERATOR_MARKER_KEEP = 512
 
 _RESOLVER_MODULE_NAME = "omniclaude_capture_redaction_by_path"
+
+#: OMN-20926: the record field carrying the producer's store-then-reference
+#: outcome, ``{stored, fallback, reason}``. Declared on the content topic.
+STORE_OUTCOME_FIELD = "stored_refs"
+
+#: OMN-20926: omnimarket's captured secret store effect, imported only when a
+#: capture actually finds a secret, so a clean capture pays nothing for it.
+_STORE_HANDLER_MODULE = (
+    "omnimarket.nodes.node_captured_secret_store_effect.handlers."
+    "handler_captured_secret_store"
+)
+_STORE_REQUEST_MODULE = (
+    "omnimarket.nodes.node_captured_secret_store_effect.models."
+    "model_captured_secret_store_request"
+)
+#: The effect is not installed in this interpreter yet (an older omnimarket).
+WRITER_UNAVAILABLE = "writer_unavailable"
+#: Reasons that mean this machine does not take part, so nothing is alarmed.
+QUIET_STORE_REFUSALS = frozenset({"not_configured", WRITER_UNAVAILABLE})
 _REPLY_MARKER_SUFFIX = ".reply"
 _OPERATOR_MARKER_SUFFIX = ".operator"
 
@@ -477,6 +497,68 @@ def record_operator_marker(turn_dir: Path, session_id: str, ids: list[str]) -> N
         _log(f"operator marker not written ({exc}); the prompts may be recorded again")
 
 
+# ---------------------------------------------------------------------------
+# the secret store (OMN-20926)
+# ---------------------------------------------------------------------------
+
+
+def onex_home() -> Path:
+    """This machine's ``~/.onex``: its private overlay and credential file."""
+    return Path("~/.onex").expanduser()
+
+
+def build_store_handler(home: Path) -> tuple[Any, Any]:
+    """omnimarket's captured secret store effect, and its request model.
+
+    Raises ``ImportError`` when this interpreter's omnimarket predates the
+    effect; :class:`StoreSink` turns that into ``writer_unavailable``.
+    """
+    handler_module = importlib.import_module(_STORE_HANDLER_MODULE)
+    request_module = importlib.import_module(_STORE_REQUEST_MODULE)
+    return (
+        handler_module.HandlerCapturedSecretStore(onex_home=home),
+        request_module.ModelCapturedSecretStoreRequest,
+    )
+
+
+class StoreSink:
+    """The resolver's ``SecretSink``, over the captured secret store effect.
+
+    Called with each secret the scrub finds; returns the reference the effect
+    minted, or ``None`` so the span crosses as the marker. ``refusal`` is the
+    effect's outcome code for the first failure, which also stops every later
+    call in this capture.
+    """
+
+    def __init__(self, *, session_id: str) -> None:
+        self.session_id = session_id
+        self.refusal: str | None = None
+        self.detail: str | None = None
+        self._handler: Any = None
+        self._request: Any = None
+
+    def __call__(self, value: str) -> str | None:
+        if self.refusal is not None:
+            return None
+        if self._handler is None:
+            try:
+                self._handler, self._request = build_store_handler(onex_home())
+            except ImportError as exc:
+                self.refusal = WRITER_UNAVAILABLE
+                self.detail = type(exc).__name__
+                return None
+        result = self._handler.handle(
+            self._request(
+                value=value, session_id=self.session_id, captured_at=datetime.now(UTC)
+            )
+        )
+        if result.reference is None:
+            self.refusal = str(result.outcome.value)
+            self.detail = result.detail
+            return None
+        return str(result.reference)
+
+
 # ---------------------------------------------------------------------------
 # entrypoint
 # ---------------------------------------------------------------------------
@@ -547,18 +629,34 @@ def capture(
             )
             items = [*items, *operator_items]
 
+    # OMN-20926: a detected secret is stored in the secret store and a
+    # reference replaces it. The writer loads only when a secret is found; a
+    # span the store did not take crosses as the marker.
+    sink = (
+        None
+        if resolver.load_contract().reference is None
+        else StoreSink(session_id=session)
+    )
+
     written = 0
+    fallback = 0
     for item in items:
         record = {"session_id": session, "hook_source": _HOOK_SOURCE[kind], **item}
-        for chunk in resolver.prepare_content_records(
+        chunks = resolver.prepare_content_records(
             record,
             topic=topic,
             content_field="content",
             redaction_field="producer_redaction",
             chunk_index_field="chunk_index",
             chunk_count_field="chunk_count",
             digest_field="content_sha256",
-        ):
+            secret_sink=sink,
+            stored_refs_field=STORE_OUTCOME_FIELD,
+        )
+        outcome = chunks[0].get(STORE_OUTCOME_FIELD) if chunks else None
+        if isinstance(outcome, dict):
+            fallback += int(outcome.get("fallback") or 0)
+        for chunk in chunks:
             appender.append_event(
                 # A literal, not CONTENT_EVENT_TYPE, on purpose (OMN-19913):
                 # validate_hook_edge_lane.py resolves only literal classes at
@@ -577,11 +675,80 @@ def capture(
                 journal_dir=str(journal_dir),
             )
             written += 1
+    if sink is not None and fallback and sink.refusal not in QUIET_STORE_REFUSALS:
+        _report_store_fallback(
+            sink,
+            fallback=fallback,
+            session=session,
+            journal_dir=journal_dir,
+            correlation_id=correlation_id,
+            cwd=cwd,
+            actor=actor,
+            host_turn_id=host_turn_id,
+            agent_id=agent_id,
+            transcript_path=transcript_path,
+        )
     if operator_ids:
         record_operator_marker(turn_dir, session, operator_ids)
     return written
 
 
+def _report_store_fallback(
+    sink: StoreSink,
+    *,
+    fallback: int,
+    session: str,
+    journal_dir: Path,
+    correlation_id: str | None,
+    cwd: str | None,
+    actor: str | None,
+    host_turn_id: str | None,
+    agent_id: str | None,
+    transcript_path: str | None,
+) -> None:
+    """Say loudly that secrets crossed as the marker because the store failed.
+
+    The hook log always; a ``hook.health.error`` record too when the local
+    drainer can publish one. Reason code and count only: never a value, a
+    digest, a store address or a path.
+    """
+    message = (
+        f"captured-secret store fallback: {fallback} secret span(s) crossed as "
+        f"the redaction marker, reason={sink.refusal}"
+    )
+    _log(message + (f" ({sink.detail})" if sink.detail else ""))
+    status = health.read_status(status_path_for(journal_dir))
+    publishable = None if status is None else status.publishable_event_types
+    if publishable is None or "hook.health.error" not in publishable:
+        return
+    emitted_at = datetime.now(UTC).isoformat()
+    appender.append_event(
+        # A literal, not a constant (OMN-19913): validate_hook_edge_lane.py
+        # resolves only literal classes at an emit call.
+        event_type="hook.health.error",
+        payload={
+            "event_id": str(uuid.uuid4()),
+            "hook_name": "hook_content_capture",
+            "error_tier": "degraded",
+            "error_category": "functional_degradation",
+            "error_message": message,
+            "session_id": session,
+            "fingerprint": hashlib.sha256(
+                f"hook_content_capture:secret_store:{sink.refusal}".encode()
+            ).hexdigest()[:16],
+            "emitted_at": emitted_at,
+        },
+        correlation_id=correlation_id or session,
+        cwd=cwd,
+        actor=actor,
+        host_turn_id=host_turn_id,
+        agent_id=agent_id,
+        transcript_path=transcript_path,
+        session_id=session,
+        journal_dir=str(journal_dir),
+    )
+
+
 def main(argv: list[str] | None = None) -> int:
     parser = argparse.ArgumentParser(
         description="Journal full hook content (OMN-19551)."

pyproject.toml

@@ -268,7 +268,7 @@ override-dependencies = [
     # pin was one release behind on the day it landed. Not a mistake in that
     # bump: sibling-lock-refresh.yml cannot move an immutable rev, so this pin
     # decays on every omnimarket release until OMN-13902 / OMN-18594 land.
-    "omnimarket @ git+https://github.com/OmniNode-ai/omnimarket.git@a44caa7efd7950d4145ca67c6c1404921ca0b221",
+    "omnimarket @ git+https://github.com/OmniNode-ai/omnimarket.git@f7ff837dfb882cad126f88a023cfc400e113de49",
 ]
 
 [tool.uv.sources]
@@ -314,7 +314,7 @@ omninode-intelligence = { git = "https://github.com/OmniNode-ai/omniintelligence
 #
 # So do not hand-bump this line. If it is stale, run `sibling-lock-refresh.yml`
 # on workflow_dispatch and review the bot PR.
-omnimarket = { git = "https://github.com/OmniNode-ai/omnimarket.git", rev = "a44caa7efd7950d4145ca67c6c1404921ca0b221" }  # pragma: allowlist secret
+omnimarket = { git = "https://github.com/OmniNode-ai/omnimarket.git", rev = "f7ff837dfb882cad126f88a023cfc400e113de49" }  # pragma: allowlist secret
 
 # OMN-18752: which surface decides the commit a git-sourced sibling must carry.
 # `clone` is claimed only where something enforces it. omnimarket's in-process

scripts/validation/generate_event_registry.py

@@ -285,6 +285,19 @@ def _resolved_posture(contract_path: Path) -> dict[str, Any]:
             )
             for topic, policy in contract.topics.items()
         },
+        # OMN-20926: the reference a stored secret is named by, the store key it
+        # maps to and the value group are posture: a drifted pattern would let
+        # one side scrub what the other side minted.
+        "secret_reference": (
+            None
+            if contract.reference is None
+            else (
+                contract.reference.pattern.pattern,
+                contract.reference.template,
+                contract.reference.store_key_template,
+                contract.reference.value_group,
+            )
+        ),
     }