diff --git a/CHANGELOG.md b/CHANGELOG.md index 0cd03b3..c925a6e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,46 @@ ## Unreleased +- Section 18.1 gates durable writes behind the watermark. Durable + incorporation of stream text, into conversation history, a session store, + or any record a later evaluation or run can read, follows the same rule as + emission: a rune is eligible for a durable write when the watermark covers + it, under every safety level, and a terminal deny or a failing settlement + forbids persisting the withheld or uncleared runes. This restates the + AGENT-HOOKS-0.1 section 6.1 discard obligation at the granularity the + profile evaluates. The released prefix is already part of the caller + visible record and may stay durable alongside the refusal that followed + it. Mirrored as a module doc obligation on `StreamSession`. +- Section 18.1 defines the caller as any consumer outside the enforcement + boundary. A host registered observer, a callback, a preview channel, or a + sink fed from the raw accumulation is a caller, and withheld runes must + not be delivered to one. The profile holds no text, so nothing structural + separates the accumulation from a channel wired ahead of the release + decision; the stated obligation is the whole of the protection. +- Section 18.1 states that the attempt boundary is not a clearance boundary. + A track resuming at an offset above zero retains the last `L - 1` runes + the earlier attempt delivered and includes them in the value it evaluates + near the boundary, since a term can straddle the attempts and no value + drawn from the new attempt alone can contain it. A host that no longer + holds that tail must not resume the track under the profile. A mediation + test covers a term straddling the resume boundary, with the host that + dropped the tail as the negative control. +- Section 18.1 states the released text identity obligation. The runes the + host releases are rune identical to the runes the recorded outcomes were + evaluated against; a host side rewrite after clearance invalidates the + clearance, and altered text belongs on the whole snapshot path or in a new + session. Added to the `StreamSession` module doc obligations. +- Section 18.1 requires settlement of every opened session, including one + the host abandons on disconnect, cancellation, or replacement by a retry. + An abandoned session settles like any other, so uncleared residue is + recorded rather than lost with the dropped session. +- Section 18.1 states the interaction with the `output` point, which section + 18 keeps on the whole snapshot path in every case. A host adopting the + profile for caller facing egress receives that verdict after runes have + reached the caller, so a deny there cannot recall them; the host records + it and does not present the stream as settled clean, per the + AGENT-HOOKS-0.1 section 6.1a record and close shape. + ## 0.4.0-alpha.2 - Section 18's requirement that a host assemble streamed model output before diff --git a/engine/src/stream_session.rs b/engine/src/stream_session.rs index aaed109..ad89f93 100644 --- a/engine/src/stream_session.rs +++ b/engine/src/stream_session.rs @@ -26,7 +26,7 @@ //! runes. So the host declares the range it evaluated, and the session tracks //! only what that clears. //! -//! This leaves the host owing three obligations that a session cannot check +//! This leaves the host owing five obligations that a session cannot check //! for it: //! //! 1. The text evaluated for a span covers at least that span, and reaches @@ -44,6 +44,27 @@ //! `evaluate_only` evaluation. A cleared span releases text, which //! specification section 20 forbids presenting an `evaluate_only` result as //! doing. No verdict carries the mode, so nothing here can check it. +//! 4. The runes released are rune identical to the runes the recorded +//! outcomes were evaluated against. A clearance vouches for the text the +//! task saw, so a host side rewrite after clearance, the host's own post +//! processing included, invalidates the clearance it would ride on. Text +//! the host must alter belongs on the whole snapshot path, or in a new +//! session over the altered value. A session holds no text, so it cannot +//! compare the two. +//! 5. Durable incorporation of stream text waits for the same watermark that +//! gates emission. A rune becomes eligible for a durable write when the +//! watermark covers it, under every safety level; on a terminal deny or a +//! failing settlement the withheld and uncleared runes are not persisted, +//! per AGENT-HOOKS-0.1 section 6.1. The released prefix is already part +//! of the caller visible record and may stay durable alongside the +//! refusal that followed it. +//! +//! The caller these obligations withhold from is any consumer outside the +//! enforcement boundary: an observer, a callback, a preview channel, a sink +//! fed from the raw accumulation. Specification section 18.1 states the +//! full set, including that a host settles every session it opens, an +//! abandoned one included, because a dropped session leaves a trail with no +//! settlement outcome. //! //! One capability limit is worth stating plainly, because it is not obvious //! from the types. A `transform` ends the session. The substitution replaces diff --git a/engine/tests/stream_session_mediation.rs b/engine/tests/stream_session_mediation.rs index 51bdeba..f6bc4ef 100644 --- a/engine/tests/stream_session_mediation.rs +++ b/engine/tests/stream_session_mediation.rs @@ -399,3 +399,115 @@ fn a_transform_verdict_records_the_host_obligation_to_substitute() { "settlement must tell the host the stream was not verbatim" ); } + +/// A session resuming attempt 2 of a response stream whose first attempt +/// delivered runes `[0, resume_at)` before it was abandoned. +fn resumed_session(resume_at: u32) -> StreamSession { + StreamSession::new(StreamSessionConfig { + safety_level: SafetyLevel::Blocking, + request_start_rune_offset: 0, + response_start_rune_offset: resume_at, + request_tasks: Vec::new(), + response_tasks: vec!["harm".to_string()], + }) + .expect("config is valid") +} + +/// Drive a resumed attempt over `payloads`, evaluating each span over the +/// accumulated attempt text prefixed by `retained`, and return what the +/// retry released. `retained` is the tail of the earlier attempt's delivered +/// text that the host kept across the boundary; passing an empty string is +/// the host that dropped it. +fn drive_resumed( + runtime: &Runtime, + session: &mut StreamSession, + resume_at: u32, + retained: &str, + payloads: &[&str], +) -> String { + let mut attempt_text = String::new(); + let mut released = String::new(); + let mut cursor = resume_at; + let mut emitted = resume_at; + for payload in payloads { + attempt_text.push_str(payload); + let end = session + .observe_text(StreamSourceType::ModelGenerated, payload) + .expect("observe"); + let span = + StreamSpan::new(StreamSourceType::ModelGenerated, cursor, end).expect("range is valid"); + let value = format!("{retained}{attempt_text}"); + let verdict = evaluate(runtime, &value); + session + .record_verdict("harm", &span, &verdict) + .expect("the outcome records, a denial included"); + cursor = end; + if session.is_ended() { + break; + } + if let Some(safe) = session.advance(StreamTrack::Response) { + released.push_str(&slice_runes( + &attempt_text, + emitted - resume_at, + safe - resume_at, + )); + emitted = safe; + } + } + released +} + +#[test] +fn a_term_straddling_a_resume_boundary_is_caught_with_the_retained_tail() { + // Section 18.1: the attempt boundary is not a clearance boundary. A + // track resuming at an offset above zero MUST retain the last `L - 1` + // runes the earlier attempt delivered and include them in the value it + // evaluates near the boundary. + // + // Attempt 1 released `xxxxforb`, 8 runes, then was abandoned. The term + // `forbidden` is 9 runes at [4, 13): it begins inside attempt 1's + // released tail and ends inside attempt 2's first spans, so no value + // drawn from attempt 2 alone can ever contain it. With `L - 1` of 8 the + // tail the host must retain happens to be the whole of what attempt 1 + // delivered. + let runtime = runtime("forbidden"); + let delivered = "xxxxforb"; + let resume_at = 8; + let mut session = resumed_session(resume_at); + let released = drive_resumed( + &runtime, + &mut session, + resume_at, + delivered, + &["idde", "nyyy"], + ); + assert!(matches!( + session.end_reason(), + Some(StreamEndReason::Denied { .. }) + )); + // The retry released only the prefix cleared before the term completed. + assert_eq!(released, "idde"); + let caller_sees = format!("{delivered}{released}"); + assert!(!caller_sees.contains("forbidden")); +} + +#[test] +fn a_resumed_host_that_drops_the_prior_tail_misses_the_straddling_term() { + // The negative control for the retention obligation. The same stream, + // resumed by a host that evaluates only what attempt 2 accumulated: + // every value it evaluates holds at most `iddenyyy`, so every span + // clears, the session settles clean, and the caller assembles the term + // across the attempts. + let runtime = runtime("forbidden"); + let delivered = "xxxxforb"; + let resume_at = 8; + let mut session = resumed_session(resume_at); + let released = drive_resumed(&runtime, &mut session, resume_at, "", &["idde", "nyyy"]); + assert_eq!(session.finish().reason, StreamEndReason::Complete); + assert_eq!(released, "iddenyyy"); + let caller_sees = format!("{delivered}{released}"); + assert!( + caller_sees.contains("forbidden"), + "dropping the prior attempt's tail is what the retention obligation exists to prevent" + ); +} diff --git a/spec/SPECIFICATION.md b/spec/SPECIFICATION.md index 819b573..93f7b57 100644 --- a/spec/SPECIFICATION.md +++ b/spec/SPECIFICATION.md @@ -408,11 +408,15 @@ This profile applies only to a text stream. It does not apply to a structured st The profile holds no stream text. A host receives the payloads, so it already accumulates them, and a host that cuts that accumulation into evaluation units already decides where those cuts fall. An implementation MUST NOT impose a second segmentation on such a host, because two accounts of what was evaluated over the same runes cannot both be authoritative. A host therefore declares the rune range it evaluated together with the outcome it obtained, and the accounting tracks what that clears. -This places three obligations on the host that an implementation cannot verify for it. First, the value the host evaluates for a span MUST contain at least that span's own text. The unit a policy reasons about crosses segment boundaries, so evaluating less is unsound: a policy that refuses a banned span sees the halves of that span separately, permits each, and the caller receives the span. A host that evaluates the whole accumulated prefix satisfies this obligation exactly. +The caller this profile withholds from is any consumer outside the enforcement boundary, not only the far end of the connection. A host registered observer, a completion callback, a preview or typing channel, a sub agent or logging sink fed from the raw accumulation: each of these is a caller, and withheld runes MUST NOT be delivered to one. The profile holds no text, so nothing structural separates the host's accumulation from a channel wired ahead of the release decision, and an implementation cannot detect one. A consumer inside the enforcement boundary, such as the segmenter or the evaluation itself, necessarily reads unreleased text, which is what the boundary means. + +This places four obligations on the host that an implementation cannot verify for it. First, the value the host evaluates for a span MUST contain at least that span's own text. The unit a policy reasons about crosses segment boundaries, so evaluating less is unsound: a policy that refuses a banned span sees the halves of that span separately, permits each, and the caller receives the span. A host that evaluates the whole accumulated prefix satisfies this obligation exactly. A host that bounds its target to control cost MUST size the bound from where the span begins, not from the length of the term. The value it evaluates for a span starting at offset `p` MUST reach at least `L - 1` runes below `p`, where `L` is the longest term the policy needs to detect, clamped at the start of the track. A term of length `L` that overlaps the span can begin as early as `p - L + 1`, so a value starting any higher can hold only part of it. For a host that evaluates a suffix window of `N` runes ending at each span's end, with spans of at most `S` runes, that is `N` of at least `S + L - 1`. -Sizing the bound merely above `L` is unsound, and satisfying the coverage obligation for every span does not rescue it. With `L` of 9, `S` of 4, and `N` of 10, a term at runes 4 through 13 is contained in no evaluated window: the window for the span at runes 12 through 16 covers runes 6 through 16 and holds only the term's tail. Every span clears and the term reaches the caller. The same stream is refused at `N` of 12. Second, the rune counts the host reports MUST match the text it accumulated. Third, the profile is an enforcement path, since its whole purpose is to decide which runes reach the caller, so a host MUST NOT feed an `evaluate_only` result into the accounting. Section 20 forbids presenting one as enforcement and a cleared span is enforcement. An implementation cannot check this, because the mode is not carried on a verdict. +Sizing the bound merely above `L` is unsound, and satisfying the coverage obligation for every span does not rescue it. With `L` of 9, `S` of 4, and `N` of 10, a term at runes 4 through 13 is contained in no evaluated window: the window for the span at runes 12 through 16 covers runes 6 through 16 and holds only the term's tail. Every span clears and the term reaches the caller. The same stream is refused at `N` of 12. Second, the rune counts the host reports MUST match the text it accumulated. Third, the profile is an enforcement path, since its whole purpose is to decide which runes reach the caller, so a host MUST NOT feed an `evaluate_only` result into the accounting. Section 20 forbids presenting one as enforcement and a cleared span is enforcement. An implementation cannot check this, because the mode is not carried on a verdict. Fourth, the runes the host releases MUST be rune identical to the runes the recorded outcomes were evaluated against. A clearance vouches for the text the task saw, so a host side rewrite after clearance, the host's own post processing included, invalidates the clearance it would ride on. Text the host must alter belongs on the whole snapshot path of section 18, or in a new session over the altered value, where what is evaluated and what is emitted are the same runes again. An implementation holding no text cannot compare the two, so this too is stated rather than checked. + +The clamp at the start of the track reads differently for a resumed session. A session resuming a partially delivered stream starts its track at the resume offset, but the text the caller sees started at zero, and a term the policy must detect can straddle the attempt boundary, beginning inside the tail the earlier attempt released and ending inside the first spans of the new one. The attempt boundary is not a clearance boundary. For a track resuming at an offset above zero the host MUST retain the last `L - 1` runes of the text already delivered and include them in the value it evaluates for any span beginning less than `L - 1` runes past the resume offset. A host that no longer holds that tail cannot satisfy the coverage obligation and MUST NOT resume the track under this profile; it assembles the remainder on the whole snapshot path of section 18 instead. A task clears a span by recording an outcome for it, and that task's offset advances only when the span starts at or below the offset the task has already cleared. A span starting past that offset MUST fail closed, because confirming it would release the gap between them, which nothing evaluated. Overlapping spans MUST be accepted, since a sliding or growing segmentation produces them. A span lying wholly below a task's frontier carries no new clearance, so a host MAY re-report a cleared one and an implementation MUST ignore it rather than treat it as an error. That covers only a clearing outcome. A refusal is terminal wherever it arrives, because the host is refusing text it has evaluated and no reading of that is safe to discard. A `transform` is governed by the rule below, since it asks to rewrite text rather than to report on it. A refusal that arrives after an earlier prefix was already released still withholds the remainder, so the complete refused span never reaches the caller even though a harmless prefix of it did. @@ -426,6 +430,8 @@ A session declares one safety level. Under `blocking` and `complete` a host MUST A `deny` denies and is terminal for the session. The host MUST stop the stream and MUST NOT release any withheld text, including runes a task had already cleared but the host had not yet emitted. An implementation MUST NOT offer a release point for a track once the session has ended, so that a host polling for one stops on its own rather than on the strength of a value it was still handed. The offset the track reached is unaffected and remains readable, since an audit record needs it. A `deny` carrying an `approval` block is liftable through the host's approval seam (AGENT-HOOKS-0.1 section 9), and resolving it is a host obligation that MUST happen before the outcome reaches the accounting, because a session cannot hold its connection open across an out of band approval. An implementation that receives one unresolved MUST take it at its word and deny, since the decision it carries is `deny` and honoring the seam is the obligation of the host that owns the connection. Withholding the text is the conservative reading, and a host that intended to lift the deny records the lifted outcome instead. +Release is not the only way a rune leaves the host's hands, and the watermark gates the other way too. Durable incorporation of stream text, into conversation history, a session store, or any record a later evaluation or a later run can read, follows the same rule as emission: a rune becomes eligible for a durable write when the watermark for its track covers it, and a host MUST NOT persist a rune the watermark has not reached, under every safety level. A `deferred` host emits on arrival because latency is its product, but a durable write has no latency to save and waits for the watermark like any withheld emission. On a terminal `deny` or a failing settlement the host MUST NOT persist the withheld or uncleared runes. AGENT-HOOKS-0.1 section 6.1 already obliges a host to discard a refused response rather than incorporate it into subsequent agent state, and this clause is that rule restated at the granularity this profile evaluates, so that a partially cleared stream cannot enter a store whole at payload arrival time. The released prefix is different: it is already part of the caller visible record, so a host MAY persist it even when the remainder is later refused, and a record that carries the refusal alongside that prefix says what happened, where one that persists the prefix and omits the refusal does not. + An `allow` clears, and the `warnings` it carries neither block nor alter that, because a warning is a recorded concern rather than a release decision. Their content still has to be well formed, under the rule on malformed verdicts below. A verdict that section 5 does not admit MUST fail closed with the reserved reason `host_error:verdict_invalid` before it clears anything, because reading a decision out of a malformed verdict would release text on the strength of a substitution or an approval the verdict never carried. A `transform` whose path leaves `$target` is one such verdict, since section 14 confines a transform to the policy target. @@ -444,6 +450,10 @@ Payload arrival and session settlement are distinct. A host MAY close the payloa Settlement MUST NOT advance a watermark on the failing path. A settlement that fails is precisely when the host must emit nothing further, so raising the release point as a side effect of failing would invite the opposite. When the session settles, any rune that no task cleared is a fail closed condition regardless of safety level. The host MUST NOT emit that residue and MUST record the reserved reason `host_error:streaming_unsupported`. That reason also covers a span that would confirm a gap nothing evaluated, payload arriving after the host closed the stream, an outcome on a settled session, an unrecognized source type, an unrecognized safety level, payload on a track the session does not mediate, a configuration mediating neither track, a session that exceeds the offset ceiling of its transport, an outcome naming a task the track was not configured with, an outcome naming an offset past the text the session was told about, an outcome whose span covers no runes, and a `transform` the session cannot honor. A malformed verdict is reported as `host_error:verdict_invalid` instead, since the fault is the shape of the verdict and not the stream. +Every session that is opened settles. A host MUST settle every session, including one it abandons: a caller disconnect, a cancellation, and a retry that replaces the session with a resumed one each end the attempt and not the obligation. An abandoned session settles like any other, so residue no task cleared fails it closed and is recorded, while a session whose every rune was cleared settles complete, which records that everything was evaluated and releasable and claims nothing about delivery. A session that is simply dropped leaves no settlement outcome, and a trail with no settlement outcome silently loses both the offset the tasks had reached, which an audit record needs, and the fact that residue existed at all. + +This profile evaluates `input` and `post_model_call`. Section 18 keeps `output` on the whole snapshot path in every case, so a host that adopts this profile for caller facing egress and also binds `output` receives that verdict only after runes have reached the caller, whatever safety level the session declared. A `deny` at `output` cannot recall a released rune. The host MUST record it and MUST NOT present the stream as having settled clean over it; the shape is the record and close obligation of AGENT-HOOKS-0.1 section 6.1a, a refusal preserved in the trail with nothing left to prevent, and not a completion. + ## 19. Telemetry and audit A host MAY supply a telemetry sink. Events are content safe and stable. Known event kinds are `decision`, `annotator_dispatch`, `policy_evaluation`, `evaluation_timing`, `intervention_point.transformed`, `annotator_failed`, and `policy_failed`. The runtime emits `intervention_point.transformed` in addition to `decision` whenever the verdict is `transform`. An event carries stable metadata such as the intervention point, enforcement mode, decision, reason code, error class, policy id, annotator names, duration, and whether a transform was applied. When a verdict carries `evidence` the event MAY include the `evidence_artefact` and the key names of `verification_pointers` recorded as `evidence_verification_pointer_keys`, and it MUST NOT include the pointer URL values. A runtime MAY include the `input_identity` and `enforced_identity` from section 13 as correlation identifiers. The runtime MUST NOT emit policy target values, tool arguments or results, annotation values, model messages, secrets, or personal data.