diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index fe287389b..60ce395e5 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -135,7 +135,7 @@ Owns validated task budgets and deterministic cumulative mitigation plans. Platf ### `originweave-evidence` -Owns universally value-redacted network evidence and source-bound provenance records. Generic network records retain only bounded method, canonical origin, unambiguous bounded path, and bounded field names. Body capture, typed metadata values, WARC serialization, object storage, retention, encryption, and legal policy remain future bounded modules. +Owns universally value-redacted network evidence and source-bound provenance records. Generic network records retain only bounded method, canonical origin, unambiguous bounded path, and bounded field names. Active PR #210 adds a bounded in-memory WARC 1.1 `resource` serializer over already-authorized bytes, with target URIs capped by the shared path ceiling and validated by the shared source-URL authority. This is active-PR evidence only; body persistence, retention, encryption, and legal policy remain future bounded modules. ## 6. Planned modules diff --git a/CHANGELOG.md b/CHANGELOG.md index f747adeae..d2b86a4ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,6 +45,9 @@ All notable changes to OriginWeave are documented in this file. The format follo - Authoritative product documentation graph spanning PRD, TRD, ADR lifecycle/index, product-wide UML, conceptual ERD, requirement/decision traceability, threat modeling, product-wide test strategy, operability, API/protocol, release/rollback, and current primary-source standards doctoring, with machine-checkable repository contracts that keep conversation-derived future work distinct from protected-main implementation claims. - Purpose-bound data-governance and privacy baseline that rejects both blanket masking and ambient raw-value propagation, defines field-scoped just-in-time disclosure, opaque-handle/trusted-broker boundaries, model/provider/region policy, retention/deletion/residency/break-glass controls, truthful CSAP/SOC 2 readiness language, and machine-checkable documentation contracts without inventing an OriginWeave-owned production database. - Proposed product-wide target-architecture ADRs for the Rust control plane, isolated execution modes, typed actions, semantic observation/stale-node authority, prompt-injection and secret separation, resource-governor priority, provenance evidence, browser/protocol adapters, crawler policy, and hourly automation operational closure; these remain Proposed rather than shipped claims until protected review and merge. +- The WARC/PROV child slice from PR #217, now integrated into the unprotected #210 feature branch, adds a bounded in-memory `WarcProvBundle` for deterministic W3C PROV-O JSON-LD projection and offline verification of exact WARC records; durable persistence, retention, and transport-specific export remain planned rather than shipped. +- Connected the active WARC/PROV bundle evidence to the primary standards doctoring record, preserving the in-memory-only and non-shipped durable-adapter boundary. +- Made WARC/PROV offline verification use the complete deterministic WARC digest as the authoritative block-digest binding, preserving fail-closed exact-record checks while keeping the public verifier surface minimal. ### Changed @@ -70,6 +73,7 @@ All notable changes to OriginWeave are documented in this file. The format follo ### Security +- Provenance source URLs now reject case-insensitive and percent-encoded credential query-field names, including recursively encoded nested query-like values, and singly or recursively percent-encoded controls before query-bearing URLs can be retained or serialized into WARC target metadata. - Explicit proxy server identifiers require ASCII decimal port tokens before numeric range parsing, preventing Rust-specific leading-plus spellings from widening proxy authority. - Raw page content cannot become a trusted instruction. - Raw secrets are rejected and secret-capable actions require an opaque broker handle. @@ -101,5 +105,7 @@ All notable changes to OriginWeave are documented in this file. The format follo - Hard RAM and VRAM pressure pauses the active agent and rejects new admission; hard VRAM pressure also offloads a resident local model. - The hourly product agent has no Git metadata or repository authority. A separate post-verification publisher opens one PR and cannot approve or merge it. - The unprivileged OpenCode user is restricted to loopback egress during model execution, preventing runner-wide allow-listed endpoints from becoming direct source-exfiltration channels. +- WARC resource-record debug output now omits untrusted Content-Type parameters while retaining the full validated media type for WARC serialization. +- Active PR #210 now bounds WARC target URIs at the shared path ceiling and reuses the shared source-URL parser, rejecting non-IP-literal bracketed authorities before provenance comparison; this remains active-PR evidence only. -[Unreleased]: https://github.com/ContextualWisdomLab/OriginWeave/compare/main...HEAD \ No newline at end of file +[Unreleased]: https://github.com/ContextualWisdomLab/OriginWeave/compare/main...HEAD diff --git a/Cargo.lock b/Cargo.lock index 848cb7320..6def48327 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -286,6 +286,7 @@ name = "originweave-evidence" version = "0.1.0" dependencies = [ "originweave-core", + "sha2", ] [[package]] diff --git a/crates/originweave-evidence/Cargo.toml b/crates/originweave-evidence/Cargo.toml index a69386c38..35c21a7fb 100644 --- a/crates/originweave-evidence/Cargo.toml +++ b/crates/originweave-evidence/Cargo.toml @@ -12,6 +12,7 @@ publish = false [dependencies] originweave-core = { path = "../originweave-core" } +sha2 = "=0.10.9" [lints] workspace = true diff --git a/crates/originweave-evidence/src/lib.rs b/crates/originweave-evidence/src/lib.rs index 17c97bec8..8847f5020 100644 --- a/crates/originweave-evidence/src/lib.rs +++ b/crates/originweave-evidence/src/lib.rs @@ -10,6 +10,8 @@ mod extraction_schema; mod sensitive_access; mod sensitive_handle_lifecycle; +mod warc_prov_bundle; +mod warc_resource_record; pub use extraction_schema::{ ExtractionCardinality, ExtractionField, ExtractionNormalizationRule, ExtractionSchema, @@ -24,6 +26,15 @@ pub use sensitive_access::{ pub use sensitive_handle_lifecycle::{ SensitiveHandleLifecycleEvidence, SensitiveHandleLifecycleEvidenceInput, }; +pub use warc_prov_bundle::{ + MAX_PROV_SOFTWARE_COMMIT_SHA_BYTES, WarcProvBundle, WarcProvBundleError, + WarcProvBundleVerificationError, +}; +pub use warc_resource_record::{ + MAX_WARC_CONTENT_TYPE_BYTES, MAX_WARC_DATE_BYTES, MAX_WARC_PAYLOAD_BYTES, + MAX_WARC_RECORD_ID_BYTES, MAX_WARC_TARGET_URI_BYTES, WarcPayloadCompleteness, + WarcResourceRecord, WarcResourceRecordError, WarcTruncationReason, +}; use std::collections::BTreeMap; @@ -289,7 +300,7 @@ fn redact_all_values(values: BTreeMap) -> BTreeMap) -> std::fmt::Result { + formatter + .debug_struct("ProvenanceRecord") + .field("source_url_byte_count", &self.source_url.len()) + .field("source_locator_byte_count", &self.source_locator.len()) + .field("source_kind", &self.source_kind) + .field("verification_result", &self.verification_result) + .finish() + } +} + impl ProvenanceRecord { /// Validate and create one provenance record. pub fn new( @@ -366,21 +389,197 @@ fn valid_source_url(source_url: &str) -> bool { || source_url .chars() .any(|character| character.is_control() || character.is_whitespace()) - || source_url.contains(['?', '#', '\\']) + || source_url.contains(['#', '\\']) { return false; } let Some((scheme, remainder)) = source_url.split_once("://") else { return false; }; - let authority_end = remainder.find('/').unwrap_or(remainder.len()); - let authority = &remainder[..authority_end]; + let (hierarchical, query) = remainder + .split_once('?') + .map_or((remainder, None), |(hierarchical, query)| { + (hierarchical, Some(query)) + }); + let authority_end = hierarchical.find('/').unwrap_or(hierarchical.len()); + let authority = &hierarchical[..authority_end]; let origin_text = format!("{scheme}://{authority}"); if Origin::parse(&origin_text).is_err() { return false; } - let path = &remainder[authority_end..]; - path.is_empty() || validate_path(path).is_ok() + let path = &hierarchical[authority_end..]; + if !path.is_empty() && validate_path(path).is_err() { + return false; + } + query.is_none_or(valid_query) +} + +fn valid_query(query: &str) -> bool { + let bytes = query.as_bytes(); + let mut decoded = Vec::with_capacity(bytes.len()); + let mut index = 0; + while index < bytes.len() { + let byte = bytes[index]; + if byte == b'%' { + let Some(high) = bytes + .get(index + 1) + .and_then(|byte| hexadecimal_value(*byte)) + else { + return false; + }; + let Some(low) = bytes + .get(index + 2) + .and_then(|byte| hexadecimal_value(*byte)) + else { + return false; + }; + let decoded_byte = high * 16 + low; + if decoded_byte.is_ascii_control() { + return false; + } + decoded.push(decoded_byte); + index += 3; + continue; + } + if !is_rfc3986_pchar(byte) && !matches!(byte, b'/' | b'?') { + return false; + } + decoded.push(byte); + index += 1; + } + if recursively_encoded_control(&decoded) { + return false; + } + query.split('&').all(|field| { + let (name, value) = field + .split_once('=') + .map_or((field, ""), |(name, value)| (name, value)); + !is_credential_query_name(name) && !nested_query_contains_credential(value) + }) +} + +fn recursively_encoded_control(bytes: &[u8]) -> bool { + let mut candidate = bytes.to_owned(); + loop { + let mut decoded = Vec::with_capacity(candidate.len()); + let mut index = 0; + let mut found_escape = false; + while index < candidate.len() { + if candidate[index] == b'%' { + let Some(high) = candidate + .get(index + 1) + .and_then(|byte| hexadecimal_value(*byte)) + else { + decoded.push(candidate[index]); + index += 1; + continue; + }; + let Some(low) = candidate + .get(index + 2) + .and_then(|byte| hexadecimal_value(*byte)) + else { + decoded.push(candidate[index]); + index += 1; + continue; + }; + let decoded_byte = high * 16 + low; + if decoded_byte.is_ascii_control() { + return true; + } + decoded.push(decoded_byte); + index += 3; + found_escape = true; + continue; + } + decoded.push(candidate[index]); + index += 1; + } + if !found_escape { + return false; + } + candidate = decoded; + } +} + +fn nested_query_contains_credential(value: &str) -> bool { + let bytes = value.as_bytes(); + let mut decoded = Vec::with_capacity(bytes.len()); + let mut index = 0; + while index < bytes.len() { + if bytes[index] == b'%' { + let high = hexadecimal_value(bytes[index + 1]).unwrap_or(0); + let low = hexadecimal_value(bytes[index + 2]).unwrap_or(0); + decoded.push(high * 16 + low); + index += 3; + } else { + decoded.push(bytes[index]); + index += 1; + } + } + decoded + .split(|byte| *byte == b'?') + .skip(1) + .any(|nested_query| { + nested_query.split(|byte| *byte == b'&').any(|field| { + let name_end = field + .iter() + .position(|byte| *byte == b'=') + .unwrap_or(field.len()); + let name = &field[..name_end]; + name.contains(&b'%') || is_credential_query_name_bytes(name) + }) + }) +} + +fn is_credential_query_name(name: &str) -> bool { + let bytes = name.as_bytes(); + let mut decoded = Vec::with_capacity(bytes.len()); + let mut index = 0; + while index < bytes.len() { + if bytes[index] == b'%' { + let high = hexadecimal_value(bytes[index + 1]).unwrap_or(0); + let low = hexadecimal_value(bytes[index + 2]).unwrap_or(0); + decoded.push(high * 16 + low); + index += 3; + } else { + decoded.push(bytes[index]); + index += 1; + } + } + decoded.contains(&b'%') || is_credential_query_name_bytes(&decoded) +} + +fn is_credential_query_name_bytes(decoded: &[u8]) -> bool { + let mut decoded = decoded.to_owned(); + decoded.make_ascii_lowercase(); + decoded.iter_mut().for_each(|byte| { + if *byte == b'-' { + *byte = b'_'; + } + }); + matches!( + decoded.as_slice(), + b"access_token" + | b"api_key" + | b"auth" + | b"authorization" + | b"client_secret" + | b"credential" + | b"key" + | b"password" + | b"secret" + | b"secret_key" + | b"session" + | b"sig" + | b"signature" + | b"token" + | b"x_api_key" + | b"x_amz_credential" + | b"x_amz_security_token" + | b"x_amz_signature" + | b"x_goog_credential" + | b"x_goog_signature" + ) } fn valid_sha256(source_hash: &str) -> bool { diff --git a/crates/originweave-evidence/src/warc_prov_bundle.rs b/crates/originweave-evidence/src/warc_prov_bundle.rs new file mode 100644 index 000000000..11788e771 --- /dev/null +++ b/crates/originweave-evidence/src/warc_prov_bundle.rs @@ -0,0 +1,273 @@ +use std::fmt; + +use sha2::{Digest, Sha256}; + +use crate::{ProvenanceRecord, WarcPayloadCompleteness, WarcResourceRecord}; + +const ORIGINWEAVE_COMMIT_URL_PREFIX: &str = + "https://github.com/ContextualWisdomLab/OriginWeave/commit/"; +const WARC_RECORD_DIGEST_IRI: &str = + "tag:contextualwisdomlab.github.io,2026:OriginWeave/warcRecordDigest"; +const WARC_PAYLOAD_COMPLETENESS_IRI: &str = + "tag:contextualwisdomlab.github.io,2026:OriginWeave/warcPayloadCompleteness"; +const WARC_TRUNCATION_REASON_IRI: &str = + "tag:contextualwisdomlab.github.io,2026:OriginWeave/warcTruncationReason"; + +/// Exact byte length accepted for a canonical Git SHA-1 software revision. +pub const MAX_PROV_SOFTWARE_COMMIT_SHA_BYTES: usize = 40; + +/// A validation failure while constructing a deterministic WARC provenance bundle. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WarcProvBundleError { + /// The software revision was not one canonical lower-case 40-byte Git SHA-1. + InvalidSoftwareCommitSha, + /// A bounded provenance field exceeded its allowed size. + LimitExceeded, +} + +impl fmt::Display for WarcProvBundleError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(match self { + Self::InvalidSoftwareCommitSha => "invalid OriginWeave software commit SHA", + Self::LimitExceeded => "WARC PROV bundle limit exceeded", + }) + } +} + +impl std::error::Error for WarcProvBundleError {} + +/// A deterministic offline verification failure between a PROV bundle and a WARC record. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WarcProvBundleVerificationError { + /// The WARC record identifier differs from the identifier bound into the PROV bundle. + RecordIdentityMismatch, + /// The independently verified source provenance differs from the provenance bound into the bundle. + SourceEvidenceMismatch, + /// The WARC capture timestamp differs from the timestamp bound into the PROV bundle. + CaptureTimeMismatch, + /// The WARC complete-versus-truncated state differs from the state bound into the bundle. + PayloadCompletenessMismatch, + /// The digest of the deterministic WARC serialization differs from the bundle binding. + WarcRecordDigestMismatch, +} + +impl fmt::Display for WarcProvBundleVerificationError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(match self { + Self::RecordIdentityMismatch => "WARC PROV record identity does not match", + Self::SourceEvidenceMismatch => "WARC PROV source evidence does not match", + Self::CaptureTimeMismatch => "WARC PROV capture time does not match", + Self::PayloadCompletenessMismatch => "WARC PROV payload completeness does not match", + Self::WarcRecordDigestMismatch => "WARC PROV serialized record digest does not match", + }) + } +} + +impl std::error::Error for WarcProvBundleVerificationError {} + +/// A deterministic PROV-O JSON-LD projection over one validated WARC resource record. +/// +/// The bundle contains identifiers, exact validated source provenance, record hashes, source +/// location, capture time, explicit WARC payload completeness, and the exact OriginWeave software +/// revision. It deliberately does not retain or emit the WARC payload. +#[derive(Clone, PartialEq, Eq)] +pub struct WarcProvBundle { + record_entity_id: String, + source_entity_id: String, + capture_activity_id: String, + software_agent_id: String, + software_commit_sha: String, + source_provenance: ProvenanceRecord, + warc_date: String, + block_digest: String, + warc_record_digest: String, + payload_completeness: WarcPayloadCompleteness, +} + +impl fmt::Debug for WarcProvBundle { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("WarcProvBundle") + .field("record_entity_id", &self.record_entity_id) + .field("source_entity_id", &self.source_entity_id) + .field("capture_activity_id", &self.capture_activity_id) + .field("software_agent_id", &self.software_agent_id) + .field("payload_completeness", &self.payload_completeness) + .finish_non_exhaustive() + } +} + +impl WarcProvBundle { + /// Construct a provenance bundle from one already-validated WARC resource record. + /// + /// `software_commit_sha` is an immutable canonical Git SHA-1 identifier. This constructor + /// does not contact GitHub and does not treat the identifier as authentication or authority. + pub fn new( + record: &WarcResourceRecord, + software_commit_sha: &str, + ) -> Result { + if software_commit_sha.len() > MAX_PROV_SOFTWARE_COMMIT_SHA_BYTES { + return Err(WarcProvBundleError::LimitExceeded); + } + if !valid_software_commit_sha(software_commit_sha) { + return Err(WarcProvBundleError::InvalidSoftwareCommitSha); + } + + let record_entity_id = record.record_id().to_owned(); + let source_entity_id = format!("{}#source", record.record_id()); + let capture_activity_id = format!("{}#capture", record.record_id()); + let software_agent_id = format!("{ORIGINWEAVE_COMMIT_URL_PREFIX}{software_commit_sha}"); + let warc_record_digest = sha256_digest(&record.to_warc_bytes()); + + Ok(Self { + record_entity_id, + source_entity_id, + capture_activity_id, + software_agent_id, + software_commit_sha: software_commit_sha.to_owned(), + source_provenance: record.provenance().clone(), + warc_date: record.warc_date().to_owned(), + block_digest: record.block_digest().to_owned(), + warc_record_digest, + payload_completeness: record.completeness(), + }) + } + + /// Return the PROV entity identifier for the WARC record. + #[must_use] + pub fn record_entity_id(&self) -> &str { + &self.record_entity_id + } + + /// Return the PROV entity identifier for the independently verified source. + #[must_use] + pub fn source_entity_id(&self) -> &str { + &self.source_entity_id + } + + /// Return the PROV activity identifier for this capture. + #[must_use] + pub fn capture_activity_id(&self) -> &str { + &self.capture_activity_id + } + + /// Return the immutable OriginWeave commit URL used as the PROV software-agent identifier. + #[must_use] + pub fn software_agent_id(&self) -> &str { + &self.software_agent_id + } + + /// Return the canonical lower-case Git SHA-1 of the OriginWeave revision. + #[must_use] + pub fn software_commit_sha(&self) -> &str { + &self.software_commit_sha + } + + /// Verify offline that one validated WARC record is exactly the record bound by this bundle. + /// + /// Verification includes the complete validated [`ProvenanceRecord`] rather than only the + /// source URL and digest, so locator or evidence-channel drift cannot collapse into a match. + /// It is deterministic and performs no network, DNS, browser, model, persistence, or authority + /// operation. A matching digest proves byte identity only; it does not authenticate the actor + /// that produced either value or establish factual correctness. + pub fn verify_record( + &self, + record: &WarcResourceRecord, + ) -> Result<(), WarcProvBundleVerificationError> { + if self.record_entity_id != record.record_id() { + return Err(WarcProvBundleVerificationError::RecordIdentityMismatch); + } + if self.source_provenance != *record.provenance() { + return Err(WarcProvBundleVerificationError::SourceEvidenceMismatch); + } + if self.warc_date != record.warc_date() { + return Err(WarcProvBundleVerificationError::CaptureTimeMismatch); + } + // `warc_record_digest` covers the complete deterministic serialization, including the + // WARC-Block-Digest header, so a block-digest drift is reported by that binding below. + if self.payload_completeness != record.completeness() { + return Err(WarcProvBundleVerificationError::PayloadCompletenessMismatch); + } + if self.warc_record_digest != sha256_digest(&record.to_warc_bytes()) { + return Err(WarcProvBundleVerificationError::WarcRecordDigestMismatch); + } + Ok(()) + } + + /// Serialize the bundle as deterministic compact W3C PROV-O JSON-LD. + /// + /// All interpolated values originate from the validated WARC record or the canonical + /// lower-case software commit identifier, so no raw payload bytes enter this document. The + /// payload block digest binds the retained resource bytes while `warcRecordDigest` binds the + /// complete deterministic WARC serialization, including its headers. WARC payload completeness + /// is retained as an OriginWeave-owned absolute-IRI attribute; truncated records also retain + /// the exact WARC truncation token. The JSON-LD projection exposes source URL and digest while + /// offline verification additionally preserves the exact validated source locator and channel. + #[must_use] + pub fn to_json_ld(&self) -> String { + let completeness_attributes = + warc_payload_completeness_attributes(self.payload_completeness); + format!( + "{{\"@context\":{{\"prov\":\"http://www.w3.org/ns/prov#\",\"xsd\":\"http://www.w3.org/2001/XMLSchema#\"}},\"@graph\":[{{\"@id\":\"{}\",\"@type\":\"prov:Entity\",\"prov:atLocation\":{{\"@id\":\"{}\"}},\"prov:value\":\"{}\"}},{{\"@id\":\"{}\",\"@type\":\"prov:Activity\",\"prov:startedAtTime\":{{\"@value\":\"{}\",\"@type\":\"xsd:dateTime\"}},\"prov:used\":{{\"@id\":\"{}\"}},\"prov:wasAssociatedWith\":{{\"@id\":\"{}\"}}}},{{\"@id\":\"{}\",\"@type\":\"prov:SoftwareAgent\"}},{{\"@id\":\"{}\",\"@type\":\"prov:Entity\",\"prov:value\":\"{}\",\"{WARC_RECORD_DIGEST_IRI}\":\"{}\",{},\"prov:wasDerivedFrom\":{{\"@id\":\"{}\"}},\"prov:wasGeneratedBy\":{{\"@id\":\"{}\"}}}}]}}", + self.source_entity_id, + self.source_provenance.source_url(), + self.source_provenance.source_hash(), + self.capture_activity_id, + self.warc_date, + self.source_entity_id, + self.software_agent_id, + self.software_agent_id, + self.record_entity_id, + self.block_digest, + self.warc_record_digest, + completeness_attributes, + self.source_entity_id, + self.capture_activity_id, + ) + } +} + +fn warc_payload_completeness_attributes(completeness: WarcPayloadCompleteness) -> String { + match completeness { + WarcPayloadCompleteness::Complete => { + format!("\"{WARC_PAYLOAD_COMPLETENESS_IRI}\":\"complete\"") + } + WarcPayloadCompleteness::Truncated(reason) => format!( + "\"{WARC_PAYLOAD_COMPLETENESS_IRI}\":\"truncated\",\"{WARC_TRUNCATION_REASON_IRI}\":\"{}\"", + reason.warc_token() + ), + } +} + +fn sha256_digest(bytes: &[u8]) -> String { + let digest = Sha256::digest(bytes); + let mut encoded = String::from("sha256:"); + for byte in digest { + encoded.push_str(&format!("{byte:02x}")); + } + encoded +} + +fn valid_software_commit_sha(software_commit_sha: &str) -> bool { + software_commit_sha.len() == MAX_PROV_SOFTWARE_COMMIT_SHA_BYTES + && software_commit_sha + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) + && software_commit_sha.bytes().any(|byte| byte != b'0') +} + +#[cfg(test)] +mod tests { + use crate::WarcTruncationReason; + + #[test] + fn truncation_tokens_use_the_shared_warc_mapping() { + assert_eq!(WarcTruncationReason::Length.warc_token(), "length"); + assert_eq!(WarcTruncationReason::Time.warc_token(), "time"); + assert_eq!(WarcTruncationReason::Disconnect.warc_token(), "disconnect"); + assert_eq!( + WarcTruncationReason::Unspecified.warc_token(), + "unspecified" + ); + } +} diff --git a/crates/originweave-evidence/src/warc_resource_record.rs b/crates/originweave-evidence/src/warc_resource_record.rs new file mode 100644 index 000000000..4fdc7a475 --- /dev/null +++ b/crates/originweave-evidence/src/warc_resource_record.rs @@ -0,0 +1,562 @@ +use std::fmt; + +use sha2::{Digest, Sha256}; + +use crate::{ProvenanceRecord, VerificationResult}; + +/// Maximum encoded size of the UUID-based WARC record identifier. +pub const MAX_WARC_RECORD_ID_BYTES: usize = 45; +/// Maximum encoded size accepted for a UTC WARC date. +pub const MAX_WARC_DATE_BYTES: usize = 30; +/// Maximum encoded size retained for a WARC content type. +pub const MAX_WARC_CONTENT_TYPE_BYTES: usize = 256; +/// Maximum encoded target URI size accepted by one immutable WARC record. +pub const MAX_WARC_TARGET_URI_BYTES: usize = crate::MAX_PATH_BYTES; +/// Maximum resource payload retained by one immutable WARC record. +pub const MAX_WARC_PAYLOAD_BYTES: usize = 1_048_576; + +/// Standard WARC 1.1 reason for a deliberately or unexpectedly truncated record block. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WarcTruncationReason { + /// Capture stopped because the configured maximum length was reached. + Length, + /// Capture stopped because the configured maximum capture time was reached. + Time, + /// Capture stopped because the network connection disconnected. + Disconnect, + /// Capture stopped for another or unknown reason. + Unspecified, +} + +impl WarcTruncationReason { + /// Return the canonical WARC 1.1 truncation token for this reason. + pub(crate) const fn warc_token(self) -> &'static str { + match self { + Self::Length => "length", + Self::Time => "time", + Self::Disconnect => "disconnect", + Self::Unspecified => "unspecified", + } + } +} + +/// Whether the retained WARC record block is complete or explicitly truncated. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WarcPayloadCompleteness { + /// The retained record block is complete for the caller-authorized capture. + Complete, + /// The retained record block is partial for the stated WARC 1.1 truncation reason. + Truncated(WarcTruncationReason), +} + +/// A validation failure while constructing an immutable WARC resource record. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum WarcResourceRecordError { + /// The record identifier was not a bounded non-nil UUID URN. + InvalidRecordId, + /// The date was not a bounded UTC RFC 3339 timestamp. + InvalidDate, + /// The content type was not a bounded, syntactically valid MIME media type. + InvalidContentType, + /// A record field or payload exceeded its retention limit. + LimitExceeded, + /// The WARC target URI contained octets outside RFC 3986 URI syntax. + InvalidTargetUri, + /// The WARC target URI differed from its provenance source URL. + TargetUriMismatch, + /// The source provenance was not independently verified. + UnverifiedProvenance, + /// The retained payload digest differed from the verified provenance digest. + PayloadProvenanceMismatch, +} + +impl fmt::Display for WarcResourceRecordError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(match self { + Self::InvalidRecordId => "invalid WARC record identifier", + Self::InvalidDate => "invalid WARC date", + Self::InvalidContentType => "invalid WARC content type", + Self::LimitExceeded => "WARC resource record limit exceeded", + Self::InvalidTargetUri => "invalid WARC target URI", + Self::TargetUriMismatch => "WARC target URI does not match provenance", + Self::UnverifiedProvenance => "WARC provenance is not independently verified", + Self::PayloadProvenanceMismatch => "WARC payload digest does not match provenance", + }) + } +} + +impl std::error::Error for WarcResourceRecordError {} + +/// An immutable, bounded WARC `resource` record over already-authorized bytes. +#[derive(Clone, PartialEq, Eq)] +pub struct WarcResourceRecord { + record_id: String, + warc_date: String, + target_uri: String, + content_type: String, + payload: Vec, + block_digest: String, + provenance: ProvenanceRecord, + completeness: WarcPayloadCompleteness, +} + +impl fmt::Debug for WarcResourceRecord { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + let content_type = self + .content_type + .split_once(';') + .map_or(self.content_type.as_str(), |(essence, _)| { + essence.trim_end_matches([' ', '\t']) + }); + formatter + .debug_struct("WarcResourceRecord") + .field("record_id", &self.record_id) + .field("warc_date", &self.warc_date) + .field("content_type", &content_type) + .field("payload_byte_count", &self.payload.len()) + .field("block_digest", &self.block_digest) + .field("completeness", &self.completeness) + .field( + "provenance_verification_result", + &self.provenance.verification_result(), + ) + .finish() + } +} + +impl WarcResourceRecord { + /// Validate and construct one complete resource record without contacting a live origin. + pub fn new( + record_id: &str, + warc_date: &str, + target_uri: &str, + content_type: &str, + payload: Vec, + provenance: ProvenanceRecord, + ) -> Result { + Self::new_with_completeness( + record_id, + warc_date, + target_uri, + content_type, + payload, + provenance, + WarcPayloadCompleteness::Complete, + ) + } + + /// Validate and construct one resource record with explicit capture completeness. + /// + /// Truncated records retain the caller-provided partial block and emit the standard WARC 1.1 + /// `WARC-Truncated` reason. This constructor never truncates an oversized payload implicitly; + /// the retained block must still satisfy [`MAX_WARC_PAYLOAD_BYTES`]. The exact retained bytes, + /// including an explicitly truncated block, must match the independently verified provenance + /// digest rather than inheriting provenance from another representation of the same URL. + pub fn new_with_completeness( + record_id: &str, + warc_date: &str, + target_uri: &str, + content_type: &str, + payload: Vec, + provenance: ProvenanceRecord, + completeness: WarcPayloadCompleteness, + ) -> Result { + if record_id.len() > MAX_WARC_RECORD_ID_BYTES { + return Err(WarcResourceRecordError::LimitExceeded); + } + if warc_date.len() > MAX_WARC_DATE_BYTES { + return Err(WarcResourceRecordError::LimitExceeded); + } + if !valid_record_id(record_id) { + return Err(WarcResourceRecordError::InvalidRecordId); + } + if !valid_utc_date(warc_date) { + return Err(WarcResourceRecordError::InvalidDate); + } + if !valid_content_type(content_type) { + return Err(if content_type.len() > MAX_WARC_CONTENT_TYPE_BYTES { + WarcResourceRecordError::LimitExceeded + } else { + WarcResourceRecordError::InvalidContentType + }); + } + if target_uri.len() > MAX_WARC_TARGET_URI_BYTES { + return Err(WarcResourceRecordError::LimitExceeded); + } + if !valid_target_uri_presentation(target_uri) { + return Err(WarcResourceRecordError::InvalidTargetUri); + } + if target_uri != provenance.source_url() { + return Err(WarcResourceRecordError::TargetUriMismatch); + } + if provenance.verification_result() != VerificationResult::Verified { + return Err(WarcResourceRecordError::UnverifiedProvenance); + } + if payload.len() > MAX_WARC_PAYLOAD_BYTES { + return Err(WarcResourceRecordError::LimitExceeded); + } + let block_digest = sha256_digest(&payload); + if block_digest != provenance.source_hash() { + return Err(WarcResourceRecordError::PayloadProvenanceMismatch); + } + + Ok(Self { + record_id: record_id.to_owned(), + warc_date: warc_date.to_owned(), + target_uri: target_uri.to_owned(), + content_type: content_type.to_owned(), + block_digest, + payload, + provenance, + completeness, + }) + } + + /// Return the UUID URN used as the WARC record identity. + #[must_use] + pub fn record_id(&self) -> &str { + &self.record_id + } + + /// Return the normalized UTC capture timestamp. + #[must_use] + pub fn warc_date(&self) -> &str { + &self.warc_date + } + + /// Return the provenance-bound target URI. + #[must_use] + pub fn target_uri(&self) -> &str { + &self.target_uri + } + + /// Return the payload media type retained in the WARC record. + #[must_use] + pub fn content_type(&self) -> &str { + &self.content_type + } + + /// Return the immutable resource bytes. + #[must_use] + pub fn payload(&self) -> &[u8] { + &self.payload + } + + /// Return the lowercase SHA-256 block digest. + #[must_use] + pub fn block_digest(&self) -> &str { + &self.block_digest + } + + /// Return the independently verified provenance bound to this record. + #[must_use] + pub const fn provenance(&self) -> &ProvenanceRecord { + &self.provenance + } + + /// Return whether the retained block is complete or explicitly truncated. + #[must_use] + pub const fn completeness(&self) -> WarcPayloadCompleteness { + self.completeness + } + + /// Serialize this bounded resource record as deterministic WARC 1.1 bytes. + #[must_use] + pub fn to_warc_bytes(&self) -> Vec { + let truncated_header = match self.completeness { + WarcPayloadCompleteness::Complete => String::new(), + WarcPayloadCompleteness::Truncated(reason) => { + format!("WARC-Truncated: {}\r\n", reason.warc_token()) + } + }; + let header = format!( + "WARC/1.1\r\nWARC-Type: resource\r\nWARC-Record-ID: <{}>\r\nWARC-Date: {}\r\nWARC-Target-URI: {}\r\n{}Content-Type: {}\r\nWARC-Block-Digest: {}\r\nContent-Length: {}\r\n\r\n", + self.record_id, + self.warc_date, + self.target_uri, + truncated_header, + self.content_type, + self.block_digest, + self.payload.len() + ); + let mut bytes = header.into_bytes(); + bytes.extend_from_slice(&self.payload); + bytes.extend_from_slice(b"\r\n\r\n"); + bytes + } +} + +fn valid_record_id(record_id: &str) -> bool { + let bytes = record_id.as_bytes(); + if bytes.len() != MAX_WARC_RECORD_ID_BYTES || !record_id.starts_with("urn:uuid:") { + return false; + } + let mut has_nonzero_hex = false; + for (index, byte) in bytes[9..].iter().copied().enumerate() { + if matches!(index, 8 | 13 | 18 | 23) { + if byte != b'-' { + return false; + } + } else if !byte.is_ascii_hexdigit() { + return false; + } else if byte != b'0' { + has_nonzero_hex = true; + } + } + has_nonzero_hex +} + +fn valid_utc_date(date: &str) -> bool { + let bytes = date.as_bytes(); + if !(20..=MAX_WARC_DATE_BYTES).contains(&bytes.len()) + || bytes.get(4) != Some(&b'-') + || bytes.get(7) != Some(&b'-') + || bytes.get(10) != Some(&b'T') + || bytes.get(13) != Some(&b':') + || bytes.get(16) != Some(&b':') + { + return false; + } + let has_fraction = bytes[19] == b'.'; + if has_fraction { + if bytes.last() != Some(&b'Z') || bytes.len() < 22 { + return false; + } + let fraction = &bytes[20..bytes.len() - 1]; + if fraction.iter().any(|byte| !byte.is_ascii_digit()) { + return false; + } + } else if bytes.len() != 20 || bytes[19] != b'Z' { + return false; + } + if !bytes[..19] + .iter() + .enumerate() + .all(|(index, byte)| matches!(index, 4 | 7 | 10 | 13 | 16) || byte.is_ascii_digit()) + { + return false; + } + let year = four_digits(bytes[0], bytes[1], bytes[2], bytes[3]); + let month = two_digits(bytes[5], bytes[6]); + let day = two_digits(bytes[8], bytes[9]); + let hour = two_digits(bytes[11], bytes[12]); + let minute = two_digits(bytes[14], bytes[15]); + let second = two_digits(bytes[17], bytes[18]); + valid_calendar_date(year, month, day) && hour < 24 && minute < 60 && second < 60 +} + +fn four_digits(first: u8, second: u8, third: u8, fourth: u8) -> u16 { + u16::from(first - b'0') * 1000 + + u16::from(second - b'0') * 100 + + u16::from(third - b'0') * 10 + + u16::from(fourth - b'0') +} + +fn two_digits(high: u8, low: u8) -> u8 { + (high - b'0') * 10 + (low - b'0') +} + +fn valid_calendar_date(year: u16, month: u8, day: u8) -> bool { + if !(1..=12).contains(&month) { + return false; + } + let days_in_month = if month == 2 { + if is_leap_year(year) { 29 } else { 28 } + } else { + 30 + ((month + month / 8) % 2) + }; + (1..=days_in_month).contains(&day) +} + +fn is_leap_year(year: u16) -> bool { + year.is_multiple_of(400) || (year.is_multiple_of(4) && !year.is_multiple_of(100)) +} + +fn valid_target_uri_presentation(target_uri: &str) -> bool { + crate::valid_source_url(target_uri) +} + +fn valid_content_type(content_type: &str) -> bool { + if content_type.is_empty() || content_type.len() > MAX_WARC_CONTENT_TYPE_BYTES { + return false; + } + + let (essence, mut parameters) = content_type + .split_once(';') + .map_or((content_type, None), |(essence, parameters)| { + (essence, Some(parameters)) + }); + let essence = trim_ows_end(essence); + let Some((media_type, media_subtype)) = essence.split_once('/') else { + return false; + }; + if !valid_mime_token(media_type) || !valid_mime_token(media_subtype) { + return false; + } + + while let Some(parameter_text) = parameters { + let (parameter, remaining) = split_mime_parameter(parameter_text); + if !valid_mime_parameter(parameter) { + return false; + } + parameters = remaining; + } + true +} + +fn split_mime_parameter(value: &str) -> (&str, Option<&str>) { + let mut quoted = false; + let mut escaped = false; + for (index, byte) in value.bytes().enumerate() { + if quoted { + if escaped { + escaped = false; + } else if byte == b'\\' { + escaped = true; + } else if byte == b'"' { + quoted = false; + } + } else if byte == b'"' { + quoted = true; + } else if byte == b';' { + return (&value[..index], Some(&value[index + 1..])); + } + } + (value, None) +} + +fn valid_mime_parameter(parameter: &str) -> bool { + let parameter = trim_ows(parameter); + let Some((attribute, value)) = parameter.split_once('=') else { + return false; + }; + valid_mime_token(attribute) && valid_mime_parameter_value(value) +} + +fn valid_mime_parameter_value(value: &str) -> bool { + if valid_mime_token(value) { + return true; + } + + let bytes = value.as_bytes(); + if bytes.len() < 2 || bytes.first() != Some(&b'"') || bytes.last() != Some(&b'"') { + return false; + } + + let mut escaped = false; + for byte in bytes[1..bytes.len() - 1].iter().copied() { + if escaped { + if !valid_quoted_pair_byte(byte) { + return false; + } + escaped = false; + } else if byte == b'\\' { + escaped = true; + } else if !valid_quoted_text_byte(byte) { + return false; + } + } + !escaped +} + +const fn valid_mime_token(value: &str) -> bool { + let bytes = value.as_bytes(); + if bytes.is_empty() { + return false; + } + let mut index = 0; + while index < bytes.len() { + if !is_mime_token_byte(bytes[index]) { + return false; + } + index += 1; + } + true +} + +const fn is_mime_token_byte(byte: u8) -> bool { + byte.is_ascii_alphanumeric() + || matches!( + byte, + b'!' | b'#' + | b'$' + | b'%' + | b'&' + | b'\'' + | b'*' + | b'+' + | b'-' + | b'.' + | b'^' + | b'_' + | b'`' + | b'|' + | b'~' + ) +} + +const fn valid_quoted_text_byte(byte: u8) -> bool { + byte == b'\t' + || byte == b' ' + || byte == b'!' + || (byte >= 0x23 && byte <= 0x5b) + || (byte >= 0x5d && byte <= 0x7e) +} + +const fn valid_quoted_pair_byte(byte: u8) -> bool { + byte == b'\t' || byte == b' ' || (byte >= 0x21 && byte <= 0x7e) +} + +fn trim_ows(value: &str) -> &str { + value.trim_matches([' ', '\t']) +} + +fn trim_ows_end(value: &str) -> &str { + value.trim_end_matches([' ', '\t']) +} + +fn sha256_digest(payload: &[u8]) -> String { + let digest = Sha256::digest(payload); + let mut encoded = String::from("sha256:"); + for byte in digest { + encoded.push_str(&format!("{byte:02x}")); + } + encoded +} + +#[cfg(test)] +#[allow(clippy::expect_used)] +mod tests { + use super::*; + use crate::{WarcProvBundle, WarcProvBundleVerificationError}; + + #[test] + fn provenance_bundle_rejects_a_tampered_block_digest() { + let provenance = ProvenanceRecord::new( + "https://example.com/item", + "body", + "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824", + crate::EvidenceSourceKind::NetworkResponse, + crate::VerificationResult::Verified, + ) + .expect("verified provenance"); + let mut record = WarcResourceRecord::new( + "urn:uuid:123e4567-e89b-12d3-a456-426614174000", + "2026-08-22T12:00:00Z", + "https://example.com/item", + "text/plain", + b"hello".to_vec(), + provenance, + ) + .expect("WARC resource record"); + let bundle = WarcProvBundle::new(&record, "0123456789abcdef0123456789abcdef01234567") + .expect("PROV bundle"); + + record.block_digest = "sha256:tampered".to_owned(); + + assert_eq!( + bundle.verify_record(&record), + Err(WarcProvBundleVerificationError::WarcRecordDigestMismatch) + ); + } +} diff --git a/crates/originweave-evidence/tests/evidence.rs b/crates/originweave-evidence/tests/evidence.rs index 48d49cbc4..56532c6c8 100644 --- a/crates/originweave-evidence/tests/evidence.rs +++ b/crates/originweave-evidence/tests/evidence.rs @@ -103,6 +103,7 @@ fn provenance_accepts_safe_root_path_and_loopback_sources() { for source_url in [ "https://example.com", "https://example.com/item/42", + "https://example.com/search?cache", "http://localhost:9222/json/version", "http://[::1]:9222/json/version", ] { @@ -126,7 +127,6 @@ fn provenance_rejects_credential_bearing_or_ambiguous_source_urls() { "ftp://example.com/path", "http://example.com/path", "https://user:password@example.com/path", - "https://example.com/path?access_token=secret", "https://example.com/path#fragment", "https://example.com/bad\\path", "https://example.com/\n", diff --git a/crates/originweave-evidence/tests/provenance_query_urls.rs b/crates/originweave-evidence/tests/provenance_query_urls.rs new file mode 100644 index 000000000..4aecd7701 --- /dev/null +++ b/crates/originweave-evidence/tests/provenance_query_urls.rs @@ -0,0 +1,128 @@ +#![allow(clippy::expect_used)] + +use originweave_evidence::{ + EvidenceError, EvidenceSourceKind, ProvenanceRecord, VerificationResult, WarcResourceRecord, +}; + +const EMPTY_SHA256: &str = + "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"; +const RECORD_ID: &str = "urn:uuid:123e4567-e89b-12d3-a456-426614174000"; +const DATE: &str = "2026-08-28T00:00:00Z"; + +#[test] +fn provenance_and_warc_preserve_query_bearing_resource_urls_without_debug_disclosure() { + let source_url = "https://example.com/search?next=/products?category=widgets&q=public%20term&literal_percent=percent%25&partial_percent=percent%252"; + let provenance = ProvenanceRecord::new( + source_url, + "body", + EMPTY_SHA256, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("query-bearing provenance URL should be accepted"); + + assert_eq!(provenance.source_url(), source_url); + let provenance_debug = format!("{provenance:?}"); + assert!(!provenance_debug.contains(source_url)); + assert!(!provenance_debug.contains("public%20term")); + assert!(!provenance_debug.contains(EMPTY_SHA256)); + assert!(!provenance_debug.contains("body")); + + let record = WarcResourceRecord::new( + RECORD_ID, + DATE, + source_url, + "text/plain", + Vec::new(), + provenance, + ) + .expect("query-bearing WARC target URI should be accepted"); + + assert_eq!(record.target_uri(), source_url); + let warc = String::from_utf8(record.to_warc_bytes()).expect("bounded WARC bytes are UTF-8"); + assert!(warc.contains(&format!("WARC-Target-URI: {source_url}\r\n"))); +} + +#[test] +fn provenance_query_support_rejects_credential_fields() { + for source_url in [ + "https://example.com/callback?access_token=secret", + "https://example.com/callback?ACCESS-TOKEN=secret", + "https://example.com/callback?access%5Ftoken=secret", + "https://example.com/callback?access%255Ftoken=secret", + "https://example.com/download?api_key=secret", + "https://example.com/download?client_secret=secret", + "https://example.com/download?X-Amz-Credential=secret", + "https://example.com/download?X-Amz-Signature=secret", + "https://example.com/download?X%2Damz%2DSignature=secret", + "https://example.com/download?x-goog-credential=secret", + "https://example.com/login?password=secret", + "https://example.com/login?auth=secret", + "https://example.com/login?sig=secret", + "https://example.com/callback?redirect=https://example.com/landing?token=secret", + "https://example.com/callback?redirect=https%3A%2F%2Fexample.com%2Flanding%3Ftoken%3Dsecret", + "https://example.com/callback?redirect=https%3A%2F%2Fexample.com%2Flanding%3Faccess%255Ftoken%3Dsecret", + ] { + assert_eq!( + ProvenanceRecord::new( + source_url, + "body", + EMPTY_SHA256, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ), + Err(EvidenceError::InvalidSourceUrl), + "credential-bearing source_url={source_url:?}" + ); + } +} + +#[test] +fn provenance_query_support_does_not_admit_fragments_or_unsafe_uri_octets() { + for source_url in [ + "https://example.com/search?q=value#fragment", + "https://example.com/search#fragment", + "https://example.com/search?q=bad value", + "https://example.com/search?q=bad\\value", + "https://example.com/search?q=raw|pipe", + "https://example.com/search?q=raw-한글", + "https://example.com/search?q=%", + "https://example.com/search?q=%2", + "https://example.com/search?q=%GG", + "https://example.com/search?q=%2G", + "https://example.com/search?q=%0A", + "https://example.com/search?q=%09", + "https://example.com/search?q=%7F", + "https://example.com/search?q=%250A", + "https://example.com/search?q=%2509", + "https://example.com/search?q=%257F", + ] { + assert_eq!( + ProvenanceRecord::new( + source_url, + "body", + EMPTY_SHA256, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ), + Err(EvidenceError::InvalidSourceUrl), + "source_url={source_url:?}" + ); + } +} + +#[test] +fn provenance_query_support_rejects_recursively_encoded_controls() { + let source_url = "https://example.com/search?q=%25250A"; + assert_eq!( + ProvenanceRecord::new( + source_url, + "body", + EMPTY_SHA256, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ), + Err(EvidenceError::InvalidSourceUrl), + "recursively encoded control source_url={source_url:?}" + ); +} diff --git a/crates/originweave-evidence/tests/warc_debug_redaction.rs b/crates/originweave-evidence/tests/warc_debug_redaction.rs new file mode 100644 index 000000000..f9f4cea5c --- /dev/null +++ b/crates/originweave-evidence/tests/warc_debug_redaction.rs @@ -0,0 +1,56 @@ +#![allow(clippy::expect_used)] + +use originweave_evidence::{ + EvidenceSourceKind, ProvenanceRecord, VerificationResult, WarcResourceRecord, +}; + +#[test] +fn warc_record_debug_does_not_disclose_payload_or_provenance_locator() { + let provenance = ProvenanceRecord::new( + "https://example.com/resource", + "private-selector-marker", + "sha256:3684d4581255ca55e94c2cb89affc7d0dc914b7462c1f496780d7f2214877709", + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("verified provenance"); + let record = WarcResourceRecord::new( + "urn:uuid:123e4567-e89b-12d3-a456-426614174000", + "2026-08-21T09:00:00Z", + "https://example.com/resource", + "application/octet-stream", + vec![254, 237, 250, 206], + provenance, + ) + .expect("WARC resource record"); + + let debug = format!("{record:?}"); + assert!(debug.contains("payload_byte_count")); + assert!(!debug.contains("254, 237, 250, 206")); + assert!(!debug.contains("private-selector-marker")); +} + +#[test] +fn warc_record_debug_does_not_disclose_content_type_parameters() { + let provenance = ProvenanceRecord::new( + "https://example.com/resource", + "body", + "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("verified provenance"); + let record = WarcResourceRecord::new( + "urn:uuid:123e4567-e89b-12d3-a456-426614174000", + "2026-08-21T09:00:00Z", + "https://example.com/resource", + "text/plain; token=secret", + Vec::new(), + provenance, + ) + .expect("WARC resource record"); + + let debug = format!("{record:?}"); + assert!(debug.contains("content_type: \"text/plain\"")); + assert!(!debug.contains("token=secret")); +} diff --git a/crates/originweave-evidence/tests/warc_field_limit_errors.rs b/crates/originweave-evidence/tests/warc_field_limit_errors.rs new file mode 100644 index 000000000..ec2581d55 --- /dev/null +++ b/crates/originweave-evidence/tests/warc_field_limit_errors.rs @@ -0,0 +1,94 @@ +#![allow(clippy::expect_used)] + +use originweave_evidence::{ + EvidenceSourceKind, MAX_WARC_TARGET_URI_BYTES, ProvenanceRecord, VerificationResult, + WarcResourceRecord, WarcResourceRecordError, +}; + +const SOURCE_HASH: &str = "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"; +const RECORD_ID: &str = "urn:uuid:123e4567-e89b-12d3-a456-426614174000"; +const DATE: &str = "2026-08-21T00:00:00Z"; +const TARGET_URI: &str = "https://example.com/item"; + +fn provenance() -> ProvenanceRecord { + ProvenanceRecord::new( + TARGET_URI, + "body", + SOURCE_HASH, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("valid provenance") +} + +#[test] +fn oversized_record_fields_report_the_bounded_limit_error() { + let oversized_record_id = format!("{RECORD_ID}x"); + assert_eq!( + WarcResourceRecord::new( + &oversized_record_id, + DATE, + TARGET_URI, + "text/plain", + Vec::new(), + provenance(), + ), + Err(WarcResourceRecordError::LimitExceeded), + ); + + let oversized_date = "2026-08-21T00:00:00.1234567890Z"; + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + oversized_date, + TARGET_URI, + "text/plain", + Vec::new(), + provenance(), + ), + Err(WarcResourceRecordError::LimitExceeded), + ); +} + +#[test] +fn target_uri_limit_is_inclusive_and_overflow_fails_closed() { + let prefix = "https://example.com/"; + let target_uri = format!( + "{prefix}{}", + "a".repeat(MAX_WARC_TARGET_URI_BYTES - prefix.len()) + ); + assert_eq!(target_uri.len(), MAX_WARC_TARGET_URI_BYTES); + + let exact_provenance = ProvenanceRecord::new( + &target_uri, + "body", + SOURCE_HASH, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("exact target URI bound remains valid provenance"); + assert!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + &target_uri, + "text/plain", + Vec::new(), + exact_provenance, + ) + .is_ok() + ); + + let oversized_target_uri = format!("{target_uri}a"); + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + &oversized_target_uri, + "text/plain", + Vec::new(), + provenance(), + ), + Err(WarcResourceRecordError::LimitExceeded), + ); +} diff --git a/crates/originweave-evidence/tests/warc_payload_provenance_binding.rs b/crates/originweave-evidence/tests/warc_payload_provenance_binding.rs new file mode 100644 index 000000000..4c4f500df --- /dev/null +++ b/crates/originweave-evidence/tests/warc_payload_provenance_binding.rs @@ -0,0 +1,65 @@ +#![allow(clippy::expect_used)] + +use originweave_evidence::{ + EvidenceSourceKind, ProvenanceRecord, VerificationResult, WarcPayloadCompleteness, + WarcResourceRecord, WarcResourceRecordError, WarcTruncationReason, +}; + +const RECORD_ID: &str = "urn:uuid:123e4567-e89b-12d3-a456-426614174000"; +const DATE: &str = "2026-08-28T00:00:00Z"; +const SOURCE_URL: &str = "https://example.com/item"; +const HELLO_HASH: &str = "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"; +const OTHER_HASH: &str = "sha256:486ea46224d1bb4fb680f34f7c9ad96a8f24ec88be73ea8e5a6c65260e9cb8a7"; + +fn provenance(source_hash: &str) -> ProvenanceRecord { + ProvenanceRecord::new( + SOURCE_URL, + "body", + source_hash, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("valid provenance") +} + +#[test] +fn complete_payload_must_match_verified_source_hash() { + let record = WarcResourceRecord::new( + RECORD_ID, + DATE, + SOURCE_URL, + "text/plain", + b"hello".to_vec(), + provenance(HELLO_HASH), + ) + .expect("matching payload provenance"); + assert_eq!(record.block_digest(), HELLO_HASH); + + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + SOURCE_URL, + "text/plain", + b"hello".to_vec(), + provenance(OTHER_HASH), + ), + Err(WarcResourceRecordError::PayloadProvenanceMismatch) + ); +} + +#[test] +fn truncated_payload_still_binds_the_exact_retained_bytes() { + assert_eq!( + WarcResourceRecord::new_with_completeness( + RECORD_ID, + DATE, + SOURCE_URL, + "text/plain", + b"hello".to_vec(), + provenance(OTHER_HASH), + WarcPayloadCompleteness::Truncated(WarcTruncationReason::Length), + ), + Err(WarcResourceRecordError::PayloadProvenanceMismatch) + ); +} diff --git a/crates/originweave-evidence/tests/warc_prov_jsonld.rs b/crates/originweave-evidence/tests/warc_prov_jsonld.rs new file mode 100644 index 000000000..bec8b803c --- /dev/null +++ b/crates/originweave-evidence/tests/warc_prov_jsonld.rs @@ -0,0 +1,218 @@ +#![allow(clippy::expect_used)] + +use originweave_evidence::{ + EvidenceSourceKind, MAX_PROV_SOFTWARE_COMMIT_SHA_BYTES, ProvenanceRecord, VerificationResult, + WarcPayloadCompleteness, WarcProvBundle, WarcProvBundleError, WarcResourceRecord, + WarcTruncationReason, +}; + +const SOURCE_HASH: &str = "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"; +const RECORD_ID: &str = "urn:uuid:123e4567-e89b-12d3-a456-426614174000"; +const DATE: &str = "2026-08-22T12:00:00Z"; +const SOFTWARE_COMMIT_SHA: &str = "0123456789abcdef0123456789abcdef01234567"; +const WARC_RECORD_DIGEST_IRI: &str = + "tag:contextualwisdomlab.github.io,2026:OriginWeave/warcRecordDigest"; +const PAYLOAD_COMPLETENESS_IRI: &str = + "tag:contextualwisdomlab.github.io,2026:OriginWeave/warcPayloadCompleteness"; +const TRUNCATION_REASON_IRI: &str = + "tag:contextualwisdomlab.github.io,2026:OriginWeave/warcTruncationReason"; + +fn resource_record() -> WarcResourceRecord { + let provenance = ProvenanceRecord::new( + "https://example.com/item", + "body", + SOURCE_HASH, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("verified provenance"); + WarcResourceRecord::new( + RECORD_ID, + DATE, + "https://example.com/item", + "text/plain", + b"hello".to_vec(), + provenance, + ) + .expect("WARC resource record") +} + +fn resource_record_with_completeness(completeness: WarcPayloadCompleteness) -> WarcResourceRecord { + let provenance = ProvenanceRecord::new( + "https://example.com/item", + "body", + SOURCE_HASH, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("verified provenance"); + WarcResourceRecord::new_with_completeness( + RECORD_ID, + DATE, + "https://example.com/item", + "text/plain", + b"hello".to_vec(), + provenance, + completeness, + ) + .expect("WARC resource record") +} + +fn assert_standard_error_contract() {} + +#[test] +fn warc_prov_bundle_exposes_stable_capture_identities_and_standard_errors() { + assert_standard_error_contract::(); + assert_eq!( + WarcProvBundleError::InvalidSoftwareCommitSha.to_string(), + "invalid OriginWeave software commit SHA" + ); + assert_eq!( + WarcProvBundleError::LimitExceeded.to_string(), + "WARC PROV bundle limit exceeded" + ); + + let bundle = WarcProvBundle::new(&resource_record(), SOFTWARE_COMMIT_SHA) + .expect("PROV bundle over a validated WARC record"); + assert_eq!(bundle.record_entity_id(), RECORD_ID); + assert_eq!( + bundle.source_entity_id(), + "urn:uuid:123e4567-e89b-12d3-a456-426614174000#source" + ); + assert_eq!( + bundle.capture_activity_id(), + "urn:uuid:123e4567-e89b-12d3-a456-426614174000#capture" + ); + assert_eq!( + bundle.software_agent_id(), + "https://github.com/ContextualWisdomLab/OriginWeave/commit/0123456789abcdef0123456789abcdef01234567" + ); + assert_eq!(bundle.software_commit_sha(), SOFTWARE_COMMIT_SHA); + + let debug = format!("{bundle:?}"); + assert!(debug.contains(RECORD_ID)); + assert!(!debug.contains("https://example.com/item")); + assert!(!debug.contains(SOURCE_HASH)); + assert!(!debug.contains("hello")); +} + +#[test] +fn warc_prov_bundle_emits_deterministic_prov_o_json_ld_without_raw_payload() { + let bundle = WarcProvBundle::new(&resource_record(), SOFTWARE_COMMIT_SHA) + .expect("PROV bundle over a validated WARC record"); + let json_ld = bundle.to_json_ld(); + + assert_eq!( + json_ld, + concat!( + "{\"@context\":{\"prov\":\"http://www.w3.org/ns/prov#\",\"xsd\":\"http://www.w3.org/2001/XMLSchema#\"},\"@graph\":[", + "{\"@id\":\"urn:uuid:123e4567-e89b-12d3-a456-426614174000#source\",\"@type\":\"prov:Entity\",\"prov:atLocation\":{\"@id\":\"https://example.com/item\"},\"prov:value\":\"sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824\"},", + "{\"@id\":\"urn:uuid:123e4567-e89b-12d3-a456-426614174000#capture\",\"@type\":\"prov:Activity\",\"prov:startedAtTime\":{\"@value\":\"2026-08-22T12:00:00Z\",\"@type\":\"xsd:dateTime\"},\"prov:used\":{\"@id\":\"urn:uuid:123e4567-e89b-12d3-a456-426614174000#source\"},\"prov:wasAssociatedWith\":{\"@id\":\"https://github.com/ContextualWisdomLab/OriginWeave/commit/0123456789abcdef0123456789abcdef01234567\"}},", + "{\"@id\":\"https://github.com/ContextualWisdomLab/OriginWeave/commit/0123456789abcdef0123456789abcdef01234567\",\"@type\":\"prov:SoftwareAgent\"},", + "{\"@id\":\"urn:uuid:123e4567-e89b-12d3-a456-426614174000\",\"@type\":\"prov:Entity\",\"prov:value\":\"sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824\",\"tag:contextualwisdomlab.github.io,2026:OriginWeave/warcRecordDigest\":\"sha256:b6ea360a1ec548527ff5ed9c03966b05c8afd5c2b882bee259e362effb0fe0a8\",\"tag:contextualwisdomlab.github.io,2026:OriginWeave/warcPayloadCompleteness\":\"complete\",\"prov:wasDerivedFrom\":{\"@id\":\"urn:uuid:123e4567-e89b-12d3-a456-426614174000#source\"},\"prov:wasGeneratedBy\":{\"@id\":\"urn:uuid:123e4567-e89b-12d3-a456-426614174000#capture\"}}", + "]}" + ) + ); + assert!(!json_ld.contains("hello")); +} + +#[test] +fn warc_prov_bundle_rejects_noncanonical_or_oversized_software_revisions() { + let record = resource_record(); + for software_commit_sha in [ + "", + "0000000000000000000000000000000000000000", + "0123456789abcdef0123456789abcdef0123456", + "0123456789abcdef0123456789abcdef0123456G", + "0123456789ABCDEF0123456789ABCDEF01234567", + "0123456789abcdef0123456789abcdef0123456 ", + ] { + assert_eq!( + WarcProvBundle::new(&record, software_commit_sha), + Err(WarcProvBundleError::InvalidSoftwareCommitSha), + "software_commit_sha={software_commit_sha:?}" + ); + } + + assert_eq!( + MAX_PROV_SOFTWARE_COMMIT_SHA_BYTES, + SOFTWARE_COMMIT_SHA.len() + ); + assert_eq!( + WarcProvBundle::new(&record, &"a".repeat(MAX_PROV_SOFTWARE_COMMIT_SHA_BYTES + 1)), + Err(WarcProvBundleError::LimitExceeded) + ); +} + +#[test] +fn warc_prov_bundle_preserves_warc_payload_completeness_for_replay() { + let complete = WarcProvBundle::new( + &resource_record_with_completeness(WarcPayloadCompleteness::Complete), + SOFTWARE_COMMIT_SHA, + ) + .expect("complete PROV bundle"); + let complete_json = complete.to_json_ld(); + assert!(complete_json.contains(&format!("\"{PAYLOAD_COMPLETENESS_IRI}\":\"complete\""))); + assert!(!complete_json.contains(TRUNCATION_REASON_IRI)); + + for (reason, token) in [ + (WarcTruncationReason::Length, "length"), + (WarcTruncationReason::Time, "time"), + (WarcTruncationReason::Disconnect, "disconnect"), + (WarcTruncationReason::Unspecified, "unspecified"), + ] { + let truncated = WarcProvBundle::new( + &resource_record_with_completeness(WarcPayloadCompleteness::Truncated(reason)), + SOFTWARE_COMMIT_SHA, + ) + .expect("truncated PROV bundle"); + let truncated_json = truncated.to_json_ld(); + assert!(truncated_json.contains(&format!("\"{PAYLOAD_COMPLETENESS_IRI}\":\"truncated\""))); + assert!(truncated_json.contains(&format!("\"{TRUNCATION_REASON_IRI}\":\"{token}\""))); + } +} + +#[test] +fn warc_prov_bundle_distinguishes_distinct_warc_serializations() { + let record_with_content_type = |content_type: &str| { + let provenance = ProvenanceRecord::new( + "https://example.com/item", + "body", + SOURCE_HASH, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("verified provenance"); + WarcResourceRecord::new( + RECORD_ID, + DATE, + "https://example.com/item", + content_type, + b"hello".to_vec(), + provenance, + ) + .expect("WARC resource record") + }; + + let text_record = record_with_content_type("text/plain"); + let binary_record = record_with_content_type("application/octet-stream"); + assert_ne!(text_record.to_warc_bytes(), binary_record.to_warc_bytes()); + + let text_prov = WarcProvBundle::new(&text_record, SOFTWARE_COMMIT_SHA) + .expect("text PROV bundle") + .to_json_ld(); + let binary_prov = WarcProvBundle::new(&binary_record, SOFTWARE_COMMIT_SHA) + .expect("binary PROV bundle") + .to_json_ld(); + + assert_ne!( + text_prov, binary_prov, + "provenance must distinguish WARC records whose serialized headers differ" + ); + assert!(text_prov.contains(&format!( + "\"{WARC_RECORD_DIGEST_IRI}\":\"sha256:b6ea360a1ec548527ff5ed9c03966b05c8afd5c2b882bee259e362effb0fe0a8\"" + ))); + assert!(binary_prov.contains(&format!( + "\"{WARC_RECORD_DIGEST_IRI}\":\"sha256:9c59979535d4a1b3589c0fe2d17837c4ddb0e4cf911854d9aae362903ff83db9\"" + ))); +} diff --git a/crates/originweave-evidence/tests/warc_prov_offline_verification.rs b/crates/originweave-evidence/tests/warc_prov_offline_verification.rs new file mode 100644 index 000000000..7222cd956 --- /dev/null +++ b/crates/originweave-evidence/tests/warc_prov_offline_verification.rs @@ -0,0 +1,203 @@ +#![allow(clippy::expect_used)] + +use originweave_evidence::{ + EvidenceSourceKind, ProvenanceRecord, VerificationResult, WarcPayloadCompleteness, + WarcProvBundle, WarcProvBundleVerificationError, WarcResourceRecord, WarcTruncationReason, +}; + +const SOURCE_HASH: &str = "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"; +const OTHER_SOURCE_HASH: &str = + "sha256:486ea46224d1bb4fb680f34f7c9ad96a8f24ec88be73ea8e5a6c65260e9cb8a7"; +const RECORD_ID: &str = "urn:uuid:123e4567-e89b-12d3-a456-426614174000"; +const OTHER_RECORD_ID: &str = "urn:uuid:123e4567-e89b-12d3-a456-426614174001"; +const DATE: &str = "2026-08-22T12:00:00Z"; +const OTHER_DATE: &str = "2026-08-22T12:00:01Z"; +const SOURCE_URL: &str = "https://example.com/item"; +const OTHER_SOURCE_URL: &str = "https://example.com/other"; +const SOFTWARE_COMMIT_SHA: &str = "0123456789abcdef0123456789abcdef01234567"; + +fn record( + record_id: &str, + date: &str, + source_url: &str, + source_hash: &str, + content_type: &str, + payload: &[u8], + completeness: WarcPayloadCompleteness, +) -> WarcResourceRecord { + let provenance = ProvenanceRecord::new( + source_url, + "body", + source_hash, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("verified provenance"); + WarcResourceRecord::new_with_completeness( + record_id, + date, + source_url, + content_type, + payload.to_vec(), + provenance, + completeness, + ) + .expect("WARC resource record") +} + +fn baseline_record_with_source_evidence( + source_locator: &str, + source_kind: EvidenceSourceKind, +) -> WarcResourceRecord { + let provenance = ProvenanceRecord::new( + SOURCE_URL, + source_locator, + SOURCE_HASH, + source_kind, + VerificationResult::Verified, + ) + .expect("verified provenance"); + WarcResourceRecord::new_with_completeness( + RECORD_ID, + DATE, + SOURCE_URL, + "text/plain", + b"hello".to_vec(), + provenance, + WarcPayloadCompleteness::Complete, + ) + .expect("WARC resource record") +} + +fn baseline_record() -> WarcResourceRecord { + record( + RECORD_ID, + DATE, + SOURCE_URL, + SOURCE_HASH, + "text/plain", + b"hello", + WarcPayloadCompleteness::Complete, + ) +} + +fn assert_standard_error_contract() {} + +#[test] +fn warc_prov_bundle_offline_verification_accepts_only_the_exact_bound_record() { + assert_standard_error_contract::(); + assert_eq!( + WarcProvBundleVerificationError::RecordIdentityMismatch.to_string(), + "WARC PROV record identity does not match" + ); + assert_eq!( + WarcProvBundleVerificationError::SourceEvidenceMismatch.to_string(), + "WARC PROV source evidence does not match" + ); + assert_eq!( + WarcProvBundleVerificationError::CaptureTimeMismatch.to_string(), + "WARC PROV capture time does not match" + ); + assert_eq!( + WarcProvBundleVerificationError::PayloadCompletenessMismatch.to_string(), + "WARC PROV payload completeness does not match" + ); + assert_eq!( + WarcProvBundleVerificationError::WarcRecordDigestMismatch.to_string(), + "WARC PROV serialized record digest does not match" + ); + + let exact = baseline_record(); + let bundle = WarcProvBundle::new(&exact, SOFTWARE_COMMIT_SHA).expect("PROV bundle"); + assert_eq!(bundle.verify_record(&exact), Ok(())); + + let mismatches = [ + ( + record( + OTHER_RECORD_ID, + DATE, + SOURCE_URL, + SOURCE_HASH, + "text/plain", + b"hello", + WarcPayloadCompleteness::Complete, + ), + WarcProvBundleVerificationError::RecordIdentityMismatch, + ), + ( + record( + RECORD_ID, + DATE, + OTHER_SOURCE_URL, + SOURCE_HASH, + "text/plain", + b"hello", + WarcPayloadCompleteness::Complete, + ), + WarcProvBundleVerificationError::SourceEvidenceMismatch, + ), + ( + record( + RECORD_ID, + DATE, + SOURCE_URL, + OTHER_SOURCE_HASH, + "text/plain", + b"world", + WarcPayloadCompleteness::Complete, + ), + WarcProvBundleVerificationError::SourceEvidenceMismatch, + ), + ( + baseline_record_with_source_evidence( + "different-body", + EvidenceSourceKind::NetworkResponse, + ), + WarcProvBundleVerificationError::SourceEvidenceMismatch, + ), + ( + baseline_record_with_source_evidence("body", EvidenceSourceKind::StructuredData), + WarcProvBundleVerificationError::SourceEvidenceMismatch, + ), + ( + record( + RECORD_ID, + OTHER_DATE, + SOURCE_URL, + SOURCE_HASH, + "text/plain", + b"hello", + WarcPayloadCompleteness::Complete, + ), + WarcProvBundleVerificationError::CaptureTimeMismatch, + ), + ( + record( + RECORD_ID, + DATE, + SOURCE_URL, + SOURCE_HASH, + "text/plain", + b"hello", + WarcPayloadCompleteness::Truncated(WarcTruncationReason::Length), + ), + WarcProvBundleVerificationError::PayloadCompletenessMismatch, + ), + ( + record( + RECORD_ID, + DATE, + SOURCE_URL, + SOURCE_HASH, + "application/octet-stream", + b"hello", + WarcPayloadCompleteness::Complete, + ), + WarcProvBundleVerificationError::WarcRecordDigestMismatch, + ), + ]; + + for (candidate, expected) in mismatches { + assert_eq!(bundle.verify_record(&candidate), Err(expected)); + } +} diff --git a/crates/originweave-evidence/tests/warc_resource_record.rs b/crates/originweave-evidence/tests/warc_resource_record.rs new file mode 100644 index 000000000..0d27ae72f --- /dev/null +++ b/crates/originweave-evidence/tests/warc_resource_record.rs @@ -0,0 +1,287 @@ +#![allow(clippy::expect_used)] + +use originweave_evidence::{ + EvidenceSourceKind, MAX_WARC_PAYLOAD_BYTES, ProvenanceRecord, VerificationResult, + WarcResourceRecord, WarcResourceRecordError, +}; + +const EMPTY_HASH: &str = "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"; +const HELLO_HASH: &str = "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"; +const RECORD_ID: &str = "urn:uuid:123e4567-e89b-12d3-a456-426614174000"; +const DATE: &str = "2026-08-21T00:00:00Z"; + +fn provenance_with_hash( + source_url: &str, + source_hash: &str, + verification: VerificationResult, +) -> ProvenanceRecord { + ProvenanceRecord::new( + source_url, + "body", + source_hash, + EvidenceSourceKind::NetworkResponse, + verification, + ) + .expect("provenance") +} + +fn provenance(source_url: &str, verification: VerificationResult) -> ProvenanceRecord { + provenance_with_hash(source_url, EMPTY_HASH, verification) +} + +fn assert_standard_error_contract() {} + +#[test] +fn warc_resource_record_error_implements_standard_error_contract() { + assert_standard_error_contract::(); + + for (error, message) in [ + ( + WarcResourceRecordError::InvalidRecordId, + "invalid WARC record identifier", + ), + (WarcResourceRecordError::InvalidDate, "invalid WARC date"), + ( + WarcResourceRecordError::InvalidContentType, + "invalid WARC content type", + ), + ( + WarcResourceRecordError::LimitExceeded, + "WARC resource record limit exceeded", + ), + ( + WarcResourceRecordError::InvalidTargetUri, + "invalid WARC target URI", + ), + ( + WarcResourceRecordError::TargetUriMismatch, + "WARC target URI does not match provenance", + ), + ( + WarcResourceRecordError::UnverifiedProvenance, + "WARC provenance is not independently verified", + ), + ( + WarcResourceRecordError::PayloadProvenanceMismatch, + "WARC payload digest does not match provenance", + ), + ] { + assert_eq!(error.to_string(), message); + } +} + +#[test] +fn resource_record_binds_verified_provenance_and_emits_deterministic_warc_bytes() { + let record = WarcResourceRecord::new( + RECORD_ID, + DATE, + "https://example.com/item", + "text/plain", + b"hello".to_vec(), + provenance_with_hash( + "https://example.com/item", + HELLO_HASH, + VerificationResult::Verified, + ), + ) + .expect("resource record"); + + assert_eq!(record.record_id(), RECORD_ID); + assert_eq!(record.warc_date(), DATE); + assert_eq!(record.target_uri(), "https://example.com/item"); + assert_eq!(record.content_type(), "text/plain"); + assert_eq!(record.payload(), b"hello"); + assert_eq!(record.block_digest(), HELLO_HASH); + assert_eq!(record.provenance().source_url(), record.target_uri()); + assert!(record.provenance().verification_result() == VerificationResult::Verified); + assert_eq!( + record.to_warc_bytes(), + b"WARC/1.1\r\nWARC-Type: resource\r\nWARC-Record-ID: \r\nWARC-Date: 2026-08-21T00:00:00Z\r\nWARC-Target-URI: https://example.com/item\r\nContent-Type: text/plain\r\nWARC-Block-Digest: sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824\r\nContent-Length: 5\r\n\r\nhello\r\n\r\n" + ); + for date in [ + "2026-08-21T00:00:00.123Z", + "2024-02-29T00:00:00Z", + "2000-02-29T00:00:00Z", + ] { + WarcResourceRecord::new( + RECORD_ID, + date, + "https://example.com/item", + "text/plain", + Vec::new(), + provenance("https://example.com/item", VerificationResult::Verified), + ) + .expect("valid UTC date"); + } +} + +#[test] +fn resource_record_rejects_invalid_identifiers_dates_content_and_limits() { + let valid = |record_id, date, content_type, payload| { + WarcResourceRecord::new( + record_id, + date, + "https://example.com/item", + content_type, + payload, + provenance("https://example.com/item", VerificationResult::Verified), + ) + }; + + for record_id in [ + "", + "http://example.com/record", + "urn:uuid:00000000-0000-0000-0000-000000000000", + "urn:uuid:123e4567-e89b-12d3-a456-42661417400", + "urn:uuid:123e4567_e89b-12d3-a456-426614174000", + "urn:uuid:123e4567-e89b-12d3-a456-42661417400z", + "xrn:uuid:123e4567-e89b-12d3-a456-426614174000", + ] { + assert_eq!( + valid(record_id, DATE, "text/plain", Vec::new()), + Err(WarcResourceRecordError::InvalidRecordId), + "record_id={record_id:?}" + ); + } + + for date in [ + "", + "2026-08-21 00:00:00Z", + "2026-13-21T00:00:00Z", + "2026-08-32T00:00:00Z", + "2026-02-29T00:00:00Z", + "2024-02-30T00:00:00Z", + "2026-04-31T00:00:00Z", + "1900-02-29T00:00:00Z", + "2026-08-21T24:00:00Z", + "2026-0x-21T00:00:00Z", + "2026-08-21T00:00:00+00:00", + "2026-08-21T00:00:00.123", + "2026-08-21T00:00:00.XZ", + "2026-08-21T00:00:00.Z", + "2026x08-21T00:00:00Z", + "2026-08x21T00:00:00Z", + "2026-08-21T00x00:00Z", + "2026-08-21T00:00x00Z", + "2026-08-21T00:00:00X", + "2026-08-21T00:61:00Z", + "2026-08-21T00:00:61Z", + "2026-08-21T12:34:60Z", + "2026-06-30T23:58:60Z", + ] { + assert_eq!( + valid(RECORD_ID, date, "text/plain", Vec::new()), + Err(WarcResourceRecordError::InvalidDate), + "date={date:?}" + ); + } + + for content_type in ["", "text plain", "text\nplain"] { + assert_eq!( + valid(RECORD_ID, DATE, content_type, Vec::new()), + Err(WarcResourceRecordError::InvalidContentType), + "content_type={content_type:?}" + ); + } + + assert_eq!( + valid( + RECORD_ID, + DATE, + "text/plain", + vec![b'x'; MAX_WARC_PAYLOAD_BYTES + 1], + ), + Err(WarcResourceRecordError::LimitExceeded) + ); + assert_eq!( + valid( + RECORD_ID, + DATE, + &"x".repeat(originweave_evidence::MAX_WARC_CONTENT_TYPE_BYTES + 1), + Vec::new(), + ), + Err(WarcResourceRecordError::LimitExceeded) + ); +} + +#[test] +fn resource_record_accepts_valid_mime_parameters_and_rejects_malformed_media_types() { + let build = |content_type| { + WarcResourceRecord::new( + RECORD_ID, + DATE, + "https://example.com/item", + content_type, + Vec::new(), + provenance("https://example.com/item", VerificationResult::Verified), + ) + }; + + for content_type in [ + "text/plain; charset=utf-8", + "application/http; msgtype=response", + "multipart/form-data; boundary=example-boundary", + "text/plain; note=\"a;b\"", + "text/plain; note=\"a;b\"; charset=utf-8", + "text/plain; note=\"a\\;b\"", + "text/plain; note=\"\t !\"", + "text/plain; note=\"\\\t\"", + "text/plain; note=\"\\ \"", + "text/plain; note=\"\\!\"", + ] { + let record = build(content_type).expect("valid WARC MIME media type"); + assert_eq!(record.content_type(), content_type); + } + + for content_type in [ + "plain", + "/plain", + "text/", + "text//plain", + "text/plain;", + "text/plain; charset", + "text/plain; =utf-8", + "text/plain; charset=", + "text/(plain)", + "text/plain; note=(x", + "text/plain; note=\"unterminated", + "text/plain; note=\"a\"b\"", + "text/plain; note=\"a\nb\"", + "text/plain; note=\"\\\nb\"", + "text/plain; note=\"\\\u{7f}\"", + ] { + assert_eq!( + build(content_type), + Err(WarcResourceRecordError::InvalidContentType), + "content_type={content_type:?}" + ); + } +} + +#[test] +fn resource_record_rejects_provenance_drift_and_unverified_sources() { + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + "https://example.com/other", + "text/plain", + Vec::new(), + provenance("https://example.com/item", VerificationResult::Verified), + ), + Err(WarcResourceRecordError::TargetUriMismatch) + ); + for verification in [VerificationResult::Unverified, VerificationResult::Rejected] { + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + "https://example.com/item", + "text/plain", + Vec::new(), + provenance("https://example.com/item", verification), + ), + Err(WarcResourceRecordError::UnverifiedProvenance) + ); + } +} diff --git a/crates/originweave-evidence/tests/warc_target_uri_presentation.rs b/crates/originweave-evidence/tests/warc_target_uri_presentation.rs new file mode 100644 index 000000000..601391032 --- /dev/null +++ b/crates/originweave-evidence/tests/warc_target_uri_presentation.rs @@ -0,0 +1,198 @@ +#![allow(clippy::expect_used)] + +use originweave_evidence::{ + EvidenceSourceKind, ProvenanceRecord, VerificationResult, WarcResourceRecord, + WarcResourceRecordError, +}; + +const SOURCE_HASH: &str = "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"; +const RECORD_ID: &str = "urn:uuid:123e4567-e89b-12d3-a456-426614174000"; +const DATE: &str = "2026-08-21T00:00:00Z"; + +fn provenance(source_url: &str) -> ProvenanceRecord { + ProvenanceRecord::new( + source_url, + "body", + SOURCE_HASH, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("provenance") +} + +#[test] +fn warc_target_uri_rejects_invisible_formatting_characters_before_serialization() { + let source_provenance = provenance("https://example.com/valid"); + for formatting_character in [ + '\u{00ad}', '\u{061c}', '\u{200b}', '\u{200e}', '\u{202e}', '\u{2066}', '\u{2060}', + '\u{feff}', + ] { + let target_uri = format!("https://example.com/item{formatting_character}shadow"); + + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + &target_uri, + "text/plain", + Vec::new(), + source_provenance.clone(), + ), + Err(WarcResourceRecordError::InvalidTargetUri), + "target_uri={target_uri:?}" + ); + } +} + +#[test] +fn warc_target_uri_rejects_control_and_whitespace_before_provenance_comparison() { + let source_provenance = provenance("https://example.com/item"); + for target_uri in [ + "https://example.com/item\rshadow", + "https://example.com/item shadow", + ] { + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + target_uri, + "text/plain", + Vec::new(), + source_provenance.clone(), + ), + Err(WarcResourceRecordError::InvalidTargetUri), + "target_uri={target_uri:?}" + ); + } +} + +#[test] +fn warc_target_uri_rejects_raw_unicode_because_warc_uses_rfc3986_uri_syntax() { + let target_uri = "https://example.com/상품/상세"; + + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + target_uri, + "text/plain", + Vec::new(), + provenance("https://example.com/valid"), + ), + Err(WarcResourceRecordError::InvalidTargetUri), + ); +} + +#[test] +fn warc_target_uri_rejects_ascii_characters_outside_rfc3986_uri_syntax() { + let source_provenance = provenance("https://example.com/valid"); + for invalid_character in ['<', '>', '"', '{', '}', '|', '^', '`'] { + let target_uri = format!("https://example.com/item{invalid_character}shadow"); + + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + &target_uri, + "text/plain", + Vec::new(), + source_provenance.clone(), + ), + Err(WarcResourceRecordError::InvalidTargetUri), + "target_uri={target_uri:?}" + ); + } +} + +#[test] +fn warc_target_uri_rejects_general_delimiters_in_path_segments() { + let source_provenance = provenance("https://example.com/valid"); + for target_uri in [ + "https://example.com/[segment]", + "https://example.com/item]shadow", + ] { + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + target_uri, + "text/plain", + Vec::new(), + source_provenance.clone(), + ), + Err(WarcResourceRecordError::InvalidTargetUri), + "target_uri={target_uri:?}" + ); + } +} + +#[test] +fn warc_target_uri_rejects_non_ip_literal_brackets_in_authority() { + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + "https://[not-an-ip]/", + "text/plain", + Vec::new(), + provenance("https://example.com/valid"), + ), + Err(WarcResourceRecordError::InvalidTargetUri), + ); +} + +#[test] +fn warc_target_uri_preserves_brackets_in_ipv6_authority() { + let target_uri = "https://[::1]:8443/path"; + let record = WarcResourceRecord::new( + RECORD_ID, + DATE, + target_uri, + "text/plain", + Vec::new(), + provenance(target_uri), + ) + .expect("RFC 3986 bracketed IPv6 authority"); + + assert_eq!(record.target_uri(), target_uri); +} + +#[test] +fn warc_target_uri_rejects_malformed_percent_encoding() { + let source_provenance = provenance("https://example.com/valid"); + for target_uri in [ + "https://example.com/%", + "https://example.com/%2", + "https://example.com/%GG", + "https://example.com/%0G", + ] { + assert_eq!( + WarcResourceRecord::new( + RECORD_ID, + DATE, + target_uri, + "text/plain", + Vec::new(), + source_provenance.clone(), + ), + Err(WarcResourceRecordError::InvalidTargetUri), + "target_uri={target_uri:?}" + ); + } +} + +#[test] +fn warc_target_uri_accepts_percent_encoded_utf8_path_octets() { + let target_uri = "https://example.com/%EC%83%81%ED%92%88/%EC%83%81%EC%84%B8"; + let record = WarcResourceRecord::new( + RECORD_ID, + DATE, + target_uri, + "text/plain", + Vec::new(), + provenance(target_uri), + ) + .expect("RFC 3986 percent-encoded target URI"); + + assert_eq!(record.target_uri(), target_uri); +} diff --git a/crates/originweave-evidence/tests/warc_truncation_state.rs b/crates/originweave-evidence/tests/warc_truncation_state.rs new file mode 100644 index 000000000..847b339fd --- /dev/null +++ b/crates/originweave-evidence/tests/warc_truncation_state.rs @@ -0,0 +1,65 @@ +#![allow(clippy::expect_used)] + +use originweave_evidence::{ + EvidenceSourceKind, ProvenanceRecord, VerificationResult, WarcPayloadCompleteness, + WarcResourceRecord, WarcTruncationReason, +}; + +const RECORD_ID: &str = "urn:uuid:01234567-89ab-cdef-0123-456789abcdef"; +const DATE: &str = "2026-08-22T00:00:00Z"; +const SOURCE_URL: &str = "https://example.com/resource"; +const SOURCE_HASH: &str = "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"; + +fn verified_provenance() -> ProvenanceRecord { + ProvenanceRecord::new( + SOURCE_URL, + "body", + SOURCE_HASH, + EvidenceSourceKind::NetworkResponse, + VerificationResult::Verified, + ) + .expect("reviewed provenance fixture must be valid") +} + +#[test] +fn warc_resource_records_preserve_explicit_completeness_and_truncation_reason() { + let complete = WarcResourceRecord::new_with_completeness( + RECORD_ID, + DATE, + SOURCE_URL, + "text/plain", + b"hello".to_vec(), + verified_provenance(), + WarcPayloadCompleteness::Complete, + ) + .expect("complete capture must be admitted"); + assert_eq!(complete.completeness(), WarcPayloadCompleteness::Complete); + let complete_bytes = String::from_utf8(complete.to_warc_bytes()) + .expect("text fixture must serialize as UTF-8 WARC bytes"); + assert!(!complete_bytes.contains("WARC-Truncated:")); + + for (reason, token) in [ + (WarcTruncationReason::Length, "length"), + (WarcTruncationReason::Time, "time"), + (WarcTruncationReason::Disconnect, "disconnect"), + (WarcTruncationReason::Unspecified, "unspecified"), + ] { + let completeness = WarcPayloadCompleteness::Truncated(reason); + let truncated = WarcResourceRecord::new_with_completeness( + RECORD_ID, + DATE, + SOURCE_URL, + "text/plain", + b"hello".to_vec(), + verified_provenance(), + completeness, + ) + .expect("typed truncated capture must be admitted"); + + assert_eq!(truncated.completeness(), completeness); + let truncated_bytes = String::from_utf8(truncated.to_warc_bytes()) + .expect("text fixture must serialize as UTF-8 WARC bytes"); + assert!(truncated_bytes.contains(&format!("WARC-Truncated: {token}\r\n"))); + assert!(truncated_bytes.contains("Content-Length: 5\r\n")); + } +} diff --git a/docs/adr/0106-provenance-evidence-model.md b/docs/adr/0106-provenance-evidence-model.md index 09cb0d7ca..5bc6147a9 100644 --- a/docs/adr/0106-provenance-evidence-model.md +++ b/docs/adr/0106-provenance-evidence-model.md @@ -9,6 +9,17 @@ OriginWeave must not merely act; it must let a user, operator, auditor, or downstream system establish what was observed, authorized, executed, and verified. Browser logs alone are not sufficient because they collapse observation, policy, action, network identity, approvals, and post-conditions. At the same time, evidence can itself contain sensitive or attacker-controlled content. The product needs a durable model compatible with web-archive and provenance concepts without claiming that every conceptual record is already persisted. +## Active implementation boundary + +The extraction lane currently implements one bounded, verified in-memory WARC 1.1 +`resource` record contract over already-authorized bytes. It binds the WARC target +URI to independently verified provenance, computes a SHA-256 block digest, and +emits deterministic record bytes. The active-PR `WarcProvBundle` projection adds +deterministic W3C PROV-O JSON-LD and offline verification of the exact bound WARC +record without retaining the payload in the bundle. This is active-PR evidence; +durable persistence remains planned, as do tenant retention, request/response +capture, and transport-specific export adapters. + ## Decision drivers - `Browse. Act. Prove.` requires evidence as a first-class product output. diff --git a/docs/doctoring.md b/docs/doctoring.md index ec51daaf3..2790c0f20 100644 --- a/docs/doctoring.md +++ b/docs/doctoring.md @@ -90,9 +90,13 @@ RFC 9309 standardizes robots parsing, matching, error handling, and caching. It W3C PROV-O supplies interoperable Entity, Activity, Agent, derivation, attribution, and responsibility concepts. ISO 28500:2017, confirmed in 2023, defines WARC storage for protocol payloads, control information, metadata, transformations, duplicate detection, integrity, and segmentation. OriginWeave uses source hashes and locators in the safety kernel, then adds WARC and PROV adapters as separately testable modules. +The WARC/PROV child slice from PR #217, now integrated into the unprotected #210 feature branch, adds a bounded in-memory `WarcProvBundle` for deterministic W3C PROV-O JSON-LD projection and offline verification of exact WARC records. The bundle emits credential-free identifiers, source location, capture time, completeness, and digests without retaining the WARC payload. Offline verification uses the complete deterministic WARC serialization digest as the authoritative record binding, so block-digest drift cannot bypass the exact-record check. This remains active-PR evidence only until #210 reaches protected `main`; durable persistence, retention, and transport-specific export remain planned and are not implied by the in-memory projection. + +The IIPC WARC 1.1 specification and its annotated guidance define the `resource` record fields used by the active capture slice: a UTC `WARC-Date`, an RFC 3986 target URI, an algorithm-prefixed block digest, `Content-Length`, and an explicit `WARC-Truncated` reason when a retained block is incomplete. Active PR #210 applies those provisions only to already-authorized bytes: record identifiers, dates, content types, target URIs, and payloads are bounded before serialization; target-URI validation reuses the provenance source-URL authority so bracketed hosts are accepted only when the shared origin parser accepts their IP-literal form. RFC 3986 permits `?` and `/` within the query component, so OriginWeave keeps safe query-bearing URLs. RFC 9700 specifically requires clients not to pass access tokens in URI query parameters. Independently, OriginWeave applies a broader evidence-retention policy: decoded credential-like field names are rejected at the top level and inside nested query-like values, and residual nested percent-encoding in a field name fails closed rather than hiding another encoded credential name. The serializer is deterministic and in-memory; persistence, retention, encryption, and third-party conformance remain unreleased adapters. + The versioned `ExtractionSchema` is an admission and interpretation contract for typed extracted fields: each field is bounded, declares a value type, cardinality, normalization rule, and a canonical duplicate-free set of reviewed source-channel classes. That declaration does not create browser, network, model, secret, storage, retention, disclosure, or governance authority. PROV/WARC interoperability is therefore layered after the schema contract rather than inferred from it. -RFC 3986 remains Internet Standard STD 66 for generic URI syntax. RFC 8820 is the current URI design-and-ownership Best Current Practice; it obsoletes RFC 7320 and updates RFC 3986 without replacing RFC 3986's path grammar. Section 3.3 of RFC 3986 defines each path segment as `*pchar`, where literal path characters are unreserved characters, sub-delimiters, `:`, or `@`; `/` separates segments and other reserved characters such as `[` and `]` are not literal `pchar`. OriginWeave's shared evidence-path validator therefore applies that literal ASCII `pchar` set plus validated percent-encoded octets and explicit slash separators to both `NetworkEvidence::capture` paths and provenance source-URL paths. Existing stricter evidence-safety rules continue to reject encoded separators, dot-segment ambiguity, controls, whitespace, query strings, fragments, backslashes, and credential-bearing authority. This fail-closed syntax tightening affects both evidence surfaces; it does not authorize the source origin, destination, network access, capture, disclosure, or retention. +RFC 3986 remains Internet Standard STD 66 for generic URI syntax. RFC 8820 is the current URI design-and-ownership Best Current Practice; it obsoletes RFC 7320 and updates RFC 3986 without replacing RFC 3986's path grammar. Section 3.3 of RFC 3986 defines each path segment as `*pchar`, where literal path characters are unreserved characters, sub-delimiters, `:`, or `@`; `/` separates segments and other reserved characters such as `[` and `]` are not literal `pchar`. OriginWeave's shared evidence-path validator therefore applies that literal ASCII `pchar` set plus validated percent-encoded octets and explicit slash separators to both `NetworkEvidence::capture` paths and provenance source-URL paths. Existing stricter evidence-safety rules continue to reject encoded separators, dot-segment ambiguity, singly or recursively encoded controls, whitespace, query strings, fragments, backslashes, and credential-bearing authority. This fail-closed syntax tightening affects both evidence surfaces; it does not authorize the source origin, destination, network access, capture, disclosure, or retention. ### AI risk and prompt injection @@ -148,6 +152,10 @@ Internet Assigned Numbers Authority. (2025, October 9). *IPv6 special-purpose ad Internet Assigned Numbers Authority. (2025, October 10). *IPv6 global unicast address space*. https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.xhtml +International Internet Preservation Consortium. (n.d.). *WARC 1.1*. https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/ + +International Internet Preservation Consortium. (n.d.). *WARC 1.1 annotated*. https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1-annotated/ + International Organization for Standardization. (2017). *Information and documentation—WARC file format* (ISO Standard No. 28500:2017). https://www.iso.org/standard/68004.html Koster, M., Illyes, G., Zeller, H., & Sassman, L. (2022). *Robots Exclusion Protocol* (RFC 9309). Internet Engineering Task Force. https://doi.org/10.17487/RFC9309 diff --git a/tests/test_doctoring_reference_contract.py b/tests/test_doctoring_reference_contract.py index bdeded44f..144f0c1e0 100644 --- a/tests/test_doctoring_reference_contract.py +++ b/tests/test_doctoring_reference_contract.py @@ -23,6 +23,18 @@ def test_rfc_5280_reference_uses_canonical_author_initials(self) -> None: ) self.assertIn(expected, text) + def test_warc_prov_decision_trace_distinguishes_in_memory_evidence(self) -> None: + """The standards record must bound the public WARC/PROV projection honestly.""" + text = DOCTORING.read_text(encoding="utf-8") + for expected in ( + "WarcProvBundle", + "deterministic W3C PROV-O JSON-LD projection", + "offline verification of exact WARC records", + "durable persistence, retention, and transport-specific export remain planned", + ): + with self.subTest(expected=expected): + self.assertIn(expected, text) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_documentation_active_pr_evidence_contract.py b/tests/test_documentation_active_pr_evidence_contract.py index bc60535a2..c30aa532b 100644 --- a/tests/test_documentation_active_pr_evidence_contract.py +++ b/tests/test_documentation_active_pr_evidence_contract.py @@ -9,6 +9,7 @@ FITNESS = DOCS / "DOCUMENTATION_FITNESS.md" MATURITY = DOCS / "evidence" / "2026-08-10-active-pr-maturity.md" BASELINE = DOCS / "product-technical-gap-baseline.md" +DOCTORING = DOCS / "doctoring.md" CHANGELOG = ROOT / "CHANGELOG.md" @@ -31,6 +32,7 @@ def setUpClass(cls) -> None: cls.fitness = FITNESS.read_text(encoding="utf-8") cls.maturity = MATURITY.read_text(encoding="utf-8") cls.baseline = BASELINE.read_text(encoding="utf-8") + cls.doctoring = DOCTORING.read_text(encoding="utf-8") cls.changelog = CHANGELOG.read_text(encoding="utf-8") def test_latest_live_pr_snapshot_is_recorded_in_the_product_baseline(self) -> None: @@ -220,6 +222,17 @@ def test_erd_stays_conceptual_without_persistence_owner(self) -> None: self.assertIn("add no OriginWeave-owned durable store", self.fitness) self.assertIn("false architecture", self.fitness) + def test_warc_prov_attribution_follows_the_stacked_merge(self) -> None: + """Merged child evidence must not remain labelled as an active child PR.""" + attribution = ( + "The WARC/PROV child slice from PR #217, now integrated into the " + "unprotected #210 feature branch" + ) + self.assertIn(attribution, self.changelog) + self.assertIn(attribution, self.doctoring) + self.assertNotIn("Active PR #217 adds a bounded in-memory `WarcProvBundle`", self.changelog) + self.assertNotIn("Active PR #217 adds a bounded in-memory `WarcProvBundle`", self.doctoring) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_product_documentation_contract.py b/tests/test_product_documentation_contract.py index f192aaa4d..412aaeba4 100644 --- a/tests/test_product_documentation_contract.py +++ b/tests/test_product_documentation_contract.py @@ -77,6 +77,19 @@ def test_product_technical_gap_baseline_records_live_delivery_state(self) -> Non vpn_status, ) + def test_warc_prov_bundle_boundary_is_documented_as_in_memory_evidence(self) -> None: + """The public WARC/PROV projection must not be mistaken for durable persistence.""" + adr = (ROOT / "docs/adr/0106-provenance-evidence-model.md").read_text(encoding="utf-8") + changelog = (ROOT / "CHANGELOG.md").read_text(encoding="utf-8") + for text in (adr, changelog): + with self.subTest(document=text[:40]): + self.assertIn("WarcProvBundle", text) + self.assertIn("deterministic", text) + self.assertIn("JSON-LD", text) + self.assertIn("offline verification", text) + self.assertIn("in-memory", text) + self.assertIn("durable persistence remains planned", adr) + def test_root_architecture_links_the_authoritative_product_graph(self) -> None: """Architecture readers must be able to reach requirements, decisions, diagrams, and data.""" architecture = (ROOT / "ARCHITECTURE.md").read_text(encoding="utf-8")