diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6fe6621b6..dca7af6a7 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -90,6 +90,19 @@ sequenceDiagram - Rust remains the psychometric arithmetic owner. Repair never substitutes Python for scoring math. +## Object-storage governance (2026-08-16) + +Central `.github` publishes a provider-neutral `object_storage` contract. +Naruon and other products keep their own adapters. The executable check is +`scripts/ci/validate_object_storage_contract.py`. HTTPS, exact-host +allowlists, tenant-purpose binding, server-side encryption, SHA-256-or-stronger +integrity, distinct lifecycle states, and non-destructive rollback are +required. Denied private-network trust also rejects special-use internal +suffixes. CSAP and SOC 2 remain design constraints, not certification claims. +Operational PII is not blanket-masked. Product adapters prove write/read/delete +timeout and partial-upload behavior with +`docs/object-storage/PRODUCT_ACCEPTANCE_TEMPLATE.md`. + ## Quality gates `scripts/ci/` ships with 100% statement/branch coverage and 100% docstrings. diff --git a/CHANGELOG.md b/CHANGELOG.md index 1de9130a5..5dd5793af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ Semantic Versioning where the repository publishes a release. ### Added +- Added a provider-neutral object-storage contract with an executable fail-closed validator, JSON Schema, naruon-shaped example, and APA 7 doctoring so AWS S3 and S3-compatible endpoints share one HTTPS, exact-host, integrity, retention, and rollback policy (ContextualWisdomLab/.github#1019, ContextualWisdomLab/naruon#1364). - Added a trusted pull-request comment router for `@cwl-noema-review` and review-only `@opencode-agent` dispatches, with an organization sweep, exact-head receipts, repository allowlisting, fixed runners, immutable checkout pins, and a permanent 100% statement/branch/docstring quality gate. - Added exact-base `uv.lock` materialization that reconstructs standalone nested projects with a checksum-pinned official `uv` exporter, isolated frozen/offline execution, strict exact-pin and SHA-256 output validation, and complete Python 3.10/3.14 quality evidence. - Added a permanent exact-head contract workflow for the hourly review-repair scheduler, immutable reusable-workflow source, NVIDIA NIM model boundary, credential isolation, and fail-closed unattended-agent permissions. @@ -26,6 +27,11 @@ Semantic Versioning where the repository publishes a release. ### Fixed +- Closed the object-storage contract around exact lowercase DNS hosts, TCP port range, finite RFC 8259 numbers, nested JSON Schema objects, and typed observability labels so hardening tests cannot pass a metadata, Unicode, or unhashable-label document. +- Removed the leftover one-shot object-storage repair workflow so Scorecard Token-Permissions no longer sees workflow-default `contents: write` after the hardening is already committed. +- Rejected decimal, hexadecimal, and dotted-numeric IPv4 aliases in object-storage host allowlists so Strix-proven metadata-service SSRF bypasses fail closed. +- Rejected multicast `.local`, metadata, and cluster-local names from object-storage allowlists, and rejected RFC 6761/6762 internal suffixes when private-network trust is denied, so a `denied` policy cannot admit `minio.internal`. +- Required tenant- and purpose-bound provider selection, aligned the portable JSON Schema with the executable value constraints, and added the product write/read/delete acceptance template so sibling adapters can prove compatibility without copying central code. - Materialized base Python locks only when every package line is an exact SHA-256 pin or a bounded relative `-r`/`--requirement` include. A lone `--require-hashes` directive, a dotted include such as `./lock.txt`, or `-r other-hashes.txt` no longer enters the trusted build context. - Refused a conflict-scope repository root whose immediate parent is a symbolic link, so a swapped parent cannot redirect the canonical worktree after the last-component check (CWE-367). - Bounded the Strix quality self-test's deterministic timeout fixtures to 3-second process and 5-second fake-sleep budgets so exact-head policy evidence completes inside the existing job limit without changing production Strix scanner timeouts, providers, credentials, or review semantics. diff --git a/docs/doctoring/object-storage-contract.md b/docs/doctoring/object-storage-contract.md new file mode 100644 index 000000000..ba1e72c88 --- /dev/null +++ b/docs/doctoring/object-storage-contract.md @@ -0,0 +1,103 @@ +# Object-storage contract: evidence and design record + +검토 기준일: **2026-08-16** + +## Incident / buyer-visible gap + +Sibling products persist customer or evidence objects in AWS S3 or +S3-compatible stores. Without a central executable contract, each repository +can invent endpoint, credential, encryption, integrity, retention, and +rollback rules. That drift is a procurement and data-loss risk for buyers who +must keep documents, even when Naruon already implements a scoped adapter in +ContextualWisdomLab/naruon#1364. + +This record does not implement Naruon storage. It records the organization +policy and the fail-closed check that leaf repositories can cite. + +## Decision + +1. Treat AWS S3 and S3-compatible HTTPS stores as one `object_storage` + capability. +2. Require HTTPS, exact-host allowlists, no wildcards, and no automatic + redirects (CWE-918). +3. Require explicit private-network trust. Implicit RFC1918, metadata-service, + multicast `.local` (Cheshire & Krochmal, 2013b), or special-use internal + suffixes (Cheshire & Krochmal, 2013a) are not authorized unless the exact + host is named after an explicit trust decision. Multicast `.local` names + are never unicast endpoints. +4. Require least-privilege object permissions and prohibit public ACLs, public + buckets, and browser-exposed long-lived credentials (CWE-798; CWE-200). +5. Require server-side encryption and fail-closed content-length plus SHA-256 + or stronger read verification (Amazon Web Services, 2024a, 2024b). +6. Keep lifecycle states distinct. `consumed` does not imply immediate + deletion unless the product explicitly configures zero retention. Legal hold + and archive remain separate from transient reprocessing retention. +7. Refuse rollback that deletes customer data after a partial + migration or backfill. +8. Forbid bucket names, object keys, credentials, and raw PII as unbounded + telemetry labels. This is not a blanket operational PII mask. +9. Record CSAP and SOC 2 only as design constraints. The contract must not + claim certification. +10. Require tenant- and purpose-bound provider selection so one credential or + bucket cannot serve unrelated tenants or jobs. +11. Ship an executable validator and a product acceptance template. Prose + alone does not close ContextualWisdomLab/.github#1019. + +## Trust boundary + +- Central `.github` owns the schema, policy, doctoring record, and validator. +- Product repositories own adapters, buckets, keys, credentials, and evidence + objects. +- NVIDIA NIM / OpenCode credentials are untouched. +- The validator never prints bucket names, object keys, or secrets from a + failing document beyond the field that violated the closed policy. + +## Verification contract + +`tests/test_object_storage_contract.py` proves the checked-in example passes, +the schema keys match production constants, and each fail-closed control has a +unique rejection. `tests/test_object_storage_contract_hardening.py` proves +nested schema objects stay closed, NaN/Infinity are rejected, exact-host +allowlists exclude localhost, metadata, cluster-local names, IPv4 and IPv6 +literals, decimal or hexadecimal IP aliases, Unicode, case aliases, and +multicast `.local` names, a denied private-network policy rejects +single-label and special-use internal hosts, custom endpoints reject ports +above 65535, schema value constraints match the validator, tenant-purpose +binding is mandatory, and malformed observability labels raise policy +errors instead of TypeError. Local quality remains 100% statement/branch +coverage and 100% docstrings. + +## Rollback + +Revert the validator, schema, example, policy, and this record together. Do +not keep a schema that the executable check no longer enforces. + +## References (APA 7th) + +Cheshire, S., & Krochmal, M. (2013a). *Special-use domain names* (RFC 6761). +Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc6761 + +Cheshire, S., & Krochmal, M. (2013b). *Multicast DNS* (RFC 6762). Internet +Engineering Task Force. https://www.rfc-editor.org/rfc/rfc6762 + +Amazon Web Services. (2024a). *Checking object integrity for data uploads in +Amazon S3*. Amazon Simple Storage Service User Guide. +https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html + +Amazon Web Services. (2024b). *Using server-side encryption with AWS KMS keys +(SSE-KMS)*. Amazon Simple Storage Service User Guide. +https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html + +Amazon Web Services. (n.d.). *Authenticating requests (AWS Signature Version +4)*. Amazon Simple Storage Service API Reference. +https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html + +MITRE. (n.d.-a). *CWE-200: Exposure of sensitive information to an +unauthorized actor*. CWE List. +https://cwe.mitre.org/data/definitions/200.html + +MITRE. (n.d.-b). *CWE-798: Use of hard-coded credentials*. CWE List. +https://cwe.mitre.org/data/definitions/798.html + +MITRE. (n.d.-c). *CWE-918: Server-side request forgery (SSRF)*. CWE List. +https://cwe.mitre.org/data/definitions/918.html diff --git a/docs/object-storage/CWL_OBJECT_STORAGE_CONTRACT.md b/docs/object-storage/CWL_OBJECT_STORAGE_CONTRACT.md new file mode 100644 index 000000000..fa8a409dd --- /dev/null +++ b/docs/object-storage/CWL_OBJECT_STORAGE_CONTRACT.md @@ -0,0 +1,78 @@ +# CWL object-storage contract + +This repository owns the reusable `object_storage` policy and the executable +check `scripts/ci/validate_object_storage_contract.py`. Product repositories +keep their own adapters, databases, and evidence objects. Naruon +[ContextualWisdomLab/naruon#1364](https://github.com/ContextualWisdomLab/naruon/pull/1364) +is the current concrete consumer and remains the owner of its document-storage +implementation. + +AWS-managed S3 and S3-compatible HTTPS endpoints are one provider-neutral +capability. They are not an AWS-only product assumption. + +## Closed controls + +A contract JSON document is valid only when every control below is true. + +| Control | Required value | +|---|---| +| Transport | `https` only | +| Hosts | exact-host allowlist; no wildcards; no automatic redirects; no multicast `.local` or metadata names | +| Private networks | `explicit_allowlist` or `denied`; never implicit RFC1918, RFC 6761/6762, or `.internal` access | +| Credentials | scoped secret registry or workload identity; never broadcast, browser-exposed, or ambient process-wide | +| Permissions | least privilege; public ACLs and public buckets prohibited | +| Encryption | server-side encryption `required` | +| Integrity | content length plus SHA-256 or stronger; fail-closed read verification | +| Lifecycle | `pending`, `available`, `consumed`, `archived`, and `held` are distinct; `consumed` does not delete unless zero retention is explicit | +| Rollback | a partial migration must not delete customer data | +| Names | persisted metadata uses multiword `snake_case` | +| Tenant binding | provider selection is tenant- and purpose-bound | +| Telemetry | bucket names, object keys, credentials, and raw PII are forbidden high-cardinality labels | +| Assurance | CSAP and SOC 2 are design constraints, not certifications | + +Operational PII stays usable through the owning product. Forbidding +high-cardinality labels is not a blanket PII mask. + +## Endpoint classes + +1. **AWS-managed S3 HTTPS endpoints** — `provider_class` is `aws_s3` and every + host is an exact Amazon S3 regional or dual-stack name. +2. **Public S3-compatible HTTPS endpoints** — `provider_class` is + `s3_compatible` and every host is an exact public DNS name. +3. **Authorized private-network endpoints** — `private_network_trust` is + `explicit_allowlist` and each private host is named. Implicit RFC1918, + link-local, metadata-service, multicast `.local`, or special-use + `.internal` / `.corp` / `.lan` access is rejected unless that exact host + is named after an explicit trust decision. Multicast `.local` names are + never valid unicast endpoints. + +Workload-identity and instance-metadata access require their own SSRF review. +This contract does not grant that access. + +## Product next actions + +On success the consumer persists or reads the object under its own schema and +records a purpose-bound audit event without bucket, key, credential, or raw PII +labels. On rejection, timeout, duplicate, or partial upload the consumer leaves +the previous durable object in place. Rollback never deletes customer data +because a backfill step only partly succeeded. Product repositories then run +the write/read/delete failure-injection lane in +`docs/object-storage/PRODUCT_ACCEPTANCE_TEMPLATE.md`. + +## Verification + +```bash +python3 scripts/ci/validate_object_storage_contract.py \ + --path schemas/examples/cwl-object-storage-v1.example.json +``` + +The checked-in example is the naruon-shaped fixture. A product repository may +commit its own contract file and run the same command in its quality job. The +issue is not closed by prose alone: the executable check is the acceptance +evidence. + +## Schema + +`schemas/cwl-object-storage-v1.schema.json` lists the closed top-level keys. +The Python validator is authoritative when a JSON Schema library is not +installed. diff --git a/docs/object-storage/PRODUCT_ACCEPTANCE_TEMPLATE.md b/docs/object-storage/PRODUCT_ACCEPTANCE_TEMPLATE.md new file mode 100644 index 000000000..8c6cb154e --- /dev/null +++ b/docs/object-storage/PRODUCT_ACCEPTANCE_TEMPLATE.md @@ -0,0 +1,46 @@ +# Object-storage product acceptance template + +Use this checklist in the product repository that owns the adapter. Central +`.github` does not store customer objects. The current consumer is +[ContextualWisdomLab/naruon#1364](https://github.com/ContextualWisdomLab/naruon/pull/1364). + +## Before you claim S3 compatibility + +1. Commit a contract JSON that passes + `python3 scripts/ci/validate_object_storage_contract.py --path `. +2. Keep provider selection tenant- and purpose-bound. Do not share one + credential or bucket across tenants or unrelated jobs. +3. Persist provider and object metadata in 3NF with multiword `snake_case` + names. Do not store credentials in events, logs, SBOMs, or model context. + +## Failure-injection lane + +A product may call itself S3-compatible only after a real object-store lane +(LocalStack or the purchased endpoint) proves write/read/delete behavior +against the current head, including: + +| Case | Required next action | +|---|---| +| write timeout | Leave the previous durable object in place and record a purpose-bound audit event | +| read timeout | Fail closed; do not treat a truncated body as valid evidence | +| delete timeout | Retry the same deterministic key; do not enumerate the bucket | +| partial or aborted upload | Do not publish a `available` object; compensate without deleting older customer data | +| digest or content-length mismatch | Reject the read; keep the prior object | + +## Rollback and retention + +A partial migration or backfill must not delete customer data. `consumed` +does not delete unless the product explicitly configures zero retention. +Legal hold and archive stay distinct from the reprocessing window. + +## Telemetry + +Emit OpenTelemetry or audit evidence without `bucket`, `object_key`, +`credential`, or `raw_pii` labels. Operational PII that the product must +process stays in the product data plane; do not blanket-mask it. + +## Close the issue only with current-head proof + +Point the product PR at this template, the central contract SHA, and the +exact-head integration run. Prose without that run does not close +ContextualWisdomLab/.github#1019. diff --git a/schemas/cwl-object-storage-v1.schema.json b/schemas/cwl-object-storage-v1.schema.json new file mode 100644 index 000000000..b46a07be6 --- /dev/null +++ b/schemas/cwl-object-storage-v1.schema.json @@ -0,0 +1,163 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/ContextualWisdomLab/.github/schemas/cwl-object-storage-v1.schema.json", + "title": "CWL object-storage contract v1", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "capability", + "repository", + "provider_class", + "endpoint_policy", + "credentials", + "permissions", + "encryption", + "integrity", + "lifecycle", + "rollback", + "observability", + "database_object_names", + "tenant_purpose_bound", + "assurance_posture" + ], + "properties": { + "schema_version": { "const": "1" }, + "capability": { "const": "object_storage" }, + "repository": { + "type": "string", + "pattern": "^ContextualWisdomLab/[A-Za-z0-9._-]*[A-Za-z0-9_-]$" + }, + "provider_class": { "enum": ["aws_s3", "s3_compatible"] }, + "endpoint_policy": { + "type": "object", + "additionalProperties": false, + "required": [ + "transport", + "host_allowlist", + "allow_wildcards", + "follow_redirects", + "private_network_trust" + ], + "properties": { + "transport": { "const": "https" }, + "host_allowlist": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { "type": "string", "minLength": 1 } + }, + "allow_wildcards": { "const": false }, + "follow_redirects": { "const": false }, + "private_network_trust": { + "enum": ["explicit_allowlist", "denied"] + }, + "custom_endpoint": { "type": "string", "pattern": "^https://" } + } + }, + "credentials": { + "type": "object", + "additionalProperties": false, + "required": [ + "broadcast", + "browser_long_lived", + "ambient_process_wide", + "mechanism" + ], + "properties": { + "broadcast": { "const": false }, + "browser_long_lived": { "const": false }, + "ambient_process_wide": { "const": false }, + "mechanism": { + "enum": ["scoped_secret_registry", "workload_identity"] + } + } + }, + "permissions": { + "type": "object", + "additionalProperties": false, + "required": ["public_acls", "public_buckets", "least_privilege"], + "properties": { + "public_acls": { "const": false }, + "public_buckets": { "const": false }, + "least_privilege": { "const": true } + } + }, + "encryption": { + "type": "object", + "additionalProperties": false, + "required": ["server_side"], + "properties": { + "server_side": { "const": "required" } + } + }, + "integrity": { + "type": "object", + "additionalProperties": false, + "required": ["content_length", "digest", "fail_closed_read"], + "properties": { + "content_length": { "const": true }, + "digest": { "enum": ["sha256", "sha384", "sha512"] }, + "fail_closed_read": { "const": true } + } + }, + "lifecycle": { + "type": "object", + "additionalProperties": false, + "required": [ + "states", + "consumed_implies_immediate_delete", + "zero_retention_explicit", + "legal_hold_distinct" + ], + "properties": { + "states": { + "type": "array", + "minItems": 5, + "uniqueItems": true, + "allOf": [ + { "contains": { "const": "pending" } }, + { "contains": { "const": "available" } }, + { "contains": { "const": "consumed" } }, + { "contains": { "const": "archived" } }, + { "contains": { "const": "held" } } + ], + "items": { "type": "string", "minLength": 1 } + }, + "consumed_implies_immediate_delete": { "type": "boolean" }, + "zero_retention_explicit": { "type": "boolean" }, + "legal_hold_distinct": { "const": true } + } + }, + "rollback": { + "type": "object", + "additionalProperties": false, + "required": ["delete_customer_data_on_partial_migration"], + "properties": { + "delete_customer_data_on_partial_migration": { "const": false } + } + }, + "observability": { + "type": "object", + "additionalProperties": false, + "required": ["high_cardinality_labels_forbid"], + "properties": { + "high_cardinality_labels_forbid": { + "type": "array", + "minItems": 4, + "uniqueItems": true, + "allOf": [ + { "contains": { "const": "bucket" } }, + { "contains": { "const": "object_key" } }, + { "contains": { "const": "credential" } }, + { "contains": { "const": "raw_pii" } } + ], + "items": { "type": "string", "minLength": 1 } + } + } + }, + "database_object_names": { "const": "multiword_snake_case" }, + "tenant_purpose_bound": { "const": true }, + "assurance_posture": { "const": "design_constraints_only" } + } +} diff --git a/schemas/examples/cwl-object-storage-v1.example.json b/schemas/examples/cwl-object-storage-v1.example.json new file mode 100644 index 000000000..f44562007 --- /dev/null +++ b/schemas/examples/cwl-object-storage-v1.example.json @@ -0,0 +1,56 @@ +{ + "schema_version": "1", + "capability": "object_storage", + "repository": "ContextualWisdomLab/naruon", + "provider_class": "s3_compatible", + "endpoint_policy": { + "transport": "https", + "host_allowlist": [ + "s3.ap-northeast-2.amazonaws.com", + "objects.example.example" + ], + "allow_wildcards": false, + "follow_redirects": false, + "private_network_trust": "explicit_allowlist", + "custom_endpoint": "https://objects.example.example" + }, + "credentials": { + "broadcast": false, + "browser_long_lived": false, + "ambient_process_wide": false, + "mechanism": "scoped_secret_registry" + }, + "permissions": { + "public_acls": false, + "public_buckets": false, + "least_privilege": true + }, + "encryption": { + "server_side": "required" + }, + "integrity": { + "content_length": true, + "digest": "sha256", + "fail_closed_read": true + }, + "lifecycle": { + "states": ["pending", "available", "consumed", "archived", "held"], + "consumed_implies_immediate_delete": false, + "zero_retention_explicit": false, + "legal_hold_distinct": true + }, + "rollback": { + "delete_customer_data_on_partial_migration": false + }, + "observability": { + "high_cardinality_labels_forbid": [ + "bucket", + "object_key", + "credential", + "raw_pii" + ] + }, + "database_object_names": "multiword_snake_case", + "tenant_purpose_bound": true, + "assurance_posture": "design_constraints_only" +} diff --git a/scripts/ci/validate_object_storage_contract.py b/scripts/ci/validate_object_storage_contract.py new file mode 100644 index 000000000..6432e81fb --- /dev/null +++ b/scripts/ci/validate_object_storage_contract.py @@ -0,0 +1,574 @@ +#!/usr/bin/env python3 +"""Fail-closed validator for organization object-storage contracts. + +The central repository owns only the reusable policy and this check. Product +repositories keep their own S3 or S3-compatible adapters. The contract is +provider-neutral: AWS-managed S3 and HTTPS S3-compatible endpoints share one +``object_storage`` capability. CSAP and SOC 2 appear only as design +constraints, never as a certification claim. +""" + +from __future__ import annotations + +import argparse +import ipaddress +import json +import sys +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any + + +SCHEMA_VERSION = "1" +CAPABILITY = "object_storage" +MAX_CONTRACT_BYTES = 65536 +ALLOWED_PROVIDER_CLASSES = frozenset({"aws_s3", "s3_compatible"}) +ALLOWED_CREDENTIAL_MECHANISMS = frozenset( + {"scoped_secret_registry", "workload_identity"} +) +ALLOWED_PRIVATE_NETWORK_TRUST = frozenset({"explicit_allowlist", "denied"}) +ALLOWED_DIGESTS = frozenset({"sha256", "sha384", "sha512"}) +REQUIRED_LIFECYCLE_STATES = ( + "pending", + "available", + "consumed", + "archived", + "held", +) +FORBIDDEN_HIGH_CARDINALITY_LABELS = frozenset( + {"bucket", "object_key", "credential", "raw_pii"} +) +FORBIDDEN_EXACT_HOSTS = frozenset( + { + "localhost", + "metadata.google.internal", + "metadata.goog", + "instance-data", + "instance-data.ec2.internal", + "kubernetes.default.svc", + } +) +ALWAYS_FORBIDDEN_HOST_SUFFIXES = (".localhost", ".local") +DENIED_PRIVATE_NETWORK_SUFFIXES = ( + ".internal", + ".corp", + ".lan", + ".home", + ".intranet", + ".private", +) +MAX_TCP_PORT = 65535 +MAX_DNS_HOST_LENGTH = 253 +ALLOWED_TOP_LEVEL = ( + "schema_version", + "capability", + "repository", + "provider_class", + "endpoint_policy", + "credentials", + "permissions", + "encryption", + "integrity", + "lifecycle", + "rollback", + "observability", + "database_object_names", + "tenant_purpose_bound", + "assurance_posture", +) +ENDPOINT_POLICY_KEYS = ( + "transport", + "host_allowlist", + "allow_wildcards", + "follow_redirects", + "private_network_trust", + "custom_endpoint", +) +CREDENTIAL_KEYS = ( + "broadcast", + "browser_long_lived", + "ambient_process_wide", + "mechanism", +) +PERMISSION_KEYS = ("public_acls", "public_buckets", "least_privilege") +ENCRYPTION_KEYS = ("server_side",) +INTEGRITY_KEYS = ("content_length", "digest", "fail_closed_read") +LIFECYCLE_KEYS = ( + "states", + "consumed_implies_immediate_delete", + "zero_retention_explicit", + "legal_hold_distinct", +) +ROLLBACK_KEYS = ("delete_customer_data_on_partial_migration",) +OBSERVABILITY_KEYS = ("high_cardinality_labels_forbid",) + + +class ObjectStorageContractError(ValueError): + """Raised when a contract file fails a fail-closed policy check.""" + + +def _reject_non_finite_json_constant(token: str) -> None: + """Reject NaN and Infinity, which are not finite RFC 8259 numbers.""" + raise ObjectStorageContractError( + f"contract contains a non-finite JSON number: {token}" + ) + + +def reject_duplicate_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + """Build an object while rejecting duplicate JSON keys.""" + seen: set[str] = set() + result: dict[str, Any] = {} + for key, value in pairs: + if key in seen: + raise ObjectStorageContractError(f"duplicate object key {key!r}") + seen.add(key) + result[key] = value + return result + + +def _is_dns_label_char(char: str) -> bool: + """Return whether *char* is an ASCII lowercase DNS label character.""" + return char.isascii() and (char.islower() or char.isdigit() or char == "-") + + +def _label_is_integer_token(label: str) -> bool: + """Return whether *label* is a decimal or ``0x`` hexadecimal integer token.""" + if label.isdigit() and label.isascii(): + return True + return ( + label.startswith("0x") + and len(label) > 2 + and all(char in "0123456789abcdef" for char in label[2:]) + ) + + +def _is_ip_literal_or_alias(host: str) -> bool: + """Return whether *host* is an IP literal or a numeric IP alias.""" + try: + ipaddress.ip_address(host) + except ValueError: + pass + else: + return True + return all(_label_is_integer_token(label) for label in host.split(".")) + + +def host_matches_suffix(host: str, suffixes: tuple[str, ...]) -> bool: + """Return whether *host* equals or ends with one DNS suffix in *suffixes*.""" + for suffix in suffixes: + bare = suffix[1:] + if host == bare or host.endswith(suffix): + return True + return False + + +def is_denied_private_network_host(host: str) -> bool: + """Return whether *host* is implicit-local under a denied private-network policy.""" + if "." not in host: + return True + return host_matches_suffix(host, DENIED_PRIVATE_NETWORK_SUFFIXES) + + +def is_exact_dns_host(host: str) -> bool: + """Return whether *host* is one exact lowercase DNS name. + + Localhost, link-local metadata names, IP literals, decimal or hexadecimal + IP aliases, Unicode, and case aliases are not exact allowlist members. + """ + if not isinstance(host, str) or not host or len(host) > MAX_DNS_HOST_LENGTH: + return False + if any(char in host for char in "*:/@ \t\\"): + return False + if host.startswith(".") or host.endswith("."): + return False + if host != host.lower(): + return False + if host in FORBIDDEN_EXACT_HOSTS or host_matches_suffix( + host, ALWAYS_FORBIDDEN_HOST_SUFFIXES + ): + return False + if _is_ip_literal_or_alias(host): + return False + labels = host.split(".") + for label in labels: + if not label or len(label) > 63: + return False + if label.startswith("-") or label.endswith("-"): + return False + if not all(_is_dns_label_char(char) for char in label): + return False + return True + + +def parse_https_endpoint_host(url: str) -> str: + """Return the exact host of an HTTPS endpoint, rejecting redirects and userinfo.""" + if not url.startswith("https://"): + raise ObjectStorageContractError( + "custom endpoints must use https:// and must not follow redirects" + ) + remainder = url[len("https://") :] + if not remainder or remainder.startswith("/"): + raise ObjectStorageContractError("custom endpoint is missing a host") + authority = remainder.split("/", 1)[0] + if "@" in authority: + raise ObjectStorageContractError( + "custom endpoints must not embed credentials" + ) + host, separator, port = authority.partition(":") + if separator and ( + not port.isdigit() + or not port.isascii() + or port == "0" + or int(port) > MAX_TCP_PORT + ): + raise ObjectStorageContractError("custom endpoint port is invalid") + if not is_exact_dns_host(host): + raise ObjectStorageContractError( + f"custom endpoint host {host!r} is not an exact DNS name" + ) + return host + + +def load_contract_bytes(raw: bytes) -> dict[str, Any]: + """Parse a bounded UTF-8 JSON object and reject duplicate keys.""" + if len(raw) > MAX_CONTRACT_BYTES: + raise ObjectStorageContractError("contract exceeds 65536 bytes") + if not raw: + raise ObjectStorageContractError("contract is empty") + try: + text = raw.decode("utf-8") + except UnicodeDecodeError as exc: + raise ObjectStorageContractError("contract is not UTF-8") from exc + if "\x00" in text: + raise ObjectStorageContractError("contract contains a NUL byte") + decoder = json.JSONDecoder( + object_pairs_hook=reject_duplicate_keys, + parse_constant=_reject_non_finite_json_constant, + ) + try: + data, index = decoder.raw_decode(text) + except json.JSONDecodeError as exc: + raise ObjectStorageContractError( + f"contract is not JSON: {exc.msg}" + ) from exc + if text[index:].strip(): + raise ObjectStorageContractError("contract has trailing JSON data") + if not isinstance(data, dict): + raise ObjectStorageContractError("contract root must be a JSON object") + return data + + +def load_contract_path(path: Path) -> dict[str, Any]: + """Read one contract file through the bounded UTF-8 loader.""" + try: + raw = path.read_bytes() + except OSError as exc: + raise ObjectStorageContractError(f"cannot read {path}: {exc}") from exc + return load_contract_bytes(raw) + + +def require_mapping(value: Any, name: str) -> dict[str, Any]: + """Return *value* when it is an object, otherwise fail closed.""" + if not isinstance(value, dict): + raise ObjectStorageContractError(f"{name} must be an object") + return value + + +def require_bool(value: Any, name: str, expected: bool) -> None: + """Require an exact boolean policy value.""" + if value is not expected: + raise ObjectStorageContractError(f"{name} must be {str(expected).lower()}") + + +def require_text(value: Any, name: str) -> str: + """Return a nonempty string or fail closed.""" + if not isinstance(value, str) or not value: + raise ObjectStorageContractError(f"{name} must be a nonempty string") + return value + + +def require_allowed(value: Any, name: str, allowed: frozenset[str]) -> str: + """Return *value* when it is one of the closed allowed strings.""" + text = require_text(value, name) + if text not in allowed: + raise ObjectStorageContractError( + f"{name} must be one of {sorted(allowed)}" + ) + return text + + +def reject_unknown_keys( + mapping: Mapping[str, Any], allowed: Sequence[str], name: str +) -> None: + """Reject keys that are not in the closed allowlist.""" + unknown = sorted(set(mapping) - set(allowed)) + if unknown: + raise ObjectStorageContractError(f"unknown {name} keys: {unknown}") + + +def validate_endpoint_policy(policy: Mapping[str, Any]) -> None: + """Validate HTTPS, exact-host allowlists, and explicit private-network trust.""" + reject_unknown_keys(policy, ENDPOINT_POLICY_KEYS, "endpoint_policy") + if require_text(policy.get("transport"), "endpoint_policy.transport") != "https": + raise ObjectStorageContractError("endpoint_policy.transport must be https") + require_bool(policy.get("allow_wildcards"), "endpoint_policy.allow_wildcards", False) + require_bool( + policy.get("follow_redirects"), "endpoint_policy.follow_redirects", False + ) + require_allowed( + policy.get("private_network_trust"), + "endpoint_policy.private_network_trust", + ALLOWED_PRIVATE_NETWORK_TRUST, + ) + hosts = policy.get("host_allowlist") + if not isinstance(hosts, list) or not hosts: + raise ObjectStorageContractError( + "endpoint_policy.host_allowlist must be a nonempty list" + ) + seen: set[str] = set() + private_network_trust = policy.get("private_network_trust") + for host in hosts: + if not isinstance(host, str) or not is_exact_dns_host(host): + raise ObjectStorageContractError( + f"endpoint host {host!r} is not an exact DNS name" + ) + if private_network_trust == "denied" and is_denied_private_network_host(host): + if "." not in host: + raise ObjectStorageContractError( + f"endpoint host {host!r} is a single-label name" + ) + raise ObjectStorageContractError( + f"endpoint host {host!r} is a private-network name" + ) + if host in seen: + raise ObjectStorageContractError(f"duplicate endpoint host {host!r}") + seen.add(host) + custom = policy.get("custom_endpoint") + if custom is None: + return + host = parse_https_endpoint_host( + require_text(custom, "endpoint_policy.custom_endpoint") + ) + if host not in seen: + raise ObjectStorageContractError( + "custom endpoint host is not on the exact allowlist" + ) + + +def validate_credentials(credentials: Mapping[str, Any]) -> None: + """Reject broadcast, browser, and ambient process-wide credentials.""" + reject_unknown_keys(credentials, CREDENTIAL_KEYS, "credentials") + require_bool(credentials.get("broadcast"), "credentials.broadcast", False) + require_bool( + credentials.get("browser_long_lived"), + "credentials.browser_long_lived", + False, + ) + require_bool( + credentials.get("ambient_process_wide"), + "credentials.ambient_process_wide", + False, + ) + require_allowed( + credentials.get("mechanism"), + "credentials.mechanism", + ALLOWED_CREDENTIAL_MECHANISMS, + ) + + +def validate_permissions(permissions: Mapping[str, Any]) -> None: + """Require least privilege and prohibit public ACLs or public buckets.""" + reject_unknown_keys(permissions, PERMISSION_KEYS, "permissions") + require_bool(permissions.get("public_acls"), "permissions.public_acls", False) + require_bool( + permissions.get("public_buckets"), "permissions.public_buckets", False + ) + require_bool( + permissions.get("least_privilege"), "permissions.least_privilege", True + ) + + +def validate_encryption(encryption: Mapping[str, Any]) -> None: + """Require server-side encryption without claiming a certification.""" + reject_unknown_keys(encryption, ENCRYPTION_KEYS, "encryption") + if ( + require_text(encryption.get("server_side"), "encryption.server_side") + != "required" + ): + raise ObjectStorageContractError("encryption.server_side must be required") + + +def validate_integrity(integrity: Mapping[str, Any]) -> None: + """Require content-length plus SHA-256 or stronger fail-closed read checks.""" + reject_unknown_keys(integrity, INTEGRITY_KEYS, "integrity") + require_bool(integrity.get("content_length"), "integrity.content_length", True) + require_allowed(integrity.get("digest"), "integrity.digest", ALLOWED_DIGESTS) + require_bool( + integrity.get("fail_closed_read"), "integrity.fail_closed_read", True + ) + + +def validate_lifecycle(lifecycle: Mapping[str, Any]) -> None: + """Keep consumed, legal-hold, and retention states distinct.""" + reject_unknown_keys(lifecycle, LIFECYCLE_KEYS, "lifecycle") + states = lifecycle.get("states") + if not isinstance(states, list): + raise ObjectStorageContractError("lifecycle.states must be a list") + missing = [state for state in REQUIRED_LIFECYCLE_STATES if state not in states] + if missing: + raise ObjectStorageContractError( + f"lifecycle.states is missing required states: {missing}" + ) + if any(not isinstance(state, str) or not state for state in states): + raise ObjectStorageContractError( + "lifecycle.states must contain nonempty strings" + ) + consumed_deletes = lifecycle.get("consumed_implies_immediate_delete") + zero_retention = lifecycle.get("zero_retention_explicit") + if consumed_deletes is True and zero_retention is not True: + raise ObjectStorageContractError( + "consumed must not imply immediate deletion unless zero retention is explicit" + ) + if consumed_deletes is not True and consumed_deletes is not False: + raise ObjectStorageContractError( + "lifecycle.consumed_implies_immediate_delete must be a boolean" + ) + if zero_retention is not True and zero_retention is not False: + raise ObjectStorageContractError( + "lifecycle.zero_retention_explicit must be a boolean" + ) + require_bool( + lifecycle.get("legal_hold_distinct"), + "lifecycle.legal_hold_distinct", + True, + ) + + +def validate_rollback(rollback: Mapping[str, Any]) -> None: + """Refuse rollback that deletes customer data after a partial migration.""" + reject_unknown_keys(rollback, ROLLBACK_KEYS, "rollback") + require_bool( + rollback.get("delete_customer_data_on_partial_migration"), + "rollback.delete_customer_data_on_partial_migration", + False, + ) + + +def validate_observability(observability: Mapping[str, Any]) -> None: + """Forbid high-cardinality bucket, key, credential, and raw-PII labels. + + Operational PII remains usable through the owning product. This check only + blocks unbounded telemetry labels; it is not a blanket PII mask. + """ + reject_unknown_keys(observability, OBSERVABILITY_KEYS, "observability") + labels = observability.get("high_cardinality_labels_forbid") + if not isinstance(labels, list): + raise ObjectStorageContractError( + "observability.high_cardinality_labels_forbid must be a list" + ) + seen_labels: set[str] = set() + for label in labels: + if not isinstance(label, str) or not label: + raise ObjectStorageContractError( + "observability.high_cardinality_labels_forbid must contain nonempty strings" + ) + if label in seen_labels: + raise ObjectStorageContractError( + f"duplicate observability label {label!r}" + ) + seen_labels.add(label) + missing = sorted(FORBIDDEN_HIGH_CARDINALITY_LABELS - seen_labels) + if missing: + raise ObjectStorageContractError( + "observability must forbid high-cardinality labels: " + f"{missing}" + ) + + +def is_cwl_repository(repository: str) -> bool: + """Return whether *repository* is an owner/name pair in this organization.""" + owner, separator, name = repository.partition("/") + if separator != "/" or owner != "ContextualWisdomLab" or not name: + return False + if "/" in name or name.endswith("."): + return False + return all(char.isalnum() or char in "._-" for char in name) + + +def validate_contract(data: Mapping[str, Any]) -> None: + """Validate one loaded object-storage contract object.""" + reject_unknown_keys(data, ALLOWED_TOP_LEVEL, "contract") + missing = [key for key in ALLOWED_TOP_LEVEL if key not in data] + if missing: + raise ObjectStorageContractError(f"contract is missing keys: {missing}") + if require_text(data.get("schema_version"), "schema_version") != SCHEMA_VERSION: + raise ObjectStorageContractError("schema_version must be 1") + if require_text(data.get("capability"), "capability") != CAPABILITY: + raise ObjectStorageContractError("capability must be object_storage") + repository = require_text(data.get("repository"), "repository") + if not is_cwl_repository(repository): + raise ObjectStorageContractError( + "repository must be ContextualWisdomLab/" + ) + require_allowed( + data.get("provider_class"), "provider_class", ALLOWED_PROVIDER_CLASSES + ) + validate_endpoint_policy( + require_mapping(data.get("endpoint_policy"), "endpoint_policy") + ) + validate_credentials(require_mapping(data.get("credentials"), "credentials")) + validate_permissions(require_mapping(data.get("permissions"), "permissions")) + validate_encryption(require_mapping(data.get("encryption"), "encryption")) + validate_integrity(require_mapping(data.get("integrity"), "integrity")) + validate_lifecycle(require_mapping(data.get("lifecycle"), "lifecycle")) + validate_rollback(require_mapping(data.get("rollback"), "rollback")) + validate_observability( + require_mapping(data.get("observability"), "observability") + ) + if ( + require_text(data.get("database_object_names"), "database_object_names") + != "multiword_snake_case" + ): + raise ObjectStorageContractError( + "database_object_names must be multiword_snake_case" + ) + require_bool( + data.get("tenant_purpose_bound"), "tenant_purpose_bound", True + ) + if ( + require_text(data.get("assurance_posture"), "assurance_posture") + != "design_constraints_only" + ): + raise ObjectStorageContractError( + "assurance_posture must be design_constraints_only; " + "CSAP and SOC 2 are design constraints, not certifications" + ) + + +def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + """Parse the validator CLI.""" + parser = argparse.ArgumentParser( + description="Validate a CWL object-storage contract." + ) + parser.add_argument( + "--path", + required=True, + help="Path to the JSON contract file.", + ) + return parser.parse_args(argv) + + +def main(argv: Sequence[str] | None = None) -> int: + """Load and validate one contract path, returning a process exit code.""" + args = parse_args(argv) + try: + validate_contract(load_contract_path(Path(args.path))) + except ObjectStorageContractError as exc: + print(f"object-storage contract failed: {exc}", file=sys.stderr) + return 1 + print("object-storage contract passed") + return 0 + + +if __name__ == "__main__": # pragma: no cover + raise SystemExit(main()) diff --git a/tests/test_object_storage_contract.py b/tests/test_object_storage_contract.py new file mode 100644 index 000000000..398d6a8c7 --- /dev/null +++ b/tests/test_object_storage_contract.py @@ -0,0 +1,346 @@ +"""Contracts for the organization-wide object-storage policy check.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from scripts.ci import validate_object_storage_contract as validator + + +ROOT = Path(__file__).resolve().parents[1] +EXAMPLE = ROOT / "schemas" / "examples" / "cwl-object-storage-v1.example.json" +SCHEMA = ROOT / "schemas" / "cwl-object-storage-v1.schema.json" +POLICY = ROOT / "docs" / "object-storage" / "CWL_OBJECT_STORAGE_CONTRACT.md" +DOCTORING = ROOT / "docs" / "doctoring" / "object-storage-contract.md" +CHANGELOG = ROOT / "CHANGELOG.md" +ARCHITECTURE = ROOT / "ARCHITECTURE.md" + + +def _valid_contract() -> dict: + """Return a mutable copy of the checked-in valid example.""" + return json.loads(EXAMPLE.read_text(encoding="utf-8")) + + +def test_checked_in_example_passes_the_executable_policy() -> None: + """The published naruon-shaped example must satisfy the fail-closed check.""" + validator.validate_contract(validator.load_contract_path(EXAMPLE)) + assert validator.main(["--path", str(EXAMPLE)]) == 0 + + +def test_schema_required_keys_match_production_constants() -> None: + """The JSON Schema must not drift from the executable allowlist.""" + schema = json.loads(SCHEMA.read_text(encoding="utf-8")) + assert schema["required"] == list(validator.ALLOWED_TOP_LEVEL) + assert set(schema["properties"]) == set(validator.ALLOWED_TOP_LEVEL) + assert schema["additionalProperties"] is False + + +def test_reject_duplicate_keys_and_non_object_root() -> None: + """Duplicate keys and array roots are untrusted evidence.""" + with pytest.raises(validator.ObjectStorageContractError, match="duplicate"): + validator.load_contract_bytes(b'{"schema_version":"1","schema_version":"2"}') + with pytest.raises(validator.ObjectStorageContractError, match="JSON object"): + validator.load_contract_bytes(b"[]") + + +def test_load_contract_bytes_rejects_empty_oversize_encoding_and_trailing() -> None: + """The loader must fail closed before any policy field is trusted.""" + with pytest.raises(validator.ObjectStorageContractError, match="empty"): + validator.load_contract_bytes(b"") + with pytest.raises(validator.ObjectStorageContractError, match="65536"): + validator.load_contract_bytes(b"{" + (b"a" * 65537)) + with pytest.raises(validator.ObjectStorageContractError, match="UTF-8"): + validator.load_contract_bytes(b"\xff\xfe") + with pytest.raises(validator.ObjectStorageContractError, match="NUL"): + validator.load_contract_bytes(b'{"a":"x\x00y"}') + with pytest.raises(validator.ObjectStorageContractError, match="not JSON"): + validator.load_contract_bytes(b"{") + with pytest.raises(validator.ObjectStorageContractError, match="trailing"): + validator.load_contract_bytes(b'{"schema_version":"1"}{}') + + +def test_load_contract_path_reports_unreadable_files(tmp_path: Path) -> None: + """A missing path is a contract failure, not a crash.""" + missing = tmp_path / "absent.json" + with pytest.raises(validator.ObjectStorageContractError, match="cannot read"): + validator.load_contract_path(missing) + assert validator.main(["--path", str(missing)]) == 1 + + +def test_is_exact_dns_host_rejects_wildcards_ports_and_empty_labels() -> None: + """Exact-host allowlists cannot contain wildcard or port syntax.""" + assert validator.is_exact_dns_host("s3.ap-northeast-2.amazonaws.com") + assert validator.is_exact_dns_host("localhost") is False + assert validator.is_exact_dns_host("") is False + assert validator.is_exact_dns_host(None) is False + assert validator.is_exact_dns_host("a..b") is False + assert validator.is_exact_dns_host("*.example.com") is False + assert validator.is_exact_dns_host("example.com:443") is False + assert validator.is_exact_dns_host(".example.com") is False + assert validator.is_exact_dns_host("example.com.") is False + assert validator.is_exact_dns_host("-bad.example.com") is False + assert validator.is_exact_dns_host("bad-.example.com") is False + assert validator.is_exact_dns_host("bad_name.example.com") is False + assert validator.is_exact_dns_host("a" * 64 + ".example.com") is False + + +def test_parse_https_endpoint_host_rejects_http_userinfo_and_bad_ports() -> None: + """Custom endpoints stay HTTPS, credential-free, and exact-host.""" + assert ( + validator.parse_https_endpoint_host("https://objects.example.example/path") + == "objects.example.example" + ) + assert ( + validator.parse_https_endpoint_host("https://objects.example.example:9000") + == "objects.example.example" + ) + with pytest.raises(validator.ObjectStorageContractError, match="https://"): + validator.parse_https_endpoint_host("http://objects.example.example") + with pytest.raises(validator.ObjectStorageContractError, match="missing a host"): + validator.parse_https_endpoint_host("https://") + with pytest.raises(validator.ObjectStorageContractError, match="missing a host"): + validator.parse_https_endpoint_host("https:///bucket") + with pytest.raises(validator.ObjectStorageContractError, match="embed credentials"): + validator.parse_https_endpoint_host("https://user:pass@objects.example.example") + with pytest.raises(validator.ObjectStorageContractError, match="port"): + validator.parse_https_endpoint_host("https://objects.example.example:0") + with pytest.raises(validator.ObjectStorageContractError, match="port"): + validator.parse_https_endpoint_host("https://objects.example.example:abc") + with pytest.raises(validator.ObjectStorageContractError, match="exact DNS"): + validator.parse_https_endpoint_host("https://*.example.example") + + +def test_validate_contract_accepts_zero_retention_only_when_explicit() -> None: + """Consumed-delete is allowed only when the product opts into zero retention.""" + contract = _valid_contract() + contract["lifecycle"]["consumed_implies_immediate_delete"] = True + with pytest.raises(validator.ObjectStorageContractError, match="zero retention"): + validator.validate_contract(contract) + contract["lifecycle"]["zero_retention_explicit"] = True + validator.validate_contract(contract) + + +def test_validate_contract_rejects_policy_regressions() -> None: + """Each required fail-closed control must have a unique rejection.""" + cases = [ + ("schema_version", "2", "schema_version"), + ("capability", "s3", "object_storage"), + ("repository", "other/naruon", "ContextualWisdomLab"), + ("provider_class", "gcs", "provider_class"), + ("database_object_names", "camelCase", "multiword_snake_case"), + ("tenant_purpose_bound", False, "tenant_purpose_bound"), + ("assurance_posture", "certified", "design_constraints_only"), + ] + for field, value, needle in cases: + contract = _valid_contract() + contract[field] = value + with pytest.raises(validator.ObjectStorageContractError, match=needle): + validator.validate_contract(contract) + + +def test_is_cwl_repository_rejects_nested_and_dot_names() -> None: + """Repository identity is a single owner/name pair in this organization.""" + assert validator.is_cwl_repository("ContextualWisdomLab/naruon") + assert validator.is_cwl_repository("ContextualWisdomLab/.github") + assert not validator.is_cwl_repository("naruon") + assert not validator.is_cwl_repository("ContextualWisdomLab/") + assert not validator.is_cwl_repository("ContextualWisdomLab/naruon/extra") + assert not validator.is_cwl_repository("ContextualWisdomLab/.hidden.") + assert not validator.is_cwl_repository("ContextualWisdomLab/bad name") + + +def test_nested_objects_must_be_objects_and_closed() -> None: + """Unknown nested keys and non-objects fail before a weaker field is read.""" + contract = _valid_contract() + contract["endpoint_policy"] = [] + with pytest.raises(validator.ObjectStorageContractError, match="endpoint_policy"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["endpoint_policy"]["extra"] = True + with pytest.raises(validator.ObjectStorageContractError, match="unknown"): + validator.validate_contract(contract) + contract = _valid_contract() + del contract["integrity"] + with pytest.raises(validator.ObjectStorageContractError, match="missing keys"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["unknown"] = True + with pytest.raises(validator.ObjectStorageContractError, match="unknown"): + validator.validate_contract(contract) + + +def test_endpoint_policy_https_exact_hosts_and_no_redirects() -> None: + """HTTPS, exact hosts, and no automatic redirects are mandatory.""" + contract = _valid_contract() + contract["endpoint_policy"]["transport"] = "http" + with pytest.raises(validator.ObjectStorageContractError, match="https"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["endpoint_policy"]["allow_wildcards"] = True + with pytest.raises(validator.ObjectStorageContractError, match="allow_wildcards"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["endpoint_policy"]["follow_redirects"] = True + with pytest.raises(validator.ObjectStorageContractError, match="follow_redirects"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["endpoint_policy"]["private_network_trust"] = "rfc1918" + with pytest.raises(validator.ObjectStorageContractError, match="private_network"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["endpoint_policy"]["host_allowlist"] = [] + with pytest.raises(validator.ObjectStorageContractError, match="host_allowlist"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["endpoint_policy"]["host_allowlist"] = ["*.example.com"] + with pytest.raises(validator.ObjectStorageContractError, match="exact DNS"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["endpoint_policy"]["host_allowlist"] = [ + "objects.example.example", + "objects.example.example", + ] + with pytest.raises(validator.ObjectStorageContractError, match="duplicate"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["endpoint_policy"]["custom_endpoint"] = "https://other.example.example" + with pytest.raises(validator.ObjectStorageContractError, match="allowlist"): + validator.validate_contract(contract) + contract = _valid_contract() + del contract["endpoint_policy"]["custom_endpoint"] + validator.validate_contract(contract) + + +def test_credentials_permissions_encryption_and_integrity_gates() -> None: + """Public buckets, ambient credentials, and weak integrity fail closed.""" + contract = _valid_contract() + contract["credentials"]["broadcast"] = True + with pytest.raises(validator.ObjectStorageContractError, match="broadcast"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["credentials"]["browser_long_lived"] = True + with pytest.raises(validator.ObjectStorageContractError, match="browser"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["credentials"]["ambient_process_wide"] = True + with pytest.raises(validator.ObjectStorageContractError, match="ambient"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["credentials"]["mechanism"] = "os.getenv" + with pytest.raises(validator.ObjectStorageContractError, match="mechanism"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["permissions"]["public_acls"] = True + with pytest.raises(validator.ObjectStorageContractError, match="public_acls"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["permissions"]["public_buckets"] = True + with pytest.raises(validator.ObjectStorageContractError, match="public_buckets"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["permissions"]["least_privilege"] = False + with pytest.raises(validator.ObjectStorageContractError, match="least_privilege"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["encryption"]["server_side"] = "optional" + with pytest.raises(validator.ObjectStorageContractError, match="server_side"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["integrity"]["content_length"] = False + with pytest.raises(validator.ObjectStorageContractError, match="content_length"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["integrity"]["digest"] = "md5" + with pytest.raises(validator.ObjectStorageContractError, match="digest"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["integrity"]["fail_closed_read"] = False + with pytest.raises(validator.ObjectStorageContractError, match="fail_closed_read"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["integrity"]["digest"] = "sha512" + validator.validate_contract(contract) + + +def test_lifecycle_rollback_and_observability_gates() -> None: + """Retention, rollback, and telemetry labels stay fail closed.""" + contract = _valid_contract() + contract["lifecycle"]["states"] = ["pending"] + with pytest.raises(validator.ObjectStorageContractError, match="missing required"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["lifecycle"]["states"] = "pending" + with pytest.raises(validator.ObjectStorageContractError, match="must be a list"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["lifecycle"]["states"] = [ + "pending", + "available", + "consumed", + "archived", + "held", + "", + ] + with pytest.raises(validator.ObjectStorageContractError, match="nonempty"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["lifecycle"]["consumed_implies_immediate_delete"] = "yes" + with pytest.raises(validator.ObjectStorageContractError, match="boolean"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["lifecycle"]["zero_retention_explicit"] = "yes" + with pytest.raises(validator.ObjectStorageContractError, match="boolean"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["lifecycle"]["legal_hold_distinct"] = False + with pytest.raises(validator.ObjectStorageContractError, match="legal_hold"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["rollback"]["delete_customer_data_on_partial_migration"] = True + with pytest.raises(validator.ObjectStorageContractError, match="partial"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["observability"]["high_cardinality_labels_forbid"] = ["bucket"] + with pytest.raises(validator.ObjectStorageContractError, match="raw_pii"): + validator.validate_contract(contract) + contract = _valid_contract() + contract["observability"]["high_cardinality_labels_forbid"] = "bucket" + with pytest.raises(validator.ObjectStorageContractError, match="must be a list"): + validator.validate_contract(contract) + + +def test_require_text_and_cli_success_message( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + """Empty strings and successful CLI output stay explicit.""" + with pytest.raises(validator.ObjectStorageContractError, match="nonempty"): + validator.require_text("", "field") + with pytest.raises(validator.ObjectStorageContractError, match="nonempty"): + validator.require_text(1, "field") + destination = tmp_path / "contract.json" + destination.write_text(EXAMPLE.read_text(encoding="utf-8"), encoding="utf-8") + assert validator.main(["--path", str(destination)]) == 0 + assert "object-storage contract passed" in capsys.readouterr().out + + +def test_policy_and_doctoring_record_buyer_visible_controls() -> None: + """Prose must name the executable check, naruon consumer, and APA 7 sources.""" + policy = POLICY.read_text(encoding="utf-8") + doctoring = DOCTORING.read_text(encoding="utf-8") + changelog = CHANGELOG.read_text(encoding="utf-8") + architecture = ARCHITECTURE.read_text(encoding="utf-8") + assert "object_storage" in policy + assert "ContextualWisdomLab/naruon#1364" in policy + assert "scripts/ci/validate_object_storage_contract.py" in policy + assert "design constraints" in policy.lower() + assert "not a blanket PII mask" in policy + assert "tenant- and purpose-bound" in policy + assert "PRODUCT_ACCEPTANCE_TEMPLATE.md" in policy + assert "APA 7" in doctoring or "References (APA 7th)" in doctoring + assert "Amazon Web Services" in doctoring + assert "CWE-918" in doctoring + assert "object-storage contract" in changelog + assert "object-storage" in architecture.lower() diff --git a/tests/test_object_storage_contract_hardening.py b/tests/test_object_storage_contract_hardening.py new file mode 100644 index 000000000..de4e094a3 --- /dev/null +++ b/tests/test_object_storage_contract_hardening.py @@ -0,0 +1,246 @@ +"""Fail-first hardening contracts for object-storage policy documents.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from scripts.ci import validate_object_storage_contract as validator + + +ROOT = Path(__file__).resolve().parents[1] +SCHEMA = ROOT / "schemas" / "cwl-object-storage-v1.schema.json" +EXAMPLE = ROOT / "schemas" / "examples" / "cwl-object-storage-v1.example.json" +ONE_SHOT_REPAIR_WORKFLOW = ( + ROOT / ".github" / "workflows" / "repair-object-storage-contract.yml" +) + + +def _valid_contract() -> dict: + """Return an independent mutable copy of the published example.""" + return json.loads(EXAMPLE.read_text(encoding="utf-8")) + + +def test_one_shot_repair_workflow_is_absent() -> None: + """Completed one-shot writers must not remain to grant contents: write.""" + assert not ONE_SHOT_REPAIR_WORKFLOW.exists() + + +def test_schema_closes_every_nested_policy_object() -> None: + """The portable schema must express the executable nested-key boundary.""" + schema = json.loads(SCHEMA.read_text(encoding="utf-8")) + expected = { + "endpoint_policy": validator.ENDPOINT_POLICY_KEYS, + "credentials": validator.CREDENTIAL_KEYS, + "permissions": validator.PERMISSION_KEYS, + "encryption": validator.ENCRYPTION_KEYS, + "integrity": validator.INTEGRITY_KEYS, + "lifecycle": validator.LIFECYCLE_KEYS, + "rollback": validator.ROLLBACK_KEYS, + "observability": validator.OBSERVABILITY_KEYS, + } + for name, keys in expected.items(): + definition = schema["properties"][name] + assert definition["type"] == "object" + assert definition["additionalProperties"] is False + assert set(definition["properties"]) == set(keys) + required = set(keys) + if name == "endpoint_policy": + required.remove("custom_endpoint") + assert set(definition["required"]) == required + + +def test_loader_rejects_non_finite_json_constants() -> None: + """NaN and infinity are not RFC 8259 JSON contract values.""" + for token in (b"NaN", b"Infinity", b"-Infinity"): + with pytest.raises(validator.ObjectStorageContractError, match="finite"): + validator.load_contract_bytes(b'{"unexpected":' + token + b"}") + + +@pytest.mark.parametrize( + "host", + [ + "localhost", + "api.localhost", + "169.254.169.254", + "127.0.0.1", + "2130706433", + "2851992574", + "127.1", + "127.0.1", + "0x7f000001", + "0x7f.0.0.1", + "a" * 254, + "metadata.google.internal", + "metadata.goog", + "OBJECTS.EXAMPLE.COM", + "objécts.example.com", + ], +) +def test_exact_dns_hosts_reject_local_metadata_literal_and_noncanonical_names( + host: str, +) -> None: + """An exact allowlist must not admit metadata, IP, Unicode, or case aliases.""" + assert validator.is_exact_dns_host(host) is False + + +def test_exact_dns_hosts_still_accept_numeric_dns_labels() -> None: + """A numeric label is allowed only when the name is not an IP alias.""" + assert validator.is_exact_dns_host("1.s3.amazonaws.com") + assert validator.is_exact_dns_host("0xz.example.com") + assert validator.is_exact_dns_host("0x.example.com") + + +def test_explicit_allowlist_rejects_decimal_metadata_ip_alias() -> None: + """Decimal IPv4 aliases must not enter an explicit host allowlist.""" + contract = _valid_contract() + contract["endpoint_policy"]["private_network_trust"] = "explicit_allowlist" + contract["endpoint_policy"]["host_allowlist"] = ["2851992574"] + contract["endpoint_policy"].pop("custom_endpoint", None) + with pytest.raises(validator.ObjectStorageContractError, match="exact DNS"): + validator.validate_contract(contract) + + +def test_denied_private_network_policy_rejects_single_label_hosts() -> None: + """A denied private-network policy cannot admit an implicitly local host.""" + contract = _valid_contract() + contract["endpoint_policy"]["private_network_trust"] = "denied" + contract["endpoint_policy"]["host_allowlist"] = ["minio"] + contract["endpoint_policy"]["custom_endpoint"] = "https://minio" + with pytest.raises(validator.ObjectStorageContractError, match="single-label"): + validator.validate_contract(contract) + + +def test_custom_endpoint_rejects_out_of_range_port() -> None: + """A syntactically numeric port still must fit the TCP port range.""" + with pytest.raises(validator.ObjectStorageContractError, match="port"): + validator.parse_https_endpoint_host("https://objects.example.example:65536") + + +def test_observability_labels_fail_closed_without_type_errors() -> None: + """Malformed or duplicate telemetry labels produce policy errors, not crashes.""" + contract = _valid_contract() + contract["observability"]["high_cardinality_labels_forbid"] = [ + "bucket", + "object_key", + "credential", + {}, + ] + with pytest.raises(validator.ObjectStorageContractError, match="nonempty strings"): + validator.validate_contract(contract) + + contract = _valid_contract() + contract["observability"]["high_cardinality_labels_forbid"].append("bucket") + with pytest.raises(validator.ObjectStorageContractError, match="duplicate"): + validator.validate_contract(contract) + + +@pytest.mark.parametrize( + "host", + [ + "s3.internal", + "objects.corp", + "minio.lan", + "storage.home", + "minio.intranet", + "objects.private", + ], +) +def test_denied_private_network_rejects_special_use_internal_suffixes( + host: str, +) -> None: + """A denied private-network policy cannot admit RFC 6761/6762 internal names.""" + contract = _valid_contract() + contract["endpoint_policy"]["private_network_trust"] = "denied" + contract["endpoint_policy"]["host_allowlist"] = [host] + contract["endpoint_policy"].pop("custom_endpoint", None) + with pytest.raises(validator.ObjectStorageContractError, match="private-network"): + validator.validate_contract(contract) + + +def test_mdns_local_suffix_is_never_an_exact_allowlist_member() -> None: + """Multicast .local names are not unicast object-storage endpoints.""" + assert validator.is_exact_dns_host("local") is False + assert validator.is_exact_dns_host("minio.local") is False + assert validator.is_denied_private_network_host("objects.example.example") is False + contract = _valid_contract() + contract["endpoint_policy"]["private_network_trust"] = "explicit_allowlist" + contract["endpoint_policy"]["host_allowlist"] = ["minio.local"] + contract["endpoint_policy"].pop("custom_endpoint", None) + with pytest.raises(validator.ObjectStorageContractError, match="exact DNS"): + validator.validate_contract(contract) + + +def test_explicit_allowlist_still_admits_a_named_internal_host() -> None: + """Operators may name one private DNS host after an explicit trust decision.""" + contract = _valid_contract() + contract["endpoint_policy"]["private_network_trust"] = "explicit_allowlist" + contract["endpoint_policy"]["host_allowlist"] = ["minio.internal"] + contract["endpoint_policy"]["custom_endpoint"] = "https://minio.internal" + validator.validate_contract(contract) + + +@pytest.mark.parametrize( + "host", + [ + "instance-data", + "instance-data.ec2.internal", + "kubernetes.default.svc", + "::1", + "[::1]", + ], +) +def test_metadata_cluster_and_ipv6_literals_are_never_exact_hosts(host: str) -> None: + """Metadata, cluster-local, and IPv6 literals stay off the exact allowlist.""" + assert validator.is_exact_dns_host(host) is False + + +def test_tenant_purpose_bound_selection_is_mandatory() -> None: + """Provider selection must stay tenant- and purpose-bound.""" + contract = _valid_contract() + contract["tenant_purpose_bound"] = False + with pytest.raises( + validator.ObjectStorageContractError, match="tenant_purpose_bound must be true" + ): + validator.validate_contract(contract) + contract = _valid_contract() + contract.pop("tenant_purpose_bound", None) + with pytest.raises(validator.ObjectStorageContractError, match="missing keys"): + validator.validate_contract(contract) + + +def test_schema_encodes_value_level_closed_controls() -> None: + """Portable schema consumers must see the same closed values as the validator.""" + schema = json.loads(SCHEMA.read_text(encoding="utf-8")) + repository = schema["properties"]["repository"] + assert repository["pattern"] == r"^ContextualWisdomLab/[A-Za-z0-9._-]*[A-Za-z0-9_-]$" + hosts = schema["properties"]["endpoint_policy"]["properties"]["host_allowlist"] + assert hosts["uniqueItems"] is True + custom = schema["properties"]["endpoint_policy"]["properties"]["custom_endpoint"] + assert custom["pattern"] == r"^https://" + states = schema["properties"]["lifecycle"]["properties"]["states"] + assert states["uniqueItems"] is True + assert {item["contains"]["const"] for item in states["allOf"]} == set( + validator.REQUIRED_LIFECYCLE_STATES + ) + labels = schema["properties"]["observability"]["properties"][ + "high_cardinality_labels_forbid" + ] + assert labels["uniqueItems"] is True + assert {item["contains"]["const"] for item in labels["allOf"]} == set( + validator.FORBIDDEN_HIGH_CARDINALITY_LABELS + ) + assert schema["properties"]["tenant_purpose_bound"]["const"] is True + + +def test_product_acceptance_template_names_the_consumer_failure_lane() -> None: + """Product repositories need the next write/read/delete proof, not more prose.""" + template = ( + ROOT / "docs" / "object-storage" / "PRODUCT_ACCEPTANCE_TEMPLATE.md" + ).read_text(encoding="utf-8") + assert "ContextualWisdomLab/naruon#1364" in template + assert "write/read/delete" in template + assert "partial" in template.lower() + assert "timeout" in template.lower()