Repository navigation
feat(storage): add executable object-storage contract #1023
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| # Object-storage contract: evidence and design record | ||
|
|
||
| 검토 기준일: **2026-08-16** | ||
|
|
||
| ## Incident / buyer-visible gap | ||
|
|
||
| Sibling products persist customer or evidence objects in AWS S3 or | ||
| S3-compatible stores. Without a central executable contract, each repository | ||
| can invent endpoint, credential, encryption, integrity, retention, and | ||
| rollback rules. That drift is a procurement and data-loss risk for buyers who | ||
| must keep documents, even when Naruon already implements a scoped adapter in | ||
| ContextualWisdomLab/naruon#1364. | ||
|
|
||
| This record does not implement Naruon storage. It records the organization | ||
| policy and the fail-closed check that leaf repositories can cite. | ||
|
|
||
| ## Decision | ||
|
|
||
| 1. Treat AWS S3 and S3-compatible HTTPS stores as one `object_storage` | ||
| capability. | ||
| 2. Require HTTPS, exact-host allowlists, no wildcards, and no automatic | ||
| redirects (CWE-918). | ||
| 3. Require explicit private-network trust. Implicit RFC1918 or metadata-service | ||
| access is not authorized by this contract. | ||
| 4. Require least-privilege object permissions and prohibit public ACLs, public | ||
| buckets, and browser-exposed long-lived credentials (CWE-798; CWE-200). | ||
| 5. Require server-side encryption and fail-closed content-length plus SHA-256 | ||
| or stronger read verification (Amazon Web Services, 2024a, 2024b). | ||
| 6. Keep lifecycle states distinct. `consumed` does not imply immediate | ||
| deletion unless the product explicitly configures zero retention. Legal hold | ||
| and archive remain separate from transient reprocessing retention. | ||
| 7. Refuse rollback that deletes customer data after a partial | ||
| migration or backfill. | ||
| 8. Forbid bucket names, object keys, credentials, and raw PII as unbounded | ||
| telemetry labels. This is not a blanket operational PII mask. | ||
| 9. Record CSAP and SOC 2 only as design constraints. The contract must not | ||
| claim certification. | ||
| 10. Ship an executable validator. Prose alone does not close | ||
| ContextualWisdomLab/.github#1019. | ||
|
|
||
| ## Trust boundary | ||
|
|
||
| - Central `.github` owns the schema, policy, doctoring record, and validator. | ||
| - Product repositories own adapters, buckets, keys, credentials, and evidence | ||
| objects. | ||
| - NVIDIA NIM / OpenCode credentials are untouched. | ||
| - The validator never prints bucket names, object keys, or secrets from a | ||
| failing document beyond the field that violated the closed policy. | ||
|
|
||
| ## Verification contract | ||
|
|
||
| `tests/test_object_storage_contract.py` proves the checked-in example passes, | ||
| the schema keys match production constants, and each fail-closed control has a | ||
| unique rejection. `tests/test_object_storage_contract_hardening.py` proves | ||
| nested schema objects stay closed, NaN/Infinity are rejected, exact-host | ||
| allowlists exclude localhost, metadata, IPv4, Unicode, and case aliases, a | ||
| denied private-network policy rejects single-label hosts, custom endpoints | ||
| reject ports above 65535, and malformed observability labels raise policy | ||
| errors instead of TypeError. Local quality remains 100% statement/branch | ||
| coverage and 100% docstrings. | ||
|
|
||
| ## Rollback | ||
|
|
||
| Revert the validator, schema, example, policy, and this record together. Do | ||
| not keep a schema that the executable check no longer enforces. | ||
|
|
||
| ## References (APA 7th) | ||
|
|
||
| Amazon Web Services. (2024a). *Checking object integrity for data uploads in | ||
| Amazon S3*. Amazon Simple Storage Service User Guide. | ||
| https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html | ||
|
|
||
| Amazon Web Services. (2024b). *Using server-side encryption with AWS KMS keys | ||
| (SSE-KMS)*. Amazon Simple Storage Service User Guide. | ||
| https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html | ||
|
|
||
| Amazon Web Services. (n.d.). *Authenticating requests (AWS Signature Version | ||
| 4)*. Amazon Simple Storage Service API Reference. | ||
| https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html | ||
|
|
||
| MITRE. (n.d.-a). *CWE-200: Exposure of sensitive information to an | ||
| unauthorized actor*. CWE List. | ||
| https://cwe.mitre.org/data/definitions/200.html | ||
|
|
||
| MITRE. (n.d.-b). *CWE-798: Use of hard-coded credentials*. CWE List. | ||
| https://cwe.mitre.org/data/definitions/798.html | ||
|
|
||
| MITRE. (n.d.-c). *CWE-918: Server-side request forgery (SSRF)*. CWE List. | ||
| https://cwe.mitre.org/data/definitions/918.html |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| # CWL object-storage contract | ||
|
|
||
| This repository owns the reusable `object_storage` policy and the executable | ||
| check `scripts/ci/validate_object_storage_contract.py`. Product repositories | ||
| keep their own adapters, databases, and evidence objects. Naruon | ||
| [ContextualWisdomLab/naruon#1364](https://github.com/ContextualWisdomLab/naruon/pull/1364) | ||
| is the current concrete consumer and remains the owner of its document-storage | ||
| implementation. | ||
|
|
||
| AWS-managed S3 and S3-compatible HTTPS endpoints are one provider-neutral | ||
| capability. They are not an AWS-only product assumption. | ||
|
|
||
| ## Closed controls | ||
|
|
||
| A contract JSON document is valid only when every control below is true. | ||
|
|
||
| | Control | Required value | | ||
| |---|---| | ||
| | Transport | `https` only | | ||
| | Hosts | exact-host allowlist; no wildcards; no automatic redirects | | ||
| | Private networks | `explicit_allowlist` or `denied`; never implicit RFC1918 access | | ||
| | Credentials | scoped secret registry or workload identity; never broadcast, browser-exposed, or ambient process-wide | | ||
| | Permissions | least privilege; public ACLs and public buckets prohibited | | ||
| | Encryption | server-side encryption `required` | | ||
| | Integrity | content length plus SHA-256 or stronger; fail-closed read verification | | ||
| | Lifecycle | `pending`, `available`, `consumed`, `archived`, and `held` are distinct; `consumed` does not delete unless zero retention is explicit | | ||
| | Rollback | a partial migration must not delete customer data | | ||
| | Names | persisted metadata uses multiword `snake_case` | | ||
| | Telemetry | bucket names, object keys, credentials, and raw PII are forbidden high-cardinality labels | | ||
| | Assurance | CSAP and SOC 2 are design constraints, not certifications | | ||
|
|
||
| Operational PII stays usable through the owning product. Forbidding | ||
| high-cardinality labels is not a blanket PII mask. | ||
|
|
||
| ## Endpoint classes | ||
|
|
||
| 1. **AWS-managed S3 HTTPS endpoints** — `provider_class` is `aws_s3` and every | ||
| host is an exact Amazon S3 regional or dual-stack name. | ||
| 2. **Public S3-compatible HTTPS endpoints** — `provider_class` is | ||
| `s3_compatible` and every host is an exact public DNS name. | ||
| 3. **Authorized private-network endpoints** — `private_network_trust` is | ||
| `explicit_allowlist` and each private host is named. Implicit RFC1918, | ||
| link-local, or metadata-service access is rejected. | ||
|
|
||
| Workload-identity and instance-metadata access require their own SSRF review. | ||
| This contract does not grant that access. | ||
|
|
||
| ## Product next actions | ||
|
|
||
| On success the consumer persists or reads the object under its own schema and | ||
| records a purpose-bound audit event without bucket, key, credential, or raw PII | ||
| labels. On rejection, timeout, duplicate, or partial upload the consumer leaves | ||
| the previous durable object in place. Rollback never deletes customer data | ||
| because a backfill step only partly succeeded. | ||
|
|
||
| ## Verification | ||
|
|
||
| ```bash | ||
| python3 scripts/ci/validate_object_storage_contract.py \ | ||
| --path schemas/examples/cwl-object-storage-v1.example.json | ||
| ``` | ||
|
|
||
| The checked-in example is the naruon-shaped fixture. A product repository may | ||
| commit its own contract file and run the same command in its quality job. The | ||
| issue is not closed by prose alone: the executable check is the acceptance | ||
| evidence. | ||
|
|
||
| ## Schema | ||
|
|
||
| `schemas/cwl-object-storage-v1.schema.json` lists the closed top-level keys. | ||
| The Python validator is authoritative when a JSON Schema library is not | ||
| installed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| { | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "$id": "https://github.com/ContextualWisdomLab/.github/schemas/cwl-object-storage-v1.schema.json", | ||
| "title": "CWL object-storage contract v1", | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": [ | ||
| "schema_version", | ||
| "capability", | ||
| "repository", | ||
| "provider_class", | ||
| "endpoint_policy", | ||
| "credentials", | ||
| "permissions", | ||
| "encryption", | ||
| "integrity", | ||
| "lifecycle", | ||
| "rollback", | ||
| "observability", | ||
| "database_object_names", | ||
| "assurance_posture" | ||
| ], | ||
| "properties": { | ||
| "schema_version": { "const": "1" }, | ||
| "capability": { "const": "object_storage" }, | ||
| "repository": { "type": "string", "minLength": 1 }, | ||
| "provider_class": { "enum": ["aws_s3", "s3_compatible"] }, | ||
| "endpoint_policy": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": [ | ||
| "transport", | ||
| "host_allowlist", | ||
| "allow_wildcards", | ||
| "follow_redirects", | ||
| "private_network_trust" | ||
| ], | ||
| "properties": { | ||
| "transport": { "const": "https" }, | ||
| "host_allowlist": { | ||
| "type": "array", | ||
| "minItems": 1, | ||
| "items": { "type": "string", "minLength": 1 } | ||
| }, | ||
| "allow_wildcards": { "const": false }, | ||
| "follow_redirects": { "const": false }, | ||
| "private_network_trust": { | ||
| "enum": ["explicit_allowlist", "denied"] | ||
| }, | ||
| "custom_endpoint": { "type": "string", "minLength": 1 } | ||
| } | ||
| }, | ||
| "credentials": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": [ | ||
| "broadcast", | ||
| "browser_long_lived", | ||
| "ambient_process_wide", | ||
| "mechanism" | ||
| ], | ||
| "properties": { | ||
| "broadcast": { "const": false }, | ||
| "browser_long_lived": { "const": false }, | ||
| "ambient_process_wide": { "const": false }, | ||
| "mechanism": { | ||
| "enum": ["scoped_secret_registry", "workload_identity"] | ||
| } | ||
| } | ||
| }, | ||
| "permissions": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": ["public_acls", "public_buckets", "least_privilege"], | ||
| "properties": { | ||
| "public_acls": { "const": false }, | ||
| "public_buckets": { "const": false }, | ||
| "least_privilege": { "const": true } | ||
| } | ||
| }, | ||
| "encryption": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": ["server_side"], | ||
| "properties": { | ||
| "server_side": { "const": "required" } | ||
| } | ||
| }, | ||
| "integrity": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": ["content_length", "digest", "fail_closed_read"], | ||
| "properties": { | ||
| "content_length": { "const": true }, | ||
| "digest": { "enum": ["sha256", "sha384", "sha512"] }, | ||
| "fail_closed_read": { "const": true } | ||
| } | ||
| }, | ||
| "lifecycle": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": [ | ||
| "states", | ||
| "consumed_implies_immediate_delete", | ||
| "zero_retention_explicit", | ||
| "legal_hold_distinct" | ||
| ], | ||
| "properties": { | ||
| "states": { | ||
| "type": "array", | ||
| "minItems": 5, | ||
| "items": { "type": "string", "minLength": 1 } | ||
| }, | ||
| "consumed_implies_immediate_delete": { "type": "boolean" }, | ||
| "zero_retention_explicit": { "type": "boolean" }, | ||
|
Contributor
There was a problem hiding this comment. The validator rejects |
||
| "legal_hold_distinct": { "const": true } | ||
| } | ||
| }, | ||
| "rollback": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": ["delete_customer_data_on_partial_migration"], | ||
| "properties": { | ||
| "delete_customer_data_on_partial_migration": { "const": false } | ||
| } | ||
| }, | ||
| "observability": { | ||
| "type": "object", | ||
| "additionalProperties": false, | ||
| "required": ["high_cardinality_labels_forbid"], | ||
| "properties": { | ||
| "high_cardinality_labels_forbid": { | ||
| "type": "array", | ||
| "minItems": 4, | ||
| "items": { "type": "string", "minLength": 1 } | ||
| } | ||
| } | ||
| }, | ||
| "database_object_names": { "const": "multiword_snake_case" }, | ||
| "assurance_posture": { "const": "design_constraints_only" } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| { | ||
| "schema_version": "1", | ||
| "capability": "object_storage", | ||
| "repository": "ContextualWisdomLab/naruon", | ||
| "provider_class": "s3_compatible", | ||
| "endpoint_policy": { | ||
| "transport": "https", | ||
| "host_allowlist": [ | ||
| "s3.ap-northeast-2.amazonaws.com", | ||
| "objects.example.example" | ||
| ], | ||
| "allow_wildcards": false, | ||
| "follow_redirects": false, | ||
| "private_network_trust": "explicit_allowlist", | ||
| "custom_endpoint": "https://objects.example.example" | ||
| }, | ||
| "credentials": { | ||
| "broadcast": false, | ||
| "browser_long_lived": false, | ||
| "ambient_process_wide": false, | ||
| "mechanism": "scoped_secret_registry" | ||
| }, | ||
| "permissions": { | ||
| "public_acls": false, | ||
| "public_buckets": false, | ||
| "least_privilege": true | ||
| }, | ||
| "encryption": { | ||
| "server_side": "required" | ||
| }, | ||
| "integrity": { | ||
| "content_length": true, | ||
| "digest": "sha256", | ||
| "fail_closed_read": true | ||
| }, | ||
| "lifecycle": { | ||
| "states": ["pending", "available", "consumed", "archived", "held"], | ||
| "consumed_implies_immediate_delete": false, | ||
| "zero_retention_explicit": false, | ||
| "legal_hold_distinct": true | ||
| }, | ||
| "rollback": { | ||
| "delete_customer_data_on_partial_migration": false | ||
| }, | ||
| "observability": { | ||
| "high_cardinality_labels_forbid": [ | ||
| "bucket", | ||
| "object_key", | ||
| "credential", | ||
| "raw_pii" | ||
| ] | ||
| }, | ||
| "database_object_names": "multiword_snake_case", | ||
| "assurance_posture": "design_constraints_only" | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.