diff --git a/CHANGELOG.md b/CHANGELOG.md index c23f6ae..8104012 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ Keep a Changelog, and releases use semantic versioning. ### Added +- A protected root-bootstrap credential transport, rejecting plaintext vault + roots in the ordinary config DB, unsafe file objects and dotenv fallback; + 42 focused regression cases cover the compatibility boundary. +- A scoped CWL Key Vault migration baseline, preserving the historical snapshot + and separating proposed workload-resolution/release gates from implemented + bootstrap repair. - An opt-in namespaced Keyvault foundation with encrypted-at-rest values, atomic secret-change audit records, metadata-only administrator APIs, and a fail-closed boundary for future workload-scoped reads. @@ -59,6 +65,9 @@ Keep a Changelog, and releases use semantic versioning. ### Changed +- Keyvault bootstrap accepts a supervisor-owned file locator instead of a + plaintext passphrase entry. ADR-0014 now remains Proposed until independently + accepted, and operations document controlled migration and residual risks. - Federation PUT and apply now report `applied_to_keycloak: true` only after a fresh live Keycloak identity-provider observation matches the desired observable representation. Keycloak's fixed mask for the known @@ -115,6 +124,8 @@ Keep a Changelog, and releases use semantic versioning. ### Fixed +- Excluded client, operator, registration and vault-root credentials from the + typed configuration's repr; arbitrary settings serialization remains unsafe. - Prevented relying-party inventory from silently accepting a KV key/body identity mismatch, rejected unsafe live or `Location`-derived client UUIDs, and aligned exact client discovery with Keycloak's documented diff --git a/docs/adr/0014-keyverse-keyvault-bounded-context.md b/docs/adr/0014-keyverse-keyvault-bounded-context.md index 14ebb8e..6da412a 100644 --- a/docs/adr/0014-keyverse-keyvault-bounded-context.md +++ b/docs/adr/0014-keyverse-keyvault-bounded-context.md @@ -1,152 +1,184 @@ -# ADR-0014: Keyvault as a separate bounded context from IdP identity/config - -**Status:** Accepted (first slice implemented) -**Date:** 2026-09-02 - -## Context - -The owner asked that Keyverse stop being only a Keycloak-fronting Identity -Provider and also become usable as a Keyvault: a namespaced, -encrypted-at-rest secrets store analogous to Azure Key Vault or HashiCorp -Vault's KV secrets engine, with write/read/delete APIs and audit logging. - -Keyverse already runs one seam that looks superficially similar: -`services/account_unification/app/kv_store.py`'s `idp_config_entries` table -(`KvStore` protocol; `InMemoryKvStore`/`SqliteKvStore`), which -`config.py` reads at startup for this service's *own* operational -configuration (Keycloak URLs, operator tokens, timeouts). Values there are -stored as plain SQLite text, never encrypted — appropriate for -`config.py`'s stated invariant ("nothing here reads process environment"; -everything is this service's own internal, operator-set configuration), but -not a safe foundation for a general-purpose secrets product surface other -CWL services would write arbitrary customer/provider secrets into. - -Identity/authentication and secrets storage are historically separate -concerns even inside mature platforms — Keycloak (an OpenID Connect -Provider) is architecturally distinct from HashiCorp Vault (a secrets -manager), and the two are typically deployed and operated independently. -Domain-Driven Design treats this as a Bounded Context question: two models -that use similar words ("store a value under a key") but serve different -Aggregates, different invariants, and different consumers should not be -collapsed into one undifferentiated model merely because they could share a -table (Evans, 2003, ch. 14; Vernon, 2013, ch. 2–3). - -A genuine motivating first consumer already exists in this ecosystem: -`contextual-orchestrator`'s `credentials.py` documents the exact same "KV, -not env" principle for its own provider API keys (`CredentialBackend` -Protocol; `InMemoryCredentialBackend` default; pgcrypto-encrypted -`PostgresCredentialBackend`). That module's docstring explicitly names -Keyverse-style KV-backed secret resolution as the org reference pattern. -A Keyverse Keyvault is not a feature built for its own sake — it is the -natural next step for that pattern to be centrally operated rather than -reimplemented per repository, and `contextual-orchestrator` could later -swap in a `KeyverseCredentialBackend` implementing its existing -`CredentialBackend` Protocol with no call-site change, exactly as its -pluggable-backend design already anticipates. - -NIST SP 800-57 Part 1 Rev. 5 sets general key-management expectations -(key separation, controlled key lifetime, protecting keys distinct from the -data they protect) that a from-scratch secrets store should satisfy rather -than inventing ad hoc practice (Barker, 2020). OWASP's Application Security -Verification Standard requires application-layer secrets to be encrypted -at rest with keys that are not co-located with the ciphertext under the -same trust boundary (OWASP Foundation, 2021, V6 Cryptography at rest). - -## Decision - -Keyvault is a **separate bounded context** from Keyverse's IdP-facing -modules (Keycloak realm/client management, `relying_party_admin.py`, the -in-flight `authorization_plane.py`/`org_authorization.py` line — see -ADR-0015). It does **not** extend `idp_config_entries` or `kv_store.py`. - -What is shared (deliberately minimal Shared Kernel, per this org's -DDD convention of keeping Shared Kernels small): - -- The **pattern** already proven twice in this service (`kv_store.py`, - `audit.py`): a small `Protocol` plus in-memory and SQLite backends, WAL - journal mode, a 10-second `busy_timeout`, and an append-only trail. - Keyvault keeps each mutation and its audit event in one transaction. -- The `operator_auth_dependency` / `admin_path_security_dependency` - router-level authentication and opaque-path-segment validation already - required for every privileged router in `main.py`. -- The "KV, not env" bootstrap discipline: `keyvault_passphrase` is read - from the *existing* `idp_config_entries` config store (never a raw - environment variable at request time), exactly like every other - `ServiceConfig` field. - -What is genuinely new (Capability #1 of the owner's three-capability -request; #2 service ABAC/RBAC and #3 login credential store are ADR-0015 -and ADR-0016): - -- `app/keyvault.py` — `SqliteKeyvaultStore`/`InMemoryKeyvaultStore` over a - dedicated `keyvault_secrets` table (`secret_namespace`, `secret_key`, - `encrypted_value`, `updated_at`) and a dedicated `keyvault_audit_log` table - (namespace/key/action/actor/`created_at` — deliberately not - `AuditEvent`'s survivor/duplicate-user shape, since a Keyvault write has - no survivor and forcing one schema onto the other would blur two - different Aggregates); `KeyvaultService`, which is the only collaborator - that ever holds plaintext (encryption/decryption happens at this service - boundary with `cryptography.fernet.Fernet`, keyed by PBKDF2-HMAC-SHA256 of - the configured passphrase — the store never sees plaintext and audit events - never contain ciphertext or plaintext). -- `app/keyvault_admin.py` — `PUT`/`DELETE /keyvault/{namespace}/{key}`, - `GET /keyvault/{namespace}` (metadata only: namespace, key, `updated_at` - — **never** a value, so an admin UI can render an inventory without ever - holding plaintext it does not need), and - `GET /keyvault/{namespace}/{key}/audit`. -- Opt-in by construction: `config.py`'s `keyvault_passphrase` defaults to - `None`. `main.py`'s `_build_keyvault_service` returns `None` when unset, - and `keyvault_admin.get_keyvault` then fails closed with **503** ("not - configured"), never a misleading 404 that would suggest the feature - exists but is empty. A namespace here identifies the *consumer* of a - secret (one CWL service or deployment-scoped concern), never an end user - or a Keycloak realm object. - -A dedicated admin *page* for Keyvault (distinct from any general "3 -admin webs" work — see the sibling multi-repo research this ADR's PR -accompanies) is designed but not built in this slice; the API above is the -complete, tested surface it will consume. - -The branch briefly used a bare SHA-256 derivation before this feature reached -protected main. No released database used that pre-release format, so there is -no ciphertext to migrate and no legacy weak-key fallback is admitted. If live -deployment evidence later contradicts that premise, migration must be a -separate recovery change that identifies legacy rows explicitly and rewrites -them once; new rows must never try the legacy derivation. - -## Consequences - -- `idp_config_entries` stays exactly what its own docstring says it is: - this service's internal configuration, never a place other services' - secrets land. -- A wrong `keyvault_passphrase` cannot silently produce garbage: Fernet - authenticates ciphertext (`cryptography.fernet.InvalidToken` on - mismatch), so a passphrase rotation without re-encrypting existing rows - fails loudly rather than returning corrupted plaintext. -- `contextual-orchestrator`'s `CredentialBackend`/`kv_config.ConfigStore` - Protocols are the natural adapter target for a future - `KeyverseCredentialBackend` — noted here as the motivating consumer, not - implemented in this PR (see the "what's left" note in the accompanying - PR description). -- Plaintext retrieval is intentionally absent from the administrator surface. - A consumer adapter cannot ship until Keyverse can verify a signed workload - identity and bind its read scope to exactly one namespace. -- 100% branch coverage and 100% docstring coverage on `app/keyvault.py` - and `app/keyvault_admin.py` (verified: `uv run coverage run --branch - --source=app -m pytest -q && uv run coverage report --fail-under=100`; - `uv run interrogate -v app`), and `uv run ruff check app tests` passes - clean, matching this service's existing gates. - -## References - -Barker, E. (2020). *Recommendation for key management: Part 1 – General* -(NIST SP 800-57 Part 1, Rev. 5). National Institute of Standards and -Technology. https://doi.org/10.6028/NIST.SP.800-57pt1r5 - -Evans, E. (2003). *Domain-driven design: Tackling complexity in the heart -of software*. Addison-Wesley. - -OWASP Foundation. (2021). *OWASP application security verification -standard 4.0.3*. https://owasp.org/www-project-application-security-verification-standard/ +# ADR-0014: Keyverse Key Vault authority and separate secret-lifecycle context + +**Status:** Proposed; foundation and bootstrap repair are unmerged PR work. +**Original date:** 2026-09-02. **Updated:** 2026-09-09. +**Owner line:** PR #129. **Organization rollout:** ContextualWisdomLab/.github#2063. + +## Problem and observed implementation + +The owner requested both a namespaced Key Vault and, on 2026-09-09, removal of +CWL code's dependence on `.env`. Keyverse must provide secret custody and +lifecycle without confusing identity, authorization, ordinary configuration and +product-specific credential meaning. + +PR #129 at `0f10ac556a318c3c3f5ce7eab0802573ecce0c4c` adds +`app/keyvault.py`, `app/keyvault_admin.py`, encrypted SQLite values and atomic +mutation/audit records. Administrator routes expose metadata and write/delete +outcomes, not plaintext reads. The vault is opt-in; absent configuration means +unavailable, not an apparently empty vault. + +The earlier ADR called this Accepted while it was still an open PR. It also +called plaintext internal configuration suitable merely because it was not +read from process environment. Both statements are corrected here: changing +transport does not establish confidentiality, and an unmerged implementation is +not accepted production evidence. + +`idp_config_entries` currently stores ordinary configuration and some legacy +service credentials as text. It is not a general-purpose credential vault. In +particular, the root unlocking credential must not be stored there alongside +configuration backups. Other legacy credentials require their own controlled +migration; this bootstrap repair does not falsely claim to have removed them. + +CO already exposes a `CredentialBackend` port in `credentials.py`. That is a +consumer seam, not justification to copy CO's storage implementation into +Keyverse or to give CO an administrator token. Authorization-plane PR #103 and +ADRs 0015/0016 remain separate owner work. + +## Alternatives + +### Retain dotenv or rename it to a plaintext KV table + +Rejected. Neither a filename nor avoiding `os.getenv` provides workload +identity, narrow authority, rotation, revocation or encryption-key separation. + +### Give every application a direct cloud-vault integration + +Rejected as the default CWL product contract. It duplicates lifecycle and +access semantics across consumers. Deployment-specific KMS/HSM and external +vault providers belong behind Keyverse-owned ports and anti-corruption layers; +consumer business data and provider selection remain in their owning domains. + +### Extend Keycloak's internal credential store as a generic secret database + +Rejected. Keycloak's supported vault integration resolves selected Keycloak +credentials. It does not supply the versioned CWL-wide workload API requested +here. Identity records, password authenticators and application secret values +have different invariants and must not share an undifferentiated model. + +### Keyverse-owned secret context with separate identity/authorization contexts + +Selected direction, pending implementation and independent acceptance. The +existing foundation is preserved, repaired and evolved rather than discarded. +New secret-service/security runtime is Rust; the Python change in this child +is limited to the already-existing bootstrap/config adapter. It does not add a +new Python cryptographic engine or duplicate the future Rust data plane. + +## Decision and responsibility boundary + +Keyverse owns secret references, encrypted custody, secret versions, authorized +resolution, rotation/revocation, leases and access audit. Identity establishes +who a human/workload is; authorization grants narrowly scoped operations; +secret custody performs them. Product consumers retain their domain truth and +what a credential is used for. Billing, ontology, configuration, and LLM +provider/model routing are not moved into the vault. + +The minimal shared kernel consists of value-free reference and verified-context +contracts. Separate persistence and public API boundaries are required. The +foundation's `keyvault_secrets`/`keyvault_audit_log` are not extensions of +`idp_config_entries`, account-merge audit, or Keycloak realm records. + +The administrator surface remains metadata-only on reads. A consumer may not +use an operator session/token, set its own trusted namespace, query another +service's DB, import an open PR's source, or obtain all secrets in a catalog. +A namespace must be bound by verified policy to tenant, environment and workload; +its name alone is not authorization. + +## Implemented bootstrap repair + +The existing configuration loader rejects any plaintext `keyvault_passphrase` +entry. It accepts only `keyvault_passphrase_file`, a non-secret locator resolved +by the existing bootstrap boundary. There is no process-environment, dotenv, +home-directory or plaintext-DB fallback. + +The POSIX reader validates every path component with descriptor-relative, +no-follow opens. The actual opened object must be a root/service-owned regular +file, mode 0400 or 0600, one link, bounded to 4096 bytes, valid UTF-8 and nonempty. +It rejects FIFOs, directories, symlinks, dot segments, invalid ownership and +unsafe permissions, detects changes during reading, and closes all descriptors. +It removes one terminal newline only. OS/decoder failures are converted to a +value-free error. Configuration repr excludes all declared credential fields; +this does not make arbitrary dataclass serialization safe. + +This root-only supervisor transport is an explicit bootstrap exception: the +vault cannot call its own locked service to retrieve its unlock credential. +The locator belongs in configuration, the credential in an independently +protected supervisor mount. A regular private file is not an HSM and does not +protect against host/root compromise or Python string retention. + +Production native runtime must support managed workload identity and external +KMS/HSM envelope-key custody. Standard Kubernetes projected Secret symlinks are +not silently accepted by the compatibility reader; a trusted controller must +provide a private regular-file snapshot or a separately reviewed adapter. + +## Required workload-resolution contract — not yet implemented + +- Verify signature, allowed algorithm, exact issuer/audience, subject, time + claims and workload profile before accepting identity. Do not trust decoded + JWT claims or caller-provided tenant headers. +- Bind secret resolution to tenant/environment, namespace, key, version, + operation and bounded lease. Deny unknown/missing/ambiguous scope. +- GitHub OIDC integrations bind repository/owner IDs, event/ref/environment, + trusted workflow identity and immutable workflow revision. PR/fork source + never receives general provider or administrator credentials. +- Store encrypted values with authenticated context binding; separate wrapping + keys from ciphertext. Parent PBKDF2 fixed salt/value-only encryption is not + the final multi-tenant cryptographic contract. +- Publish durable immutable versions, audited rotation/revocation, idempotent + updates, concurrency control and recoverable migrations. Preserve ciphertext + and key-version relationships through backup/restore and rollback. +- Return no-store responses. Audit denial and permitted access without secret + values; audit-store failure cannot silently become successful resolution. +- A consumer may use a still-valid lease only under its explicit revocation + policy. Expired cache and dotenv fallback are not allowed during outages. +- Emit no credentials into frontend bundles, event buses, logs, traces, + screenshots, command arguments, generated artifacts or model context. + +CO consumes only an immutable owner API release through its existing port. +Provider keys stay inside CO's approved execution boundary; other CWL products +use CO rather than holding those keys. Model-backed Actions retain +`orchestrator/free`; routing and provider discovery stay in CO. + +## Migration and consequences + +The child preserves canonical PR #129 and never force-pushes over another writer. +No consumer is switched before owner release. A private operator must transfer +the existing root value unchanged to the supervisor before removing the legacy +DB entry. Replacing its value without rewrap would break old ciphertext. +Deleting the row is not secure erasure of SQLite pages, WAL or backups. Retire +those copies and rotate/rewrap only under tested recovery procedures. + +No prior bare-SHA key derivation fallback is introduced. The parent's premise +that that format never reached a release must be checked against actual private +deployment evidence before any legacy migration is needed. + +Standalone operation remains an explicit deployment profile, not an invisible +fallback. Failure modes and risk are visible: unavailable authority, expired +lease, wrong key, missing audit store and unsupported bootstrap platform are +errors, not empty inventories or synthetic success. The result adds operational +complexity but avoids a hidden second authority and uncontrolled key fan-out. + +## Verification and acceptance + +The child has observed RED tests and 42 passing focused bootstrap/config tests. +Changed executable-line coverage is 61/61 with no missing changed-line arcs. +Full service coverage, docstrings, locked dependencies, security review and +hosted Checks remain independent gates. These results do not validate the +unimplemented remote API, KMS/HSM, deployment or organization-wide migration. +See the gap baseline and operations guide for exact source blobs and limitations. + +Before promotion: full exact-head suite and 100% production coverage/docstrings, +independent review, protected merge, immutable release, recovery and consumer +contract tests. No claim of SOC 2/CSAP certification or vault completeness is +made from this ADR or from tests of a single bootstrap component. + +## References — APA 7th + +Evans, E. (2003). *Domain-driven design: Tackling complexity in the heart of software*. Addison-Wesley. Vernon, V. (2013). *Implementing domain-driven design*. Addison-Wesley. + +OWASP Foundation. (n.d.). *Secrets management cheat sheet*. https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html + +GitHub. (n.d.). *OpenID Connect reference*. https://docs.github.com/en/actions/reference/security/oidc + +Keycloak. (n.d.). *Using a vault*. https://www.keycloak.org/server/vault diff --git a/docs/doctoring/keyvault_protected_bootstrap.md b/docs/doctoring/keyvault_protected_bootstrap.md new file mode 100644 index 0000000..a0cc84e --- /dev/null +++ b/docs/doctoring/keyvault_protected_bootstrap.md @@ -0,0 +1,66 @@ +# Keyverse protected bootstrap: source and control evidence + +Date: 2026-09-09. Status: focused legacy-adapter repair; no release claim. + +## Primary-source interpretation + +OWASP's secrets-management guidance supports centralized policy, least +privilege, lifecycle/rotation, auditing and explicit bootstrap choices. Avoiding +`.env` by moving the root credential into plaintext configuration does not meet +those goals. The child separates the root locator from its protected transport; +it does not claim that a file provides hardware key custody. + +GitHub's OIDC reference documents workload token claims and reusable-workflow +identity. The future Keyverse verifier must cryptographically validate them and +bind them to approved repository, environment and immutable workflow identity. +This bootstrap child does not implement JWT validation or change Actions tokens. + +Keycloak documents vault providers for supported Keycloak credential settings. +That integration is a downstream adapter concern, not evidence of a complete +CWL secret-management service. + +## Exact evidence + +Parent: `0f10ac556a318c3c3f5ce7eab0802573ecce0c4c` in canonical PR #129. +Implementation: `4caafd0fa56b9ca377c93d78299bfe82dbec8faf`. + +Original bootstrap/config/kv_store blobs were reconstructed exactly and verified +with Git blob hashes. Tests first observed absent protected-reader behavior, +plaintext config acceptance and credential-bearing repr. Additional regressions +caught read-time mutation and false-positive atime detection before repair. + +Reproduction from the full repository: + +```sh +cd services/account_unification +uv sync --locked --extra dev +uv run pytest -q tests/test_keyvault_bootstrap_credentials.py +uv run ruff check app tests tools +uv run interrogate . +uv run coverage run --branch --source=app -m pytest -q +uv run coverage report --show-missing --fail-under=100 +``` + +Only the first focused test command's equivalent was executed in the +reconstructed local environment: 42 passed. Python compilation and diff +whitespace checks passed. Changed executable statements are 61/61 covered; +no missing branch arcs originate on changed executable lines. Full repository +coverage, Ruff, locked installation, independent review and hosted gates must +still execute; historical parent results are not reused for this child. + +## Residual risks and nonclaims + +The Python adapter cannot guarantee secret zeroization. Host/root access can +read mounted credentials. Other legacy configuration credentials are not yet +migrated. Namespace/key context binding, immutable secret versions, workload +reads, revocation, no-store metadata, managed identity/KMS/HSM, durable rewrap and +organization-wide rollout remain in the gap register. Deleting a SQLite row +cannot prove erasure from WAL, pages or backups. + +## References — APA 7th + +OWASP Foundation. (n.d.). *Secrets management cheat sheet*. https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html + +GitHub. (n.d.). *OpenID Connect reference*. https://docs.github.com/en/actions/reference/security/oidc + +Keycloak. (n.d.). *Using a vault*. https://www.keycloak.org/server/vault diff --git a/docs/keyvault_credential_gap_baseline.md b/docs/keyvault_credential_gap_baseline.md new file mode 100644 index 0000000..8837fd8 --- /dev/null +++ b/docs/keyvault_credential_gap_baseline.md @@ -0,0 +1,114 @@ +# Keyverse product and technical gap baseline + +**Scope:** CWL credential-authority migration, 2026-09-09. +**Status:** active-PR implementation and observed gaps; not release acceptance. +**Canonical owner:** `ContextualWisdomLab/keyverse`. +**Organization migration:** [central #2063](https://github.com/ContextualWisdomLab/.github/issues/2063). + +The previous complete 2026-08-21 snapshot is preserved byte-for-byte in +[the historical baseline](product-technical-gap-baseline-2026-08-21.md). +Its PR counts, approvals and Checks are historical evidence, not current gates. +This scoped update does not pretend to have re-audited every identity capability +or every CWL repository. + +## Current evidence and decision + +Canonical vault PR [#129](https://github.com/ContextualWisdomLab/keyverse/pull/129) +was read at `0f10ac556a318c3c3f5ce7eab0802573ecce0c4c`. It has an encrypted +namespaced store and metadata/write administrator APIs. It does not have a +released, signed-workload credential-resolution API. Its authorization sibling +[#103](https://github.com/ContextualWisdomLab/keyverse/pull/103) remains a separate +owner line; no duplicate authorization implementation is introduced here. + +The bounded child implementation at +`4caafd0fa56b9ca377c93d78299bfe82dbec8faf` removes plaintext vault-root loading +from the ordinary config DB, reads a protected supervisor credential, and hides +credential fields from configuration repr. It preserves #129's entire delta. +No consumer is switched to this unreleased branch and no deployed secret is +read, migrated, deleted or rotated by this change. + +Keyverse is to be the CWL secret-lifecycle authority, not merely a replacement +name for a plaintext key/value table. Non-secret configuration remains typed +configuration. Identity, service authorization and secret custody remain +separate bounded contexts within the owner product. + +## Gap register + +| Gap | Evidence / risk | Status and acceptance | +|---|---|---| +| Vault root stored as plaintext configuration | Parent `config.py` reads `keyvault_passphrase` from `idp_config_entries` | Child implementation rejects that entry, accepts only a protected locator; full service/hosted acceptance pending | +| Secret-bearing configuration repr | Parent dataclass includes client/operator/registration/root credentials | Child `field(repr=False)` plus regression tests; arbitrary serialization is still prohibited | +| Safe self-bootstrap | A locked vault cannot obtain its unlock credential through its own API | Explicit root-only supervisor file transport implemented; production KMS/HSM and managed workload identity still missing | +| Cryptographic context separation | Parent fixed installation-wide PBKDF2 salt and value-only ciphertext do not bind tenant/namespace/key/version | Rust envelope-encryption owner work must prove context binding, key separation and cross-context tamper rejection | +| Workload authorization | Administrator access is not consumer read authority | Signature/issuer/audience/subject/tenant/environment/key/version/operation validation and wrong-scope denial required; not implemented here | +| Version and lease lifecycle | Parent overwrites one encrypted value; no immutable secret versions or leases | Add durable version/rotation/revocation/lease records, audit and bounded cache policy | +| Other legacy Keyverse credentials | Client/operator tokens and deployment templates still use historical config/transport | Migrate separately without losing bootstrap/recovery; this PR fixes only vault-root storage and repr | +| Operational confidentiality | Parent review flags metadata caching; backups/WAL may retain retired plaintext root | No-store handling, key rewrap, secure retirement, backup/restore and incident evidence remain mandatory | +| Organization consumers | CO has a CredentialBackend seam; Naruon, LineageWeave and others still expose dotenv entry points in indexed code snapshots | Central #2063 records explicit evidence and owner work; no consumer cutover claimed | +| Release | Feature-head focused tests are not an immutable release | Exact-head full suite, 100% service coverage/docstrings, security checks, independent review, protected merge and release/rollback evidence | + +## Verification evidence + +The environment reconstructed the exact upstream `bootstrap.py`, `config.py` +and `kv_store.py` blobs and confirmed their Git hashes before changing them. +Initial hostile tests observed 36 failures and one pass. Additional read-race +and atime regressions were observed failing before their corresponding fixes. +The final focused suite has **42 passed** with no failures. + +| Published implementation blob | Git blob SHA | +|---|---| +| `services/account_unification/app/bootstrap.py` | `4d368a781f0bfe2fe03f1a9440a1619d9be50d82` | +| `services/account_unification/app/config.py` | `7e45e8e6cc2e127389e912e6d6252c7578b11434` | +| `services/account_unification/tests/test_keyvault_bootstrap_credentials.py` | `c8a51c5423660fd498dca6c2606359d6835ef47c` | + +Coverage intersection with changed executable lines is 61/61, with no missing +changed-line branch arcs. This is NOT a claim of 100% coverage of the complete +account-unification service. Python compilation and `git diff --check` pass. +The complete locked dependency suite, Ruff, hosted security Checks, independent +review, deployment acceptance, KMS/HSM and release have not been verified here. + +## Product contract and migration sequence + +1. Repair and integrate the existing vault foundation through normal reviewed + PRs; never replace a valid predecessor by closing it without delta transfer. +2. Implement the native Rust secret data plane, trusted workload verification, + context-bound storage and versioned lease/rotation/audit contracts in Keyverse. +3. Publish immutable owner artifacts and conformance evidence. Cross-domain + contract metadata may be published through context-graph-contracts; no secret + value or product-owned private schema is copied there. +4. Connect CO through its existing credential port. Other products call CO for + provider operations rather than receiving copies of model-provider keys. +5. For each consumer, prove clean startup without `.env`, access denial, + authority outage, lease expiry/revocation, rotation, recovery and rollback. + Then retire its old dotenv and plaintext credential entry points. +6. Central `.github` uses released AppGuardrail detection and exact-head + inventories. Include unreadable/unscanned repositories in the denominator. + Missing evidence must not be converted into a zero-gap or complete result. + +Configuration endpoint/locator values can be transported by environment where +required by an external runtime; application credentials must not depend on +`.env` loading, home-directory discovery or environment fallback. The root-only +exception is documented in [operations](operations/keyvault.md). An authorized +still-valid lease may be used only within its explicit revocation policy; +expired cache, operator-token reuse and fallback secret stores are forbidden. + +The required runtime sequence is: + +```mermaid +sequenceDiagram + participant W as Consumer workload + participant I as Trusted identity verifier + participant K as Keyverse secret service + participant E as External KMS/HSM + W->>I: Short-lived workload identity + I-->>K: Verified principal and constrained scope + W->>K: Versioned opaque secret reference + K->>K: Tenant/key/operation/lease authorization + K->>E: Authorized envelope-key operation + E-->>K: Protected key operation result + K->>K: Durable audit and version check + K-->>W: Scoped lease or fail-closed error +``` + +This diagram is the target contract; the implemented child is only the legacy +root-bootstrap repair. It is not a working remote secret-service deployment. diff --git a/docs/operations/keyvault.md b/docs/operations/keyvault.md index aa2fc8c..69bfb6a 100644 --- a/docs/operations/keyvault.md +++ b/docs/operations/keyvault.md @@ -1,22 +1,80 @@ # Keyvault operations -Keyvault is opt-in. Configure its database path and passphrase in Keyverse's -existing private configuration store. If the passphrase is absent, every -Keyvault administrator operation fails closed as unavailable. - -Administrators can list secret metadata, set or rotate a value, inspect its -audit history, and delete it. The API never returns plaintext. Use the product -form only for values being created or rotated; after submission, show presence, -last change time, and outcome. Do not echo the submitted value in UI state, -logs, screenshots, notifications, or error text. - -Each set or delete and its audit event commit in one database transaction. On -failure, retry only after checking metadata and audit history. Back up and -restore the Keyvault database as one unit. A wrong passphrase fails -authentication during decryption; it is not an empty vault. - -Noema and contextual-orchestrator keep their current stores until a separate -consumer PR proves all of these together: signed workload identity, one -namespace-bound read scope, denied cross-namespace access, rotation, Keyverse -outage behavior, and rollback. An administrator session is never a substitute -for workload identity. +## Current delivery boundary + +PR #129 is the canonical encrypted-store foundation, not a released workload +credential service. The administrator API exposes metadata and mutation +outcomes, never plaintext. Consumers must not use an operator token as workload +identity, query the Keyverse database, or import this branch as a dependency. + +## Root bootstrap without a plaintext config-store key + +The account-unification compatibility service now accepts the non-secret +`keyvault_passphrase_file` locator in its private config namespace. It rejects +any `keyvault_passphrase` entry, including an empty entry, instead of silently +falling back. The process environment and `.env` are not alternative credential +sources. With no locator the vault remains disabled; an invalid locator aborts +startup rather than making an encrypted vault look empty. + +The supervisor must supply a regular POSIX file with mode `0400` or `0600`, +owned by root or the service's effective UID, and with one hard link. Every +path component must be absolute and free of symlinks, dot segments, and empty +segments. The reader uses descriptor-relative opens, no-follow and nonblocking +flags, rejects FIFOs/directories, bounds content to 4096 bytes, validates UTF-8, +and detects in-place changes while reading. One final LF or CRLF is removed; +other whitespace remains part of the credential. Errors do not echo the path, +raw content, or an underlying decoder/OS diagnostic. All opened descriptors +are closed on both success and failure. + +Use a supervisor-managed, read-only private tmpfs credential mount outside the +repository, image, config DB, application secret DB, logs, and backup of those +DBs. Standard Kubernetes projected Secret symlinks are deliberately NOT accepted +by this compatibility reader: a trusted deployment controller must materialize +a private regular-file snapshot, or use a separately reviewed native adapter. +Do not weaken the reader to follow an arbitrary symlink. A filesystem file is +not an HSM: host/root compromise, crash dumps and Python string retention remain +risks. This change does not claim hardware custody or complete zeroization. + +This root-only exception breaks the self-bootstrap cycle: a vault cannot fetch +the key needed to open itself from its own unavailable API. Production Rust +vault work must replace it with managed workload identity and an external +KMS/HSM envelope-key provider. Ordinary product credentials do not qualify for +this exception. + +## Controlled migration + +Before adopting this unreleased change, stop the affected vault service and +have an authorized operator transfer the existing root credential through a +private administrative channel to the supervisor. Keep its exact value: merely +changing the root key makes existing ciphertext unreadable. Configure only the +new locator, remove the plaintext entry, rehearse decryption and restart against +a private restored copy, then cut over. SQL DELETE alone is not secure erasure: +old SQLite pages, WALs and backups may retain the former value. Retire those +copies under the recovery/retention policy after a tested rewrap/rotation. This +session does not transfer, delete, rotate, or deploy real credentials. + +Rollback must retain the protected bootstrap path; it must not recreate `.env` +or put the root credential back into the ordinary config DB. Actual key +rotation, durable rewrap/rollback and KMS integration remain release gates. + +## Administration and consumer adoption + +Administrators can list metadata, set or rotate a stored value, inspect audit +history and delete it. Never echo submitted values into UI state, logs, +screenshots, notifications, exceptions or exports. Configuration repr excludes +client, operator, registration and vault bootstrap credentials; this is not +permission to serialize configuration with `dataclasses.asdict`. + +Each secret mutation and its audit event commits in one transaction. Back up +and restore that database as one unit. Wrong-key decryption is an error, not an +empty namespace. Root-key storage protection does not fix the parent's +installation-wide KDF salt, ciphertext context binding, or missing immutable +secret-version/lease model; those findings remain explicit in the gap register. + +Noema, contextual-orchestrator and other consumers require signed workload +identity, namespace/key/version-scoped authorization, cross-tenant denial, +rotation/revocation, bounded lease behavior, outage and rollback tests, plus an +immutable Keyverse release before production adoption. A cache may be used only +within its issued lease and revocation policy; no expired-cache or dotenv +fallback is permitted. Application settings which are not secrets stay in typed +configuration, not in the secret-value store. diff --git a/docs/product-technical-gap-baseline-2026-08-21.md b/docs/product-technical-gap-baseline-2026-08-21.md new file mode 100644 index 0000000..9604326 --- /dev/null +++ b/docs/product-technical-gap-baseline-2026-08-21.md @@ -0,0 +1,173 @@ +# Keyverse product and technical gap baseline + +**Evidence snapshot:** 2026-08-21T16:47:10Z (UTC) +**Repository:** `ContextualWisdomLab/keyverse` +**Protected-main head observed:** `ce207dfd42975db61c82a5963e206fc1db14ac2b` +**Status:** live inventory and gap register; not a release acceptance record + +This baseline joins the product, architecture, ADR, standards, operations, +and exact-head GitHub evidence into one executable backlog. It distinguishes +protected-main evidence from open-PR work, accepted contracts, and claims that +remain intentionally unverified. + +## Product and authority boundary + +Keyverse is a standalone and embeddable identity control plane for CWL, Naruon, +and sibling products. It owns passwordless-first Keycloak policy, federation +and directory preflight/reconciliation, account unification, SCIM lifecycle, +relying-party desired state, audit, and safe deployment operations. + +Downstream applications own token signature/issuer/audience validation, +tenant/resource/purpose ABAC, and bounded RBAC. A Keycloak mapper receipt is +issuer-side configuration evidence, never proof that a relying party accepts a +token or enforces authorization. + +## Evidence vocabulary + +| Classification | Meaning | +|---|---| +| `implemented-main` | Source and representative tests are on protected `main`. | +| `active-PR` | Work exists only in an open PR and is not released evidence. | +| `active-issue` | An open issue records a product or operational gap. | +| `accepted-contract` | An ADR or standard defines policy; runtime acceptance may still be absent. | +| `gap-not-claimed` | The repository makes no success claim until stronger evidence exists. | + +Queued, cancelled, skipped-required, stale, predecessor-head, and +rate-limited checks are not successful evidence. Formal approval must bind to +the exact current head and satisfy the latest-pusher and independent-review +rules. + +## Capability and buyer acceptance map + +| Capability | Current maturity | Buyer-visible boundary | +|---|---|---| +| Passwordless local identity | `implemented-main` | Realm validators and tests protect WebAuthn/passwordless policy; live login remains separate evidence. | +| Federation and LDAP preflight | `implemented-main` | Validators are side-effect-free; external bind/discovery and apply remain separate. | +| Account merge and SCIM full replacement | `implemented-main` | Verified identity matching, tombstones, audit, and shared merge/PUT locking are covered on main. | +| SCIM `PATCH active=false` lock parity | `active-PR` | PR #113 is not protected-main evidence until its current head passes all gates and merges. | +| Closed RP mapper profile | `implemented-main` / `accepted-contract` | Canonical `role`, `org`, and `workspace` claims remain closed; consumers must prove their own authorization. | +| Real login and token acceptance | `gap-not-claimed` | No live controlled passwordless browser flow, token exchange, downstream ABAC/RBAC, or revocation acceptance is claimed. | +| Standalone Compose/Helm operation | `implemented-main` / `gap-not-claimed` | Repository validators exist; deployment secret/configuration, rollback, and immutable artifact evidence remain required. | +| Product loop and protected merge | `active-PR` | The scheduler and review path must bind every decision to a current exact head. | +| Release artifact acceptance | `gap-not-claimed` | Version, immutable image digest, SBOM/provenance, rollback, and exact-main regression are still release gates. | + +## Current exact-head PR inventory + +This table was queried from the live GitHub state at the snapshot time. Counts +exclude informational CodeRabbit/Devin contexts and count only CheckRun +success, skipped, or non-terminal results. + +| PR | Scope | Base | Exact head | Checks | Gate / next safe action | +|---:|---|---|---|---|---| +| [#113](https://github.com/ContextualWisdomLab/keyverse/pull/113) | SCIM deactivation shared lock | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `9bd33ee0d00ef1874fd5efabac3462f678a256ed` | 22 success / 8 skipped | `REVIEW_REQUIRED`; obtain exact-head independent approval. | +| [#112](https://github.com/ContextualWisdomLab/keyverse/pull/112) | Account-unification lockfile and stacked contract updates | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `44c2adb18687f8df457bd4bafade551533cee5b9` | 14 queued / 7 skipped | `REVIEW_REQUIRED`; six valid unresolved review threads were observed on this head and are being dispositioned; no approval. | +| [#103](https://github.com/ContextualWisdomLab/keyverse/pull/103) | Hierarchical authorization, login helper, and PATs | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `77b8f4ea9995329f1c55b916d110b460b4bc7649` | 22 success / 8 skipped | `REVIEW_REQUIRED`; retain fail-closed security boundary and obtain current approval. | +| [#101](https://github.com/ContextualWisdomLab/keyverse/pull/101) | Coupled Python dependency updates | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `50dd9c96cab5c230f775685e8baea939fba390dd` | 22 success / 8 skipped | `REVIEW_REQUIRED`; obtain exact-head approval. | +| [#100](https://github.com/ContextualWisdomLab/keyverse/pull/100) | LineageWeave account-derived RP profile | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `2fd5a77cf3765f933debd244f457e13241726929` | 14 queued / 7 skipped | `REVIEW_REQUIRED`; downstream issuer/audience/tenant acceptance remains unclaimed. | +| [#83](https://github.com/ContextualWisdomLab/keyverse/pull/83) | Remove runtime application RPs from portable realm | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `dd1ab7444a75342b42e3af013ccda6d1dbfb359d` | 22 success / 8 skipped | `REVIEW_REQUIRED`; confirm exact-head approval and latest-pusher policy before merge. | + +PR #104 is closed by squash merge at +`44c2adb18687f8df457bd4bafade551533cee5b9`, which advanced the #112 feature +base without changing protected `main`. Its feature-base merge was outside the +default-branch ruleset scope, so it is retained as governance audit evidence, +not as a protected approval or force-merge precedent. + +The central coordination PR [`.github#1203`](https://github.com/ContextualWisdomLab/.github/pull/1203) +is open at exact head `94c09152a843db1a0d3a3463900ef4d30467f085` against +`dd58a88391e44a32fb399f7407f508d8e73cc1c7`; `pip-audit` and `strix` are failed +while the provider outage and shared pip root are remediated. Central #1198 is +at `d2490ad594bd2ab8cccd5ff9e0b6f2a3fa8e23d4` with no failed Checks and its +normal auto-merge armed, but it still lacks required approval. Central #1026 +is at `71c0cc890bd06a0ff97aa10267cb075b02c62f9e` with no failed Checks and +running/queued jobs. None supplies D1–D5 emergency evidence. + +This review update is prepared from exact #112 head `44c2adb`; its successor +will invalidate the table's predecessor Checks and review evidence and must be +re-queried before any merge decision. + +## Open issue inventory + +| Issue | Signal | Classification | Required outcome | +|---:|---|---|---| +| [#114](https://github.com/ContextualWisdomLab/keyverse/issues/114) | MCP-compatible OAuth authorization for headless agents | `active-issue` | Independently review the design, then prove a real resource-bound client flow before runtime implementation. | +| [#102](https://github.com/ContextualWisdomLab/keyverse/issues/102) | Hierarchical authorization plane and PATs | `active-PR` | Prove tenant/resource fail-closed behavior and current-head security review. | +| [#99](https://github.com/ContextualWisdomLab/keyverse/issues/99) | Orphaned federation and product-loop identities | `active-issue` | Preserve the registry recurrence detector and central coordination evidence. | +| [#71](https://github.com/ContextualWisdomLab/keyverse/issues/71) | Remove runtime application RPs from portable import | `active-PR` | Merge #83 only after exact protected evidence. | +| [#2](https://github.com/ContextualWisdomLab/keyverse/issues/2) | Central IdP and external-IdP federation | `accepted-contract` | Complete approved-environment acceptance without weakening preflight boundaries. | + +## Buyer-visible gap order + +### G0 — Protected queue convergence + +The repository must distinguish current, reviewed, passing artifacts from stale +or coupled proposals. The loop is inventory, review disposition, focused fix, +exact-head local and hosted checks, independent approval, protected merge, merge +SHA verification, and re-listing. Never self-approve, force-push, admin-merge, +publish fake status, or reuse predecessor evidence. + +### G1 — Controlled real login and authorization acceptance + +In an approved environment, prove discovery/issuer, JWKS signature and allowed +algorithm, authorization-code + PKCE `S256`, passwordless browser login, token +`iss`/`sub`/`aud`/time claims, logout, tenant/resource ABAC, role/scope RBAC, +cross-tenant denial, and verifier-unavailable fail-closed behavior. An +unavailable issuer stays `unavailable`; it is never replaced with a synthetic +success. + +### G2 — Downstream tenant semantics + +For `lineageweave-web`, `org` is one opaque external tenant key and `workspace` +is one child namespace. Ambiguous or missing membership denies before ABAC/RBAC; +membership changes require a new token or session. Generic tenant claims must +not be added to the closed mapper profile. + +### G3 — SCIM concurrency and database evidence + +After #113, prove real concurrent PATCH/merge behavior on protected main. For +production storage, add PostgreSQL migration/rollback, tenant-qualified +constraints, concentrated-tenant skew measurements, partition/index decisions, +backup/restore, and recovery evidence. Local SQLite tests are not that proof. + +### G4 — MCP resource authorization + +The design-only ADR requires Keycloak authorization code + PKCE, exact redirects, +RFC 8707 resource binding, RFC 9728 protected-resource metadata, RFC 9207 +callback issuer comparison, RFC 9068 JWT validation, revocation, and negative +evidence. Runtime MCP acceptance remains `gap-not-claimed`. + +### G5 — Release and module acceptance + +On exact protected main, complete regression and controlled deployment +acceptance, publish immutable image digest plus SBOM/provenance, and prove +rollback. A green feature PR is not a release. + +## Loop and design boundary + +The hourly PR steward may advance only trusted same-repository PRs with exact +head, independent approval, and required Checks. The hourly product loop may +create at most one bounded draft product-gap PR only after the open queue is +empty and protected-main evidence is healthy. GitHub review/check waiting is not +a reason to stop independent review, documentation, or test design, but queued +results are never promoted to success. + +This repository has no current frontend change in this baseline. Therefore no +Figma file or Storybook inventory is claimed. If a future buyer gap changes a +web surface, its ADR must record the Figma File ID, design tokens, reusable +components, Storybook scene/edge events, and accessibility/interaction/ +performance/responsive/form/navigation/chart acceptance before implementation +is claimed. + +## APA 7th references + +- OpenID Foundation. (2014). *OpenID Connect Core 1.0*. https://openid.net/specs/openid-connect-core-1_0-18.html +- Internet Engineering Task Force. (2020). *JSON Web Token best current practices* (RFC 8725). https://www.rfc-editor.org/rfc/rfc8725.html +- Lodderstedt, T., Bradley, J., Labunets, A., & Fett, D. (2025). *Best current practice for OAuth 2.0 security* (RFC 9700). https://www.rfc-editor.org/rfc/rfc9700.html +- Internet Engineering Task Force. (2018). *OAuth 2.0 authorization server metadata* (RFC 8414). https://doi.org/10.17487/RFC8414 +- Internet Engineering Task Force. (2020). *Resource indicators for OAuth 2.0* (RFC 8707). https://doi.org/10.17487/RFC8707 +- Bertocci, V. (2021). *JSON Web Token (JWT) profile for OAuth 2.0 access tokens* (RFC 9068). https://doi.org/10.17487/RFC9068 +- Meyer zu Selhausen, K., & Fett, D. (2022). *OAuth 2.0 authorization server issuer identification* (RFC 9207). https://doi.org/10.17487/RFC9207 +- Internet Engineering Task Force. (2025). *OAuth 2.0 protected resource metadata* (RFC 9728). https://doi.org/10.17487/RFC9728 +- Model Context Protocol. (2026, July 28). *Authorization*. https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization + +Interpretations and evidence boundaries are maintained in +[`docs/doctoring/product-technical-gap-baseline.md`](doctoring/product-technical-gap-baseline.md). diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md index 9604326..5aca54a 100644 --- a/docs/product-technical-gap-baseline.md +++ b/docs/product-technical-gap-baseline.md @@ -1,173 +1,115 @@ # Keyverse product and technical gap baseline -**Evidence snapshot:** 2026-08-21T16:47:10Z (UTC) -**Repository:** `ContextualWisdomLab/keyverse` -**Protected-main head observed:** `ce207dfd42975db61c82a5963e206fc1db14ac2b` -**Status:** live inventory and gap register; not a release acceptance record - -This baseline joins the product, architecture, ADR, standards, operations, -and exact-head GitHub evidence into one executable backlog. It distinguishes -protected-main evidence from open-PR work, accepted contracts, and claims that -remain intentionally unverified. - -## Product and authority boundary - -Keyverse is a standalone and embeddable identity control plane for CWL, Naruon, -and sibling products. It owns passwordless-first Keycloak policy, federation -and directory preflight/reconciliation, account unification, SCIM lifecycle, -relying-party desired state, audit, and safe deployment operations. - -Downstream applications own token signature/issuer/audience validation, -tenant/resource/purpose ABAC, and bounded RBAC. A Keycloak mapper receipt is -issuer-side configuration evidence, never proof that a relying party accepts a -token or enforces authorization. - -## Evidence vocabulary - -| Classification | Meaning | -|---|---| -| `implemented-main` | Source and representative tests are on protected `main`. | -| `active-PR` | Work exists only in an open PR and is not released evidence. | -| `active-issue` | An open issue records a product or operational gap. | -| `accepted-contract` | An ADR or standard defines policy; runtime acceptance may still be absent. | -| `gap-not-claimed` | The repository makes no success claim until stronger evidence exists. | - -Queued, cancelled, skipped-required, stale, predecessor-head, and -rate-limited checks are not successful evidence. Formal approval must bind to -the exact current head and satisfy the latest-pusher and independent-review -rules. - -## Capability and buyer acceptance map - -| Capability | Current maturity | Buyer-visible boundary | -|---|---|---| -| Passwordless local identity | `implemented-main` | Realm validators and tests protect WebAuthn/passwordless policy; live login remains separate evidence. | -| Federation and LDAP preflight | `implemented-main` | Validators are side-effect-free; external bind/discovery and apply remain separate. | -| Account merge and SCIM full replacement | `implemented-main` | Verified identity matching, tombstones, audit, and shared merge/PUT locking are covered on main. | -| SCIM `PATCH active=false` lock parity | `active-PR` | PR #113 is not protected-main evidence until its current head passes all gates and merges. | -| Closed RP mapper profile | `implemented-main` / `accepted-contract` | Canonical `role`, `org`, and `workspace` claims remain closed; consumers must prove their own authorization. | -| Real login and token acceptance | `gap-not-claimed` | No live controlled passwordless browser flow, token exchange, downstream ABAC/RBAC, or revocation acceptance is claimed. | -| Standalone Compose/Helm operation | `implemented-main` / `gap-not-claimed` | Repository validators exist; deployment secret/configuration, rollback, and immutable artifact evidence remain required. | -| Product loop and protected merge | `active-PR` | The scheduler and review path must bind every decision to a current exact head. | -| Release artifact acceptance | `gap-not-claimed` | Version, immutable image digest, SBOM/provenance, rollback, and exact-main regression are still release gates. | - -## Current exact-head PR inventory - -This table was queried from the live GitHub state at the snapshot time. Counts -exclude informational CodeRabbit/Devin contexts and count only CheckRun -success, skipped, or non-terminal results. - -| PR | Scope | Base | Exact head | Checks | Gate / next safe action | -|---:|---|---|---|---|---| -| [#113](https://github.com/ContextualWisdomLab/keyverse/pull/113) | SCIM deactivation shared lock | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `9bd33ee0d00ef1874fd5efabac3462f678a256ed` | 22 success / 8 skipped | `REVIEW_REQUIRED`; obtain exact-head independent approval. | -| [#112](https://github.com/ContextualWisdomLab/keyverse/pull/112) | Account-unification lockfile and stacked contract updates | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `44c2adb18687f8df457bd4bafade551533cee5b9` | 14 queued / 7 skipped | `REVIEW_REQUIRED`; six valid unresolved review threads were observed on this head and are being dispositioned; no approval. | -| [#103](https://github.com/ContextualWisdomLab/keyverse/pull/103) | Hierarchical authorization, login helper, and PATs | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `77b8f4ea9995329f1c55b916d110b460b4bc7649` | 22 success / 8 skipped | `REVIEW_REQUIRED`; retain fail-closed security boundary and obtain current approval. | -| [#101](https://github.com/ContextualWisdomLab/keyverse/pull/101) | Coupled Python dependency updates | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `50dd9c96cab5c230f775685e8baea939fba390dd` | 22 success / 8 skipped | `REVIEW_REQUIRED`; obtain exact-head approval. | -| [#100](https://github.com/ContextualWisdomLab/keyverse/pull/100) | LineageWeave account-derived RP profile | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `2fd5a77cf3765f933debd244f457e13241726929` | 14 queued / 7 skipped | `REVIEW_REQUIRED`; downstream issuer/audience/tenant acceptance remains unclaimed. | -| [#83](https://github.com/ContextualWisdomLab/keyverse/pull/83) | Remove runtime application RPs from portable realm | `ce207dfd42975db61c82a5963e206fc1db14ac2b` | `dd1ab7444a75342b42e3af013ccda6d1dbfb359d` | 22 success / 8 skipped | `REVIEW_REQUIRED`; confirm exact-head approval and latest-pusher policy before merge. | - -PR #104 is closed by squash merge at -`44c2adb18687f8df457bd4bafade551533cee5b9`, which advanced the #112 feature -base without changing protected `main`. Its feature-base merge was outside the -default-branch ruleset scope, so it is retained as governance audit evidence, -not as a protected approval or force-merge precedent. - -The central coordination PR [`.github#1203`](https://github.com/ContextualWisdomLab/.github/pull/1203) -is open at exact head `94c09152a843db1a0d3a3463900ef4d30467f085` against -`dd58a88391e44a32fb399f7407f508d8e73cc1c7`; `pip-audit` and `strix` are failed -while the provider outage and shared pip root are remediated. Central #1198 is -at `d2490ad594bd2ab8cccd5ff9e0b6f2a3fa8e23d4` with no failed Checks and its -normal auto-merge armed, but it still lacks required approval. Central #1026 -is at `71c0cc890bd06a0ff97aa10267cb075b02c62f9e` with no failed Checks and -running/queued jobs. None supplies D1–D5 emergency evidence. - -This review update is prepared from exact #112 head `44c2adb`; its successor -will invalidate the table's predecessor Checks and review evidence and must be -re-queried before any merge decision. - -## Open issue inventory - -| Issue | Signal | Classification | Required outcome | -|---:|---|---|---| -| [#114](https://github.com/ContextualWisdomLab/keyverse/issues/114) | MCP-compatible OAuth authorization for headless agents | `active-issue` | Independently review the design, then prove a real resource-bound client flow before runtime implementation. | -| [#102](https://github.com/ContextualWisdomLab/keyverse/issues/102) | Hierarchical authorization plane and PATs | `active-PR` | Prove tenant/resource fail-closed behavior and current-head security review. | -| [#99](https://github.com/ContextualWisdomLab/keyverse/issues/99) | Orphaned federation and product-loop identities | `active-issue` | Preserve the registry recurrence detector and central coordination evidence. | -| [#71](https://github.com/ContextualWisdomLab/keyverse/issues/71) | Remove runtime application RPs from portable import | `active-PR` | Merge #83 only after exact protected evidence. | -| [#2](https://github.com/ContextualWisdomLab/keyverse/issues/2) | Central IdP and external-IdP federation | `accepted-contract` | Complete approved-environment acceptance without weakening preflight boundaries. | - -## Buyer-visible gap order - -### G0 — Protected queue convergence - -The repository must distinguish current, reviewed, passing artifacts from stale -or coupled proposals. The loop is inventory, review disposition, focused fix, -exact-head local and hosted checks, independent approval, protected merge, merge -SHA verification, and re-listing. Never self-approve, force-push, admin-merge, -publish fake status, or reuse predecessor evidence. - -### G1 — Controlled real login and authorization acceptance - -In an approved environment, prove discovery/issuer, JWKS signature and allowed -algorithm, authorization-code + PKCE `S256`, passwordless browser login, token -`iss`/`sub`/`aud`/time claims, logout, tenant/resource ABAC, role/scope RBAC, -cross-tenant denial, and verifier-unavailable fail-closed behavior. An -unavailable issuer stays `unavailable`; it is never replaced with a synthetic -success. - -### G2 — Downstream tenant semantics - -For `lineageweave-web`, `org` is one opaque external tenant key and `workspace` -is one child namespace. Ambiguous or missing membership denies before ABAC/RBAC; -membership changes require a new token or session. Generic tenant claims must -not be added to the closed mapper profile. - -### G3 — SCIM concurrency and database evidence - -After #113, prove real concurrent PATCH/merge behavior on protected main. For -production storage, add PostgreSQL migration/rollback, tenant-qualified -constraints, concentrated-tenant skew measurements, partition/index decisions, -backup/restore, and recovery evidence. Local SQLite tests are not that proof. - -### G4 — MCP resource authorization - -The design-only ADR requires Keycloak authorization code + PKCE, exact redirects, -RFC 8707 resource binding, RFC 9728 protected-resource metadata, RFC 9207 -callback issuer comparison, RFC 9068 JWT validation, revocation, and negative -evidence. Runtime MCP acceptance remains `gap-not-claimed`. - -### G5 — Release and module acceptance - -On exact protected main, complete regression and controlled deployment -acceptance, publish immutable image digest plus SBOM/provenance, and prove -rollback. A green feature PR is not a release. - -## Loop and design boundary - -The hourly PR steward may advance only trusted same-repository PRs with exact -head, independent approval, and required Checks. The hourly product loop may -create at most one bounded draft product-gap PR only after the open queue is -empty and protected-main evidence is healthy. GitHub review/check waiting is not -a reason to stop independent review, documentation, or test design, but queued -results are never promoted to success. - -This repository has no current frontend change in this baseline. Therefore no -Figma file or Storybook inventory is claimed. If a future buyer gap changes a -web surface, its ADR must record the Figma File ID, design tokens, reusable -components, Storybook scene/edge events, and accessibility/interaction/ -performance/responsive/form/navigation/chart acceptance before implementation -is claimed. - -## APA 7th references - -- OpenID Foundation. (2014). *OpenID Connect Core 1.0*. https://openid.net/specs/openid-connect-core-1_0-18.html -- Internet Engineering Task Force. (2020). *JSON Web Token best current practices* (RFC 8725). https://www.rfc-editor.org/rfc/rfc8725.html -- Lodderstedt, T., Bradley, J., Labunets, A., & Fett, D. (2025). *Best current practice for OAuth 2.0 security* (RFC 9700). https://www.rfc-editor.org/rfc/rfc9700.html -- Internet Engineering Task Force. (2018). *OAuth 2.0 authorization server metadata* (RFC 8414). https://doi.org/10.17487/RFC8414 -- Internet Engineering Task Force. (2020). *Resource indicators for OAuth 2.0* (RFC 8707). https://doi.org/10.17487/RFC8707 -- Bertocci, V. (2021). *JSON Web Token (JWT) profile for OAuth 2.0 access tokens* (RFC 9068). https://doi.org/10.17487/RFC9068 -- Meyer zu Selhausen, K., & Fett, D. (2022). *OAuth 2.0 authorization server issuer identification* (RFC 9207). https://doi.org/10.17487/RFC9207 -- Internet Engineering Task Force. (2025). *OAuth 2.0 protected resource metadata* (RFC 9728). https://doi.org/10.17487/RFC9728 -- Model Context Protocol. (2026, July 28). *Authorization*. https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization - -Interpretations and evidence boundaries are maintained in -[`docs/doctoring/product-technical-gap-baseline.md`](doctoring/product-technical-gap-baseline.md). +**Updated scope:** CWL Key Vault migration, 2026-09-09. +**Status:** observed evidence and open gaps, not a release acceptance record. +**Organization migration:** ContextualWisdomLab/.github#2063. + +## Evidence map + +The product baseline retains identity, federation, relying-party authorization, +SCIM, operational and release obligations alongside the new secret context. +A scoped vault repair must not remove those continuing acceptance obligations. + +- [Key Vault implementation and gap register](keyvault_credential_gap_baseline.md): + exact parent/source blobs, 42 local focused cases, root bootstrap and remaining + workload/KMS/version/rotation/consumer gates. +- [Historical 2026-08-21 baseline](product-technical-gap-baseline-2026-08-21.md): + preserved byte-for-byte; its PR counts, Checks and approval observations are + historical, not fresh release evidence. +- [Identity and standards doctoring](doctoring/product-technical-gap-baseline.md): + continuing protocol interpretation; it does not certify a running deployment. +- [Vault decision](adr/0014-keyverse-keyvault-bounded-context.md) and + [operations](operations/keyvault.md): Proposed design and controlled migration. + +## Current implementation evidence + +Canonical vault #129 was inspected at +`0f10ac556a318c3c3f5ce7eab0802573ecce0c4c`. Its encrypted store and +metadata/write administrator API are not a released workload read API. +Child #151 implemented protected root bootstrap at +`4caafd0fa56b9ca377c93d78299bfe82dbec8faf`, with documentation at +`bc81fed1c3e9f431bc17c6f24ab36eb518cd2286`. No consumer was switched to this +unmerged branch and no production credentials were inspected or changed. + +The first hosted run for that child was `34366116385`, merge revision +`02408cf12f940a2bd0cb55ccd9d0295b446e819b`. Realm and Compose validation, +locked dependency installation, Ruff, docstrings (100%) and compilation passed. +Documentation contracts failed 1/7 because the scoped rewrite removed explicit +RFC 9068/RFC 9207 obligations from this root index. Full service tests and +coverage were therefore skipped, not passed. This documentation repair restores +those active obligations without weakening or deleting the test. Any new head +requires fresh verification; predecessor successes are not reused as approval. + +## Continuing product gates + +### Identity and relying-party acceptance + +Prove a controlled real login and token lifecycle, not just a Keycloak mapper +receipt. Every relying party verifies exact issuer, signature, allowed algorithm, +audience, subject, expiry, `iat`, exact resource and tenant before applying its +own purpose-bound access-control policy. Keyverse identity, the authorization +plane and secret custody retain separate bounded contexts. Unknown or ambiguous +membership fails closed; a raw header or decoded JWT is never verified identity. + +### MCP protocol contract + +The retained version-pinned baseline is MCP Authorization 2026-07-28; this is +not a claim that a later specification has been re-audited in this vault change. +RFC 9068 defines the JWT access-token profile used by the existing design: +verify the allowed token/header type, claims, signature, issuer, audience and +time bounds. RFC 9207 requires comparison of the authorization-response issuer +before accepting its code; an issuer mismatch rejects the authorization code. + +Retain authorization code plus PKCE S256, exact redirects, RFC 8707 resource +binding, RFC 9728 protected-resource metadata, discovery, revocation and +verifier-unavailable negative tests. A future vault workload verifier must not +weaken this identity boundary or treat a generic administrator token as a +namespace-scoped workload credential. Real end-to-end MCP/resource acceptance +remains unclaimed by the bootstrap repair. + +### Tenant, SCIM and persistent state + +Downstream applications own tenant/resource/purpose enforcement and bounded +RBAC, not merely issuer-side role mappers. Preserve cross-tenant denial, +concurrent SCIM deactivation/merge locking, durable audit, schema compatibility, +PostgreSQL migration/rollback and backup/restore evidence. Local SQLite tests do +not establish those deployment guarantees. + +### Key Vault and consumers + +Keyverse owns secret custody, authorized resolution, versions, leases, +rotation/revocation and audit. Non-secret settings remain typed configuration. +The legacy bootstrap child rejects plaintext config-store roots and dotenv +fallback; its protected supervisor file is a root-only self-bootstrap exception, +not KMS/HSM custody or a second application-secret authority. + +Complete Rust-native workload resolution, cryptographic context binding, +external KMS/HSM, versioned storage and lifecycle/restore tests before releasing +the owner API. Only then may CO use its existing CredentialBackend port and other +consumers retire their dotenv loaders with clean-install, outage, revocation, +rotation and rollback evidence. No sibling DB/source reads or mutable PR-head +runtime dependencies are permitted. + +### Review, operation and release + +Inventory, review, causal RED/fix/GREEN, exact-head checks, independent approval, +protected merge and immutable release remain the order of work. Preserve valid +predecessor delta and active writers. Queued, skipped, cancelled, stale and +rate-limited results are not GREEN evidence. Waiting for review is not permission +to bypass it, nor a reason to stop independent safe repair. + +A release needs immutable artifacts, SBOM/provenance, deployment and recovery +acceptance, current coverage/docstrings and downstream contract verification. +Neither a document, an open PR, a local test count nor an issue is certification +of SOC 2/CSAP, commercial readiness or organization-wide migration completion. + +## Protocol references retained from the governing baseline + +Bertocci, V. (2021). *JSON Web Token (JWT) profile for OAuth 2.0 access tokens* +(RFC 9068). https://doi.org/10.17487/RFC9068 + +Meyer zu Selhausen, K., & Fett, D. (2022). *OAuth 2.0 authorization server issuer +identification* (RFC 9207). https://doi.org/10.17487/RFC9207 + +The source interpretations and remaining runtime limitations stay linked in the +identity and vault doctoring records above. diff --git a/services/account_unification/app/bootstrap.py b/services/account_unification/app/bootstrap.py index d171197..f66460f 100644 --- a/services/account_unification/app/bootstrap.py +++ b/services/account_unification/app/bootstrap.py @@ -1,13 +1,16 @@ """Bootstrap: the single, clearly-marked place an environment variable is read. ``CWL_IDP_BOOTSTRAP`` is *bootstrap transport only* — it names a small YAML file -whose sole job is to point at the real config/secret store (KV or DB). Once the -store is opened, all application config and secrets come from -:mod:`app.kv_store`. No other module reads process environment for config. +that locates the existing configuration store. The vault root credential is +separate: a supervisor supplies a private file, and only its non-secret locator +may be stored in configuration. This legacy bootstrap transport does not make +the ordinary config DB a Key Vault, nor does it implement KMS/HSM custody. """ from __future__ import annotations import os +import stat +from contextlib import ExitStack from dataclasses import dataclass from pathlib import Path @@ -69,3 +72,106 @@ def open_config_store(descriptor: BootstrapDescriptor) -> KvStore: "standalone image; use the sqlite backend or ship a PgKvStore adapter " "with this deployment image." ) + + +MAX_BOOTSTRAP_CREDENTIAL_BYTES = 4096 +_BOOTSTRAP_CREDENTIAL_ERROR = "bootstrap credential is unavailable or unsafe" + + +def _validate_bootstrap_directory(directory_descriptor: int) -> None: + """Reject path components another untrusted principal can replace.""" + directory_state = os.fstat(directory_descriptor) + directory_mode = stat.S_IMODE(directory_state.st_mode) + trusted_owner = directory_state.st_uid in {0, os.geteuid()} + shared_writable = bool(directory_mode & (stat.S_IWGRP | stat.S_IWOTH)) + root_sticky_directory = ( + directory_state.st_uid == 0 and bool(directory_mode & stat.S_ISVTX) + ) + if ( + not stat.S_ISDIR(directory_state.st_mode) + or not trusted_owner + or (shared_writable and not root_sticky_directory) + ): + raise ValueError + + +def read_bootstrap_credential(credential_path: str) -> str: + """Read a supervisor-owned POSIX credential without dotenv or DB fallback. + + All path components are opened relative to held directory descriptors with + no symlink following. Validate directory trust plus file ownership, + permissions, type, and size on the actual descriptors rather than on an + earlier path lookup. Root-owned sticky directories such as ``/tmp`` are an + explicit shared-parent exception because their sticky bit prevents another + unprivileged principal from replacing entries it does not own. The only + allowed plaintext-file exception is Keyverse's own root bootstrap: + application secrets still require the separately released workload API. + """ + try: + required_flags = ("O_NOFOLLOW", "O_DIRECTORY", "O_CLOEXEC", "O_NONBLOCK") + if not all(hasattr(os, flag_name) for flag_name in required_flags): + raise ValueError + if ( + not isinstance(credential_path, str) + or not credential_path.startswith("/") + or "\x00" in credential_path + ): + raise ValueError + path_parts = credential_path.split("/")[1:] + if any(part in {"", ".", ".."} for part in path_parts): + raise ValueError + directory_flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW | os.O_CLOEXEC + file_flags = os.O_RDONLY | os.O_NOFOLLOW | os.O_CLOEXEC | os.O_NONBLOCK + with ExitStack() as descriptor_stack: + parent_descriptor = os.open("/", directory_flags) + descriptor_stack.callback(os.close, parent_descriptor) + _validate_bootstrap_directory(parent_descriptor) + for path_part in path_parts[:-1]: + parent_descriptor = os.open( + path_part, directory_flags, dir_fd=parent_descriptor + ) + descriptor_stack.callback(os.close, parent_descriptor) + _validate_bootstrap_directory(parent_descriptor) + file_descriptor = os.open( + path_parts[-1], file_flags, dir_fd=parent_descriptor + ) + descriptor_stack.callback(os.close, file_descriptor) + file_state = os.fstat(file_descriptor) + if ( + not stat.S_ISREG(file_state.st_mode) + or stat.S_IMODE(file_state.st_mode) not in {0o400, 0o600} + or file_state.st_uid not in {0, os.geteuid()} + or file_state.st_nlink != 1 + or not 0 < file_state.st_size <= MAX_BOOTSTRAP_CREDENTIAL_BYTES + ): + raise ValueError + credential_bytes = bytearray() + while len(credential_bytes) <= MAX_BOOTSTRAP_CREDENTIAL_BYTES: + chunk_bytes = os.read( + file_descriptor, + MAX_BOOTSTRAP_CREDENTIAL_BYTES + 1 - len(credential_bytes), + ) + if not chunk_bytes: + break + credential_bytes.extend(chunk_bytes) + final_state = os.fstat(file_descriptor) + stable_fields = ( + "st_dev", "st_ino", "st_mode", "st_uid", "st_gid", "st_nlink", + "st_size", "st_mtime_ns", "st_ctime_ns", + ) + if len(credential_bytes) != file_state.st_size or any( + getattr(final_state, field_name) != getattr(file_state, field_name) + for field_name in stable_fields + ): + raise ValueError + credential_value = credential_bytes.decode("utf-8") + if credential_value.endswith("\r\n"): + credential_value = credential_value[:-2] + elif credential_value.endswith("\n"): + credential_value = credential_value[:-1] + if not credential_value.strip() or "\x00" in credential_value: + raise ValueError + return credential_value + except (OSError, ValueError): + # OS and decoder diagnostics may contain a private path or raw bytes. + raise RuntimeError(_BOOTSTRAP_CREDENTIAL_ERROR) from None diff --git a/services/account_unification/app/config.py b/services/account_unification/app/config.py index ef78937..7e45e8e 100644 --- a/services/account_unification/app/config.py +++ b/services/account_unification/app/config.py @@ -1,4 +1,4 @@ -"""Typed service configuration, loaded entirely from the KV/DB store. +"""Typed service configuration with a separate protected vault bootstrap. Nothing here reads process environment. :func:`load_service_config` takes an opened :class:`~app.kv_store.KvStore` (from :mod:`app.bootstrap`) and returns a @@ -7,9 +7,10 @@ from __future__ import annotations import math -from dataclasses import dataclass +from dataclasses import dataclass, field from urllib.parse import urlsplit +from .bootstrap import read_bootstrap_credential from .kv_store import KvStore # Keys expected in the KV namespace. Two-word snake_case where they map to @@ -31,6 +32,7 @@ KEY_AUDIT_DATABASE_PATH = "audit_database_path" KEY_KEYVAULT_DATABASE_PATH = "keyvault_database_path" KEY_KEYVAULT_PASSPHRASE = "keyvault_passphrase" +KEY_KEYVAULT_PASSPHRASE_FILE = "keyvault_passphrase_file" MAX_REGISTRATION_ACTION_LIFESPAN_SECONDS = 3600 @@ -44,11 +46,11 @@ class ServiceConfig: keycloak_server_url: str keycloak_realm: str keycloak_client_id: str - keycloak_client_secret: str + keycloak_client_secret: str = field(repr=False) # Privileged and product registration surfaces deliberately use different # bearer credentials so relying products never acquire operator authority. - operator_api_token: str - registration_api_token: str | None = None + operator_api_token: str = field(repr=False) + registration_api_token: str | None = field(default=None, repr=False) registration_client_id: str | None = None registration_redirect_uri: str | None = None registration_action_lifespan_seconds: int = 900 @@ -57,7 +59,7 @@ class ServiceConfig: # keyvault_service=None (503 "not configured"), never a silently-open # secret store. See app/keyvault.py and app/keyvault_admin.py. keyvault_database_path: str = "/var/lib/account-unification/keyvault.db" - keyvault_passphrase: str | None = None + keyvault_passphrase: str | None = field(default=None, repr=False) merge_conflict_policy: str = "survivor_wins" # This is an invariant, not a deployer-selectable feature. The field remains # so audit/config evidence can prove it was explicitly disabled. @@ -185,6 +187,19 @@ def _registration_settings( return client_id, redirect_uri, lifespan_seconds +def _load_keyvault_bootstrap(store: KvStore, namespace: str) -> str | None: + """Resolve only a protected locator; never accept a plaintext DB root key.""" + if store.get(namespace, KEY_KEYVAULT_PASSPHRASE) is not None: + raise RuntimeError( + "plaintext Keyvault bootstrap is forbidden; migrate the root " + "credential outside the config store and configure its file reference" + ) + credential_path = store.get(namespace, KEY_KEYVAULT_PASSPHRASE_FILE) + if credential_path is None: + return None + return read_bootstrap_credential(credential_path) + + def load_service_config(store: KvStore, namespace: str) -> ServiceConfig: """Build and validate the :class:`ServiceConfig` from the KV store.""" operator_api_token = _require(store, namespace, KEY_OPERATOR_API_TOKEN) @@ -246,7 +261,7 @@ def load_service_config(store: KvStore, namespace: str) -> ServiceConfig: store.get(namespace, KEY_KEYVAULT_DATABASE_PATH) or "/var/lib/account-unification/keyvault.db" ), - keyvault_passphrase=store.get(namespace, KEY_KEYVAULT_PASSPHRASE) or None, + keyvault_passphrase=_load_keyvault_bootstrap(store, namespace), merge_conflict_policy=merge_conflict_policy, allow_unverified_email_link=allow_unverified_email_link, request_timeout_seconds=_as_finite_float( diff --git a/services/account_unification/tests/test_keyvault_bootstrap_credentials.py b/services/account_unification/tests/test_keyvault_bootstrap_credentials.py new file mode 100644 index 0000000..c8a51c5 --- /dev/null +++ b/services/account_unification/tests/test_keyvault_bootstrap_credentials.py @@ -0,0 +1,256 @@ +"""Root credential transport rejects plaintext config and unsafe file objects.""" +from __future__ import annotations + +import dataclasses +import os +import traceback +from types import SimpleNamespace + +import pytest + +from app import bootstrap, config +from app.kv_store import InMemoryKvStore + + +@pytest.fixture +def config_store(): + """Seed only the existing account-unification startup requirements.""" + return InMemoryKvStore({"service_config": { + "keycloak_server_url": "https://identity.example.invalid", + "keycloak_realm": "cwl", + "keycloak_client_id": "account_unification", + "keycloak_client_secret": "fixture-client-secret", + "operator_api_token": "fixture-operator-token", + }}) + + +@pytest.fixture +def credential_path(tmp_path): + """Represent a supervisor-provided private credential mount.""" + credential_file = tmp_path / "vault_credential" + credential_file.write_bytes(b"fixture-vault-credential\n") + credential_file.chmod(0o400) + return credential_file + + +def read_credential(credential_path): + """Invoke the existing bootstrap boundary with a deliberately public locator.""" + assert hasattr(bootstrap, "read_bootstrap_credential"), ( + "bootstrap needs a protected-credential reader, not a plaintext DB secret" + ) + return bootstrap.read_bootstrap_credential(str(credential_path)) + + +@pytest.mark.parametrize("raw_secret", ["fixture-legacy-secret", ""]) +def test_plaintext_config_entry_is_rejected(config_store, raw_secret): + """Even an empty legacy entry requires explicit migration rather than fallback.""" + config_store.put("service_config", "keyvault_passphrase", raw_secret) + with pytest.raises(RuntimeError, match="plaintext Keyvault bootstrap"): + config.load_service_config(config_store, "service_config") + + +def test_config_resolves_only_the_protected_reference(config_store, credential_path): + """The typed value reaches the existing vault constructor without persisting it.""" + config_store.put("service_config", "keyvault_passphrase_file", str(credential_path)) + service_config = config.load_service_config(config_store, "service_config") + assert service_config.keyvault_passphrase == "fixture-vault-credential" + assert "fixture-vault-credential" not in str(config_store.get_all("service_config")) + + +def test_legacy_value_cannot_override_a_valid_reference(config_store, credential_path): + """A conflicting root credential has no silent precedence rule.""" + config_store.put("service_config", "keyvault_passphrase_file", str(credential_path)) + config_store.put("service_config", "keyvault_passphrase", "fixture-legacy-secret") + with pytest.raises(RuntimeError, match="plaintext Keyvault bootstrap"): + config.load_service_config(config_store, "service_config") + + +def test_process_environment_is_not_a_credential_fallback(config_store, monkeypatch): + """Scattered environment values cannot silently enable an unconfigured vault.""" + monkeypatch.setenv("KEYVAULT_PASSPHRASE", "fixture-environment-secret") + monkeypatch.setenv("KEYVAULT_PASSPHRASE_FILE", "/ignored/credential") + assert config.load_service_config(config_store, "service_config").keyvault_passphrase is None + + +def test_dataclass_repr_hides_all_credential_fields(config_store): + """Logging the typed config must not leak unrelated bootstrap credentials either.""" + service_config = dataclasses.replace( + config.load_service_config(config_store, "service_config"), + registration_api_token="fixture-registration-token", + keyvault_passphrase="fixture-vault-secret", + ) + rendered = repr(service_config) + for value in ["fixture-client-secret", "fixture-operator-token", + "fixture-registration-token", "fixture-vault-secret"]: + assert value not in rendered + assert "account_unification" in rendered + + +@pytest.mark.parametrize("file_mode", [0o400, 0o600]) +def test_private_regular_file_can_be_read(credential_path, file_mode): + """Read-only and owner-writable supervisor files are both supported.""" + credential_path.chmod(file_mode) + assert read_credential(credential_path) == "fixture-vault-credential" + + +@pytest.mark.parametrize("file_mode", [0o644, 0o640, 0o604, 0o660, 0o444]) +def test_group_or_other_permissions_are_rejected(credential_path, file_mode): + """A readable credential is insufficient when another principal can access it.""" + credential_path.chmod(file_mode) + with pytest.raises(RuntimeError, match="bootstrap credential is unavailable or unsafe"): + read_credential(credential_path) + + +@pytest.mark.parametrize("raw_bytes", [b"", b"\n", b" \t\n", b"invalid\x00secret", b"\xff", b"x" * 4097]) +def test_invalid_credential_content_is_rejected(credential_path, raw_bytes): + """Malformed or oversized credential bytes fail without reflecting content.""" + credential_path.chmod(0o600) + credential_path.write_bytes(raw_bytes) + with pytest.raises(RuntimeError, match="bootstrap credential is unavailable or unsafe"): + read_credential(credential_path) + + +@pytest.mark.parametrize("raw_bytes,expected", [ + (b" exact-spaces ", " exact-spaces "), + (b"exact\r\n", "exact"), + (b"exact\n\n", "exact\n"), + (b"x" * 4096, "x" * 4096), + ("비밀-시험".encode(), "비밀-시험"), +]) +def test_only_one_terminal_newline_is_removed(credential_path, raw_bytes, expected): + """Do not normalize away meaningful credential bytes.""" + credential_path.chmod(0o600) + credential_path.write_bytes(raw_bytes) + assert read_credential(credential_path) == expected + + +def test_symlink_leaf_is_rejected(credential_path, tmp_path): + """Checking and opening the same descriptor prevents leaf-link races.""" + link_path = tmp_path / "credential_link" + link_path.symlink_to(credential_path) + with pytest.raises(RuntimeError): + read_credential(link_path) + + +def test_symlink_parent_is_rejected(credential_path, tmp_path): + """No path component may redirect the bootstrap credential to another tree.""" + link_path = tmp_path / "directory_link" + link_path.symlink_to(tmp_path, target_is_directory=True) + with pytest.raises(RuntimeError): + read_credential(link_path / credential_path.name) + + +def test_hardlinked_file_is_rejected(credential_path, tmp_path): + """The credential cannot also be exposed under an ungoverned second name.""" + link_path = tmp_path / "second_name" + os.link(credential_path, link_path) + with pytest.raises(RuntimeError): + read_credential(credential_path) + + +@pytest.mark.parametrize("unsafe_path", ["relative_secret", "/", "/tmp/../secret", "", "/tmp/secret\x00"]) +def test_invalid_locator_is_rejected(unsafe_path): + """Reject ambiguous path syntax before opening any object.""" + with pytest.raises(RuntimeError): + read_credential(unsafe_path) + + +def test_directory_is_not_a_credential(tmp_path): + """Directories and other non-regular objects cannot enter the key derivation.""" + with pytest.raises(RuntimeError): + read_credential(tmp_path) + + +def test_fifo_is_rejected_without_waiting_for_a_writer(tmp_path): + """O_NONBLOCK permits fstat rejection instead of a blocking FIFO open.""" + fifo_path = tmp_path / "credential_fifo" + os.mkfifo(fifo_path, 0o600) + with pytest.raises(RuntimeError): + read_credential(fifo_path) + + +def test_missing_file_and_parent_fail_without_path_disclosure(tmp_path): + """Operational failures do not echo private path components.""" + private_path = tmp_path / "private_customer_name" / "credential" + with pytest.raises(RuntimeError) as raised_error: + read_credential(private_path) + rendered = "".join(traceback.format_exception(raised_error.value)) + assert "private_customer_name" not in rendered.split("RuntimeError:")[-1] + + +def test_foreign_owner_is_rejected(credential_path, monkeypatch): + """Permission bits alone do not establish the credential owner.""" + real_fstat = os.fstat + def foreign_owner(file_descriptor): + """Vary only ownership in the already-open file metadata.""" + result = real_fstat(file_descriptor) + return SimpleNamespace(st_mode=result.st_mode, st_uid=os.geteuid() + 10001, + st_nlink=result.st_nlink, st_size=result.st_size) + monkeypatch.setattr(os, "fstat", foreign_owner) + with pytest.raises(RuntimeError): + read_credential(credential_path) + + +def test_read_error_closes_descriptors(credential_path, monkeypatch): + """An interrupted read releases all opened descriptors and exposes no raw error.""" + before_count = len(os.listdir("/proc/self/fd")) + def interrupted_read(file_descriptor, byte_count): + """Represent a kernel read failure with a sensitive diagnostic payload.""" + raise OSError("fixture-private-diagnostic") + monkeypatch.setattr(os, "read", interrupted_read) + with pytest.raises(RuntimeError) as raised_error: + read_credential(credential_path) + assert "fixture-private-diagnostic" not in "".join(traceback.format_exception(raised_error.value)) + assert len(os.listdir("/proc/self/fd")) == before_count + + +def test_dotenv_in_working_directory_is_not_loaded(config_store, tmp_path, monkeypatch): + """A local dotenv file cannot silently become Keyverse's root credential source.""" + (tmp_path / ".env").write_text("KEYVAULT_PASSPHRASE=fixture-dotenv-secret\n") + monkeypatch.chdir(tmp_path) + assert config.load_service_config(config_store, "service_config").keyvault_passphrase is None + + +def test_platform_without_no_follow_fails_closed(credential_path, monkeypatch): + """Unsupported platforms need another reviewed transport, not weaker file I/O.""" + monkeypatch.delattr(os, "O_NOFOLLOW") + with pytest.raises(RuntimeError): + read_credential(credential_path) + + +def test_file_growth_after_open_is_rejected(credential_path, monkeypatch): + """The reader detects a file that grows past the maximum after fstat.""" + real_read = os.read + def growing_read(file_descriptor, byte_count): + """Model growth between fstat and read without allocating unbounded data.""" + if byte_count == 4097: + return b"x" * byte_count + return real_read(file_descriptor, byte_count) + monkeypatch.setattr(os, "read", growing_read) + with pytest.raises(RuntimeError): + read_credential(credential_path) + + +def test_non_atomic_same_size_rewrite_is_rejected(credential_path, monkeypatch): + """An in-place rotation cannot produce a mixed but plausible credential.""" + real_read = os.read + did_replace = False + def replaced_read(file_descriptor, byte_count): + """Represent a supervisor incorrectly rewriting instead of replacing.""" + nonlocal did_replace + result = real_read(file_descriptor, byte_count) + if not did_replace: + did_replace = True + old_time = credential_path.stat().st_mtime_ns + os.utime(credential_path, ns=(old_time + 1_000_000, old_time + 1_000_000)) + return result + monkeypatch.setattr(os, "read", replaced_read) + with pytest.raises(RuntimeError): + read_credential(credential_path) + + +def test_access_time_update_is_not_mistaken_for_credential_rotation(credential_path): + """Reading a file may update atime without changing its protected content.""" + current_modified = credential_path.stat().st_mtime_ns + os.utime(credential_path, ns=(0, current_modified)) + assert read_credential(credential_path) == "fixture-vault-credential" diff --git a/services/account_unification/tests/test_keyvault_bootstrap_parent_permissions.py b/services/account_unification/tests/test_keyvault_bootstrap_parent_permissions.py new file mode 100644 index 0000000..6b5354a --- /dev/null +++ b/services/account_unification/tests/test_keyvault_bootstrap_parent_permissions.py @@ -0,0 +1,56 @@ +"""Parent-directory trust tests for the vault root bootstrap path.""" +from __future__ import annotations + +import os +import stat + +import pytest + +from app.bootstrap import read_bootstrap_credential + + +def _private_credential(directory_path): + """Create one private regular credential below the supplied directory.""" + credential_path = directory_path / "vault_credential" + credential_path.write_text("fixture-vault-credential\n", encoding="utf-8") + credential_path.chmod(0o400) + return credential_path + + +def test_group_writable_parent_directory_is_rejected(tmp_path): + """A group member must not be able to replace a trusted path component.""" + credential_path = _private_credential(tmp_path) + tmp_path.chmod(0o770) + + with pytest.raises(RuntimeError, match="bootstrap credential is unavailable or unsafe"): + read_bootstrap_credential(str(credential_path)) + + +def test_world_writable_parent_directory_is_rejected(tmp_path): + """World-writable application-owned directories are not trusted bootstrap roots.""" + credential_path = _private_credential(tmp_path) + tmp_path.chmod(0o777) + + with pytest.raises(RuntimeError, match="bootstrap credential is unavailable or unsafe"): + read_bootstrap_credential(str(credential_path)) + + +def test_owner_private_parent_directory_remains_supported(tmp_path): + """An owner-controlled directory retains the intended standalone bootstrap path.""" + credential_path = _private_credential(tmp_path) + tmp_path.chmod(0o700) + + assert read_bootstrap_credential(str(credential_path)) == "fixture-vault-credential" + + +def test_root_sticky_tmp_component_does_not_break_private_descendant(tmp_path): + """A root-owned sticky /tmp ancestor is allowed when descendants are private.""" + if not str(tmp_path).startswith("/tmp/"): + pytest.skip("runner temporary directory is not below /tmp") + tmp_state = os.stat("/tmp") + if tmp_state.st_uid != 0 or not stat.S_ISVTX & tmp_state.st_mode: + pytest.skip("runner /tmp is not the root-owned sticky-directory profile") + credential_path = _private_credential(tmp_path) + tmp_path.chmod(0o700) + + assert read_bootstrap_credential(str(credential_path)) == "fixture-vault-credential" diff --git a/services/account_unification/tests/test_keyvault_sqlite_namespace_lifecycle.py b/services/account_unification/tests/test_keyvault_sqlite_namespace_lifecycle.py new file mode 100644 index 0000000..93a4c9e --- /dev/null +++ b/services/account_unification/tests/test_keyvault_sqlite_namespace_lifecycle.py @@ -0,0 +1,51 @@ +"""Durable namespace inventory follows actual encrypted-secret lifecycle.""" +from contextlib import closing + +from cryptography.fernet import Fernet + +from app.keyvault import KeyvaultService, SqliteKeyvaultStore + + +def test_empty_sqlite_namespace_inventory(tmp_path): + """An initialized database reports no namespaces until a secret exists.""" + database_path = str(tmp_path / "namespace_inventory.db") + with closing(SqliteKeyvaultStore(database_path)) as secret_store: + assert secret_store.list_namespaces() == [] + + +def test_sqlite_namespace_inventory_is_unique_sorted_and_value_free(tmp_path): + """Multiple keys and a replacement value cannot duplicate a namespace.""" + database_path = str(tmp_path / "namespace_inventory.db") + with closing(SqliteKeyvaultStore(database_path)) as secret_store: + secret_service = KeyvaultService(secret_store, Fernet.generate_key()) + secret_service.put_secret("zeta_service", "first_key", "fixture-first-value", actor="test_operator") + secret_service.put_secret("alpha_service", "first_key", "fixture-second-value", actor="test_operator") + secret_service.put_secret("zeta_service", "second_key", "fixture-third-value", actor="test_operator") + secret_service.put_secret("zeta_service", "first_key", "fixture-replacement-value", actor="test_operator") + assert secret_service.list_namespaces() == ["alpha_service", "zeta_service"] + assert b"fixture-replacement-value" not in secret_store.get("zeta_service", "first_key") + assert secret_service.get_secret("zeta_service", "first_key", actor="test_reader") == "fixture-replacement-value" + + +def test_sqlite_namespace_inventory_survives_restart_and_last_key_deletion(tmp_path): + """Retired namespaces disappear while deletion audit remains durable.""" + database_path = str(tmp_path / "namespace_inventory.db") + encryption_key = Fernet.generate_key() + with closing(SqliteKeyvaultStore(database_path)) as secret_store: + secret_service = KeyvaultService(secret_store, encryption_key) + secret_service.put_secret("consumer_service", "first_key", "fixture-first-value", actor="test_operator") + secret_service.put_secret("consumer_service", "second_key", "fixture-second-value", actor="test_operator") + assert secret_service.list_namespaces() == ["consumer_service"] + + with closing(SqliteKeyvaultStore(database_path)) as secret_store: + secret_service = KeyvaultService(secret_store, encryption_key) + assert secret_service.list_namespaces() == ["consumer_service"] + assert secret_service.get_secret("consumer_service", "first_key", actor="test_reader") == "fixture-first-value" + secret_service.delete_secret("consumer_service", "first_key", actor="test_operator") + assert secret_service.list_namespaces() == ["consumer_service"] + secret_service.delete_secret("consumer_service", "second_key", actor="test_operator") + assert secret_service.list_namespaces() == [] + + with closing(SqliteKeyvaultStore(database_path)) as secret_store: + assert secret_store.list_namespaces() == [] + assert [event["action"] for event in secret_store.events_for("consumer_service", "second_key")] == ["secret_set", "secret_deleted"]