Skip to content
Draft
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
aa53a8e
feat(plane): tenant-bound ontology/catalog plane above the document KG
cursoragent Aug 18, 2026
b4e6895
fix(plane): close subject-header spoof and harden in-memory store
cursoragent Aug 18, 2026
0a6ff72
feat(plane): persist catalog/ontology plane on SDP_DATABASE_DSN
cursoragent Aug 18, 2026
9c0a0e5
fix(plane): rewrite DEFAULT now() so SQLite can host 0002 tables
cursoragent Aug 18, 2026
0bdd518
test: declare explicit demo subject-header opt-in
seonghobae Aug 18, 2026
667371c
test: reject unverified subject headers by default
seonghobae Aug 18, 2026
fdd5459
fix(auth): disable raw subject headers by default
seonghobae Aug 18, 2026
e2447e2
docs: record fail-closed subject-header default
seonghobae Aug 18, 2026
830a551
docs: make catalog identity trust boundary explicit
seonghobae Aug 18, 2026
bfa409f
docs: record subject-header security boundary
seonghobae Aug 18, 2026
f592e7d
docs: assign corporate-master resolution owner (#83)
seonghobae Aug 26, 2026
684bbe0
docs: keep catalog evidence under CAT-010
seonghobae Aug 26, 2026
e507bff
fix: migrate deprecated framework contracts
seonghobae Aug 26, 2026
ac539f5
Merge deprecation-free runtime contract
seonghobae Aug 26, 2026
d427fff
build: align lock with portal version
seonghobae Aug 26, 2026
ac9fd49
fix: harden catalog plane review boundaries
seonghobae Aug 26, 2026
311668e
fix(authz): honor verified catalog-plane roles
seonghobae Aug 26, 2026
1e22ab3
perf(catalog): batch relational child hydration
seonghobae Sep 9, 2026
3f408ba
fix(auth): centralize oidc policy configuration
seonghobae Sep 9, 2026
851ebff
fix(policy): preserve verified roles across actions
seonghobae Sep 9, 2026
6de1447
fix(auth): keep subject-header guidance configuration-neutral
seonghobae Sep 9, 2026
512c89f
test(auth): reuse oidc configuration callable
seonghobae Sep 9, 2026
1681a7f
fix(security): pin cryptography security release
seonghobae Sep 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Changelog

## Unreleased

### Added

- Accepted Semantic Data Portal as the corporate-master entity-resolution
owner and published the target `sdp.corporate-master-resolution/v1` contract.
The contract preserves distinct `unique`, `miss`, and `tie` outcomes and adds
no matching heuristic, threshold, weight, or executable endpoint claim.

## 0.3.1 — 2026-08-18

### Added

- Buyer-facing ontology/catalog plane above the document KG (`/plane/catalog-objects`,
`/plane/query`) for issue #13: create, list, get, query, document-KG links, and
concept bindings.
- Fail-closed Keyverse binding on `X-CWL-Oidc-Subject` / Bearer JWT,
`X-CWL-Tenant-Reference`, and `X-CWL-Access-Purpose`.
- 3NF migration `0002_ontology_catalog_plane.sql` and ADR 0001 (OWL/RDF/PROV
cited in APA 7th only).
- Catalog/ontology plane persistence: in-memory remains the CI default;
`SDP_DATABASE_DSN` reads and writes the 0002 tables so a paid-pilot restart
keeps glossary/catalog rows.

### Security

- Disabled raw `X-CWL-Oidc-Subject` authentication by default. Local demo and CI
must explicitly set `SDP_ALLOW_UNVERIFIED_SUBJECT_HEADER=true`; production and
paid-pilot deployments use verified Keyverse Bearer JWTs, and JWKS-configured
deployments reject the raw header even when the demo flag is present.
29 changes: 28 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ PYTHONPATH=src uvicorn sdp.api:app --reload
`SDP_SQLITE_PATH=.local/sdp-evidence.sqlite3`를 지정하면 production credential 없이도 같은 evidence store protocol을 로컬 SQLite fallback으로 검증할 수 있습니다. `SDP_DATABASE_URL`이 있으면 Postgres가 SQLite보다 우선합니다.
`SDP_LOG_SINK_URL=file://.local/sdp-requests.jsonl`과 `SDP_REQUEST_ID_HEADER=X-Request-Id`를 지정하면 request id, tenant, actor, route, status, latency, evidence ids만 body 없이 request observation log로 기록됩니다.
REST connector secret은 `SDP_CONNECTOR_SECRET_REF_PREFIX=SDP_CONNECTOR_SECRET_` 기준의 env secret reference로 조회합니다. 예: `SDP_CONNECTOR_SECRET_REST_CONNECTOR_MARKETING_CAMPAIGN_TOKEN` 값은 presence만 검증하며 API 응답에는 노출하지 않습니다.
OIDC token verification은 `SDP_OIDC_ISSUER`, `SDP_OIDC_AUDIENCE`, `SDP_OIDC_JWKS_URL`, `SDP_OIDC_GROUP_ROLE_MAP`를 사용합니다.
OIDC token verification은 `SDP_OIDC_ISSUER`, `SDP_OIDC_AUDIENCE`, `SDP_OIDC_JWKS_URL`, `SDP_OIDC_GROUP_ROLE_MAP`를 사용합니다. Raw `X-CWL-Oidc-Subject` 인증은 기본적으로 거부되며, 외부에 노출되지 않은 local demo 또는 CI에서만 `SDP_ALLOW_UNVERIFIED_SUBJECT_HEADER=true`로 명시적으로 opt-in할 수 있습니다. Paid pilot과 production에서는 이 flag를 사용하지 말고 Keyverse Bearer JWT를 검증해야 합니다.

Docker 기반 로컬 데모는 다음 명령으로 실행합니다.

Expand Down Expand Up @@ -105,6 +105,32 @@ SDP_DATABASE_DSN='postgresql+psycopg://sdp_graph_app:<url-encoded-password>@loca
- `GET /ontology/term/{term}/graph` — 개념 그래프 (그래프 스토어 백엔드)
- `POST /search/semantic` — pgvector KNN 시맨틱 검색 (kind 필터)

### Ontology/catalog plane (문서 KG 위 카탈로그 평면, issue #13)

Keyverse identity + `X-CWL-Tenant-Reference` + `X-CWL-Access-Purpose`가 모두
있어야 하며, tenant가 불일치하면 fail-closed(401/403)입니다. 기본 identity 경로는
`Authorization: Bearer <Keyverse access token>`입니다. Raw `X-CWL-Oidc-Subject`는
JWKS 설정 여부와 무관하게 기본적으로 거부되며, 외부에 노출되지 않은 local demo/CI에서만
`SDP_ALLOW_UNVERIFIED_SUBJECT_HEADER=true`로 opt-in할 수 있습니다. JWKS가 구성된
환경에서는 이 flag가 있어도 raw subject header가 거부됩니다. 지원하지 않는 purpose는
400입니다. 문서 본문·DiskSage ingest/preview·TEPP 채점은 하지 않고, naruon
`content_node` 등 **opaque 참조**만 저장합니다. Steward 표시 이름은 purpose-limited로
그대로 노출되며 마스킹하지 않습니다. CI/pytest 기본은 in-memory이고,
`SDP_DATABASE_DSN`이 있으면 `0002` 3NF 테이블에 저장되어 재시작 후에도
glossary/catalog row가 유지됩니다.

- `POST /plane/catalog-objects` — glossary term / catalog dataset / concept asset 등록
- `GET /plane/catalog-objects` — 동일 tenant 목록
- `GET /plane/catalog-objects/{catalog_object_id}` — 단건 조회
- `POST /plane/catalog-objects/{catalog_object_id}/document-kg-links` — 문서 KG 참조 연결
- `POST /plane/catalog-objects/{catalog_object_id}/concept-bindings` — 온톨로지 개념 연결
- `GET /plane/query?q=...` — title/alias/concept/document-KG id 검색

Production 필수 헤더: `Authorization: Bearer <Keyverse access token>`,
`X-CWL-Tenant-Reference`, `X-CWL-Access-Purpose` (`catalog_browse` |
`glossary_stewardship` | `ontology_query` | `document_kg_alignment`). Local demo/CI의
명시적 opt-in에서만 Bearer 대신 `X-CWL-Oidc-Subject`를 사용할 수 있습니다.

### Catalog / governance / enterprise (기존)

- `GET /health`
Expand Down Expand Up @@ -150,6 +176,7 @@ PYTHONPATH=src python -m sdp.demo_smoke
| PRD/TRD 항목 | 구현 |
|---|---|
| Catalog Service | `src/sdp/catalog.py`, `/catalog/*` |
| Ontology / catalog plane (above doc KG) | `src/sdp/catalog_plane.py`, `src/sdp/catalog_plane_store.py`, `src/sdp/tenant_binding.py`, `/plane/*` (in-memory CI default; `SDP_DATABASE_DSN` → 0002) |
| Ontology / Terminology | `src/sdp/ontology.py`, `/ontology/*` |
| Browse/Query | `src/sdp/browse.py`, `/browse/*` |
| Policy Service | `src/sdp/policy.py`, `/policy/decision` |
Expand Down
72 changes: 72 additions & 0 deletions docs/adr/0001-ontology-catalog-plane.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# ADR 0001 — Ontology/catalog plane above the document knowledge graph

**Status:** Accepted
**Date:** 2026-08-18
**Issue:** ContextualWisdomLab/semantic-data-portal#13

## Context

naruon owns the document knowledge graph (`content_graph`, `project_graph`).
Semantic Data Portal (SDP) is the upper ontology and catalog plane: buyers
browse glossary terms and catalog objects that *resolve* document-KG identity,
they do not store email/file bodies here (ContextualWisdomLab/naruon#974).

DiskSage catalog ingest and preview live in open PRs #59 and #61. This plane
must consume those objects as opaque references, not duplicate the adapters.
Keyverse remains the identity provider; SDP only consumes the OIDC subject and
`X-CWL-Tenant-Reference`.

## Decision

1. Expose a tenant-bound plane (`/plane/catalog-objects`, `/plane/query`) that
supports create, list, get, query, document-KG linking, and concept binding.
2. Fail closed when the Keyverse OIDC subject or `X-CWL-Tenant-Reference` is
missing or when the header does not match the OIDC tenant claim. SDP does
not provision tenants, an IdP, or SCIM.
3. Persist catalog identity in 3NF tables with two-or-more-word `snake_case`
names (`catalog_objects`, `object_definitions`, `object_aliases`,
`document_kg_links`, `concept_object_bindings`, `commons_score_references`,
`object_stewards`). Repeating groups are rows, not JSON blobs.
4. Keep steward display names usable. Access is purpose-limited
(`X-CWL-Access-Purpose`). The plane reuses existing `policy.evaluate()`
(create/search) and records `policy_decision_id`; it does not add a local
GRC policy registry and does not mask PII.
5. Plane contracts live in `sdp_core.catalog_plane` (not `contracts.py`) so the
library layer stays split: demo Dataset contracts vs tenant-bound plane
objects.
6. If a psychometric or commons score is needed, store only an https TEPP or
commons REST pointer. SDP does not compute item scores.
7. Cite OWL, RDF, and PROV as vocabulary references only. Deep doctoring stays
with CWL Researcher.
8. `Authorization: Bearer <Keyverse access token>` is the production identity
path. Raw `X-CWL-Oidc-Subject` authentication is disabled by default even
when JWKS coordinates are absent. A local demo or CI environment may opt in
only by setting `SDP_ALLOW_UNVERIFIED_SUBJECT_HEADER=true`; the code still
rejects that header whenever Keyverse JWKS is configured. Externally
reachable gateways must strip caller-supplied subject headers, and a paid
pilot must not enable the demo flag.

## Standards cited (APA 7th)

Lebo, T., Sahoo, S., & McGuinness, D. (Eds.). (2013). *PROV-O: The PROV
ontology*. World Wide Web Consortium. https://www.w3.org/TR/prov-o/

World Wide Web Consortium. (2012). *OWL 2 web ontology language document
overview* (2nd ed.). https://www.w3.org/TR/owl2-overview/

World Wide Web Consortium. (2014). *RDF 1.1 concepts and abstract syntax*.
https://www.w3.org/TR/rdf11-concepts/

## Consequences

- Buyers can register and browse catalog/ontology objects for one Keyverse
tenant without waiting for DiskSage ingest PRs to merge.
- Document content, people/team/legal-entity KG, weekly lineage reports, and
IRT scoring remain outside this repository.
- Postgres AGE + pgvector remain the graph engine under the plane; the new
tables are the relational catalog identity layer above that engine.
- In-memory is the CI / pytest default. When ``SDP_DATABASE_DSN`` is set the
plane reads and writes the 0002 tables so a paid-pilot restart keeps
glossary and catalog rows. This plane does not use Apache AGE or pgvector.
- Test fixtures must opt into the demo subject-header path explicitly; deleting
the opt-in exercises the same fail-closed behavior as an external deployment.
91 changes: 91 additions & 0 deletions docs/adr/0002-corporate-master-resolution-owner.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# ADR 0002 — Corporate-master entity resolution ownership

**Status:** Accepted target contract
**Date:** 2026-08-26
**Consumers:** ContextualWisdomLab/LineageWeave and other authorized catalog clients

## Context

Consumers need to bind an observed organization label to an authoritative
corporate-master entity. LineageWeave reconstructs record lineage and presents
authorized evidence; it does not own the corporate master, alias lifecycle, or
cross-product entity identity. Keyverse owns account identity, not business
entity master data. TEPP owns temporal and psychometric measurement, and
RankWeave owns retrieval fusion. None of those responsibilities authorizes a
consumer to decide corporate identity locally.

Semantic Data Portal already owns the tenant-scoped ontology, glossary, catalog
identity, aliases, concept bindings, stewardship, and governance plane under
ADR 0001 and PRD ONT-001–ONT-006. Corporate-master resolution is therefore a
catalog identity decision and belongs at this boundary.

## Decision

Semantic Data Portal is the ecosystem owner of corporate-master entity
resolution. It will publish the versioned contract in
[`docs/corporate-master-resolution-api.md`](../corporate-master-resolution-api.md).

The public outcome is exactly one of:

- `unique`: one catalog entity is supported by the accepted owner method;
- `miss`: no catalog entity is supported; or
- `tie`: multiple catalog entities remain equally admissible.

Only `unique` may return a binding. `miss` and `tie` leave the caller's source
label unbound. A tie is not collapsed into a miss, and no catalog row is
created as a side effect of resolution.

The envelope binds the request, tenant, catalog snapshot, method and contract
versions, source evidence references, outcome, candidates where applicable,
limitations, and result digest. The source representation remains caller-owned;
the API does not ingest document bodies or expose cross-tenant aliases.

## Scientific and policy boundary

This decision introduces no matching algorithm, suffix vocabulary, similarity
threshold, score weight, or fallback. An implementation requires a separate
accepted ADR with authoritative evidence, synthetic unique/miss/tie recovery
tests, and an explicit abstention contract. Until then the endpoint is a target
contract and consumers must report resolution as unavailable rather than run a
local heuristic.

TEPP may provide provenance-bearing temporal/event evidence, and RankWeave may
return a versioned ranking artifact, but neither artifact is corporate identity
authority. Semantic Data Portal validates any such reference under the accepted
resolution method and remains the outcome owner.

## Consequences

- LineageWeave can remove local corporate-name similarity and suffix rules once
an implemented, released contract is pinned.
- Other products reuse the same tenant-scoped master identity and preserve the
same unique/miss/tie semantics.
- Catalog stewardship, alias changes, merge/split corrections, and resolution
audit evidence remain in one governance plane.
- The contract does not expand Keyverse, TEPP, RankWeave, or LineageWeave into
corporate-master stores.

## Alternatives considered

1. **Keep resolution in each consumer** — rejected because aliases, ties, and
catalog corrections would diverge across products.
2. **Assign it to Keyverse** — rejected because account authentication and
business-entity master data are different products and trust boundaries.
3. **Assign it to TEPP or RankWeave** — rejected because measurement/ranking
evidence does not own canonical catalog identity.
4. **Create a new repository** — rejected because Semantic Data Portal already
owns the reusable ontology/catalog identity and persistence boundary.

## Rollback

Consumers keep source labels unbound and record the service as unavailable.
They must not restore local heuristic binding or fabricate an `AUTO-*` entity.

## Standards references — APA 7th

Miles, A., & Bechhofer, S. (Eds.). (2009). *SKOS simple knowledge
organization system reference*. World Wide Web Consortium.
https://www.w3.org/TR/skos-reference/

World Wide Web Consortium. (2024). *Data catalog vocabulary (DCAT)—Version
3*. https://www.w3.org/TR/vocab-dcat-3/
86 changes: 86 additions & 0 deletions docs/corporate-master-resolution-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Corporate-master resolution API contract

**Status:** Accepted target; no executable endpoint is claimed
**Contract:** `sdp.corporate-master-resolution/v1`

## Responsibility

Semantic Data Portal resolves an authorized source organization label against
one tenant's governed corporate-master catalog. The caller owns source access
and authorization. The portal owns catalog snapshot identity, aliases,
stewardship state, and the final `unique` / `miss` / `tie` outcome.

## Target resource

```text
POST /api/v1/corporate-master-resolutions
```

### Request envelope

```json
{
"contract_version": "sdp.corporate-master-resolution/v1",
"resolution_request_id": "opaque-request-id",
"tenant_reference": "opaque-tenant-reference",
"source_label": "Synthetic Organization",
"source_language": "en",
"source_evidence_references": ["https://evidence.example.invalid/ref/1"],
"catalog_snapshot_id": "opaque-snapshot-id"
}
```

The production identity and purpose headers remain governed by ADR 0001. The
request is bounded, rejects unknown fields, and carries references rather than
document bodies. The snapshot is explicit; the service never silently switches
to a newer catalog during one resolution.

### Result envelope

```json
{
"contract_version": "sdp.corporate-master-resolution/v1",
"resolution_request_id": "opaque-request-id",
"catalog_snapshot_id": "opaque-snapshot-id",
"resolution_method_id": "accepted-method-id",
"resolution_method_version": "method-version",
"outcome": "tie",
"bound_entity_reference": null,
"candidate_entity_references": ["opaque-entity-a", "opaque-entity-b"],
"source_evidence_references": ["https://evidence.example.invalid/ref/1"],
"limitations": ["multiple_catalog_entities_remain_admissible"],
"result_digest": "sha256:hex-digest"
}
```

## Outcome invariants

| Outcome | Bound entity | Candidate references | Required caller action |
| --- | --- | --- | --- |
| `unique` | exactly one | the same one entity | Use the returned reference with its result digest. |
| `miss` | `null` | empty | Keep the source label unbound and request stewardship if needed. |
| `tie` | `null` | two or more distinct entities | Keep the source label unbound and request disambiguating evidence or stewardship. |

Candidate order is evidence, not a hidden tie-break. No numeric confidence is
part of v1: a later calibrated measure requires its own method contract and may
not change these three outcomes or authorize a binding by itself.

## Failure contract

Malformed, oversized, cross-tenant, unauthorized, stale-snapshot, unknown
method, unimplemented, or unavailable requests fail closed with a stable error
code and a customer-action message. They do not become `miss`, create an entity,
or invoke a consumer-local fallback.

## Implementation acceptance

An executable route is not accepted until a follow-up ADR identifies the
authoritative method and proves, with synthetic fixtures:

1. unique, miss, and tie recovery without an arbitrary threshold;
2. tenant and snapshot isolation;
3. alias merge/split and catalog-revision behavior;
4. exact request/result digest binding and replay behavior;
5. bounded input/output and redacted errors; and
6. consumer contract tests showing unavailable resolution remains unbound.

22 changes: 22 additions & 0 deletions docs/implementation-compliance.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,28 @@
- 핵심 코드: [src/sdp/catalog.py: get_related_datasets](src/sdp/catalog.py), [get_join_candidates](src/sdp/catalog.py)
- 증빙 테스트: `tests/test_api.py::test_join_candidate_endpoint`

### CAT-010 Ontology/catalog plane above the document KG (issue #13)
- 대응: `POST/GET /plane/catalog-objects`, `GET /plane/catalog-objects/{catalog_object_id}`, `POST .../document-kg-links`, `POST .../concept-bindings`, `GET /plane/query`
- 핵심 코드:
- [src/sdp/catalog_plane.py](src/sdp/catalog_plane.py)
- [src/sdp/catalog_plane_store.py](src/sdp/catalog_plane_store.py)
- [src/sdp/tenant_binding.py](src/sdp/tenant_binding.py)
- [src/sdp_core/catalog_plane.py](src/sdp_core/catalog_plane.py)
- [migrations/0002_ontology_catalog_plane.sql](migrations/0002_ontology_catalog_plane.sql)
- 경계: Keyverse OIDC subject + `X-CWL-Tenant-Reference` fail-closed. DiskSage ingest/preview(#59/#61)를 복제하지 않고 opaque document-KG 참조만 저장. 로컬 policy registry/TEPP 채점/tenant 프로비저닝 없음. Steward PII는 purpose-limited로 사용 가능(마스킹 없음).
- ADR: [docs/adr/0001-ontology-catalog-plane.md](docs/adr/0001-ontology-catalog-plane.md)
- 증빙 테스트: `tests/test_catalog_plane.py`, `tests/test_catalog_plane_sql.py`
(SQLite mapping; live DSN in `tests/test_integration_catalog_plane.py`, skip
unless `SDP_DATABASE_DSN` and `catalog_objects` exist)

### ONT-009 Corporate-master resolution ownership

- 상태: accepted target contract; executable endpoint는 아직 구현 증거가 아님.
- Owner ADR: [docs/adr/0002-corporate-master-resolution-owner.md](docs/adr/0002-corporate-master-resolution-owner.md)
- API contract: [docs/corporate-master-resolution-api.md](docs/corporate-master-resolution-api.md)
- 수용 경계: `unique`만 binding; `miss`와 `tie`는 unbound. 구현 전까지
consumer-local heuristic 또는 자동 catalog row 생성 금지.

## 2) Ontology / Terminology

- ONT-001 동의어/다국어: [src/sdp/ontology.py: _build_index, search_concepts]
Expand Down
4 changes: 4 additions & 0 deletions docs/papers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,3 +63,7 @@ are cited with a link + summary, respecting copyright.
| Graph traversal for retrieval | `POST /graph/query`, `GET /ontology/term/{t}/graph` (openCypher) |
| Vector similarity retrieval | `embedding_vectors` (pgvector), `POST /search/semantic` |
| Hybrid graph + vector | single Postgres instance running AGE **and** pgvector |

OWL 2, RDF 1.1, and PROV-O are cited (APA 7th, link + summary only) in
[`docs/adr/0001-ontology-catalog-plane.md`](../adr/0001-ontology-catalog-plane.md).
CWL Researcher owns deeper doctoring; this repo does not attach those W3C PDFs.
Loading
Loading