Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ flowchart LR
| `temporal_expressions.py` | Pure Korean relative-time resolver for Global Ask (ADR 0150) |
| `ask_time_axis.py` | Event-time vs ingestion-time clock choice for that window (ADR 0202) |
| `ontology.py` | Loads the governed Turtle source tree (`lineageweave-kg.ttl` plus generated fragments), the formal OWL 2/RDFS/SKOS vocabulary for the Knowledge Graph's node/edge types, source taxonomies, and published O*NET linkages (ADR 0004, ADR 0252, ADR 0255, ADR 0256) |
| `backend/app/occupation_rating_ingestion.py` | Projects authenticated, bounded occupation-rating source evidence with exact decimal and artifact provenance semantics (ADR 0258) |
| `ontology_neighborhood.py` | Bounded typed ontology/provenance neighborhood (ADR 0184); PostgreSQL stays authoritative, OWL subclass is not an instance edge |
| `ontology_source_cursor.py` | Opaque HMAC source-window continuation (ADR 0124); keyset pagination, never OFFSET |
| `period_report.py` | Fit GRM/GPCM on persisted IRT rows, FIPC-select, EAP-score a period (ADR 0003 slice 3; Bock & Mislevy, 1982) |
Expand Down
12 changes: 10 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,21 @@ All notable changes to this project are documented here. Format follows

### Added

- Authenticated occupation profiles can now read one exact imported
release/source with rating and scale artifact provenance, exact decimal
strings, uncertainty, suppression/relevance flags, and explicit unavailable
versus empty outcomes. The endpoint derives no ranking or recommendation
(ADR 0258).
- O*NET occupation-rating source evidence now has a replay-safe PostgreSQL
contract with normalized release, source-table, scale, occupation, element,
and observation tables. Exact release/source LIST partitions fail closed;
nullable categories remain idempotent identities, divergent duplicates and
truncation fail closed, and no source value is promoted to a local weight or
person score. Task Ratings remain outside this content-model-element store
pending their own normalized identity contract (ADR 0257).
person score. A pinned CSV importer validates rating and scale-reference
digests, row counts, source identities, scale bounds, uncertainty, flags,
and exact update months before immutable transactional insertion. Task
Ratings remain outside this content-model-element store pending their own
normalized identity contract (ADR 0257).
- All eight O*NET 31.0 published linkage tables now contribute 1,417 directed
Ability/Essential Skill/Transferable Skill/Work Style relations to Work
Activities and Work Context. Every direct relation has an exact reified
Expand Down
29 changes: 28 additions & 1 deletion backend/app/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@

import asyncpg
import redis.asyncio as redis
from fastapi import Depends, FastAPI, HTTPException, Query, status
from fastapi import Depends, FastAPI, HTTPException, Path, Query, status
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel

Expand Down Expand Up @@ -98,6 +98,7 @@
upsert_commitment_ticket,
)
from backend.app.operations_dashboard import fetch_operations_dashboard
from backend.app.occupation_rating_ingestion import fetch_occupation_ratings
from backend.app.keyman_ingestion import ingest_post_keymen
from backend.app.knowledge_graph import (
corporate_entity_exists,
Expand Down Expand Up @@ -2275,6 +2276,32 @@ async def read_ontology_neighborhood(
return payload


@app.get("/api/occupations/{onetsoc_code}/ratings")
async def read_occupation_ratings(
onetsoc_code: str = Path(..., pattern=r"^[0-9]{2}-[0-9]{4}\.[0-9]{2}$"),
data_release_code: str = Query(
..., min_length=1, max_length=63, pattern=r"^[a-z0-9][a-z0-9.-]*$"
),
source_table_code: str = Query(
..., min_length=1, max_length=63, pattern=r"^[a-z][a-z0-9_]*$"
),
limit: int = Query(100, ge=1, le=500),
offset: int = Query(0, ge=0, le=10000),
_account: CurrentAccount = Depends(get_current_account),
pool: asyncpg.Pool = Depends(get_pool),
) -> dict[str, object]:
"""Return one authenticated, provenance-bearing occupation source profile."""
async with pool.acquire() as conn:
return await fetch_occupation_ratings(
conn,
data_release_code=data_release_code,
source_table_code=source_table_code,
onetsoc_code=onetsoc_code,
limit=limit,
offset=offset,
)
Comment thread
seonghobae marked this conversation as resolved.


@app.get("/api/posts/{post_id}/counterparties")
async def read_post_counterparties(
post_id: str,
Expand Down
127 changes: 127 additions & 0 deletions backend/app/occupation_rating_ingestion.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
"""Read exact imported occupation ratings without deriving a score or weight."""

from __future__ import annotations

from decimal import Decimal
from typing import Any, Protocol


class RatingReadConnection(Protocol):
"""Small asyncpg-compatible surface used by the rating read projection."""

async def fetchrow(self, query: str, *args: object) -> Any:
"""Return one row or ``None``."""

async def fetch(self, query: str, *args: object) -> list[Any]:
"""Return ordered rows."""


def _decimal_text(value: Decimal | None) -> str | None:
"""Return the exact database decimal representation or honest absence."""
return str(value) if value is not None else None


async def fetch_occupation_ratings(
conn: RatingReadConnection,
*,
data_release_code: str,
source_table_code: str,
onetsoc_code: str,
limit: int,
offset: int,
) -> dict[str, object]:
"""Return one bounded source profile and explicit artifact availability."""
source = await conn.fetchrow(
"""select rating_source.source_table_name,
rating_source.source_artifact_url,
rating_source.source_artifact_sha256,
rating_source.source_row_count,
scale_source.source_artifact_url as scale_artifact_url,
scale_source.source_artifact_sha256 as scale_artifact_sha256,
scale_source.source_row_count as scale_source_row_count
from occupational_source_table rating_source
left join occupational_source_table scale_source
on scale_source.data_release_code = rating_source.data_release_code
and scale_source.source_table_code = 'scales_reference'
where rating_source.data_release_code = $1
and rating_source.source_table_code = $2""",
data_release_code,
source_table_code,
)
if source is None:
return {
"data_release_code": data_release_code,
"source_table_code": source_table_code,
"onetsoc_code": onetsoc_code,
"source_available": False,
"source": None,
"items": [],
"next_offset": None,
}
rows = await conn.fetch(
"""select observation.element_id, element.element_name,
observation.scale_id, scale.scale_name,
scale.minimum_value, scale.maximum_value,
observation.category_value, observation.data_value,
observation.sample_size, observation.standard_error,
observation.lower_ci_bound, observation.upper_ci_bound,
observation.recommend_suppress, observation.not_relevant,
observation.source_updated_month, observation.domain_source_code
from occupational_rating_observation observation
join occupational_element_definition element
on element.data_release_code = observation.data_release_code
and element.element_id = observation.element_id
join occupational_scale_definition scale
on scale.data_release_code = observation.data_release_code
and scale.scale_id = observation.scale_id
where observation.data_release_code = $1
and observation.source_table_code = $2
and observation.onetsoc_code = $3
order by observation.element_id, observation.scale_id,
observation.category_value nulls first
limit $4 offset $5""",
data_release_code,
source_table_code,
onetsoc_code,
limit + 1,
offset,
)
page = rows[:limit]
items = [
{
"element_id": row["element_id"],
"element_name": row["element_name"],
"scale_id": row["scale_id"],
"scale_name": row["scale_name"],
"minimum_value": _decimal_text(row["minimum_value"]),
"maximum_value": _decimal_text(row["maximum_value"]),
"category_value": row["category_value"],
"data_value": _decimal_text(row["data_value"]),
"sample_size": row["sample_size"],
"standard_error": _decimal_text(row["standard_error"]),
"lower_ci_bound": _decimal_text(row["lower_ci_bound"]),
"upper_ci_bound": _decimal_text(row["upper_ci_bound"]),
"recommend_suppress": row["recommend_suppress"],
"not_relevant": row["not_relevant"],
"source_updated_month": row["source_updated_month"],
"domain_source_code": row["domain_source_code"],
}
for row in page
]
return {
"data_release_code": data_release_code,
"source_table_code": source_table_code,
"onetsoc_code": onetsoc_code,
"source_available": True,
"source": {
"source_table_name": source["source_table_name"],
"source_artifact_url": source["source_artifact_url"],
"source_artifact_sha256": source["source_artifact_sha256"],
"source_row_count": source["source_row_count"],
"scale_artifact_url": source["scale_artifact_url"],
"scale_artifact_sha256": source["scale_artifact_sha256"],
"scale_source_row_count": source["scale_source_row_count"],
},
"items": items,
"next_offset": offset + limit if len(rows) > limit else None,
Comment thread
seonghobae marked this conversation as resolved.
}
11 changes: 10 additions & 1 deletion docs/adr/0257-onet-occupation-rating-observation-store.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,22 @@ also violate LineageWeave's externalized-compute boundary.
truncation fail closed. Any later aggregation, comparison,
temporal model, multilevel model, or occupational recommendation belongs to
TEPP/fast-mlsirm or another owning Rust service and must cite these rows.
10. `scripts/import_onet_ratings.py` accepts one caller-pinned official CSV and
Scales Reference file. It verifies both artifact SHA-256 values and row
counts, exact scale names and bounds, reference-name consistency, finite
decimals, Y/N/blank flags, optional Category and Not Relevant columns,
exact `MM/YYYY` source months, and observation-key uniqueness before opening the
target connection. A transaction-scoped advisory lock serializes one
release's partition DDL.

## Consequences

LineageWeave can import the governed public O*NET content-model rating corpus without
manufacturing semantics or embedding large production datasets in git.
Release/source partitions localize hot imports and permit exact detach/archive
operations. A separate API/UI ADR is still required before exposing ratings.
operations. The same pinned artifact is idempotent; a reused release, source,
scale, occupation, or element identity with different source metadata fails
closed. A separate API/UI ADR is still required before exposing ratings.

## References

Expand Down
50 changes: 50 additions & 0 deletions docs/adr/0258-occupation-rating-read-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# ADR 0258: Authenticated occupation-rating source read API

**Status:** Accepted

**Date:** 2026-08-27
**Extends:** ADR 0120, ADR 0184, ADR 0257

## Context

ADR 0257 preserves released occupation-to-element observations, but a database
import alone does not let a product user inspect what a job profile says. A
read contract must distinguish an unimported source from an imported source
with no row for one occupation, preserve low-precision and not-relevant flags,
and avoid presenting a published rating as a local weight or recommendation.

## Decision

1. Add an authenticated, read-only occupation-rating endpoint. O*NET source
observations are licensed public reference data and are not tenant records;
any authenticated LineageWeave account may read an imported artifact.
2. Require exact release, source-table, and O*NET-SOC codes. Return
`source_available=false` when that pinned artifact is not imported; return
`source_available=true` with an empty item list when it is imported but has
no observation for the requested occupation.
3. Return the rating and Scales Reference artifact URLs, SHA-256 values, and
row counts. Every observation retains element/scale identity, declared
bounds, optional category, exact decimal strings, sample/error/interval,
suppression, relevance, source month, and domain source.
4. Order by element, scale, and category and use bounded offset pagination.
Per-occupation source partitions bound this projection; a cursor needs a
later decision only if measured production latency requires it.
5. Do not aggregate, rank, normalize, infer person traits, or recommend an
occupation. Suppressed values remain visible with the suppression flag so a
user can audit the source without mistaking low precision for absence.
6. API and frontend copy describe the evidence and the user's next action,
never importer, partition, model-provider, or orchestration internals.

## Consequences

The semantic layer gains an honest product read boundary without duplicating
psychometric arithmetic. An accessible UI and its Storybook states remain a
separate delivery step after this API has authenticated runtime evidence.

## References

National Center for O*NET Development. (2026). *O*NET 31.0 database* [Data
set]. https://www.onetcenter.org/database.html

PostgreSQL Global Development Group. (2026). *PostgreSQL 18 documentation:
Queries—limit and offset*. https://www.postgresql.org/docs/current/queries-limit.html
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ decision from them.
| [`SOC_2018_HIERARCHY_REFERENCES.md`](../doctoring/SOC_2018_HIERARCHY_REFERENCES.md) | [0252](0252-complete-2018-soc-hierarchy.md) |
| [`ONET_31_LINKAGE_REFERENCES.md`](../doctoring/ONET_31_LINKAGE_REFERENCES.md) | [0256](0256-onet-content-model-published-linkages.md) |
| [`ONET_RATING_STORE_REFERENCES.md`](../doctoring/ONET_RATING_STORE_REFERENCES.md) | [0257](0257-onet-occupation-rating-observation-store.md) |
| Occupation-rating authenticated read projection | [0258](0258-occupation-rating-read-api.md) |

[0011](0011-prov-o-standard-relations.md) and [0065](0065-prov-o-provenance-boundary.md) cite the dated W3C PROV-O and PROV-DM Recommendations (https://www.w3.org/TR/2013/REC-prov-o-20130430/ and https://www.w3.org/TR/2013/REC-prov-dm-20130430/).

Expand Down
24 changes: 20 additions & 4 deletions docs/product-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,10 +140,26 @@ persistence and UI remain unavailable until their separate ADR acceptance.
Ratings remain unavailable until their integer Task IDs and statements have
a separate normalized source-target contract.

Acceptance: the replay-safe migration creates the normalized store; PostgreSQL
integration proves missing partitions fail closed, null-category UPSERT is
idempotent, and invalid uncertainty is rejected. Corpus import, API, UI, and
derived modeling remain unavailable until separate accepted delivery records.
Acceptance: the replay-safe migration creates the normalized store; the pinned
CSV importer validates both rating and scale-reference digests and row counts,
reference identity, source scale, uncertainty, flags, and dates before
persistence; PostgreSQL integration proves missing partitions fail closed and
repeated null-category UPSERT is idempotent.
API, UI, and derived modeling remain unavailable until separate accepted
delivery records.

### PRD-FR-2E — Occupation-rating evidence read

- Let an authenticated user open one exact release/source/occupation profile
with both rating and scale artifact provenance (ADR 0258).
- Distinguish an unavailable imported source from an available source with no
observation for the occupation.
- Preserve exact decimal text, uncertainty, suppression, relevance, source
month, domain source, and declared bounds; derive no ranking or recommendation.

Acceptance: invalid identifiers and unbounded pages are rejected; an unavailable
source never appears as a negative profile; pagination is deterministic; and a
suppressed observation retains its value and warning flag together.

### PRD-FR-3 — Bounded ontology exploration

Expand Down
Loading