Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
bc53272
test(lineage): define dynamic evaluation provenance RED
seonghobae Sep 2, 2026
3517991
test(lineage): harden dynamic evaluation projection RED boundaries
seonghobae Sep 2, 2026
1ff49a8
feat(lineage): project dynamic evaluation provenance GREEN
seonghobae Sep 2, 2026
873f700
test(api): require dynamic evaluation lineage exports RED
seonghobae Sep 2, 2026
09adf1c
feat(api): export dynamic evaluation lineage GREEN
seonghobae Sep 2, 2026
48fd949
docs(adr): govern dynamic evaluation lineage projection
seonghobae Sep 2, 2026
66fe4f7
docs(domain): add dynamic evaluation ubiquitous language
seonghobae Sep 2, 2026
3e1cf55
docs(changelog): record dynamic evaluation lineage projection
seonghobae Sep 2, 2026
b52bcc9
test(lineage): reject oversized run payload before item decoding
seonghobae Sep 2, 2026
3b76d0e
fix(lineage): enforce run payload budget before decoding
seonghobae Sep 2, 2026
dd739ca
test(lineage): reject collapsed adjudication identity
seonghobae Sep 2, 2026
91da3a2
fix(lineage): keep adjudication identities distinct
seonghobae Sep 2, 2026
fc73f70
test(lineage): reject invisible provenance reference controls
seonghobae Sep 2, 2026
8fbb21a
fix(lineage): reject invisible Unicode format controls
seonghobae Sep 2, 2026
bdf00db
docs(adr): record Unicode provenance-reference boundary
seonghobae Sep 2, 2026
fd514e8
docs(changelog): note Unicode reference hardening
seonghobae Sep 2, 2026
e4b5de6
test(lineage): require substantive criteria before evaluation
seonghobae Sep 2, 2026
8cfbd10
feat(lineage): model substantive evaluation criteria
seonghobae Sep 2, 2026
284d497
fix(lineage): keep criterion meaning source-text-free
seonghobae Sep 2, 2026
b497b20
fix(lineage): bind dynamic items to substantive criteria
seonghobae Sep 2, 2026
c17ffce
test(lineage): reject supersession cycles
seonghobae Sep 2, 2026
96ade92
fix(lineage): reject supersession cycles
seonghobae Sep 2, 2026
c51cd56
test(lineage): bound supersession admission cost
seonghobae Sep 2, 2026
fa6711d
fix(lineage): bound supersession graph traversal
seonghobae Sep 2, 2026
1b7820e
test(lineage): make supersession cost regression deterministic
seonghobae Sep 2, 2026
ad354c3
test(lineage): count supersession walk operations
seonghobae Sep 2, 2026
b9d092b
docs(lineage): record bounded supersession admission
seonghobae Sep 2, 2026
f68441d
docs(adr): move dynamic evaluation lineage to ADR 0355
seonghobae Sep 2, 2026
c95e9af
docs(adr): release conflicting ADR 0352 allocation
seonghobae Sep 2, 2026
e33ad56
docs(adr): align dynamic lineage with criterion contract
seonghobae Sep 2, 2026
6e83d1f
test(lineage): reject collapsed anchor evidence identity
seonghobae Sep 2, 2026
1061fb3
fix(lineage): keep anchor evidence roles distinct
seonghobae Sep 2, 2026
8c73220
test(lineage): reject linking evidence identity collisions
seonghobae Sep 2, 2026
9683ca6
fix(lineage): separate linking from anchor evidence identity
seonghobae Sep 2, 2026
94b8852
test(lineage): stop implying an unreleased owner contract
seonghobae Sep 2, 2026
493476b
docs(lineage): align evidence identity and released-owner boundaries
seonghobae Sep 2, 2026
9a78c0a
test(lineage): reject falsy malformed criterion reference payloads
seonghobae Sep 2, 2026
3066c12
fix(lineage): type-check criterion references before binding inference
seonghobae Sep 2, 2026
05ce15a
test(lineage): remove nonexistent fast-mlsirm contract fixture
seonghobae Sep 2, 2026
79c1c32
test(lineage): reject Unicode line separators in provenance refs
seonghobae Sep 2, 2026
290da3a
fix(lineage): reject Unicode separators in provenance refs
seonghobae Sep 2, 2026
9a72be3
docs(lineage): record Unicode separator admission boundary
seonghobae Sep 2, 2026
b83cc0f
docs(adr): define Unicode separator reference boundary
seonghobae Sep 2, 2026
6235834
test(lineage): reject Unicode noncharacters in provenance refs
seonghobae Sep 2, 2026
0e1a413
fix(lineage): reject Unicode noncharacters in provenance refs
seonghobae Sep 2, 2026
e977ba3
docs(lineage): record provenance noncharacter boundary
seonghobae Sep 2, 2026
958e68d
docs(adr): define noncharacter provenance admission
seonghobae Sep 2, 2026
7be85d7
chore(lineage): restack dynamic evaluation on current policy parent
seonghobae Sep 3, 2026
d973701
chore(lineage): converge dynamic evaluation on current policy parent
seonghobae Sep 3, 2026
8c74712
chore(stack): converge dynamic evaluation on moved policy parent
seonghobae Sep 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.d/dynamic-evaluation-lineage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
## Added

- Added the versioned `lineageweave_dynamic_evaluation_lineage/v1` projection for dynamically resolved evaluation item and run snapshots.
- Preserved generator, rater, adjudication-case/resolution, calibration, anchor-promotion, linking, and supersession references as separate immutable evidence instead of overwriting source observations or inventing decision authority.
- Permitted zero-anchor cold-start and within-run projections while requiring separate calibration, promotion, and linking evidence before an item/run can be represented as an anchor or cross-version linked.
- Rejected provider credentials/endpoints, scores, embedded adjudication decisions, mixed-blueprint item sets, duplicate identities, unsupported linking claims, Unicode format controls, Unicode line/paragraph separators, and the 66 Unicode noncharacters in opaque provenance references at the LineageWeave Anti-Corruption Layer. Other unassigned `Cn` code points are not rejected merely for being unassigned.
- Rejected directed cycles among in-run supersession references and bounded cycle admission to linear graph work at the 10,000-item contract limit without claiming a runner-specific wall-clock SLO.
312 changes: 312 additions & 0 deletions docs/adr/0355-dynamic-evaluation-lineage.md

Large diffs are not rendered by default.

34 changes: 33 additions & 1 deletion docs/ubiquitous-language.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,38 @@ A non-operational instrument state in which observations may be collected for si
**Activation**
The governed transition from pilot to operational scoring after the instrument's declared evidence criteria are satisfied. Activation is fail-closed: insufficient evidence preserves observations without issuing a latent score.

## Dynamic evaluation vocabulary

**Evaluation Blueprint Revision**
An immutable plan that governs how evaluation items are sourced, generated, sampled, covered, and reviewed. It is not the concrete item set administered in a run and may legitimately declare zero fixed anchors during cold start.

**Dynamic Evaluation Item Snapshot**
The exact, immutable identity of one concrete authored, sampled, perturbed, adversarial, or generated item resolved under a blueprint revision. It preserves a content reference/digest and foreign evidence references without storing provider credentials or granting the generator decision authority.

**Evaluation Run Snapshot**
The immutable set of concrete item snapshots resolved before observations are interpreted for one run. Replacing an item creates a new run-snapshot revision; a mutable blueprint or later regeneration cannot silently rewrite the administered set.

**Reference Semantics**
The evidence form used to judge a response, such as exact, constraint, acceptable-set, rubric, pairwise, or open-ended semantics. It is distinct from the current governance status of that reference.

**Reference Status**
The independent state of an item's reference evidence: unresolved, provisional, adjudication-required, adjudicated, validated, or invalidated. `adjudicated` does not mean calibrated, approved, validated, active, or anchor-eligible.

**Validated Anchor**
An item snapshot explicitly promoted for an intended linking/use scope after separate validation and calibration evidence. An adjudication resolution or repeated model agreement alone does not create an anchor.

**Anchor Promotion Decision**
A separate immutable governance artifact that changes an item's evaluation role for a declared use without rewriting the prior item snapshot, source observations, or adjudication case.

**Run Comparability**
The claim supported for one run: unavailable, within-run-only, or linked. `linked` requires at least one validated/promoted anchor plus independent linking evidence; using the same nominal score range or rubric name is insufficient.

**Content Replay**
Retrieval of the exact frozen content reference/digest used in a prior run. It is distinct from invoking a generator again.

**Content Regeneration**
A new generation execution from recorded inputs. A seed, prompt revision, or provider/model identity is provenance only and does not establish deterministic regeneration without independent evidence.

## Model families

**Rasch**
Expand Down Expand Up @@ -69,7 +101,7 @@ The LineageWeave-facing reference to a contextual-orchestrator orchestration/pro
A reproducible judging condition that may affect observations: judge model, provider observation identity/provenance, prompt/policy revision, language, occasion, agent role, and other declared method conditions. Scientifically material facets are retained for severity/leniency, interaction, repeatability, calibration, and DIF/invariance analysis.

**Adjudication**
The governed use of evidence and one or more fallible observations to support a product decision. Adjudication is not synonymous with accepting an LLM answer.
The governed use of evidence and one or more fallible observations to support a product decision. Adjudication is not synonymous with accepting an LLM answer, validating an item, promoting an anchor, or rewriting its source observations.

**Disagreement**
A reproducible difference among judge observations or between judge and independent criterion evidence. Disagreement is retained/analyzed; it is not automatically resolved by majority vote.
Expand Down
16 changes: 16 additions & 0 deletions lineageweave/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,15 @@
from .affiliate_tree import build_affiliate_forest
from .corporate_hierarchy_resolution import resolve_corporate_entity
from .entity_relationship_classification import OrganizationRelationship
from .evaluation_lineage import (
DYNAMIC_EVALUATION_LINEAGE_CONTRACT_ID,
DynamicEvaluationItemLineage,
DynamicEvaluationLineageError,
DynamicEvaluationRunLineage,
RunComparabilityStatus,
build_dynamic_evaluation_item_lineage,
build_dynamic_evaluation_run_lineage,
)
from .external_lineage_analysis import analyze_external_lineage
from .external_lineage_contract import (
CONTRACT_VERSION,
Expand Down Expand Up @@ -73,7 +82,11 @@
__all__ = [
"CHANNEL_EVIDENCE_TOLERANCE",
"CONTRACT_VERSION",
"DYNAMIC_EVALUATION_LINEAGE_CONTRACT_ID",
"ChannelEvidence",
"DynamicEvaluationItemLineage",
"DynamicEvaluationLineageError",
"DynamicEvaluationRunLineage",
"ExplicitParent",
"LineageAnalysisPolicy",
"LineageAnalysisRequest",
Expand Down Expand Up @@ -106,9 +119,12 @@
"ProvLiteral",
"ProvValidationError",
"Record",
"RunComparabilityStatus",
"Tree",
"analyze_external_lineage",
"build_affiliate_forest",
"build_dynamic_evaluation_item_lineage",
"build_dynamic_evaluation_run_lineage",
"build_workspace_naruon_client",
"cited_post_summaries",
"default_calendar_window",
Expand Down
Loading