Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,9 @@ contextual-orchestrator owns model discovery and selection.
`NullKeymanExtractionClient`, `NullEntityRelationshipClient`,
`NullPostSummaryClient`, `NullPostChatClient`, and
`NullCommitmentExtractionClient` (and any new channel client you add)
must set `available = False` and make their channel dropped +
renormalized (`reconstruct.active_weights`), never silently return a
must set `available = False`. The calling reconstruction drops that channel and
loads the separately calibrated vector for the exact active-channel set;
`reconstruct.active_weights` fails closed on any mismatch. Never silently return a
placeholder score, invented Keyman, guessed relationship, fabricated
summary/chat, or invented commitment. A missing signal and a
confidently-negative signal are different things. Keyman extraction,
Expand Down
5 changes: 3 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,9 @@ flowchart LR
## Design decisions worth naming

- **Pluggable, never faked, channels.** `NullEmbeddingClient` and
`NullAdjudicationClient` make a channel *unavailable* (dropped and
renormalized in `active_weights()`), never silently scored as 0. A
`NullAdjudicationClient` make a channel *unavailable*. The caller loads the
calibrated vector for that exact active-channel set; `active_weights()`
rejects a mismatched vector instead of repairing or renormalizing it. A
missing signal and a confidently-negative signal are different things and
must not be conflated.
- **A minimum fused-score floor** (`DEFAULT_MIN_FUSED_SCORE`). Without it,
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.d/external-lineage-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# External email/project lineage contract

- Add a strict, versioned external analysis contract for future Naruon and separately governed consumer use.
- Export immutable request/result types, strict parsing, canonical serialization, deterministic digests, stable errors, and the store-agnostic `analyze_external_lineage` package entry point.
- Accept only bounded caller-authorized opaque evidence references; no provider credentials, mailbox access, persistence, provider mutation, or direct application-database integration is introduced.
- Preserve caller-observed RFC/provider/manual parent relations separately from inferred reconstructed continuation.
- Exclude caller-observed children from alternative inferred-parent scoring, optional model disclosure, and inferred-pair budget while retaining them as candidate history for later records.
- Enforce available-time knowledge cutoffs and disclose excluded evidence without substituting later facts.
- Reject explicit-parent cycles and candidate-pair work above the caller-approved limit before optional LLM/provider activity.
- Expose exact active channel scores, weights, contributions, LLM availability state, proposed project groupings, and deterministic result digests.
- Add JSON Schema Draft 2020-12, one canonical ADR 0214, APA 7th doctoring, and focused TDD coverage.
9 changes: 6 additions & 3 deletions docs/adr/0064-lineage-evidence-and-tree-assembly.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

- Status: Accepted
- Date: 2026-08-19
- Amended by: [ADR 0200](0200-channel-weight-reconciliation.md) and
[ADR 0205](0205-tepp-lineage-anchor.md)

## Context

Expand All @@ -16,9 +18,10 @@ or promoting an inferred relation to fact.
- Treat every input record as a fallible mention and every accepted edge as a
lineage instance supported by evidence, not as a proven business fact.
- Fuse independent temporal, secondary-key, text/embedding, and optional LLM
channels through the RankWeave weighted convex fusion contract. A missing
channel is dropped and weights are renormalized; it is never replaced with a
fabricated negative or score.
channels through the RankWeave weighted convex fusion contract. Resolve a
missing channel before loading the exact calibrated active-channel vector;
never repair or renormalize a vector estimated for another channel set, and
never replace a missing channel with a fabricated negative or score.
- Keep the channel-score breakdown and provenance on every candidate decision.
Candidates below the minimum fused-score floor remain roots rather than being
force-attached.
Expand Down
20 changes: 12 additions & 8 deletions docs/adr/0172-event-lineage-channel-evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
**Status:** Accepted
**Date:** 2026-08-21
**Issue:** [#274](https://github.com/ContextualWisdomLab/LineageWeave/issues/274)
**Amended by:** [ADR 0200](0200-channel-weight-reconciliation.md) and
[ADR 0205](0205-tepp-lineage-anchor.md)

## Context

Expand Down Expand Up @@ -33,8 +35,9 @@ authoritative; PROV-O/RDF export is a projection.
`lineage_signal_temporal`, `lineage_signal_secondary_key`,
`lineage_signal_text`, `lineage_signal_llm`. The LLM row is omitted
when the adjudication client is unavailable; it is never fabricated.
3. Weights are the normalized active weights actually used
(`reconstruct.active_weights`). Contribution is `weight * score` and
3. Weights are the exact calibrated active-channel vector validated by
`reconstruct.active_weights`; no different channel set is renormalized into
it. Contribution is `weight * score` and
must reconcile with `fused_score` within
`CHANNEL_EVIDENCE_TOLERANCE` (`1e-6`).
4. Live Event Lineage is replaced atomically. A singleton
Expand All @@ -46,10 +49,10 @@ authoritative; PROV-O/RDF export is a projection.
The administrator-triggered live rebuild and PostgreSQL import pass the
configured contextual-orchestrator adjudication client through the same
reconstruction boundary only when the exact candidate-pair count is at
most 5,000. Larger snapshots drop the LLM channel before any provider call
and renormalize the remaining weights. This is an operational work bound,
not a model-quality or provider-ranking heuristic. One rebuild never mixes
LLM and non-LLM weight profiles across edges.
most 5,000. Larger snapshots drop the LLM channel before loading the
separately calibrated three-channel profile. This is an operational work
bound, not a model-quality or provider-ranking heuristic. One rebuild never
mixes LLM and non-LLM weight profiles across edges.
5. `GET /api/lineage` returns an additive `channel_evidence` collection
on each visible edge (`signal_code`, `signal_label`, `score`,
`weight`, `contribution`, `rank`) ordered by contribution, then
Expand All @@ -67,8 +70,9 @@ authoritative; PROV-O/RDF export is a projection.
inference from source evidence.
- A later rebuild rewrites live Event Lineage as a whole; historic
meaning is not silently mutated in place.
- Completeness is lower when the LLM channel is unavailable, matching
ADR 0064: missing channels are dropped and weights renormalize.
- Completeness is lower when the LLM channel is unavailable; the exact
three-channel profile is loaded instead of renormalizing a four-channel
estimate.

## References

Expand Down
63 changes: 63 additions & 0 deletions docs/adr/0214-external-email-project-lineage-contract.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# ADR 0214: Publish a bounded external email/project lineage contract

- Status: Accepted
- Date: 2026-08-21

## Context

Naruon owns customer mail/calendar/file access, canonical message/thread identities, projects, tasks, commitments, provider credentials, authorization, and provider mutations. LineageWeave owns evidence-fused lineage reconstruction and the provenance explaining that reconstruction. Future integration must not give either product direct SQL access to the other's application database, duplicate source authority, or depend on a mutable branch/submodule.

Email thread facts also have different truth semantics from reconstructed semantic continuation. RFC `Message-ID`, `References`, and `In-Reply-To` evidence may establish a caller-observed reply relation, while LineageWeave text/temporal/project signals produce an inferred relation. Flattening both into one unexplained score would make buyer correction and audit impossible.

## Decision

LineageWeave publishes contract version `1.0.0` through:

- `lineageweave.external_lineage_contract` for strict immutable request/result shapes, canonical serialization, bounds, and deterministic digests;
- `lineageweave.external_lineage_analysis` for adapting caller-authorized evidence to the existing reconstruction kernel.

The initial implementation is a store-agnostic Python package boundary. It performs no database, mailbox, provider, or network operation. A later service or Naruon plugin adapter must preserve the same JSON Schema and truth boundaries.

Execution requires an explicit calibrated convex channel-weight vector. The
adapter validates but never estimates, repairs, or invents that vector. The
three core channels are mandatory; the optional LLM channel is admitted only
when both a calibrated LLM weight and an available orchestrator client exist.
The calibrated vector must exactly match the channels executed for the request;
an inactive LLM channel makes a four-channel vector invalid rather than causing
the core weights to be silently renormalized.
Weight estimation and provenance remain governed by ADR 0200 and the owning
product loader.

The caller supplies opaque evidence references, bounded text labels, occurrence and availability clocks, an optional secondary key, an optional project reference, and an optional caller-observed parent relation. Explicit observed parent relations replace an inferred parent for the same child and must form an acyclic graph. Reconstructed continuation remains `inferred`. Project groupings remain `proposed`.

An admitted child with an explicit observed parent is not rescored for an alternative inferred parent and consumes no optional LLM/provider call or inferred-pair budget. The record remains in temporal history and may still be an eligible candidate parent for a later record. This preserves observed authority without weakening downstream lineage reconstruction.

The caller also supplies `maximum_pair_evaluations` in the bounded policy. The package computes the exact inferred candidate-parent pair count after knowledge-cutoff filtering, excluding children whose parent is already caller-observed, and rejects work above the declared budget before any optional LLM/provider call. Contract v1 caps the declared budget at 5,000 pairs.

Historical requests include evidence only when:

```text
available_at <= knowledge_cutoff
```

Evidence becoming available after the cutoff is excluded even when it describes an earlier occurrence.

## Consequences

- Naruon can eventually consume a released artifact without exposing credentials or application tables.
- RFC reply/thread evidence stays distinguishable from semantic lineage.
- Caller-observed children are never disclosed to an optional model merely to calculate an inferred edge that would be discarded.
- The optional LLM channel is explicit as `not_requested`, `unavailable`,
`not_used`, or `completed`; admission without a provider call is never
reported as completion, and missing output is never zero.
- A missing or malformed calibrated weight vector fails closed before scoring.
- A calibrated vector containing an inactive channel fails closed instead of
being repaired or renormalized into a different measurement.
- Canonical serialization and SHA-256 digesting are deterministic for a given request or result. Repeatability of model-backed scores additionally requires a pinned LineageWeave release, adjudicator implementation, provider/model revision, and model-side determinism policy.
- Explicit parent cycles and analysis work above the caller-approved pair budget fail closed before inference.
- Project evidence can inform Naruon without mutating authoritative project/task/provider state.
- The single generic secondary key reflects the current core kernel. Multiple independent typed secondary-key channels remain a future contract revision rather than being silently flattened.

## References

See `docs/doctoring/EXTERNAL_LINEAGE_CONTRACT_REFERENCES.md`.
2 changes: 2 additions & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ decision from them.
| Evidence operations Dashboard (`/`) | [0206](0206-evidence-operations-dashboard.md) |
| [`temporal-topic-context-influence-research.md`](../temporal-topic-context-influence-research.md) | [0210](0210-temporal-topic-context-influence-dashboard.md) |
| [`python-mathematical-compute-boundary-audit.md`](../doctoring/python-mathematical-compute-boundary-audit.md) | [0208](0208-externalize-local-mathematical-compute.md) |
| [`contracts/README.md`](../contracts/README.md) and external-lineage v1 contract set | [0214](0214-external-email-project-lineage-contract.md), [0200](0200-channel-weight-reconciliation.md) |
| [`doctoring/EXTERNAL_LINEAGE_CONTRACT_TRACEABILITY.md`](../doctoring/EXTERNAL_LINEAGE_CONTRACT_TRACEABILITY.md) | [0214](0214-external-email-project-lineage-contract.md) |

[0011](0011-prov-o-standard-relations.md) and [0065](0065-prov-o-provenance-boundary.md) cite the dated W3C PROV-O and PROV-DM Recommendations (https://www.w3.org/TR/2013/REC-prov-o-20130430/ and https://www.w3.org/TR/2013/REC-prov-dm-20130430/).

Expand Down
15 changes: 15 additions & 0 deletions docs/contracts/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Integration contracts

LineageWeave publishes strict, versioned contracts for separately governed consumers. These contracts do not grant source access and do not replace each consumer's authorization, persistence, provider, or audit authority.

## External lineage analysis v1

- JSON Schema: `external-lineage-analysis-v1.schema.json`
- Synthetic request: `external-lineage-analysis-v1.example.json`
- Python parser and immutable types: `lineageweave.external_lineage_contract`
- Store-agnostic execution adapter: `lineageweave.external_lineage_analysis`
- Decision record: `docs/adr/0214-external-email-project-lineage-contract.md`

A consumer must submit only bounded evidence it is already authorized to disclose. Outputs retain opaque caller references and explicit `observed`, `inferred`, or `proposed` truth boundaries. The contract performs no source-system access or provider mutation.
The execution adapter additionally requires an ADR-0200-compliant calibrated
channel-weight vector supplied by the host; it has no default weights.
5 changes: 5 additions & 0 deletions docs/contracts/external-lineage-analysis-v1.authorization.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# External lineage analysis v1 authorization contract

LineageWeave does not infer authorization from an opaque reference, source kind, group, project, or caller identity. The caller must authorize evidence before projection and must reauthorize any source drill-through after receiving a result.

The package does not accept provider bearer tokens, browser cookies, mailbox credentials, database DSNs, or caller SQL. A future remote service must use its own audience-scoped service credential and may not forward an end-user token to model providers.
11 changes: 11 additions & 0 deletions docs/contracts/external-lineage-analysis-v1.consumer-checklist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# External lineage analysis v1 consumer checklist

- Validate the published JSON Schema before sending or accepting payloads.
- Submit only evidence the calling principal is authorized to disclose for the declared purpose.
- Use opaque caller-owned references; never send provider credentials or database locators.
- Bind historical work to a knowledge cutoff and preserve each record's availability time.
- Keep RFC/provider thread observations separate from inferred semantic/project lineage.
- Treat project projections as proposals until the caller's own policy or reviewer accepts them.
- Preserve the returned artifact digest, LineageWeave version, limitations, and channel evidence.
- Fail closed on incompatible contract versions.
- Keep normal caller operation available when LineageWeave is unavailable.
12 changes: 12 additions & 0 deletions docs/contracts/external-lineage-analysis-v1.data-minimization.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# External lineage analysis v1 data minimization

Consumers should prefer the minimum evidence needed for a declared analysis scope:

- opaque evidence and grouping references;
- offset-aware occurrence and availability times;
- RFC/provider relation evidence when present;
- bounded subject/title labels or caller-computed text features;
- optional project or secondary-key references;
- optional participant, body, or attachment evidence only when the caller's purpose and policy explicitly permit it.

The contract does not require a mailbox dump, full thread body, recipient list, provider URL, or attachment bytes. Omitted evidence is unavailable and cannot appear in output.
41 changes: 41 additions & 0 deletions docs/contracts/external-lineage-analysis-v1.example.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
{
"contract_version": "1.0.0",
"analysis_id": "analysis:synthetic-email-lineage-001",
"analysis_scope_code": "email_lineage",
"knowledge_cutoff": "2026-08-20T09:30:00Z",
"policy": {
"candidate_window": 50,
"maximum_pair_evaluations": 1000,
"minimum_fused_score": 0.3,
"allow_llm": false
},
"records": [
{
"evidence_ref": "email:synthetic-001",
"group_ref": "workspace:synthetic",
"source_kind_code": "email",
"truth_status_code": "observed",
"label": "Synthetic proposal review",
"occurred_at": "2026-08-20T09:00:00Z",
"available_at": "2026-08-20T09:01:00Z",
"secondary_key": "provider-thread:synthetic",
"project_ref": "project:synthetic",
"explicit_parent": null
},
{
"evidence_ref": "email:synthetic-002",
"group_ref": "workspace:synthetic",
"source_kind_code": "email",
"truth_status_code": "observed",
"label": "Re: Synthetic proposal review",
"occurred_at": "2026-08-20T09:05:00Z",
"available_at": "2026-08-20T09:06:00Z",
"secondary_key": "provider-thread:synthetic",
"project_ref": "project:synthetic",
"explicit_parent": {
"evidence_ref": "email:synthetic-001",
"relation_code": "rfc_reply"
}
}
]
}
11 changes: 11 additions & 0 deletions docs/contracts/external-lineage-analysis-v1.limitations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# External lineage analysis v1 limitations

- The contract does not read IMAP, JMAP, CalDAV, Naruon, or other provider systems.
- It does not authenticate users, authorize tenant access, persist jobs, or retry remote work.
- It does not make semantic lineage equivalent to RFC reply/thread identity.
- It does not turn project groupings, responsibility context, or reconstructed edges into authoritative caller facts.
- It does not infer unavailable evidence as a zero-valued channel.
- It does not guarantee causal relations; reconstructed continuation is an evidence-weighted related-history hypothesis.
- Canonical request/result serialization and digests are deterministic, but an optional remote adjudication channel is not automatically repeatable unless the consumer pins the LineageWeave artifact, adjudicator, provider/model revision, and determinism policy.
- Contract v1 does not carry a remote provider/model receipt inside the result; production wrappers must retain that provenance alongside the result digest before model-backed integration is enabled.
- It does not replace Naruon's canonical email identity, project/task/commitment state, provider mutation, or reconciliation authority.
5 changes: 5 additions & 0 deletions docs/contracts/external-lineage-analysis-v1.operability.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# External lineage analysis v1 operability boundary

The pure package entry point is synchronous and bounded. Remote or model-backed production use must wrap it in a separately reviewed service or plugin lifecycle with durable idempotency, cancellation, timeout, retry classification, rate limiting, resource budgets, artifact retention, OpenTelemetry signals, and user-visible degraded states.

A consumer must not call optional model-backed pair adjudication directly on an unbounded web request path. LineageWeave #289 tracks the durable asynchronous reconstruction requirement for product persistence, and Naruon #1437 requires an equivalent consumer-side job receipt before integration is enabled.
Loading
Loading