Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
- Add a Noema-owned exact-claim evidence receipt contract whose execution and research producers serialize one canonical artifact that binds every receipt semantic field, including command/result/isolation/network or source revision/excerpt/retrieval policy. Admission accepts only a receipt ID from untrusted model output. The owner API first verifies the exact authenticated OpenCode-handoff manifest digest, canonical envelope bytes, reviewed producer-to-kind policy, and repository/head/workflow/run/attempt identity before it can construct an immutable typed index; admission then reconstructs each canonical artifact and verifies time/claim/artifact identity. The version-2 manifest now binds a separate producer-authenticated `ClaimEvidenceRequirement` containing the exact claim, independently required evidence kind, and `context` or `finding` publication authority. Raw current-head source lines are context only: they are withheld from finding-reference prompts and cannot publish a finding or `request_changes`; an explicitly producer-authorized source finding remains usable and retains exact path/line checks. Finding-free model `request_changes` and `blocked` verdicts cannot bypass receipt admission to publish a vacuous blocking review. Requirement/receipt kind mismatch, fixed-artifact semantic substitution, caller-supplied receipt dictionaries, model self-classification, stale identities, cross-kind receipts, marker-only sandbox output, noncanonical artifact bytes, and expired receipts fail closed before the GitHub publisher. This remains the owner prerequisite for ContextualWisdomLab/.github#1641 and issue #555. The reviewed `sandboxed_verify` adapter exists in owner source, but its actual central stdout/stderr/marker-to-manifest wiring and the trusted research producer are not yet integrated; exact-head hosted GREEN, immutable release, and the verified central consumer bump remain required.

## Unreleased
- Agent Runtime의 procedural candidate screening decision을 process-local provenance로 제한한다. `assessProceduralCandidate()`가 기존 lineage·held-out·safety·score 검증을 마친 결과만 locally admitted decision으로 발행하고, State / Checkpoint나 Policy / Approval 경계가 구조만 복사·직렬화·프록시·직접 생성한 lookalike를 `assertProceduralCandidateDecision()`으로 실패-폐쇄한다. 이 brand는 evaluator authentication·persistence·approval·activation authority가 아니며 모든 decision의 `activationAuthorized:false`는 유지된다. issue #584, PR #591.
- Agent Runtime의 screened procedural decision에 baseline/candidate paired evaluation receipt의 canonical SHA-256 identity를 결합한다. exact graph/context/case-set/score/safety validation 뒤 admitted holdout 순서로 정규화한 증거 semantics를 해시하므로 observation 배열 순서만 바꿔 identity를 우회할 수 없고 실제 score/safety evidence가 바뀌면 identity도 바뀐다. 이 digest는 향후 authenticated evaluator evidence와 durable State / Checkpoint retention을 정확히 결합하기 위한 local evidence identity일 뿐 signature·producer authentication·Policy / Approval·persistence·activation authority를 부여하지 않는다. issue #584, PR #592.
- Agent Runtime의 workflow-backed procedural guidance가 매 판단마다 기존 execution-scoped `NOEMA_WORKFLOW_STATE`의 current Workflow / Task Execution evidence를 다시 읽는다. re-admitted plan과 locally admitted procedural session의 canonical execution identity가 다르면 Durable Object를 선택하거나 읽기 전에 실패-폐쇄하고, 현재 cancellation·terminal·pre-start evidence는 guidance를 억제한다. 이 ACL은 두 번째 lifecycle DB, task/lifecycle mutation, retry, tool, Policy / Approval, provider routing 또는 activation authority를 만들지 않으며 non-workflow lifecycle freshness와 deployed Durable Object compatibility/p95/recovery는 별도 acceptance로 남긴다. issue #584, ADR 0017.
- Agent Runtime에 tenant/task/execution-scoped immutable procedural graph와 bounded advisory context, paired held-out candidate screening을 추가한다. 모든 candidate decision은 `activationAuthorized: false`를 유지하고 tool·retry·Policy/Approval·provider routing·credential·foreign-domain authority를 부여하지 않는다. 그래프/평가 wire contract는 아직 Noema-local이며 cross-service publication은 context-graph-contracts의 immutable release를 기다린다. issue #584, ADR 0017.
- Agent Runtime의 procedural guidance를 locally admitted session brand와 canonical execution lifecycle에 결합한다. 구조만 흉내 낸 session은 callback/property를 읽기 전에 거부하고, guidance는 동일 execution의 `running` 상태에서만 반환하며 accepted·cancellation-requested·terminal 상태에서는 context request를 읽지 않고 억제한다. 결과는 계속 `advisory_only`이고 tool·retry·Policy/Approval·transition authority를 만들지 않는다. issue #584.
Expand Down
22 changes: 21 additions & 1 deletion src/agent-runtime/procedural-evolution.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ export interface ProceduralCandidateDecision {
readonly baselineDigest: string;
readonly candidateDigest: string;
readonly contextDigest: string;
readonly baselineReceiptDigest: string;
readonly candidateReceiptDigest: string;
readonly rejectionKey: string;
readonly baselineMean: number;
readonly candidateMean: number;
Expand Down Expand Up @@ -61,11 +63,27 @@ function observations(input: unknown, graphDigest: string, contextDigest: string
return holdout.map(id => byId.get(id)!);
}

async function evaluationReceiptDigest(
graphDigest: string,
contextDigest: string,
holdout: readonly string[],
rows: readonly Observation[],
): Promise<string> {
return proceduralHash([
"noema.procedural-evaluation-receipt/v1",
graphDigest,
contextDigest,
holdout.map((caseId, index) => [caseId, rows[index].score, rows[index].safetyViolations]),
]);
}

/**
* Screens supplied paired held-out evidence for a direct child graph while keeping activation and
* publication outside this pure port. The validator owner must authenticate receipts and pre-register
* the evaluation context; this function checks exact identities, complete paired cases, leakage,
* finite normalized scores, reported safety violations, mean non-regression, and contextual rejection.
* Returned receipt digests bind the exact validated paired evidence semantics for later authentication
* and durable retention, but a digest alone does not authenticate its producer or authorize approval.
* @param input Exact-key baseline, candidate, evaluation plan, paired receipts, and prior rejection keys.
* @returns Promise resolving to a frozen non-authoritative screening decision with activation always false.
*/
Expand All @@ -84,6 +102,8 @@ export async function assessProceduralCandidate(input: unknown): Promise<Procedu
if (holdout.some(id => training.has(id))) rejectProceduralInput("holdout_leakage");
const oldRows = observations(value.baselineReceipt, baseline.digest, contextDigest, holdout);
const newRows = observations(value.candidateReceipt, candidate.digest, contextDigest, holdout);
const baselineReceiptDigest = await evaluationReceiptDigest(baseline.digest, contextDigest, holdout, oldRows);
const candidateReceiptDigest = await evaluationReceiptDigest(candidate.digest, contextDigest, holdout, newRows);
const rejectedKeys = new Set(readProceduralArray(value.rejectedKeys, 0, 10_000).map(proceduralDigest));
const baselineMean = oldRows.reduce((total, row) => total + row.score, 0) / holdout.length;
const candidateMean = newRows.reduce((total, row) => total + row.score, 0) / holdout.length;
Expand All @@ -94,7 +114,7 @@ export async function assessProceduralCandidate(input: unknown): Promise<Procedu
else if (newRows.some(row => row.safetyViolations > 0)) reason = "safety_violation";
else if (candidateMean < baselineMean) reason = "score_regression";
else reason = "validation_non_regression";
const decision = Object.freeze({eligibleForApproval: reason === "validation_non_regression", activationAuthorized: false as const, reason, baselineDigest: baseline.digest, candidateDigest: candidate.digest, contextDigest, rejectionKey, baselineMean, candidateMean});
const decision = Object.freeze({eligibleForApproval: reason === "validation_non_regression", activationAuthorized: false as const, reason, baselineDigest: baseline.digest, candidateDigest: candidate.digest, contextDigest, baselineReceiptDigest, candidateReceiptDigest, rejectionKey, baselineMean, candidateMean});
admittedCandidateDecisions.add(decision);
return decision;
} catch (error) { return normalizeProceduralError(error); }
Expand Down
87 changes: 87 additions & 0 deletions test/procedural-evidence-identity.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
import { test } from "vitest";
import assert from "node:assert/strict";
import { createProceduralGraph } from "../src/agent-runtime/procedural-graph.ts";
import { assessProceduralCandidate } from "../src/agent-runtime/procedural-evolution.ts";

const raw = () => ({
schemaVersion: "noema.procedural-graph/v1",
tenantId: "tenant-a",
taskType: "repair",
graphId: "graph-a",
revision: 1,
parentDigest: null,
nodes: ["Start", "check"],
edges: [{
from: "Start",
relation: "requires",
to: "check",
condition: "",
guidance: "Check evidence",
pitfalls: "",
}],
});

async function fixture() {
const baseline = await createProceduralGraph(raw());
const candidateInput = raw();
candidateInput.revision = 2;
candidateInput.parentDigest = baseline.digest;
candidateInput.edges[0].guidance = "Check exact-head evidence";
const candidate = await createProceduralGraph(candidateInput);
const contextDigest = "c".repeat(64);
const plan = {
contextDigest,
minimumCases: 2,
trainingCaseIds: ["train-1"],
holdoutCaseIds: ["case-1", "case-2"],
};
const receipt = (graph, firstScore, secondScore) => ({
graphDigest: graph.digest,
contextDigest,
observations: [
{ caseId: "case-1", score: firstScore, safetyViolations: 0 },
{ caseId: "case-2", score: secondScore, safetyViolations: 0 },
],
});
return {
baseline,
candidate,
plan,
baselineReceipt: receipt(baseline, 0.5, 0.6),
candidateReceipt: receipt(candidate, 0.7, 0.8),
rejectedKeys: [],
};
}

test("screening decision binds canonical identities of both paired evaluation receipts", async () => {
const data = await fixture();
const first = await assessProceduralCandidate(data);
assert.match(first.baselineReceiptDigest, /^[0-9a-f]{64}$/);
assert.match(first.candidateReceiptDigest, /^[0-9a-f]{64}$/);
assert.notEqual(first.baselineReceiptDigest, first.candidateReceiptDigest);

data.baselineReceipt.observations.reverse();
data.candidateReceipt.observations.reverse();
const reordered = await assessProceduralCandidate(data);
assert.equal(reordered.baselineReceiptDigest, first.baselineReceiptDigest);
assert.equal(reordered.candidateReceiptDigest, first.candidateReceiptDigest);
});

test("receipt identity changes when validated score evidence changes", async () => {
const data = await fixture();
const first = await assessProceduralCandidate(data);
data.candidateReceipt.observations[0].score = 0.71;
const changed = await assessProceduralCandidate(data);
assert.notEqual(changed.candidateReceiptDigest, first.candidateReceiptDigest);
assert.equal(changed.baselineReceiptDigest, first.baselineReceiptDigest);
});

test("receipt identity changes when validated safety evidence changes", async () => {
const data = await fixture();
const first = await assessProceduralCandidate(data);
data.candidateReceipt.observations[0].safetyViolations = 1;
const changed = await assessProceduralCandidate(data);
assert.equal(changed.reason, "safety_violation");
assert.notEqual(changed.candidateReceiptDigest, first.candidateReceiptDigest);
assert.equal(changed.baselineReceiptDigest, first.baselineReceiptDigest);
});
Loading