Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
fd5044a
feat(eps): establish telemetry-first Evidence Projection Surface cont…
timerloggedout-spec Sep 22, 2026
a52909e
feat(eps): establish telemetry-first Evidence Projection Surface cont…
timerloggedout-spec Sep 22, 2026
14eca29
feat(eps): establish telemetry-first Evidence Projection Surface cont…
timerloggedout-spec Sep 22, 2026
9cd05d7
feat(eps): establish telemetry-first Evidence Projection Surface cont…
timerloggedout-spec Sep 22, 2026
de96477
feat(eps): add versioned EPS event schema
timerloggedout-spec Sep 22, 2026
9599264
feat(eps): add deterministic metadata-only emitter
timerloggedout-spec Sep 22, 2026
d6226ec
test(eps): cover emitter idempotency and safety
timerloggedout-spec Sep 22, 2026
d8bd23d
feat(eps): emit SHA-bound telemetry from production ledger
timerloggedout-spec Sep 22, 2026
e57190d
fix(eps): make ledger event identity replay-stable
timerloggedout-spec Sep 22, 2026
e7911ea
feat(eps): add validation and dashboard projection contracts
timerloggedout-spec Sep 22, 2026
0e3dbc4
feat(eps): add validation and dashboard projection contracts
timerloggedout-spec Sep 22, 2026
959b3e7
feat(eps): add validation and dashboard projection contracts
timerloggedout-spec Sep 22, 2026
e7edcf5
feat(eps): add validation and dashboard projection contracts
timerloggedout-spec Sep 22, 2026
663c713
ci(eps): validate telemetry before artifact publication
timerloggedout-spec Sep 22, 2026
2c02476
fix(eps): validate trusted ledger payload without PR checkout
timerloggedout-spec Sep 22, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .github/workflows/pr-production-ledger.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,45 @@ jobs:
`checks_pending: ${pendingChecks.length}`,
`checks_failed: ${failedChecks.length}`,
].join('\n');
const epsEvent = {
schema_version: 'eps.v1',
event_type: 'pr.ledger_observed',
occurred_at: new Date().toISOString(),
source: 'github.actions.pr-production-ledger',
repo: `${context.repo.owner}/${context.repo.repo}`,
git_sha: pr.head.sha,
run_id: context.runId,
run_attempt: Number(process.env.GITHUB_RUN_ATTEMPT || 1),
entity_type: 'pull_request',
entity_id: `pr:${prNumber}`,
status: 'observed',
provenance: { source_ref: pr.html_url, source_kind: 'github_api', attribution_confidence: 1 },
attributes: {
base_sha: pr.base.sha, commits: commits.length, changed_files: changedFiles,
additions: pr.additions, deletions: pr.deletions, ahead_of_base: ahead,
behind_base: behind, checks_total: checks.length,
checks_pending: pendingChecks.length, checks_failed: failedChecks.length,
branch_alignment: behind > 0 ? 'BEHIND_MASTER' : 'ALIGNED_OR_AHEAD'
}
};
const stable = JSON.stringify([
epsEvent.event_type, epsEvent.source, epsEvent.repo,
epsEvent.git_sha, epsEvent.run_id, epsEvent.run_attempt,
epsEvent.entity_type, epsEvent.entity_id, epsEvent.status
]);
epsEvent.event_id = `eps-${require('crypto').createHash('sha256').update(stable).digest('hex').slice(0, 32)}`;
require('fs').writeFileSync(
`${process.env.GITHUB_WORKSPACE}/eps-events.ndjson`,
JSON.stringify(epsEvent) + '\n',
{ encoding: 'utf8', flag: 'w' }
);
// Validate the emitted envelope before artifact publication. This job intentionally
// does not checkout PR code, so validation stays inside the trusted ledger step.
if (epsEvent.schema_version !== 'eps.v1' || !epsEvent.event_id || !epsEvent.provenance?.source_ref || !epsEvent.entity_id || !epsEvent.entity_type) {
core.setFailed('EPS envelope validation failed');
return;
}

await core.summary
.addHeading(`PR #${prNumber} production ledger`)
.addRaw(ledger + '\n\n')
Expand All @@ -116,3 +155,13 @@ jobs:
.addRaw(failedChecks.length ? `❌ ${failedChecks.length} failed check(s).\n` : '✅ No completed failing checks observed.\n')
.addRaw(pendingChecks.length ? `⏳ ${pendingChecks.length} check(s) still pending.\n` : '✅ No pending checks observed.\n')
.write();


- name: Upload EPS telemetry artifact
if: ${{ always() && hashFiles('eps-events.ndjson') != '' }}
uses: actions/upload-artifact@v4.6.2
with:
name: eps-events-${{ github.run_id }}-${{ github.run_attempt }}
path: eps-events.ndjson
if-no-files-found: error
retention-days: 14
55 changes: 55 additions & 0 deletions docs/ops/EPS-CONCEPT-TEMPLATES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# EPS Concept Templates

These templates are intentionally implementation-neutral. They describe future surfaces without making them authoritative.

## Concept: Evidence Timeline

**Inputs:** EPS events
**Interaction:** time window, entity, source, status
**Output:** linked event sequence
**Authority:** source evidence
**Validation:** every node resolves to event_id + source reference

## Concept: Agent Work Graph

**Inputs:** agent/task/attempt/provider/manager EPS events
**Interaction:** actor, task, handoff, retry, intervention
**Output:** relationship projection
**Authority:** EPS + source evidence
**Validation:** unknown attribution remains unknown

## Concept: Repository Activity Projection

**Inputs:** Git/PR/Actions EPS events
**Renderer:** Gource or future equivalent
**Interaction:** temporal playback, path/actor filters
**Authority:** EPS/source evidence
**Validation:** rendered event count reconciles to projection input

## Concept: Operations Observatory

**Inputs:** validated EPS + derived metrics
**Renderer:** Grafana / SHE / Vercel
**Interaction:** time-series, drill-down, alert → evidence
**Authority:** EPS/source evidence
**Validation:** freshness + provenance + completeness gates

## Concept: Research Observatory

**Inputs:** repository observations, starred/forked/added seeds, provider/model experiments
**Renderer:** Hex / future interactive surface
**Interaction:** cohort, similarity, disposition, experiment comparison
**Authority:** source evidence + frozen snapshots
**Validation:** reproducible snapshot and schema version

## Concept: ML Experiment Loop

**Inputs:** frozen EPS snapshot
**Process:** feature extraction → model → evaluation → derived evidence
**Output:** model/version/result record
**Authority:** source EPS snapshot
**Validation:** deterministic fixture + uncertainty + source IDs

## Design rule

A custom interactive surface is a **projection of EPS**, never an alternative telemetry authority.
26 changes: 26 additions & 0 deletions docs/ops/EPS-DASHBOARD-CONTRACT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# EPS Dashboard Contract

The dashboard is a consumer of EPS, never its producer.

## Required primitives

- freshness and last observed event
- ingestion and validation rate
- volume by source/type/status
- provenance drill-down
- run and attempt timeline
- attribution-confidence distribution
- stale/partial/unverified evidence
- replay/duplicate rejection
- ML/Hex/Grafana derived links
- projection health including Gource/render adapters

Every panel declares: metric, EPS event types, reducer, freshness, provenance drill-down, and failure semantics.

Missing evidence must never silently become zero.

Gource, SHE/Vercel and future WebGL/WASM interaction consume the same EPS projections. A visual interaction may link to event IDs/source refs but cannot author telemetry.

## Acceptance

A dashboard feature is accepted only when its metric has an EPS source/reducer, missing/stale/partial evidence is visible, source evidence is reachable, replay does not inflate it, and a non-visual evidence path reproduces the finding.
37 changes: 37 additions & 0 deletions docs/ops/EPS-EVENT.schema.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://github.com/timerloggedout-spec/termux-monorepo/blob/master/docs/ops/EPS-EVENT.schema.json",
"title": "Evidence Projection Surface event",
"type": "object",
"additionalProperties": false,
"required": ["schema_version","event_id","event_type","occurred_at","source","repo","entity_type","entity_id","status","provenance","attributes"],
"properties": {
"schema_version": {"const": "eps.v1"},
"event_id": {"type": "string","minLength": 8,"maxLength": 256},
"event_type": {"type": "string","pattern": "^[a-z][a-z0-9_.-]{2,127}$"},
"occurred_at": {"type": "string","format": "date-time"},
"source": {"type": "string","minLength": 1,"maxLength": 256},
"repo": {"type": "string","pattern": "^[^/\\s]+/[^/\\s]+$"},
"git_sha": {"type": ["string","null"],"pattern": "^[0-9a-f]{40}$"},
"run_id": {"type": ["integer","null"],"minimum": 1},
"run_attempt": {"type": ["integer","null"],"minimum": 1},
"entity_type": {"type": "string","minLength": 1,"maxLength": 128},
"entity_id": {"type": "string","minLength": 1,"maxLength": 512},
"status": {"type": "string","enum": ["observed","queued","running","completed","failed","cancelled","partial","unverified"]},
"provenance": {
"type": "object",
"additionalProperties": false,
"required": ["source_ref"],
"properties": {
"source_ref": {"type": "string","minLength": 1,"maxLength": 2048},
"source_kind": {"type": "string","minLength": 1,"maxLength": 128},
"attribution_confidence": {"type": ["number","null"],"minimum": 0,"maximum": 1}
}
},
"attributes": {
"type": "object",
"additionalProperties": true,
"propertyNames": {"maxLength": 128}
}
}
}
70 changes: 70 additions & 0 deletions docs/ops/EPS-IMPLEMENTATION-MATRIX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# EPS Implementation Matrix

| Initiative | Current disposition | EPS relationship | Next useful increment |
|---|---|---|---|
| #701 | LANDED | SSOT/session decisions are evidence metadata | ingest lane disposition + source SHA/run refs |
| #682 | EXTRACT / WAIT | ML DAG is a major EPS consumer/producer candidate | emit run/attempt/decision events only after dual-gate evidence |
| #432 / #549 / #601 | EXTRACT | ML wholesale family is historical evidence and reusable component source | extract stable schemas/reducers, not branches wholesale |
| #630 | EXTRACT | Jules provenance + dashboard behavior can become agent-event fixtures | capture agent/task/attempt provenance |
| #684 | HOLD / observe | cadence, concurrency, taint boundaries are telemetry dimensions | emit schedule/concurrency/provider-state events |
| #685 / #695 / #543 / #680 / #702 | OBSERVE | proposals, optimization, skill quality, catalog changes, and research seeds are heterogeneous evidence sources | normalize only metadata needed for reproducible evaluation |
| #48 / #69 / #73 | HOLD | connector/routing/debate history supplies provenance and boundary cases | preserve as historical fixtures; do not revive wrong-base code |

## Gravitee

#702 is relevant to EPS, but as a **research-seed / integration-observatory producer**, not as a telemetry dependency.

Its current proposal registers the Gravitee upstream research seed, preserves the local fork as a comparison surface, and explicitly avoids vendoring/submodules. That makes it useful for:

- API-management capability classification;
- gateway/plugin/policy/MCP capability observations;
- upstream-versus-fork provenance;
- Kubernetes/platform integration evidence;
- future API gateway telemetry experiments.

The useful extraction is the **observation contract**, not the upstream implementation wholesale.

### Gravitee + EPS boundary

`Repository Observatory → EPS research_observation → ML/Hex/Grafana → optional visualization`

Do not let Gravitee become a new execution plane or require its runtime before the observation contract is proven.

## Grafana

Grafana is complementary when EPS is projected into operational time series. Prefer derived metrics such as:

- event ingestion rate;
- validation failures;
- stale-event age;
- run/attempt completion latency;
- provider-state counts;
- action/effect yield;
- attribution-confidence distributions.

Grafana should alert on validated telemetry and link back to evidence identifiers.

## Hex

The existing `3l0.moneyball.v1` contract already has compatible grains: experiment run, agent task attempt, provider call, outcome score, and manager decision. EPS should provide the provenance envelope around those records rather than create a competing Moneyball schema.

Hex remains analytical/evidence presentation, with durable retention outside Hex where required.

## ML

ML should consume frozen EPS snapshots, never mutate the evidence plane. Candidate work includes:

- anomaly detection;
- cohort/regression analysis;
- provider/model routing evaluation;
- attribution-confidence calibration;
- graph/community analysis;
- cost/performance frontiers.

Any learned result is derived evidence and must retain source event IDs, source SHA/ref, model/version, timestamp, and uncertainty.

## Gource / visualization

Gource is a secondary renderer. Its compact custom-log format cannot carry EPS provenance, so EPS must remain canonical and a projection adapter should map EPS events into Gource events.

Future interactive surfaces may reuse the same adapter model without coupling telemetry collection to a particular renderer.
48 changes: 48 additions & 0 deletions docs/ops/EPS-KNOWLEDGE-EVALUATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# EPS Knowledge Evaluation

## Goal

Measure whether telemetry improves repository knowledge work before investing heavily in custom visualization.

## Baselines

1. GitHub/Git-only source inspection.
2. Existing ledger/evidence artifacts.
3. EPS normalized events.
4. EPS + ML/Hex/Grafana derived views.
5. EPS + visualization (secondary experiment).

## Tasks

Use reproducible repository questions:

- What happened to a specific PR and why?
- Which agent/provider/action produced an observed effect?
- Can the event be traced to an exact SHA/run/attempt?
- Where did retries, stalls, conflicts, or provider-state interruptions occur?
- Is a metric complete, partial, stale, or unverified?
- Which observations can be safely joined without multiplying denominators?
- Can an anomaly discovered visually be recovered from source evidence?

## Measures

Do not collapse these into a single opaque score.

- evidence retrieval accuracy;
- provenance reconstruction accuracy;
- time-to-answer;
- missing-evidence detection;
- false attribution rate;
- stale-data detection;
- duplicate/replay resistance;
- context consumed;
- human intervention;
- downstream reducer agreement.

An optional experimental **Knowledge Yield** can be reported as useful, evidence-backed discoveries per unit of analyst time/context. It is a research metric, not a governance score.

## Acceptance

EPS is useful when it measurably reduces evidence-retrieval work without increasing false attribution or hiding incomplete coverage.

A visualization experiment only advances when the same evidence tasks remain reproducible without the visualization.
Loading
Loading