Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion docs/ops/AGENT-OBSERVABILITY-RESEARCH-MATRIX.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ The research candidates should be compared as **parallel adapters/providers agai
|---|---|---|---|---|
| **Canonical JSONL + receipts** | repository-owned evidence source | runtime watcher / artifact ledger | **P0** | canonical evidence authority |
| OpenTelemetry | neutral trace/span interoperability | agent invocation + Actions boundary | **P0** | transport/interoperability layer |
| Temporal (self-host FOSS) | durable Workflow/Activity execution for long agents | mcp-docker/temporal + GHA temporal-self-host-smoke + Codespaces | P1 | optional runtime substrate; not dual-gate required |
| LangSmith Trajectories | flat chronological agent session path + online evals + SFT export | observational adapter over OTEL/JSONL; Temporal LangSmithPlugin bridge | P1 | observational; no source-of-truth authority |
| **ATES runtime evidence** | run/job/task throughput observations | workflow watcher + canonical JSONL | **P0** | observational execution spine |
| **Complexity contract** | normalized task/code difficulty | structural + provider feature pipeline | **P0** | measurement foundation |
| **Docker** | reproducible CI/research execution | instrument/evaluation parity harness | **P0** | environment substrate, not evidence source |
Expand All @@ -26,6 +28,7 @@ The research candidates should be compared as **parallel adapters/providers agai
| MASEval / agent benchmark suites | multi-agent evaluation patterns | future manager tournament cohorts | P2 | research input; adopt only after local schema mapping |
| Langfuse/Phoenix persistence UX | hosted/vendor-specific persistence | adapters over canonical events | P2 | never replace canonical corpus |
| Edge/ARM/Android inference | local inference efficiency | device/runtime benchmark lane | P2 | benchmark/research only until reproducibility is demonstrated |

## TDQS lane

Glama's **Tool Definition Quality Score (TDQS)** is now explicitly represented
Expand Down Expand Up @@ -80,6 +83,16 @@ shadowing-risk work; the monorepo keeps the deterministic portion locally so
inputs, hashes, and historical evidence remain under the repository's own
contracts.

## Why Temporal + LangSmith Trajectories

**Temporal** (MIT) is the durable execution layer. Self-host first via GHA `.github/workflows/temporal-self-host-smoke.yml` (SUCCESS on master), `mcp-docker/temporal/`, or Codespaces. Cloud Free Tier optional after secrets exist.

**LangSmith Trajectories** project a multi-turn thread into a readable ordered message path. Online evaluators score the path; export to datasets / SFT.

**Bridge:** Temporal Python SDK experimental `LangSmithPlugin` for Worker-boundary continuity.

Ops: [`docs/ops/TEMPORAL-LANGSMITH-ADAPTER.md`](TEMPORAL-LANGSMITH-ADAPTER.md)

## Why Langfuse deserves attention

Langfuse supports datasets, experiments, scores, live traces, and OpenTelemetry-based experiment ingestion. Its experiment model is especially relevant to manager tournaments because the same dataset can be replayed across conditions and evaluated consistently.
Expand Down Expand Up @@ -178,6 +191,7 @@ The repository-owned evidence contract is defined in [`AGENT-EVIDENCE-SUBSTRATE.
## Current-state rule

Historical PR state must not be presented as current execution state. Re-fetch the live branch/commit before describing a PR, workflow, corpus, or promotion state. Observer/status commits are evidence epochs, not substitutes for the execution artifact they describe.

## Phase gates

### Phase A — contract + reducer **[IMPLEMENTED]**
Expand All @@ -204,7 +218,7 @@ providers. Provider observations remain separate from ATES truth.
### Phase E — parallel evaluation

Run equivalent cohorts through GitHub-only, TDQS, Langfuse-adapter, and
Phoenix-adapter paths.
Phoenix-adapter paths (optional LangSmith Trajectories adapter).

### Phase F — environment parity

Expand Down
14 changes: 14 additions & 0 deletions docs/ops/INTEGRATION-GRAPH-MATRIX.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,3 +255,17 @@ Does **not** alter live catalog → router → MB edges; feeds **roster / attrib
| Certifications template | Issue #506 |

Agent-Identity: Grok (Administrator) · RECON consolidation 2026-09-19

---

## Temporal self-host + LangSmith Trajectories (2026-09-24 / residual 2026-09-30)

| Node | Edge | Notes |
|------|------|-------|
| Temporal self-host | durable execution substrate | FOSS MIT; GHA `temporal-self-host-smoke` SUCCESS; `mcp-docker/temporal/`; Codespaces |
| LangSmith Trajectories | observational path view | flat ordered messages; online evals; SFT export |
| LangSmithPlugin (temporalio) | Worker-boundary trace continuity | experimental; capability-gated |
| OTEL + ATES/JSONL | canonical evidence | unchanged P0 |

Proposal: `docs/proposals/active/temporal-langsmith-adapter/`
Runbook: `docs/ops/TEMPORAL-LANGSMITH-ADAPTER.md`
10 changes: 6 additions & 4 deletions docs/proposals/active/temporal-langsmith-adapter/ITEMS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,13 @@
| TLS-001 | RECON LangSmith Trajectories + Temporal FOSS/Free Tier + official LangSmithPlugin | **done** | 2026-09-24 session |
| TLS-002 | Proposal MANIFEST + registry row | **done** | this PR |
| TLS-003 | Update AGENT-OBSERVABILITY-PRIORITY-DECISION (P1 Temporal + LangSmith Trajectories) | **done** | this PR |
| TLS-004 | Update AGENT-OBSERVABILITY-RESEARCH-MATRIX rows | **done** | this PR |
| TLS-004 | Update AGENT-OBSERVABILITY-RESEARCH-MATRIX rows | **done** | residual matrix PR |
| TLS-005 | Ops runbook `docs/ops/TEMPORAL-LANGSMITH-ADAPTER.md` | **done** | self-host 1st |
| TLS-006 | Docker Compose self-host under `mcp-docker/temporal/` | **done** | official temporalio images |
| TLS-007 | Minimal Python smoke (Workflow + Activity + optional LangSmithPlugin) | **done** | scripts/temporal/ |
| TLS-008 | Integration Graph Matrix row | **done** | this PR |
| TLS-009 | Dual-gate green on tip before promote | backlog | adaptive-wait; no YOLO merge |
| TLS-010 | Optional Codespace worker runbook note | backlog | after TLS-009 |
| TLS-008 | Integration Graph Matrix row | **done** | residual matrix PR |
| TLS-009 | Dual-gate green on tip before promote | **done** | Temporal smoke SUCCESS master run 36092934569; FA-ADE on master |
| TLS-010 | Optional Codespace worker runbook note | backlog | optional |
| TLS-011 | Optional Free Tier Cloud path (operator-confirmed only) | backlog | secrets external |
| TLS-012 | GHA temporal-self-host-smoke | **done** | SUCCESS on master |
| TLS-013 | CLAUDE.md FA-ADE + regression guard | **done** | on master; #827 superseded dirty |
Loading