Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,12 @@ concurrency:
jobs:
validate:
runs-on: ubuntu-24.04
# Was 5 minutes with a fetch-depth: 1 checkout; raised to cover the added full-history
# Was 5 minutes with a fetch-depth: 1 checkout; raised to 10 to cover the added full-history
# clone (the secret-scan job's comment below records ~34s locally for the same clone).
timeout-minutes: 10
# Raised to 20 on 2026-09-26: the unittest step alone ran 534s in run 36265075411, main's
# job took 598s in run 36263674462, and runs 36265075411, 36265091632 and 36269293122
# exceeded the 10-minute limit ("exceeded the maximum execution time of 10m0s").
timeout-minutes: 20
steps:
- name: Harden the runner (audit-only network egress)
uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1
Expand Down
4 changes: 4 additions & 0 deletions .gitleaks.toml
Original file line number Diff line number Diff line change
Expand Up @@ -327,11 +327,15 @@ id = "sourcegraph-access-token"
# this rule only, and only a whole anchored "pin" line whose value is a 40-hex id with an optional
# "<version> @ " prefix; any other field holding a 40-hex value in the same file is still detected
# (tests/test_gitleaks_config.py test_d/test_d2).
# 2026-09-26: the landscape-sweep-20260926 record retains the run's layer input for code-navigation exactly
# as read (inputs/SHA256SUMS). It names Sourcegraph (the SCIP index) and holds the same catalog "pin" lines,
# so the same anchored pin-line exemption covers that one file too (test_d5).
[[rules.allowlists]]
description = "Git commit ids under the \"pin\" field of the dated SOTA manifest"
condition = "AND"
paths = [
'''^catalogs/sota-convergence/manifest-20260923\.json$''',
'''^evidence/artifacts/landscape-sweep-20260926/inputs/code-navigation\.json$''',
]
regexTarget = "line"
regexes = [
Expand Down
2 changes: 2 additions & 0 deletions evidence/artifacts/landscape-sweep-20260926/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,8 @@ returns list any skill, so these counts do not measure GPT-6 skill use.

## Files

- `inputs/` (added 2026-09-26 after the record merged): the 32 layer-input files exactly as this run read them (written 2026-09-25 21:30-22:44 local, before the run), with `inputs/SHA256SUMS` (`cd inputs && sha256sum -c SHA256SUMS`). They show lane limit 2's truncated known-repository slugs as the workers saw them. The verdict wave's GPT-6 fit re-votes for the 27 not-adjudicated entries use these originals rather than rebuilt inputs.

- **`returns.json`.** The output of `convert.py`, redacted as described below. It holds the discovery returns with
the frozen scope hashes, the facts and two-family fit votes, the raw family returns, the retained failures,
`skills_usage` and `gpt6_usage`. The ledger's `returns_ref` points to it.
Expand Down
32 changes: 32 additions & 0 deletions evidence/artifacts/landscape-sweep-20260926/inputs/SHA256SUMS
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
7c4f1eeb9ea3aa74c9288c10d05802edee640bb9ff633ad7a466c9f038bb7dea agent-sdks.json
8037e618a0b516c2918d64d9c5dc65546060f5fb0594ddb4da2c3b2faf1e3b7f agents-models-workers.json
26e0fb77d7921a572bd3d47a77f0bfd84d7d376182b53f73f457a3f1b3c2e72c backtesting-engine.json
9a5f61f9649a1ee6319ab3203f24ee936fc498520309992657d16561dbb75362 ci-supply-chain.json
24b52b2a76adf530b43ea24bb8217872d44dc5d51e93ba85658cc3980f3c8254 code-navigation.json
cca7206bce26686fd932b11c6e20773e22b6f8fa5c679be821484813e2b6e1ea data-quality-orchestration.json
96aab9e8a66ba7807083285fc48464cdd7b8df0a885917fb61c87dbbf419629d document-retrieval.json
b5dad1db5a24534df6d5dfc35ae9ea7e15ab3283dd06562edccfbc2791f0715f durable-memory.json
d784ad58726c563a77ed48e492aee82253eabde50af455d3c6106b87e28db1da evaluation-experiments.json
7e38decf48aed4c541ca812a449b4aee492f438998f16033627fb00d55716f53 execution-broker.json
91d2edd2c135e92cb5f71da922c12ca8d77a67fcc8a442f4b19d2e66e7cfc356 git-github-automation.json
6e04a36d216974159f5d68c863a3607deb865287311ac37f8c6f7cdfdbea7db6 hosting-services.json
95086a2fb61cc03344117b0cf11ddb47c822e3271280f7c0b01a921930452258 identity-provenance.json
9df1adbd661dc2d950801fbf5550deffc192eec256859d8d92578866ab13061f instructions-skills.json
151cf43901bb1c289918075787142f79545ccdce4faff629bcb27ecf69f61d77 isolation.json
e506bfd42e94b6d381f4ab52ec90e05f0245ebbcc4670b2945872a24e8977c20 market-data-reference.json
f92ea6cd00782ecbe4561ef289932288457493fb60393a05065fdb25b7f88324 mcp-surfaces.json
5392a70f60bab1e93c9ea4cfd9b28ba63cc5bbd61f78202337241db4a888445d native-clients.json
17f1e04ca07ceccdcc3d28ac1372cc9de5a3fc0ab42a429db4f60c42c8f33941 observability-hosting.json
70988c23ac44461529c02f6f9c40e15cf0add7c22a73a736f40876c940fa2fe8 observation-inference.json
d53d1e4a04b4ef050e617013a1be43fe540ec872c476f9a756af70a387180937 portfolio-risk.json
cfdcf508cd3ee149b3f52616af4e65ed2103e93b07d8f89a10a4867d633289aa quality-evaluation.json
354adf9685dd3e249f4285d7f70309aa5df6f4085c8df7e250371b14a498beca recovery-portability.json
2e13695ab1eefcc03f5d9fdfd29f85a9ba4296f96c8af342c2a46569315e251f research-factors-ml.json
6ef7b1cd4a4d9d781ba5dbe0a2f442aa6acf451263782ffecd628dfae7a70186 scheduling-supervision.json
4eabe9d79a5b1455a03661dad2ba2fcf50adf74c00b192c32e473319611bbafb secrets-credentials.json
e055147a459841afdf06d47a54f91196b5f4d676897f313b66238f9cb1e6c31c security-supply-chain.json
50b17e0d8e409a637e77f272ff80b551f65a4ff8f14b267cbe92454adcd3234a semantic-rag.json
4fd50cf2a8915bf883562494cd75b7425214f9042f7dfcac223caad96068a47f storage-compute.json
a6e48d749059a28cf0b95b829888f53426879bfcaed20ce80b7deb3414d89ace token-efficiency.json
4fe346ad204784a0e4490fd21794a758ede0ddd16ead253145af011f4f491a88 web-research.json
66632940ce41135ad47c7096749bfdc2c89f79267a4e2d332116ab0cbb4f2f73 workers.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
{
"catalog": "foundation",
"layer_id": "agent-sdks",
"title": "Agent SDKs and runtime workers",
"requirement": "Give a worker structured events, custom tools or session control through a supported SDK when the native CLI's bounded review is not enough.",
"research_status": "on_requirement_change",
"next_action": "Qualify a concrete application need (durable multi-host state, explicit graph checkpoints, a remote/container worker or programmatic Claude session control) before promoting a deferred SDK alternative; the SDK table in docs/foundation-closure-20260921.md (lines 78-84) has not been revisited by any lane since 2026-09-21.",
"requirement_sha256": "e2a4bef5dcc9f4e1c0e06df41f5bf6ba27d9a18fb81ded371ac77bed163f36c4",
"platform_profiles_sha256": "8b0a136c3c0c8c99e727bb84244b9eba789b03d607d06ff1de5fb7aee0b3fece",
"winners": [
{
"component_id": "codex",
"repository": "https://github.com/openai/codex",
"pin": "0.155.1",
"evidence_class": "native_proven",
"platform_status": {
"linux-wsl2-x86_64": "accepted",
"macos-arm64": "untested"
}
}
],
"alternatives": [
{
"name": "Claude Agent SDK",
"repository": "https://github.com/anthropics/claude-agent-sdk-python",
"disposition": "conditional"
},
{
"name": "OpenHands SDK",
"repository": "https://github.com/openhands/software-agent-sdk",
"disposition": "conditional"
},
{
"name": "Temporal",
"repository": "https://github.com/temporalio/temporal",
"disposition": "out_of_scope"
},
{
"name": "LangGraph",
"repository": "https://github.com/langchain-ai/langgraph",
"disposition": "unqualified"
}
],
"components_vs_upstream_20260926": [
{
"id": "codex",
"repository": "https://github.com/openai/codex",
"pin": "0.155.1",
"upstream": {
"latest": "rust-v0.157.0",
"released_at": "2026-09-25",
"prerelease": false,
"pushed_at": "2026-09-26",
"stars": 126487,
"license": "Apache-2.0",
"archived": false,
"renamed_to": null
},
"pin_behind_upstream": true,
"pin_comparison": "compared"
}
],
"open_gaps": [
"The only executed Codex SDK vs Claude Agent SDK comparison is unresolved and publication-ineligible, with 'no arm accepted as a winner' (evidence/artifacts/comparison-progress-20260922/summary.json lines 133-151). Recovery was not run and artifact acceptance has no pinned judge.",
"The Codex cancel submission has no native returned receipt: arm_status no_receipt, usage not_requested (worker-receipts-followup.json lines 152-176). Claude Agent SDK interruption reports usage as unavailable. Neither shows actual provider consumption.",
"c4 isolation is unverified. The SDK overlays env onto the parent environment, and read-only permissions do not make inherited MCP servers or hooks read-only (blueprints/us-equities/workers/README.md lines 53-57; receipt.json line 110). The recorded tool events came from the host's Context Mode serve",
"native_worker.py line 77 creates ephemeral threads, so the c4 integration has not shown multi-turn resume. Custom-tool registration through the SDK is also unobserved. The 'start/run/resume' wording in the foundation closure is a project description, not an observation.",
"In the c4 receipt, two of three native turns failed their extraction task, blocked by the project root and by approval policy (receipt.json lines 132-220)."
],
"overturn_when": "A concrete application needs durable multi-host state/replay (Temporal), explicit graph checkpoints (LangGraph), a remote/container worker (OpenHands SDK) or programmatic Claude session control (Claude Agent SDK), and a matched comparison passes.",
"sweep_20260923": {
"survived": [],
"refuted": [
"https://github.com/google/adk-python",
"https://github.com/pydantic/pydantic-ai",
"https://github.com/strands-agents/harness-sdk"
]
},
"known_repositories": [
"anthropics/claude-agent-sdk-python",
"google/adk-python",
"langchain-ai/langgraph",
"openai/codex",
"openhands/software-agent-sdk",
"pydantic/pydantic-a",
"strands-agents/harness-sdk",
"temporalio/temporal"
],
"seeded_candidates_20260926": []
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
{
"catalog": "us-equities",
"layer_id": "agents-models-workers",
"title": "Agents, models and workers",
"requirement": "Keep durable project knowledge, exact code and dated financial sources retrievable across native clients, with source recovery, explicit project scope and measurable retrieval quality.",
"research_status": "comparison_required",
"next_action": "Create held-out finance/document/code relevance judgments; compare fidelity, abstention and total task cost while preserving source timestamps, project isolation and original recovery.",
"requirement_sha256": "c2358d8f11c2f15c2f4494c8de4cbe8ffb4dcac77f0af2a098163b354e5e7b01",
"platform_profiles_sha256": "8b0a136c3c0c8c99e727bb84244b9eba789b03d607d06ff1de5fb7aee0b3fece",
"winners": [
{
"component_id": "codex-native-sdk",
"repository": "https://github.com/openai/codex",
"pin": "CLI rust-v0.155.1; Python openai-codex 0.154.0",
"evidence_class": "native_proven",
"platform_status": {
"linux-wsl2-x86_64": "accepted",
"macos-arm64": "untested"
}
},
{
"component_id": "foundation-ai-memory",
"repository": "https://github.com/akitaonrails/ai-memory",
"pin": "v2.3.1",
"evidence_class": "native_proven",
"platform_status": {
"linux-wsl2-x86_64": "accepted",
"macos-arm64": "untested"
}
},
{
"component_id": "foundation-socraticode",
"repository": "https://github.com/giancarloerra/SocratiCode",
"pin": "v1.14.0",
"evidence_class": "native_proven",
"platform_status": {
"linux-wsl2-x86_64": "accepted",
"macos-arm64": "untested"
}
}
],
"alternatives": [
{
"name": "foundation-toon",
"repository": "https://github.com/toon-format/toon",
"disposition": "out_of_scope"
},
{
"name": "foundation-pageindex",
"repository": "https://github.com/VectifyAI/PageIndex",
"disposition": "unqualified"
},
{
"name": "langgraph",
"repository": "https://github.com/langchain-ai/langgraph",
"disposition": "unqualified"
},
{
"name": "vllm",
"repository": "https://github.com/vllm-project/vllm",
"disposition": "conditional"
},
{
"name": "omniroute",
"repository": "https://github.com/diegosouzapw/OmniRoute",
"disposition": "conditional"
},
{
"name": "foundation-repomix",
"repository": "https://github.com/yamadashy/repomix",
"disposition": "conditional"
},
{
"name": "Serena",
"repository": "https://github.com/oraios/serena",
"disposition": "overlap"
},
{
"name": "foundation-pgvector",
"repository": "https://github.com/pgvector/pgvector",
"disposition": "unqualified"
},
{
"name": "Context Mode",
"repository": "https://github.com/mksglu/context-mode",
"disposition": "conditional"
},
{
"name": "RTK",
"repository": "https://github.com/rtk-ai/rtk",
"disposition": "overlap"
},
{
"name": "foundation-codebase-memory-mcp",
"repository": "https://github.com/DeusData/codebase-memory-mcp",
"disposition": "overlap"
},
{
"name": "Graphiti",
"repository": "https://github.com/getzep/graphiti",
"disposition": "unqualified"
},
{
"name": "Qdrant",
"repository": "https://github.com/qdrant/qdrant",
"disposition": "overlap"
},
{
"name": "codex-acp",
"repository": "https://github.com/agentclientprotocol/codex-acp",
"disposition": "conditional"
},
{
"name": "deerflow",
"repository": "https://github.com/bytedance/deer-flow",
"disposition": "conditional"
},
{
"name": "foundation-agent-retrieval-bench",
"repository": "https://github.com/eyuansu62/agent-retrieval-bench",
"disposition": "unqualified"
},
{
"name": "loopx-project/loopx",
"repository": "https://github.com/loopx-project/loopx",
"disposition": "unqualified"
},
{
"name": "gastownhall/beads",
"repository": "https://github.com/gastownhall/beads",
"disposition": "unqualified"
},
{
"name": "jdx/mise",
"repository": "https://github.com/jdx/mise",
"disposition": "out_of_scope"
}
],
"components_vs_upstream_20260926": [],
"open_gaps": [
"No measured head-to-head comparison exists between the worker backends (codex-native-sdk, DeerFlow/codex-acp, OmniRoute). Each ran separately on a different task. The c19+c21 run returned the same facts in 1 of 1 tasks, with 0 blocked and 42345 cumulative tokens. c20 used 135311 tokens (204091 acros",
"The codex-native-sdk worker completed 1 research task. 2 follow-up extraction tasks were blocked, first by the Context Mode ctx_execute_file project-root restriction, then by the worker's own deny_all approval mode ('approval policy is \"never\"', workers/receipt.json line 196). For c20, strict read-o",
"c20's read-only filesystem mode does not restrict external MCP mutations (workers/receipt.json line 110). The recorded environment supplied no broker keys or broker tools.",
"The configured model/provider identity in the worker, ACP and OmniRoute receipts is not independent per-request provider attestation.",
"Retrieval quality rests on a single limit-1 query (score 0.6429) in native-rag.json. That is not a holdout or recall measurement. The group's 8/12 exact-source recall figure comes from the packet limitations, not from a file I opened."
],
"overturn_when": "Run a preregistered, representative held-out financial/document and code retrieval set at the same context budget; replace or supplement a lane only if a candidate improves source/answer fidelity, abstention and measured task cost while preserving isolation, exact source recovery and native client behavior.",
"sweep_20260923": {
"survived": [],
"refuted": [
"https://github.com/confident-ai/deepeval",
"https://github.com/infiniflow/ragflow",
"https://github.com/google-gemini/gemini-cli",
"https://github.com/modelcontextprotocol/servers"
]
},
"known_repositories": [
"agentclientprotocol/codex-acp",
"akitaonrails/ai-memory",
"basicmachines-co/basic-memory",
"bytedance/deer-flow",
"confident-ai/deepeval",
"deepset-ai/haystack",
"deusdata/codebase-memory-mcp",
"diegosouzapw/omniroute",
"docling-project/doclin",
"eyuansu62/agent-retrieval-bench",
"gastownhall/beads",
"getzep/graph",
"giancarloerra/socraticode",
"google-gemini/gemini-cl",
"headroomlabs-ai/headroom",
"hkuds/lightra",
"https://huggingface.co/ibm-granite/granite-4.2-8b",
"https://huggingface.co/qwen/qwen3.5-9b",
"infiniflow/ragflow",
"jdx/mise",
"langchain-ai/langgraph",
"letta-ai/letta-code",
"loopx-project/loopx",
"mem0ai/mem0",
"mksglu/context-mode",
"modelcontextprotocol/servers",
"msitarzewski/agency-agents",
"openai/codex",
"oraios/serena",
"pgvector/pgvector",
"qdrant/qdran",
"rtk-ai/rtk",
"run-llama/llama_index",
"supermemoryai/supermemory",
"thedotmack/claude-mem",
"tobi/qmd",
"toon-format/toon",
"topoteretes/cognee",
"vectifyai/pageindex",
"vectorize-io/hindsigh",
"vllm-project/vllm",
"volcengine/openvikin",
"yamadashy/repomix"
],
"seeded_candidates_20260926": [
"Local models (latest only, user 2026-09-26): Qwen/Qwen3.8-27B (production), prism-ml/Ternary-Bonsai-2-27B-gguf (2026-09-16, Apache-2.0, 1.72 bpw), XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (2026-09-21, MIT), XingChen-AGI/Xing4.0-29B-A4B (2026-09-16, Apache-2.0), Qwen3.8-Flash-Next (resource_infeasible on the workstation)",
"OpenAI GPT-6 family via Codex CLI: gpt-6-astra / gpt-6-sol / gpt-6-luna (codex debug models 2026-09-26)"
]
}
Loading
Loading