feat(search): make large local corpora fast and fail-safe

benchmarks/search/manifest.json

@@ -0,0 +1,128 @@
+{
+  "schemaVersion": "dotaios-search-benchmark/v1",
+  "benchmarkId": "search-baseline-2026-08-13",
+  "referenceMachine": {
+    "identifier": "mac16-1-m4-10c-16gb",
+    "modelIdentifier": "Mac16,1",
+    "chip": "Apple M4",
+    "cpuCores": 10,
+    "memoryGiB": 16,
+    "operatingSystem": "macOS 26.6.1 build 25G76 (Darwin 25.6.0)",
+    "powerProfile": {
+      "source": "battery",
+      "lowPowerMode": false,
+      "requireIdleMachine": true
+    }
+  },
+  "runtime": {
+    "supportedNodeMajors": [
+      20,
+      22
+    ],
+    "baselineNode": "22.22.3"
+  },
+  "corpus": {
+    "generator": {
+      "version": "dotaios-search-fixture-v1",
+      "seed": 20260813,
+      "fixedMtime": "2026-08-13T00:00:00.000Z"
+    },
+    "fileCounts": [
+      500,
+      2500,
+      10000
+    ],
+    "layouts": {
+      "shallow": {
+        "kind": "bucketed",
+        "directoryDepth": 1,
+        "bucketCount": 2
+      },
+      "nested": {
+        "kind": "tree",
+        "directoryDepth": 3,
+        "branchingFactor": 8
+      }
+    },
+    "distributions": {
+      "prose": {
+        "kind": "representative-prose",
+        "targetBytes": {
+          "min": 768,
+          "max": 1280
+        },
+        "vocabularySize": 192,
+        "frontmatterEvery": 4
+      },
+      "high-entropy": {
+        "kind": "adversarial-high-entropy",
+        "targetBytes": {
+          "min": 768,
+          "max": 1280
+        },
+        "tokenLength": 18,
+        "frontmatterEvery": 4
+      }
+    },
+    "scenarioMatrix": [
+      {
+        "layout": "shallow",
+        "distribution": "prose"
+      },
+      {
+        "layout": "nested",
+        "distribution": "high-entropy"
+      }
+    ]
+  },
+  "queries": [
+    {
+      "id": "no-hit",
+      "text": "zqxj-unfindable-20260813",
+      "expectation": {
+        "kind": "none",
+        "hitCount": 0
+      }
+    },
+    {
+      "id": "low-hit",
+      "text": "controlled-peregrine-benchmark-needle",
+      "expectation": {
+        "kind": "fixed-indices",
+        "fileIndices": [
+          3,
+          17,
+          101,
+          307
+        ],
+        "hitCount": 4
+      }
+    },
+    {
+      "id": "high-hit",
+      "text": "common-benchmark-marker-needle",
+      "expectation": {
+        "kind": "modulo",
+        "modulo": 25,
+        "remainder": 0,
+        "resultLimit": 20
+      }
+    }
+  ],
+  "protocol": {
+    "coldSamples": 3,
+    "warmupSamples": 3,
+    "measuredSamples": 20,
+    "concurrency": 32,
+    "resultLimit": 20,
+    "rssPollIntervalMs": 5,
+    "coldDefinition": "fresh request-scoped reader before harness warm-up; operating-system file cache is uncontrolled",
+    "warmDefinition": "fresh request-scoped reader after warm-up in the same Node process",
+    "rawReadControl": {
+      "enabled": true,
+      "method": "open-handle-read-all",
+      "validation": "exact file count and byte total",
+      "concurrency": 32
+    }
+  }
+}

docs/adr/0003-keep-canonical-memory-separate-from-derived-views.md

@@ -15,7 +15,7 @@ host-specific views without making their loss or corruption a loss of memory.
 | --- | --- | --- | --- |
 | Durable user context | User-authored files under `context/`, project records, decisions, and daily memory | The person directly, or a command they explicitly invoke for that exact record | Working context, search results, generated summaries |
 | Source material | Provenance-bearing files under `vault/` and other explicit imports | Explicit ingest/capture commands; later edits remain the person's | Search snippets, source indexes |
-| Recent event and signal memory | Append-only records under `memory/` | Explicit capture/log workflows and configured local automations with a named write contract | Bounded startup selection, search results, archives produced by explicit maintenance |
+| Recent event and signal memory | Append-only live records and bounded archive shards under `memory/` | Explicit capture/log workflows and configured local automations with a named write contract; maintenance may move complete records into canonical cold-storage shards | Bounded startup selection and search results |
 | Session evidence | Readable session Markdown under `memory/sessions/<date>/` | Explicit save/import or a separately enabled host-capture workflow | `memory/sessions/index.jsonl`, working-context selections, search results |
 | Managed scaffold | Files or marked regions DotAIOS can prove it owns | Previewed setup, activation, migration, repair, disconnect, or removal operations | Installation inventory and health reports |
 | Operational evidence | Receipts, recovery metadata, locks, metrics, and quarantine material | The exact operation that owns the artifact | Status and doctor summaries |
@@ -35,6 +35,14 @@ the Markdown durably before its index entry becomes visible, reconcile must
 recover orphans without deleting evidence, and delete must prove ownership of
 the exact canonical file before changing either representation.
 
+For event and signal memory, an archive shard is canonical cold storage once
+maintenance removes those records from the live file. It is not a disposable
+index or cache: search must include it, crash recovery must preserve exact
+record multiplicity, and maintenance may delete or replace it only through the
+documented ownership-checked archive protocol. Rotation markers, transaction
+envelopes, and format witnesses remain operational evidence; they prove or
+recover a state transition but are not user memory themselves.
+
 Rejected alternatives:
 
 - Treat the session index as authoritative: a torn or tampered row could hide

docs/advanced-memory.md

@@ -32,6 +32,30 @@ The audit never deletes memory. By default it follows DotAIOS memory routing:
 the last 50 `memory/events.jsonl` entries plus today/yesterday signal files. Use
 `--all-memory` only when you want a deeper forensic pass over older history.
 
+Older events and trimmed signals remain canonical JSONL. DotAIOS keeps each
+active archive below 2 MiB, rotating complete records into immutable numbered
+files (`events-archive.000001.jsonl`, for example). Search reads those shards
+automatically; there is no index to rebuild and no archived record becomes
+write-only. A single record may occupy its own shard up to the 4 MiB safe-read
+ceiling. Anything larger stops maintenance before the source is removed.
+Rotation uses a durable format witness and crash marker. An ambiguous archive
+created by the older markerless rotator is left byte-for-byte unchanged and
+reports `DOTAIOS_ARCHIVE_LEGACY_RECOVERY_REQUIRED` for explicit inspection;
+DotAIOS never guesses whether an identical shard prefix is a retry or a
+legitimate duplicate. A restart also repairs the narrowly proven two-name
+hard-link state left when a process dies during exclusive publication, while
+rejecting unrelated links.
+
+Execution-time runway check (2026-08-13): the live AIOS event archive is
+199,021 bytes / 725 lines and the signal archive is 66,177 bytes / 203 lines.
+At the observed Git-visible rates of roughly 30 event lines/day and 10 signal
+lines/day since 2026-07-27, the coarse per-file runways were about 480 and
+1,240 days under the former 4 MiB single-file limit, or about 230 and 620 days
+to the new 2 MiB rotation point from this snapshot. Rotation is therefore a
+durability bound, not a reason to add a database or persistent search index.
+The existing signal archive was an eligible legacy 0644 file; the next locked
+maintenance run narrows that exact safe case to 0600 before publication.
+
 `--write-queue` writes proposed skill patches to
 `memory/skill-patches/queue.md` with stable IDs, so cleanup or compaction does
 not duplicate the same lesson. If the queue is intentionally capped, the report

docs/architecture.md

@@ -122,6 +122,101 @@ a bounded `operational.migration` sibling;
 selection, and `resolve_skill` routes
 workflow intent. There are no compatibility aliases.
 
+### On-demand search
+
+Markdown search is a request-scoped safe corpus transaction. The evidence
+reader enumerates eligible regular files and returns transaction-owned
+`filePath`, UTF-8 `content`, and `mtimeMs` observations to one callback.
+Canonical matching, snippet construction, whole-corpus IDF statistics, recency
+ranking, stable ordering, and the result limit all run inside that callback.
+The search promise cannot resolve until the evidence reader has performed its
+final root, ancestor, and observed-directory generation validation; a changed
+generation rejects the request without publishing a partial result.
+
+The transaction preserves each logical corpus boundary and its source policy:
+daily and inbox notes remain separate from memory streams, plugin search accepts
+Markdown plus `manifest.json`, project search reads only the resolved selector,
+and an external vault remains its own explicitly authorized root. Hidden and
+secret-like entries remain ineligible. Linked, non-regular, changed, invalid
+UTF-8, unauthorized, misconfigured, or unexpectedly unreadable observed
+evidence rejects the whole request.
+
+Resource ceilings are different. One request-owned discovery transaction uses
+phase-local fair ledgers before metadata inspection or catalog reads can spend
+the shared, non-releasable physical ledger. Half of each currently available
+byte, file, and entry ceiling is reserved as equal protected shares; unused
+capacity is redistributed in declared order. The same rule is applied to the
+bounded catalog discovery needed for exact JSONL entry counts or session
+membership. Session discovery replays the public reverse-order filters, query,
+and limit, so it retains each body at most once and never charges a body that a
+title, agent, or project hit makes unnecessary. Retained catalog/body bytes are
+never reread. Only scopes whose remaining work fits have their ordinary content
+read and are tokenized and ranked. Otherwise the whole scope is omitted so
+partial-corpus IDF and ranking are never presented as complete. Every inspected
+file plus each directory, ancestor, and root observation remains
+transaction-owned and is revalidated before results resolve; all phase readers
+and prepared capabilities close on success or failure.
+
+Successful search arrays retain their iterable group shape and expose frozen,
+non-enumerable `scope` and `omissions` metadata. Omissions use the five primary
+closed reason codes `file_too_large`, `directory_entries_exceeded`,
+`aggregate_bytes_exceeded`, `file_count_exceeded`, and
+`entry_count_exceeded`; the explicit aggregate-remainder reason is
+`omissions_truncated`; contain bounded counts and path-free recovery text; and
+are capped at 32 records plus one defensive aggregate remainder. A directory
+ceiling is `partially_enumerated`; other ceiling omissions are `not_searched`.
+Every observed directory, including a directory stopped at its ceiling, is
+revalidated after ranking and before results resolve. The CLI prints valid
+results to stdout, warnings to stderr, and exits 2 for incomplete searches.
+Exit 0 is complete, including zero hits, while integrity and configuration
+failures remain exit 1.
+
+Search writes no index, cache, manifest, or other derived state. Each request
+enumerates the current canonical files, so additions, edits, and deletions are
+visible on the next request. The optimization amortizes repeated containment
+checks only for the lifetime of that request; the AIOS folder remains the sole
+search authority.
+
+### Bounded memory archives
+
+Event compaction and stale-signal trimming keep the unsuffixed
+`events-archive.jsonl` and `signals-archive.jsonl` files as active append
+targets. Before an append would cross 2 MiB, maintenance publishes complete
+JSONL records into immutable, zero-padded shards such as
+`events-archive.000001.jsonl`. Numbered shards are searched in numeric order,
+then the active archive. Exact retry overlap is deduplicated before corpus
+statistics and ranking, so an interruption cannot turn one event into two
+search results.
+
+One valid record above 2 MiB but no larger than the 4 MiB evidence-file ceiling
+occupies a shard by itself. A larger record stops maintenance before the live
+event generation is replaced or a stale signal source is removed. The pending
+batch remains recovery authority until shard and active-file publication have
+been fsynced. A durable `*.rotation-format` witness separates new
+marker-protocol generations from ambiguous overlap left by the older
+markerless rotator. If the witness is absent and the newest shard is an exact
+prefix of the active archive, maintenance fails before mutation with
+`DOTAIOS_ARCHIVE_LEGACY_RECOVERY_REQUIRED`; an operator can inspect both
+authoritative copies instead of DotAIOS guessing whether equal records are a
+retry or legitimate duplicates. Each shard is created exclusively at mode 0600
+and is never overwritten; active and pending files must be owned, regular,
+single-link files. Maintenance narrowly secures an eligible legacy 0644 active
+archive to 0600, but rejects links, wrong ownership, broader modes, and unsafe
+pre-existing shard targets.
+
+Exclusive publication links an owned UUID temporary into its final name. If a
+real process death leaves those two names on the same inode, restart recovery
+removes only the single proven temporary, fsyncs the directory, and revalidates
+the final file as the same owned, mode-0600, single-link object. Any different
+hard-link state remains fatal.
+
+Search observes the memory directory before reading the numbered generation
+and revalidates it before results resolve. A concurrent rotation therefore
+returns the complete old generation, the complete new generation, or a fatal
+source-changed retry—never an accepted mixture. Eventually, many valid shards
+can exhaust the request-wide search ceiling; the resource-ceiling contract
+above then reports the whole memory scope as an explicit omission.
+
 ## Vault
 
 `vault/` is long-term knowledge, loaded on demand. Users may keep it inside `~/aios/vault` or configure an external `vault_path` in `aios.json`, such as an Obsidian vault.