o11ylogsdb: in-memory engine design + 27 experiments validating 20× target

[strawgate]

Summary

Adds packages/o11ylogsdb — a browser-native (also Node, Bun, Edge) logs database for OpenTelemetry, sister to o11ytsdb. Goal: 20× storage efficiency vs raw OTLP/JSON.

• Engine: chunk format v1 with pluggable ChunkPolicy, per-(resource, scope) stream registry, baseline + columnar + Drain + typed-columnar policies, streaming query executor with header-only chunk pruning.
• M2 prototype: in-house Drain Rust port at packages/o11ylogsdb/rust-prototype/drain/. 6.7 KB gz, 0.9–3.3 M logs/s native, ARI = 1.0 vs the published Python reference on five public log corpora. TS port at src/drain.ts is bit-identical.
• M4 first cut: ColumnarDrainPolicy + TypedColumnarDrainPolicy with generic byte-shape detectors (SIGNED_INT, UUID, UUID_NODASH, PREFIXED_INT64, PREFIXED_UUID, TIMESTAMP_DELTA). Engine doesn't assume the log format.
• Bench harness: 26 reproducible bench modules under bench/, public Loghub-2k corpora + a synthetic structured-JSON corpus checked in.
• Dev-docs: dev-docs/findings.md, dev-docs/techniques.md, dev-docs/drain-prototype.md capture the validated/refuted assumptions and the codec/index/query stack we ship. No vendor coupling — algorithms are described by their published names; product names of comparator systems are not used.

PR is no longer a draft. Milestones M0/M1/M5/M6/M7 are partially started or pending — see README for the status table. Ship is the design foundation: engine round-trips, query executor works against zone maps, M2 prototype is validation-complete.

Key files

• packages/o11ylogsdb/PLAN.md — milestones, gates, architectural decisions.
• packages/o11ylogsdb/dev-docs/findings.md — what we measured.
• packages/o11ylogsdb/dev-docs/techniques.md — what we ship and why.
• packages/o11ylogsdb/dev-docs/drain-prototype.md — the M2 Rust port.
• packages/o11ylogsdb/bench/README.md — bench modules and corpora.

Review-pass fixes

• LogStore.stats() now uses a cheap chunkWireSize() helper instead of materializing each chunk's full wire buffer (bot-flagged perf bottleneck).
• package-lock.json regenerated so npm ci picks up the new package.
• Dangling "Experiment X" labels stripped from src/, bench/, PLAN.md.

Test plan

• npm run typecheck:packages — passes
• npm run build:packages — passes
• npx vitest run --coverage — 32 files, 479 tests passed (sibling packages; o11ylogsdb has no unit tests yet — design-validated via bench)
• npm run check:release — exit 0
• npx biome check packages/o11ylogsdb — 0 warnings

🤖 Generated with Claude Code

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

Walkthrough

This pull request introduces o11ylogsdb, a new OpenTelemetry logs database package for browser and Node.js environments. The addition includes a complete TypeScript implementation with codec plugins (gzip, ZSTD), templated compression via Drain, columnar and typed-columnar encoding policies, chunk serialization, streaming queries, and a Rust prototype of the Drain algorithm compiled to WebAssembly. The package includes extensive benchmarks across compression strategies, ingest performance, query latency, memory usage, and specialized indexing techniques. Documentation covers architecture, techniques, findings, and execution milestones. Supporting changes update workspace configuration files to include the new package in build scripts and ignore benchmark output directories.

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gitar-bot]

⚠️ Performance: stats() re-serializes every chunk on each call

The stats() method calls serializeChunk(c).length inside a loop over all chunks, performing a full wire-format serialization (header encoding + buffer allocation) just to measure byte count. Since ChunkHeader already carries payloadBytes, the total serialized size can be computed from header metadata without re-serializing. This is called from benchmarks (e.g., engine-roundtrip.bench.ts) and will become a bottleneck as chunk count grows.

Suggested fix:

Cache the serialized byte length when a chunk is frozen (e.g., store it alongside the chunk in the stream registry), or compute it from `chunk.header.payloadBytes` plus deterministic wire overhead, avoiding the re-serialization entirely:

// In stats():
for (const c of chunkList) {
  totalLogs += c.header.nLogs;
  totalBytes += c.header.payloadBytes + WIRE_HEADER_OVERHEAD;
}

_{Was this helpful? React with 👍 / 👎 | Reply gitar fix to apply this suggestion}

[gitar-bot]

⚠️ Performance: stats() calls JSON.stringify on every chunk header each invocation

The previous review flagged that stats() re-serializes every chunk. The commit message says "stats() perf" was fixed, but the current code still calls chunkWireSize(c) inside a nested loop, which per chunk.ts:211-214 runs JSON.stringify on each chunk's header on every call. For a store with thousands of chunks, this is O(n) stringification work on a read-only summary method.

Consider caching wireSize on the Chunk object at freeze time (it's immutable after that), or maintaining a running total in appendChunk()/flush() so stats() is O(1).

Suggested fix:

// In engine.ts, maintain running totals:
private _totalChunkBytes = 0;
private _totalLogs = 0;
private _chunkCount = 0;

// In appendChunk calls (flush + append):
this._totalChunkBytes += chunkWireSize(chunk);
this._totalLogs += chunk.header.nLogs;
this._chunkCount++;

// stats() becomes O(1):
stats(): StoreStats {
  return { streams: this.streams.size(), chunks: this._chunkCount,
    totalLogs: this._totalLogs, totalChunkBytes: this._totalChunkBytes,
    bytesPerLog: this._totalLogs > 0 ? this._totalChunkBytes / this._totalLogs : 0 };
}

_{Was this helpful? React with 👍 / 👎 | Reply gitar fix to apply this suggestion}

[gitar-bot]

💡 Edge Case: iterRecords() silently drops unflushed in-flight records

iterRecords() only yields records from frozen chunks in the stream registry. Any records appended but not yet flushed (i.e. the current in-flight ChunkBuilder) are silently omitted. The bench code in engine-columnar.bench.ts:117 correctly calls store.flush() before iterating, but the engine API doesn't document or enforce this — a caller using iterRecords() without flush() will get incomplete data with no warning.

Consider either auto-flushing at the start of iterRecords(), or adding a JSDoc note that flush() must be called first.

Suggested fix:

// Option A: auto-flush
*iterRecords(): Generator<{ streamId: StreamId; records: LogRecord[] }> {
  this.flush();
  for (const id of this.streams.ids()) {
    // ...
  }
}

_{Was this helpful? React with 👍 / 👎 | Reply gitar fix to apply this suggestion}

[gitar-bot]

Note

Your trial team has used its Gitar budget, so automatic reviews are paused. Upgrade now to unlock full capacity. Comment "Gitar review" to trigger a review manually.
Learn more about usage limits

Code Review ⚠️ Changes requested 1 resolved / 4 findings

Implements an in-memory engine with high-performance experimental validation, but requires addressing excessive serialization overhead in stats() and potential data loss in iterRecords(). Resolved inconsistencies between the TL;DR and results tables in experiment V.

⚠️ Performance: stats() re-serializes every chunk on each call

📄 packages/o11ylogsdb/src/engine.ts:146-160

The stats() method calls serializeChunk(c).length inside a loop over all chunks, performing a full wire-format serialization (header encoding + buffer allocation) just to measure byte count. Since ChunkHeader already carries payloadBytes, the total serialized size can be computed from header metadata without re-serializing. This is called from benchmarks (e.g., engine-roundtrip.bench.ts) and will become a bottleneck as chunk count grows.

Suggested fix

Cache the serialized byte length when a chunk is frozen (e.g., store it alongside the chunk in the stream registry), or compute it from `chunk.header.payloadBytes` plus deterministic wire overhead, avoiding the re-serialization entirely:

// In stats():
for (const c of chunkList) {
  totalLogs += c.header.nLogs;
  totalBytes += c.header.payloadBytes + WIRE_HEADER_OVERHEAD;
}

⚠️ Performance: stats() calls JSON.stringify on every chunk header each invocation

📄 packages/o11ylogsdb/src/engine.ts:146-157

The previous review flagged that stats() re-serializes every chunk. The commit message says "stats() perf" was fixed, but the current code still calls chunkWireSize(c) inside a nested loop, which per chunk.ts:211-214 runs JSON.stringify on each chunk's header on every call. For a store with thousands of chunks, this is O(n) stringification work on a read-only summary method.

Consider caching wireSize on the Chunk object at freeze time (it's immutable after that), or maintaining a running total in appendChunk()/flush() so stats() is O(1).

Suggested fix

// In engine.ts, maintain running totals:
private _totalChunkBytes = 0;
private _totalLogs = 0;
private _chunkCount = 0;

// In appendChunk calls (flush + append):
this._totalChunkBytes += chunkWireSize(chunk);
this._totalLogs += chunk.header.nLogs;
this._chunkCount++;

// stats() becomes O(1):
stats(): StoreStats {
  return { streams: this.streams.size(), chunks: this._chunkCount,
    totalLogs: this._totalLogs, totalChunkBytes: this._totalChunkBytes,
    bytesPerLog: this._totalLogs > 0 ? this._totalChunkBytes / this._totalLogs : 0 };
}

💡 Edge Case: iterRecords() silently drops unflushed in-flight records

📄 packages/o11ylogsdb/src/engine.ts:172-179

iterRecords() only yields records from frozen chunks in the stream registry. Any records appended but not yet flushed (i.e. the current in-flight ChunkBuilder) are silently omitted. The bench code in engine-columnar.bench.ts:117 correctly calls store.flush() before iterating, but the engine API doesn't document or enforce this — a caller using iterRecords() without flush() will get incomplete data with no warning.

Consider either auto-flushing at the start of iterRecords(), or adding a JSDoc note that flush() must be called first.

Suggested fix

// Option A: auto-flush
*iterRecords(): Generator<{ streamId: StreamId; records: LogRecord[] }> {
  this.flush();
  for (const id of this.streams.ids()) {
    // ...
  }
}

✅ 1 resolved

✅ Bug: TL;DR and Results tables contradict each other in Exp V

📄 research/experiments/engine-typed/results.md:22-31 📄 research/experiments/engine-typed/results.md:111-120
The TL;DR table (lines 22-29) reports BGL at 20.81 B/log (-6.7%) and OpenStack at 19.51 B/log (-6.7%), with a mean of -4.1%. However, the actual Results table (lines 111-119) shows BGL at 22.43 B/log (+0.6%) and OpenStack at 21.01 B/log (+0.4%), with a mean of -1.5%. These are significantly different numbers in the same document. It appears the TL;DR describes a later implementation iteration (with UUID and BGL_TS slot types active) while the Results table reflects the measured baseline. This makes the experiment's conclusions unreliable for readers trying to reproduce or build on these results.

🤖 Prompt for agents

Code Review: Implements an in-memory engine with high-performance experimental validation, but requires addressing excessive serialization overhead in stats() and potential data loss in iterRecords(). Resolved inconsistencies between the TL;DR and results tables in experiment V.

1. ⚠️ Performance: stats() re-serializes every chunk on each call
   Files: packages/o11ylogsdb/src/engine.ts:146-160

   The `stats()` method calls `serializeChunk(c).length` inside a loop over all chunks, performing a full wire-format serialization (header encoding + buffer allocation) just to measure byte count. Since `ChunkHeader` already carries `payloadBytes`, the total serialized size can be computed from header metadata without re-serializing. This is called from benchmarks (e.g., `engine-roundtrip.bench.ts`) and will become a bottleneck as chunk count grows.

   Suggested fix:
   Cache the serialized byte length when a chunk is frozen (e.g., store it alongside the chunk in the stream registry), or compute it from `chunk.header.payloadBytes` plus deterministic wire overhead, avoiding the re-serialization entirely:
   
   // In stats():
   for (const c of chunkList) {
     totalLogs += c.header.nLogs;
     totalBytes += c.header.payloadBytes + WIRE_HEADER_OVERHEAD;
   }

2. ⚠️ Performance: stats() calls JSON.stringify on every chunk header each invocation
   Files: packages/o11ylogsdb/src/engine.ts:146-157

   The previous review flagged that `stats()` re-serializes every chunk. The commit message says "stats() perf" was fixed, but the current code still calls `chunkWireSize(c)` inside a nested loop, which per `chunk.ts:211-214` runs `JSON.stringify` on each chunk's header on every call. For a store with thousands of chunks, this is O(n) stringification work on a read-only summary method.
   
   Consider caching `wireSize` on the `Chunk` object at freeze time (it's immutable after that), or maintaining a running total in `appendChunk()`/`flush()` so `stats()` is O(1).

   Suggested fix:
   // In engine.ts, maintain running totals:
   private _totalChunkBytes = 0;
   private _totalLogs = 0;
   private _chunkCount = 0;
   
   // In appendChunk calls (flush + append):
   this._totalChunkBytes += chunkWireSize(chunk);
   this._totalLogs += chunk.header.nLogs;
   this._chunkCount++;
   
   // stats() becomes O(1):
   stats(): StoreStats {
     return { streams: this.streams.size(), chunks: this._chunkCount,
       totalLogs: this._totalLogs, totalChunkBytes: this._totalChunkBytes,
       bytesPerLog: this._totalLogs > 0 ? this._totalChunkBytes / this._totalLogs : 0 };
   }

3. 💡 Edge Case: iterRecords() silently drops unflushed in-flight records
   Files: packages/o11ylogsdb/src/engine.ts:172-179

   `iterRecords()` only yields records from frozen chunks in the stream registry. Any records appended but not yet flushed (i.e. the current in-flight `ChunkBuilder`) are silently omitted. The bench code in `engine-columnar.bench.ts:117` correctly calls `store.flush()` before iterating, but the engine API doesn't document or enforce this — a caller using `iterRecords()` without `flush()` will get incomplete data with no warning.
   
   Consider either auto-flushing at the start of `iterRecords()`, or adding a JSDoc note that `flush()` must be called first.

   Suggested fix:
   // Option A: auto-flush
   *iterRecords(): Generator<{ streamId: StreamId; records: LogRecord[] }> {
     this.flush();
     for (const id of this.streams.ids()) {
       // ...
     }
   }

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

---

Important

Your trial ends in 6 days — upgrade now to keep code review, CI analysis, auto-apply, custom automations, and more.

_{Was this helpful? React with 👍 / 👎 | Gitar}