Skip to content

feat(hook,cli): injection log + aelf tail (#321) - #352

Merged
robotrocketscience merged 4 commits into
mainfrom
feat/issue-321-injection-log
May 2, 2026
Merged

feat(hook,cli): injection log + aelf tail (#321)#352
robotrocketscience merged 4 commits into
mainfrom
feat/issue-321-injection-log

Conversation

@robotrocketscience

@robotrocketscience robotrocketscience commented May 2, 2026

Copy link
Copy Markdown
Owner

Closes #321.

Approach

Per the pre-implementation comment, this PR takes path (a): extend the existing hook_audit.jsonl (shipped in #280 mitigation 3) rather than ship a parallel injections.ndjson. The rationale and three-path breakdown is in the issue thread; the user picked (a).

Net result is a single audit log on the file system — same path, same rotation, same writer entry-points — with additive structured fields and a new reader CLI.

Schema additions to hook_audit.jsonl

All additive and optional, so older readers keep working unchanged:

Field Type Source
beliefs list[{id, lane, locked, content_hash, alpha, beta, posterior_mean, snippet}] per-hit projection of the rendered belief list
latency_ms int wall-clock around retrieve+format span
tokens int rendered_block sized via the same 4-chars-per-token estimator retrieval uses for budgeting

lane is derived from lock_level (LOCK_USER"L0", else "L1"). Per-hit BM25 / ranking scores are intentionally not included — retrieve() does not propagate per-hit scores through to the hook caller, and adding that plumbing was out of scope for #321 (replay harness over historical logs is explicitly out-of-scope in the issue body too). posterior_mean is computed from the belief's alpha/beta so the user can still see Bayesian confidence at a glance.

snippet is the first line of belief.content capped at 120 chars; the full rendered_block is still on the record (existing field), so nothing is lost — the snippet is for at-a-glance scanning in aelf tail output.

aelf tail

$ aelf tail --help
usage: aelf tail [-h] [--filter key=value] [--since DUR] [--no-blob] [--no-follow]

  --filter key=value  hook=<name> | lane=L0 | lane=L1. Repeatable, AND-joined.
  --since DUR         30s | 5m | 2h | 1d — backfill rotated + live before tailing
  --no-blob           suppress per-belief snippet bodies (header + ids only)
  --no-follow         dump current contents once and exit

Header line: <HH:MM:SS> <hook> <tokens> tok <latency> ms L0×N L1×M. Then one indented line per belief from beliefs[].

Default behaviour (no --since, follow): start at end-of-file and stream new records. Rotation is detected via inode change so tail survives a single-slot rotation. Filter semantics are falsifying (records missing the queried field never match) so the count of matched records is well-defined.

The reader lives in a new aelfrice.hook_tail module — splitting it from aelfrice.hook keeps the hook write path import-cheap (every UserPromptSubmit fire pays its import cost).

Tests

  • 27 new tests in tests/test_hook_tail.py covering: filter parser (valid / unknown key / missing value), --since parser (s/m/h/d, malformed), record_matches_filters (hook + lane semantics + AND), format_record (header + per-belief + --no-blob + missing-optional-fields back-compat), one-shot tail, follow mode (new-line pickup + rotation survival), --since (rotated + live backfill), and an end-to-end hook-fire-then-tail integration test.
  • tests/test_hook_audit.py::test_rotation_at_max_bytes threshold bumped 500 → 1000 to accommodate the larger record (~650B per fire vs. ~250B previously).
  • tests/test_slash_commands.py updated: added tail to EXPECTED_COMMANDS. New slash_commands/tail.md.
  • pytest -q → 2013 passed, 14 skipped (pre-existing skips, unrelated).

Out of scope (deferred)

  • Per-hit BM25 score plumbing through retrieve() → audit record.
  • aelf statusline-inject (Optional, separate commit per issue body).
  • Replay harness over historical logs.

Discretion

Reserved-vocab grep over the full diff is empty.

Summary by Sourcery

Extend hook injection auditing with richer per-belief metadata and add a CLI subcommand to live-tail and inspect these audit logs.

New Features:

  • Add per-belief audit metadata (beliefs list, token estimates, latency) to hook audit records for UserPromptSubmit and SessionStart hooks.
  • Introduce the aelf tail CLI subcommand and backing hook_tail module to pretty-print and live-tail hook audit logs with filtering and time-based backfill.
  • Define a new slash command aelf:tail to expose the tail functionality via slash commands.

Enhancements:

  • Estimate and persist token counts for rendered hook injection blocks using the existing 4-chars-per-token heuristic.
  • Capture and record end-to-end retrieval+format latency for each hook fire in the audit log.
  • Serialize beliefs into a compact, lane-annotated snippet form to support quick scanning in tail output.
  • Adjust hook audit rotation threshold in tests to account for larger audit records produced by the new schema additions.

Documentation:

  • Document the aelf tail slash command usage, arguments, and behavior.

Tests:

  • Add comprehensive tests for the new hook_tail module, including filter and since parsing, matching semantics, formatting, follow mode, rotation handling, and end-to-end hook-to-tail flows.
  • Update existing tests to reflect the new tail subcommand and increased audit record size.

Summary by CodeRabbit

Release Notes

  • New Features

    • Added aelf tail CLI command for live-tailing hook injection audit logs with optional record filtering, time-window filtering, and output control options (--no-follow for one-shot mode, --no-blob to exclude content snippets).
    • Enhanced audit logs to include per-turn latency measurements and token counts.
  • Documentation

    • Added reference documentation for the aelf tail command.
  • Tests

    • Added comprehensive test coverage for tail functionality and audit logging enhancements.

@coderabbitai

coderabbitai Bot commented May 2, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

This PR introduces live observability for hook-injected audit logs via a new aelf tail CLI command. Hook-side code enriches audit records with belief lists, latency measurements, and token counts. A new tail-reader module filters and formats these records, supporting live following with rotation detection and time-window backfill. CLI wiring and comprehensive tests complete the feature.

Changes

Hook Audit Enrichment & Live-Tail Feature

Layer / File(s) Summary
Audit Schema Enrichment
src/aelfrice/hook.py
Hook audit records now include serialized beliefs (lane/locked/hash/posterior fields), latency_ms wall-clock retrieval time, and estimated tokens count derived from the rendered block. Introduced helpers to cap belief snippets and serialize Belief objects; both user_prompt_submit and session_start measure latency and pass hits to the audit writer.
Tail Reader Implementation
src/aelfrice/hook_tail.py
New module implementing the core tail-audit reader: parse_filter() and parse_since() for CLI argument validation; record_matches_filters() for hook/lane membership filtering; format_record() for multi-line textual rendering; tail_audit() supporting one-shot dump, follow mode with polling and rotation detection via inode changes, and --since backfill from rotated file suffix and live file.
CLI Wiring
src/aelfrice/cli.py
New _cmd_tail subcommand handler parses filter/since/output flags, validates inputs (exit code 2 on parse error), and delegates to tail_audit(...) with computed parameters. Registered in build_parser() with all supported arguments.
Tests & Documentation
tests/test_hook_tail.py, tests/test_hook_audit.py, tests/test_slash_commands.py, src/aelfrice/slash_commands/tail.md
Comprehensive test coverage for filter/since parsing, record matching, record formatting, one-shot/follow/rotation/backfill modes, and end-to-end hook-to-tail flow. Rotation test max_bytes adjusted from 500 to 1000 bytes to reflect larger records. Documentation page and slash-command registry updated.

Sequence Diagram

sequenceDiagram
    participant U as User/Prompt
    participant H as Hook (user_prompt_submit)
    participant R as Retrieval System
    participant A as Audit File
    participant T as aelf tail
    participant O as Output

    U->>H: Incoming prompt
    activate H
    H->>R: time.monotonic() start
    R->>R: Match beliefs
    R-->>H: hits (Belief list)
    H->>H: Measure latency_ms
    H->>H: Serialize beliefs (lane/locked/hash/posterior)
    H->>H: Count tokens from rendered block
    H->>A: Write record (ts/hook/beliefs/tokens/latency_ms)
    deactivate H

    User->>T: aelf tail --filter lane=L0
    activate T
    T->>A: Open audit file, seek to end (follow=True)
    T->>A: Poll for appended lines
    A-->>T: New JSON record
    T->>T: Parse JSON
    T->>T: record_matches_filters(lane=L0)
    T->>T: format_record (header + per-belief lines)
    T->>O: Write formatted output
    T->>O: Flush
    Note over T: Detect rotation via inode change<br/>Reset to start, re-read
    deactivate T
Loading
sequenceDiagram
    participant U as User<br/>(aelf tail --since 5m)
    participant T as tail_audit()
    participant R as Rotated audit.1
    participant L as Live audit
    participant O as Output

    U->>T: tail_audit(since=timedelta(minutes=5), follow=False)
    activate T
    T->>T: Compute since_cutoff (now - 5m)
    T->>R: _read_records(audit_path.1)
    R-->>T: [records]
    T->>T: Filter by ts >= since_cutoff
    T->>L: _read_records(audit_path)
    L-->>T: [records]
    T->>T: Filter by ts >= since_cutoff
    T->>T: Merge & sort records chronologically
    T->>T: Apply user filters (hook/lane)
    T->>O: _emit_records (format + write)
    T-->>U: return 0
    deactivate T
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.91% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'feat(hook,cli): injection log + aelf tail (#321)' clearly summarizes the main changes: adding injection logging and a new CLI tail command.
Description check ✅ Passed The description covers all template sections: summary, linked issues, type of change (feat), verification checklist completion, comprehensive test plan, and detailed notes for reviewers.
Linked Issues check ✅ Passed The PR successfully implements the core requirements from issue #321: extended hook_audit.jsonl schema with beliefs/latency/tokens, new aelf tail CLI with filtering (--filter, --since, --no-blob), and comprehensive tests (27 new tests with parsing, matching, formatting, follow-mode, rotation, and end-to-end integration).
Out of Scope Changes check ✅ Passed All changes are directly aligned with issue #321 requirements: hook audit schema extensions, CLI implementation, reader module, tests, and documentation. Out-of-scope items (per-hit BM25, statusline-inject, replay harness) are explicitly noted as deferred.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/issue-321-injection-log

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
Review rate limit: 0/1 reviews remaining, refill in 60 minutes.

Comment @coderabbitai help to get the list of available commands and usage tips.

@sourcery-ai

sourcery-ai Bot commented May 2, 2026

Copy link
Copy Markdown

Reviewer's Guide

Extends the hook audit logging to include structured per-belief metadata, latency, and token estimates, and introduces an aelf tail CLI subcommand with a dedicated reader module to pretty-print and live-tail the enriched audit log, including filters and since-based backfill, along with comprehensive tests and minor config updates.

Sequence diagram for aelf tail CLI reading the hook audit log

sequenceDiagram
    actor Operator
    participant CLI as aelf_cli
    participant Tail as HookTailReader
    participant Hook as HookAuditWriter
    participant FS as FileSystem

    Operator->>CLI: invoke aelf tail [--filter] [--since] [--no-blob] [--no-follow]
    CLI->>Tail: parse_filter(spec) for each --filter
    Tail-->>CLI: (key, value) filters or ValueError
    CLI->>Tail: parse_since(spec) for --since (optional)
    Tail-->>CLI: timedelta since or ValueError
    CLI->>Tail: tail_audit(filters, since, include_blob, follow, out)

    alt since provided
        Tail->>Hook: _audit_path_for_db(db_path())
        Hook-->>Tail: audit_path
        Tail->>FS: read rotated + live audit files
        FS-->>Tail: JSONL records
        Tail->>Tail: _read_records + _parse_record_ts
        Tail->>Tail: record_matches_filters(record, filters)
        Tail->>Tail: format_record(record, include_blob)
        Tail->>CLI: write formatted records to out
    end

    alt follow is False
        Tail-->>CLI: return after one-shot emit
    else follow is True
        loop poll until interrupted
            Tail->>FS: stat audit_path (size, inode)
            FS-->>Tail: st_size, st_ino
            alt inode changed
                Tail->>Tail: reset offset (rotation detected)
            end
            Tail->>FS: read new bytes from audit_path
            FS-->>Tail: new JSONL lines
            Tail->>Tail: decode JSON, record_matches_filters
            Tail->>Tail: format_record(record, include_blob)
            Tail->>CLI: write formatted records to out
        end
    end
Loading

Entity-relationship diagram for extended hook_audit.jsonl schema

erDiagram
    AUDIT_RECORD {
        string ts
        string hook
        string prompt
        string rendered_block
        int n_beliefs
        int n_locked
        string session_id
        int tokens
        int latency_ms
    }

    BELIEF_ENTRY {
        string id
        string lane
        boolean locked
        string content_hash
        float alpha
        float beta
        float posterior_mean
        string snippet
    }

    AUDIT_RECORD ||--o{ BELIEF_ENTRY : beliefs
Loading

Updated class diagram for hook audit writer and hook_tail reader modules

classDiagram
    class Belief {
        string id
        string lock_level
        string content_hash
        float alpha
        float beta
        string content
    }

    class HookAuditWriter {
        <<module>>
        +int AUDIT_BELIEF_SNIPPET_CAP
        +_belief_snippet(content str) str
        +_serialize_belief_for_audit(b Belief) dict~str, object~
        +_audit_tokens_from_block(block str) int
        +_write_hook_audit_record(hook str, prompt str, rendered_block str, n_beliefs int, n_locked int, session_id str, beliefs list~Belief~, latency_ms int, config HookAuditConfig, stderr IO_str) void
    }

    class HookTailReader {
        <<module>>
        +parse_filter(spec str) tuple~str, str~
        +parse_since(spec str) timedelta
        +record_matches_filters(record dict~str, object~, filters list~tuple~str, str~~) bool
        +format_record(record dict~str, object~, include_blob bool) str
        +tail_audit(audit_path Path, filters list~tuple~str, str~~, since timedelta, include_blob bool, follow bool, out IO_str, poll_interval float, max_iters int) int
    }

    class AelfCLI {
        <<module>>
        +_cmd_tail(args Namespace, out object) int
        +build_parser(show_advanced bool) ArgumentParser
    }

    class HookAuditConfig {
        bool enabled
        int max_bytes
    }

    HookAuditWriter --> Belief : serializes
    HookAuditWriter --> HookAuditConfig : uses
    HookTailReader --> HookAuditWriter : uses AUDIT_FILENAME, AUDIT_ROTATED_SUFFIX, _audit_path_for_db
    AelfCLI --> HookTailReader : calls tail_audit
Loading

File-Level Changes

Change Details Files
Extend hook audit records with per-belief structured data, latency, and token estimation, and wire it into existing hooks.
  • Add helpers to compute belief snippets and serialize Belief objects into audit-friendly dicts including lane, lock status, Bayesian posterior, and snippet.
  • Update _write_hook_audit_record to accept optional beliefs and latency parameters, compute token counts from the rendered block, and emit the new fields when present while preserving backward compatibility.
  • Measure retrieval+format latency in user_prompt_submit and session_start, and pass hits and latency into _write_hook_audit_record for each fire.
src/aelfrice/hook.py
Add aelf tail CLI subcommand backed by a new audit-log reader/pretty-printer module that supports filtering, since-based backfill, and follow-mode with rotation handling.
  • Introduce _cmd_tail wiring in the CLI to parse --filter, --since, --no-blob, and --no-follow flags, map them to typed parameters, and invoke tail_audit.
  • Register the new tail subcommand in the main argument parser, including help text for filters, since durations, and blob suppression options.
  • Implement hook_tail module with parsing helpers for filters and since durations, record filtering logic over hook and lane, record formatting into human-readable lines, and a tailing loop that supports optional backfill, live follow, inode-based rotation detection, and test hooks for polling interval/iterations.
src/aelfrice/cli.py
src/aelfrice/hook_tail.py
Adjust existing tests and add a dedicated test suite for the new tail functionality and enriched audit records.
  • Increase the audit rotation max_bytes in the rotation test to account for larger enriched records and document the rationale in comments.
  • Add tail to the expected slash commands list and document the new slash command in a markdown file for tool invocation.
  • Create test_hook_tail.py with unit and integration tests covering filter/since parsers, record filtering, formatting, one-shot and follow-mode tailing, rotation survival, since-based backfill across rotated and live files, and an end-to-end hook-fire-then-tail scenario.
tests/test_hook_audit.py
tests/test_slash_commands.py
tests/test_hook_tail.py
src/aelfrice/slash_commands/tail.md

Assessment against linked issues

Issue Objective Addressed Explanation
#321 Add a structured, append-only per-injection log for UserPromptSubmit and SessionStart hooks (including beliefs with lanes/locked status, token count, latency) with size-based rotation, stored under the repo’s .git/aelfrice area.
#321 Implement an aelf tail CLI command that provides a tail -f-style live pretty-printer over the injection log, with filters (e.g., by lane or hook), optional suppression of belief bodies, and support for backfilling recent history before live tailing.

Possibly linked issues

  • #[v1.5.x]: Yes. The PR implements the issue’s injection observability: structured per-injection audit fields plus the aelf tail CLI.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@robotrocketscience robotrocketscience added the attn:review Needs review (PR open, awaiting reviewer) label May 2, 2026
Comment thread src/aelfrice/hook_tail.py
from __future__ import annotations

import json
import os
Comment thread src/aelfrice/hook_tail.py
Comment on lines +26 to +30
from aelfrice.hook import (
AUDIT_FILENAME,
AUDIT_ROTATED_SUFFIX,
_audit_path_for_db,
)
Comment thread tests/test_hook_tail.py
Comment on lines +26 to +31
from aelfrice.hook import (
AUDIT_FILENAME,
AUDIT_ROTATED_SUFFIX,
_audit_path_for_db,
_write_hook_audit_record,
)

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue, and left some high level feedback:

  • There are two separate JSONL parsing paths in hook_tail (_read_records and the follow-mode loop) that implement slightly different but overlapping logic; consider extracting a shared iter_audit_records(path, since_cutoff=None) helper and reusing it in both places to keep behavior and edge-case handling (e.g. malformed lines) consistent and easier to maintain.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- There are two separate JSONL parsing paths in `hook_tail` (`_read_records` and the follow-mode loop) that implement slightly different but overlapping logic; consider extracting a shared `iter_audit_records(path, since_cutoff=None)` helper and reusing it in both places to keep behavior and edge-case handling (e.g. malformed lines) consistent and easier to maintain.

## Individual Comments

### Comment 1
<location path="src/aelfrice/slash_commands/tail.md" line_range="10" />
<code_context>
+---
+<objective>
+Stream the per-turn hook audit log so the operator can see exactly
+which beliefs each UserPromptSubmit / SessionStart fire injected — id,
+lane (L0 locked / L1 retrieved), token count, latency, and a snippet.
+By default tails forever; pass `--no-follow` for a one-shot dump.
</code_context>
<issue_to_address>
**suggestion (typo):** Consider rephrasing "fire" here to something like "firing" for grammatical clarity.

The construction "each UserPromptSubmit / SessionStart fire injected" is ungrammatical. Consider something like "each UserPromptSubmit / SessionStart firing injected" or "each UserPromptSubmit / SessionStart hook invocation injected" for smoother wording.

```suggestion
which beliefs each UserPromptSubmit / SessionStart hook invocation injected — id,
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

---
<objective>
Stream the per-turn hook audit log so the operator can see exactly
which beliefs each UserPromptSubmit / SessionStart fire injected — id,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (typo): Consider rephrasing "fire" here to something like "firing" for grammatical clarity.

The construction "each UserPromptSubmit / SessionStart fire injected" is ungrammatical. Consider something like "each UserPromptSubmit / SessionStart firing injected" or "each UserPromptSubmit / SessionStart hook invocation injected" for smoother wording.

Suggested change
which beliefs each UserPromptSubmit / SessionStart fire injected — id,
which beliefs each UserPromptSubmit / SessionStart hook invocation injected — id,

@robotrocketscience

Copy link
Copy Markdown
Owner Author

[claim:review:Kulili:2026-05-02T19:39:47Z]

@robotrocketscience

Copy link
Copy Markdown
Owner Author

Reviewed. Diff is clean: discretion grep empty, all 5 required status checks (secrets-scan, pattern-scan, history-scan, pytest 3.12/3.13) green, all 3 commits signed, additive schema with optional fields preserves back-compat for older readers.

One blocker on the way to a clean merge: typos is failing on src/aelfrice/hook_tail.py:78 — the help string "--since must look like Ns / Nm / Nh / Nd (got {spec!r})" trips the checker on NdAnd. typos isn't a required status check on main, but landing red CI when the fix is one line is avoidable. Two options:

  • Add Nd = "Nd" (and arguably Ns/Nm/Nh for symmetry, though only Nd currently fires) under [default.extend-words] in .typos.toml.
  • Rephrase the help string to use concrete examples instead of the N<unit> placeholder, e.g. "--since must look like 30s / 5m / 2h / 1d (got {spec!r})". The error message gets clearer too.

Minor cosmetic in format_record (non-blocking): f"[{lane}{locked_mark:7s}]" pads locked_mark to width 7, so unlocked rows render as [L1 ] with seven trailing spaces inside the brackets. Not wrong — just looks odd next to [L0 locked]. Up to you whether to tighten.

Will merge once typos goes green.

@robotrocketscience

Copy link
Copy Markdown
Owner Author

[release:review:Kulili:2026-05-02T19:41:10Z]

@robotrocketscience robotrocketscience added attn:unblock Needs answer from another session and removed attn:review Needs review (PR open, awaiting reviewer) labels May 2, 2026
@github-actions github-actions Bot added the attn:merge-conflict PR branch needs rebase label May 2, 2026
@github-actions

github-actions Bot commented May 2, 2026

Copy link
Copy Markdown

This PR is now behind main. Rebase locally so your commit signatures stay intact:

git fetch origin && git checkout 'feat/issue-321-injection-log' && git rebase origin/main
# resolve conflicts if any, then
git push --force-with-lease

Auto-rebase was removed because the bot has no signing key; rebasing as the bot strips author signatures and the required_signatures rule on main then blocks the merge. See #341.

)

Additive fields on hook_audit.jsonl per #321 path (a) — extend the
existing #280 audit log rather than ship a parallel injections.ndjson.

- beliefs[]: per-hit {id, lane (L0/L1), locked, content_hash, alpha,
  beta, posterior_mean, snippet}. Lane derived from lock_level; score
  intentionally absent (retrieve() does not propagate per-hit scores).
- latency_ms: wall-clock around retrieve+format span on both
  user_prompt_submit and session_start.
- tokens: derived from rendered_block via the same 4-chars-per-token
  estimator retrieval uses for budgeting.

All fields are optional in the writer signature, so callers that don't
pass them produce records readable by older readers. test_rotation
threshold bumped 500 → 1000 to accommodate the larger record.
`aelf tail` is the reader half of #321: a tail -f-style pretty-printer
over the per-turn hook audit log (extended in the previous commit with
beliefs[], latency_ms, tokens). New module `hook_tail.py` keeps the
formatting helpers out of the import-cheap hook write path.

CLI surface:
  --filter key=value (repeatable, AND-joined): hook=<name>, lane=L0|L1
  --since DUR (Ns / Nm / Nh / Nd): backfill from rotated + live
  --no-blob: suppress per-belief snippet bodies
  --no-follow: dump current contents and exit (one-shot)

Default behaviour (no --since, --follow): start at end-of-file and
stream new records. Rotation is detected via inode change so tail
survives a single-slot rotation seamlessly.

Filter semantics: hook= matches the record-level field; lane= matches
iff at least one belief in beliefs[] has that lane. Records missing
the queried field never match — falsifying, not best-effort. Records
written before #321 (no beliefs[] / tokens / latency_ms) render
without those fields and are filtered out by lane= queries.

27 new unit tests in tests/test_hook_tail.py covering parsers, filter
semantics, format_record (header + per-belief lines + --no-blob),
one-shot tail, follow mode (new-line pickup + rotation survival),
--since (rotated + live backfill), and an end-to-end hook-fire-then-
tail integration test.
typos flags 'Nd' (placeholder) as a misspelling of 'And'. Rephrasing to
concrete examples (30s / 5m / 2h / 1d) clears CI and reads better.
@robotrocketscience
robotrocketscience force-pushed the feat/issue-321-injection-log branch from 5717d14 to 6480b9f Compare May 2, 2026 19:47
@robotrocketscience

Copy link
Copy Markdown
Owner Author

Fixed via 6480b9f. Picked option (b) — rephrased the help string to use concrete examples (30s / 5m / 2h / 1d). Reads cleaner and dodges the typos checker without polluting .typos.toml. Left the format_record cosmetic alone — agree it's odd but not worth a churn commit.

Rebased onto github/main (was behind by 4 commits), force-pushed. All 4 commits signed. Clearing attn:unblock; attn:merge-conflict will clear when the rebase propagates. Ready for re-review.

@robotrocketscience robotrocketscience added attn:review Needs review (PR open, awaiting reviewer) author-Setr PR coordination mutex and removed attn:unblock Needs answer from another session attn:merge-conflict PR branch needs rebase labels May 2, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/aelfrice/hook_tail.py`:
- Around line 308-336: When a rotation is detected (st.st_ino != last_ino) we
must drain the rotated file from the previous offset before resetting pos;
locate the rotated file (audit_path.with_name(audit_path.name + ".1")), open it,
seek to the current pos, read remaining bytes, parse and emit those records via
_emit_records(...) (use the same parsing/append logic as for the live file),
then set pos = 0 and continue to read the new live file; update the code paths
that use audit_path, last_ino, pos, _emit_records, include_blob, sink and flt
accordingly and add a regression test that performs append-then-rotate to ensure
the record that triggered rotation is emitted.
- Around line 149-160: When beliefs_obj is missing or not a list, fall back to
legacy counters: read record.get("n_beliefs") and record.get("n_locked") and
compute n_l1 = int(n_locked) if present else 0 and n_l0 = max(0, int(n_beliefs)
- n_l1) if n_beliefs present else 0; otherwise keep the existing computation
from the parsed beliefs list. Update the variables n_l0 and n_l1 (used to build
parts) so they come from parsed beliefs when beliefs_obj is a list of dicts, and
from these legacy fields when beliefs_obj is absent or invalid, ensuring integer
conversion and non-negative results.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 39945046-aa79-4331-a23a-dbd6a1c9a645

📥 Commits

Reviewing files that changed from the base of the PR and between 38f7952 and 6480b9f.

📒 Files selected for processing (7)
  • src/aelfrice/cli.py
  • src/aelfrice/hook.py
  • src/aelfrice/hook_tail.py
  • src/aelfrice/slash_commands/tail.md
  • tests/test_hook_audit.py
  • tests/test_hook_tail.py
  • tests/test_slash_commands.py

Comment thread src/aelfrice/hook_tail.py
Comment on lines +149 to +160
beliefs_obj = record.get("beliefs")
beliefs: list[dict[str, object]] = []
if isinstance(beliefs_obj, list):
beliefs = [b for b in beliefs_obj if isinstance(b, dict)]
n_l0 = sum(1 for b in beliefs if b.get("lane") == "L0")
n_l1 = sum(1 for b in beliefs if b.get("lane") == "L1")
parts: list[str] = [short_ts, hook]
if isinstance(tokens, int):
parts.append(f"{tokens} tok")
if isinstance(latency_ms, int):
parts.append(f"{latency_ms} ms")
parts.append(f"L0×{n_l0} L1×{n_l1}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fall back to legacy counters when beliefs[] is missing.

Pre-#321 audit rows still have n_beliefs / n_locked. Right now those render as L0×0 L1×0, which makes historical injections look empty even when the record says hits were injected. Please derive the header counts from the legacy fields when beliefs is absent.

Suggested adjustment
     beliefs_obj = record.get("beliefs")
     beliefs: list[dict[str, object]] = []
     if isinstance(beliefs_obj, list):
         beliefs = [b for b in beliefs_obj if isinstance(b, dict)]
-    n_l0 = sum(1 for b in beliefs if b.get("lane") == "L0")
-    n_l1 = sum(1 for b in beliefs if b.get("lane") == "L1")
+        n_l0 = sum(1 for b in beliefs if b.get("lane") == "L0")
+        n_l1 = sum(1 for b in beliefs if b.get("lane") == "L1")
+    else:
+        n_locked = record.get("n_locked")
+        n_beliefs = record.get("n_beliefs")
+        if isinstance(n_locked, int) and isinstance(n_beliefs, int):
+            n_l0 = max(0, n_locked)
+            n_l1 = max(0, n_beliefs - n_locked)
+        else:
+            n_l0 = 0
+            n_l1 = 0
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
beliefs_obj = record.get("beliefs")
beliefs: list[dict[str, object]] = []
if isinstance(beliefs_obj, list):
beliefs = [b for b in beliefs_obj if isinstance(b, dict)]
n_l0 = sum(1 for b in beliefs if b.get("lane") == "L0")
n_l1 = sum(1 for b in beliefs if b.get("lane") == "L1")
parts: list[str] = [short_ts, hook]
if isinstance(tokens, int):
parts.append(f"{tokens} tok")
if isinstance(latency_ms, int):
parts.append(f"{latency_ms} ms")
parts.append(f"L0×{n_l0} L1×{n_l1}")
beliefs_obj = record.get("beliefs")
beliefs: list[dict[str, object]] = []
if isinstance(beliefs_obj, list):
beliefs = [b for b in beliefs_obj if isinstance(b, dict)]
n_l0 = sum(1 for b in beliefs if b.get("lane") == "L0")
n_l1 = sum(1 for b in beliefs if b.get("lane") == "L1")
else:
n_locked = record.get("n_locked")
n_beliefs = record.get("n_beliefs")
if isinstance(n_locked, int) and isinstance(n_beliefs, int):
n_l0 = max(0, n_locked)
n_l1 = max(0, n_beliefs - n_locked)
else:
n_l0 = 0
n_l1 = 0
parts: list[str] = [short_ts, hook]
if isinstance(tokens, int):
parts.append(f"{tokens} tok")
if isinstance(latency_ms, int):
parts.append(f"{latency_ms} ms")
parts.append(f"L0×{n_l0} L1×{n_l1}")
🧰 Tools
🪛 Ruff (0.15.12)

[warning] 160-160: String contains ambiguous × (MULTIPLICATION SIGN). Did you mean x (LATIN SMALL LETTER X)?

(RUF001)


[warning] 160-160: String contains ambiguous × (MULTIPLICATION SIGN). Did you mean x (LATIN SMALL LETTER X)?

(RUF001)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/aelfrice/hook_tail.py` around lines 149 - 160, When beliefs_obj is
missing or not a list, fall back to legacy counters: read
record.get("n_beliefs") and record.get("n_locked") and compute n_l1 =
int(n_locked) if present else 0 and n_l0 = max(0, int(n_beliefs) - n_l1) if
n_beliefs present else 0; otherwise keep the existing computation from the
parsed beliefs list. Update the variables n_l0 and n_l1 (used to build parts) so
they come from parsed beliefs when beliefs_obj is a list of dicts, and from
these legacy fields when beliefs_obj is absent or invalid, ensuring integer
conversion and non-negative results.

Comment thread src/aelfrice/hook_tail.py
Comment on lines +308 to +336
st = audit_path.stat()
if last_ino is not None and st.st_ino != last_ino:
# Rotation detected: live file was renamed to .1 and a new
# one was created. Reset position to read from the start.
pos = 0
last_ino = st.st_ino

if st.st_size <= pos:
time.sleep(poll_interval)
continue

with audit_path.open("r", encoding="utf-8") as f:
f.seek(pos)
new_text = f.read()
pos = f.tell()

new_records: list[dict[str, object]] = []
for line in new_text.splitlines():
stripped = line.strip()
if not stripped:
continue
try:
parsed = json.loads(stripped)
except json.JSONDecodeError:
continue
if not isinstance(parsed, dict):
continue
new_records.append(parsed)
_emit_records(new_records, flt, include_blob=include_blob, out=sink)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Don't drop the record that triggers rotation.

src/aelfrice/hook.py:_append_audit() writes the new line and only then renames the whole live file to .1. If that happens between polls, this branch sees the inode flip, resets pos, and starts reading only the new live file. Any unread bytes that moved into .1 are never emitted, so aelf tail can miss the exact injection that caused rotation.

Please drain the rotated file from the previous offset before switching to the new live file, and add a regression test for the append-then-rotate case.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/aelfrice/hook_tail.py` around lines 308 - 336, When a rotation is
detected (st.st_ino != last_ino) we must drain the rotated file from the
previous offset before resetting pos; locate the rotated file
(audit_path.with_name(audit_path.name + ".1")), open it, seek to the current
pos, read remaining bytes, parse and emit those records via _emit_records(...)
(use the same parsing/append logic as for the live file), then set pos = 0 and
continue to read the new live file; update the code paths that use audit_path,
last_ino, pos, _emit_records, include_blob, sink and flt accordingly and add a
regression test that performs append-then-rotate to ensure the record that
triggered rotation is emitted.

@robotrocketscience

Copy link
Copy Markdown
Owner Author

[claim:review:Gylf:2026-05-02T20:02:56Z]

@robotrocketscience
robotrocketscience merged commit 6480b9f into main May 2, 2026
22 checks passed
@robotrocketscience
robotrocketscience deleted the feat/issue-321-injection-log branch May 2, 2026 20:03
@robotrocketscience

Copy link
Copy Markdown
Owner Author

[release:review:Gylf:2026-05-02T20:03:52Z]

@robotrocketscience robotrocketscience removed the attn:review Needs review (PR open, awaiting reviewer) label May 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author-Setr PR coordination mutex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v1.5.x] Injection log + aelf tail — live observability for hook-injected memory blocks

2 participants