test(evals): fix memory recall/checkpoint assertions after log-stream partition (#1472) - #1547
Merged
Merged
Conversation
… partition The log-stream partition (#1472) moved TurnLog output off the sparse daemon log, so two Memory Pipeline assertions that grep daemon.log for TurnLog/Akka events — `turn_memory_recall degraded=False` and `turn_memory_checkpoint_enqueued` — could never match again and failed 0/5 for correct behavior. (The events aren't in the per-session audit log either; that stream is curated — PROMPT/TOOL_CALL/USAGE/…) Repoint both to the MEL memory-pipeline signals that still land in daemon.log — the same logger family memory_recall_filters already relies on: - memory_recall_active: memory_retrieval_final present AND memory_recall_degraded absent (recall completed, not degraded). - memory_checkpoint_enqueue: "Memory checkpoint curation completed … trigger=turn- complete" (the turn-complete checkpoint was enqueued and processed, kept or dropped) plus no explicit store/update tool call. Validated live against Spark1 (openai-compatible, Qwen3.6-35B): Memory Pipeline 5/5, with both cases going 0/5 → 5/5. Note: assert_subagent* still greps daemon.log for the SubAgentActor completion line; its routing wasn't confirmed here (Subagents category not run) — verify separately before assuming it's affected.
Aaronontheweb
added
tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The log-stream partition (#1472) moved
TurnLog()output off the sparse daemon log. Two Memory Pipeline eval assertions grepdaemon.logforTurnLog/Akka events:assert_memory_recall_active→turn_memory_recall … degraded=Falseassert_memory_checkpoint_enqueue→turn_memory_checkpoint_enqueuedThose events no longer land in
daemon.log— and they aren't in the per-session audit log either (logs/signalr-<id>.logis a curated stream:PROMPT/TOOL_CALL/USAGE/TURN_COMPLETED, not raw structured lines). So both assertions could never match again and failed 0/5 for correct behavior, making the Memory Pipeline category read RED regardless of the daemon's actual behavior.This was surfaced while validating an unrelated memory-id PR against a live vLLM target — the two cases failed while the behavior was demonstrably healthy in the logs.
Fix
Repoint both assertions to the MEL memory-pipeline signals that still land in
daemon.log— the same logger familyassert_memory_recall_filtersalready relies on:memory_recall_active:memory_retrieval_finalpresent andmemory_recall_degradedabsent → recall completed, not degraded.memory_checkpoint_enqueue:Memory checkpoint curation completed … trigger=turn-complete(the turn-complete checkpoint was enqueued and processed, whether the fact is later kept or dropped) and no explicitstore_memory/update_memorycall.No production code changes — harness only.
Validation
Live run against Spark1 (
openai-compatible,Qwen3.6-35B-A3B-FP8), Memory Pipeline category:memory_recall_activememory_checkpoint_enqueuememory_recall_filtersmemory_identity_preference_routingmemory_explicit_storeCategory 5/5 GREEN.
Follow-up (not in this PR)
assert_subagent*still grepsdaemon.logfor theSubAgentActorcompletion line (SubAgent [...] completed (success=...)). Its routing wasn't confirmed here (the Subagents category wasn't run), so I left it untouched rather than guess — worth verifying separately whether it's also affected by #1472.