Feature/dream summarization

HANDOFF.md

@@ -963,10 +963,10 @@ Cross-conversation, agent-scoped fact retention with LLM tools, REST API, MCP ma
 | **LLM Tool** | `UserMemoryTool.java` (`@Vetoed`, 4 tools: remember/recall/search/forget) |
 | **REST API** | `IRestUserMemoryStore.java`, `RestUserMemoryStore.java` (9 endpoints + validation) |
 | **MCP Tools** | `McpMemoryTools.java` (8 tools, GDPR-compliant) |
-| **Dream** | `DreamService.java` (stale pruning, contradiction detection, Micrometer metrics) |
+| **Dream** | `DreamService.java` (stale pruning, contradiction detection, LLM-driven summarization, Micrometer metrics) |
 | **Integration** | `AgentOrchestrator.java` (groupId extraction), `GroupConversationService.java` (groupId context) |
 | **Docs** | `docs/user-memory.md`, `docs/changelog.md`, `docs/SUMMARY.md` |
-| **Tests** | 45 new: `UserMemoryToolTest` (16), `DreamServiceTest` (9), `UserMemoryEntryTest` (22), `RestUserMemoryStoreTest` (15) |
+| **Tests** | 107 new: `UserMemoryToolTest` (16), `DreamServiceTest` (51), `UserMemoryEntryTest` (22), `RestUserMemoryStoreTest` (15), `SummarizationServiceTest` (3 new) |
 
 **Total tests:** 1406 (all pass). **Last commit:** Phase 11a code review fixes.
 

docs/changelog.md

@@ -13,8 +13,122 @@ Each entry follows this format:
 - **Decision** — Key design decisions and their reasoning
 - **Files** — Links to modified files
 
+## 🔍 DreamService PR Review Remediation — Pass 2 (2026-05-16)
+
+**Repo:** EDDI (`feature/dream-summarization`)
+**What changed:** 9 findings from Copilot (8) + CodeRabbit (1) review, all resolved.
+
+### High Severity (3 — data loss / data unreachability)
+- **Multi-agent `self` visibility upgrade** — When consolidating entries from multiple agents (preserveAgentProvenance=false), self-scoped visibility is upgraded to `global` so no agent loses its memories
+- **GroupIds preserved** — Consolidated entries now inherit the union of all groupIds from originals, fixing group-scoped entries becoming unreachable after consolidation
+- **`summarizeTargetEntries` validation** — Setter now rejects `<1` (was silently accepting `0`, which would cap to empty list, insert nothing, then delete all originals)
+
+### Medium Severity (5 — atomicity, metrics, resilience)
+- **Partial insert rollback** — If any consolidated entry fails to insert, already-inserted entries are rolled back before preserving originals (was leaving orphaned consolidated entries)
+- **Accurate metrics** — `entriesSummarized` counter now tracks actual successful deletes minus inserts (was tracking intent, overstating when deletes failed)
+- **Soft cost ceiling documented** — Added comment explaining the pre-check design is intentional (can't pre-estimate output tokens). This is not a bug.
+- **Null category NPE fixed** — `Collectors.groupingBy` now uses null-safe lambda defaulting to "fact" (legacy Mongo entries may have null category)
+- **LLM output guardrails** — `parseConsolidatedEntries` now rejects blank keys/values and truncates to `MAX_KEY_LENGTH=100`/`MAX_VALUE_LENGTH=1000` (matches UserMemoryConfig guardrails)
+
+### Low Severity (1 — log level)
+- **SummarizationService log level** — Changed `warnf` → `errorf` in both exception handlers (RuntimeException + checked) per coding guidelines
+
+### New Tests (11 added: 51 DreamService total)
+- `summarize_multiAgentSelfScope_upgradesVisibility` — visibility upgrade to global
+- `summarize_preservesGroupIds` — merged groupIds on consolidated entries
+- `summarize_nullCategory_defaultsToFact` — null-safe grouping
+- `parseConsolidatedEntries_blankKeyFiltered` — blank key rejection
+- `parseConsolidatedEntries_longKeyTruncated` — key length guardrail
+- `truncate_shortString_unchanged`, `truncate_longString_truncated`, `truncate_null_returnsNull` — truncate utility
+- `summarize_partialInsertFails_rollsBack` — rollback on partial insert failure
+- `setSummarizeTargetEntries_rejectsZero`, `setSummarizeTargetEntries_rejectsNegative` — config validation
+
+### Verification
+- `./mvnw clean test -Dtest=DreamServiceTest,ConversationSummarizerTest,SummarizationServiceTest` → 71 tests, 0 failures
+- JaCoCo: DreamService 91.9% line / 86.1% branch, SummarizationService 100% line
+
+---
+
+## 🔍 DreamService PR Review Remediation — Pass 1 (2026-05-16)
+
+**Repo:** EDDI (`feature/dream-summarization`)
+**What changed:** Initial review — 11 findings from self-review, all resolved.
+
+### Must-Fix (3)
+- **Triple DB reload eliminated** — `process()` was calling `getAllEntries()` three times when pruning + contradiction + summarization were all enabled. Hoisted the post-prune reload so it's shared (contradiction detection is read-only)
+- **`maxCostPerRun` default aligned** — Java default changed from `$5.00` to `$0.50` to match `user-memory.md` and `scheduling.md` documentation. Prevents a 10× cost surprise for operators
+- **`scheduling.md` contradiction claim fixed** — Changed "Identifies and resolves" to "Identifies and logs for review"
+
+### Should-Fix (5)
+- **Cost estimator input undercount fixed** — `estimateCost()` now takes `inputContentLength` parameter and estimates from input+output chars when providers don't report tokens (was output-only, underestimating by 5-10×)
+- **Dead exception catch block fixed** — `SummarizationService.summarizeWithUsage()` now re-throws exceptions (was swallowing them, making `DreamService`'s catch block unreachable). `summarize()` wrapper retains swallow-and-return-empty behavior for backward compat with `ConversationSummarizer`
+- **`contradictionResolution` field annotated** — Added Javadoc noting it's reserved for future use (V1 detector only counts/logs)
+- **HANDOFF.md test counts corrected** — DreamServiceTest 37→40, SummarizationServiceTest +1, total 90→94
+- **`SummarizationResult.hasContent()` removed** — Unused convenience method
+
+### Nitpicks (3)
+- **`buildEntriesJson` now uses injected ObjectMapper** — Replaced hand-rolled `StringBuilder` JSON with `objectMapper.writerWithDefaultPrettyPrinter()`, keeping manual fallback for resilience
+- **Stale Javadoc fixed** — `SummarizationService` class doc: "future Dream consolidation" → "Dream memory consolidation"
+- **`enableSummarization()` test helper** — Now also sets `maxCostPerRun` to explicit value for clarity
+
+### New Tests (6 added: 40 DreamService + 8 SummarizationService)
+- `estimateCost_withTokenUsage` — token-based cost calculation
+- `estimateCost_withoutTokenUsage_fallsBackToCharEstimate` — input+output char fallback
+- `summarize_costCeilingReached_stopsEarly` — loop stops at cost ceiling
+- `summarizeWithUsage_llmError_propagatesException` — verifies re-throw (vs `summarize()` which swallows)
+- `summarizeWithUsage_returnsTokenCounts` — token usage extraction from LLM response
+- `summarizeWithUsage_checkedExceptionWrappedInRuntime` — checked exception wrapping
+
+### Verification
+- `./mvnw clean test -Dtest=DreamServiceTest,ConversationSummarizerTest,SummarizationServiceTest` → 60 tests, 0 failures
+- JaCoCo coverage: DreamService 92% line / 88% branch, SummarizationService 100% line
+
+
+## 🧠 DreamService: LLM-Driven Memory Summarization (2026-05-15)
+
+**Repo:** EDDI (`feature/dream-summarization`)
+**What changed:** Implemented `summarizeInteractions()` in `DreamService` — config-driven LLM memory consolidation that compresses related user memory entries via SummarizationService.
+
+### DreamConfig (AgentConfiguration.java)
+- Added 6 new config fields: `summarizeMinEntries` (5), `summarizeTargetEntries` (2), `summarizeGroupBy` ("category"/"all"), `preserveAgentProvenance` (false), `maxSummarizationCalls` (10), `summarizationPrompt` (customizable default)
+- All fields have sensible defaults; existing configs with `summarizeInteractions=false` are unaffected
+
+### DreamService
+- Added `SummarizationService` as constructor dependency (CDI injection)
+- Added `entriesSummarizedCounter` metric
+- Refactored `process()` to reload entries only after pruning (contradiction detection is read-only)
+- Implemented `summarizeInteractions()` with insert-before-delete safety pattern
+- LLM call wrapped in try-catch — failure skips the group, does not kill the dream cycle
+- `escapeJson()` now uses Jackson's `JsonStringEncoder` for complete RFC 8259 compliance
+- Helpers: `buildGroups()` (category/all grouping + agent provenance sub-grouping), `parseConsolidatedEntries()` (markdown fence stripping, JSON array extraction), `mostRestrictiveVisibility()`, `buildEntriesJson()`
+
+### Safety Guarantees
+- LLM returns empty/garbage → group skipped, originals untouched
+- LLM throws exception → group skipped, originals untouched, dream cycle continues
+- LLM returns ≥ original count → group skipped
+- LLM returns > target count → result capped to `summarizeTargetEntries`
+- Insert fails → originals never deleted
+- Delete partially fails → duplicates may remain until next dream cycle (contradiction detector currently only counts/logs; dedup cleanup is a future enhancement)
+- Cost bounded by `maxSummarizationCalls`
+
+### Tests (37 total: 8 existing + 29 new)
+- Updated `setUp()` for new constructor signature
+- 12 summarization behavior tests: threshold, consolidation, empty/garbage LLM, markdown fences, count validation, insert failure, call limit, groupBy all, agent provenance, custom prompt, visibility merge
+- 9 coverage-hardening tests: null updatedAt, prune delete failure, same-key-same-value no contradiction, LLM result capping, delete partial failure, LLM exception isolation, summarize-after-pruning reload, missing key field filtering, escapeJson control chars/null
+- 8 unit tests: `parseConsolidatedEntries` (valid/null/blank/fences/missing-key), `mostRestrictiveVisibility` (self/global/group), `escapeJson` (control chars/null)
+
+### Documentation Updates
+- `docs/user-memory.md` — Dream config table expanded (6 new fields), config example updated, removed "V2, not yet active" label, added `dream.entries.summarized` metric
+- `docs/scheduling.md` — Dream config example updated with new fields
+- `HANDOFF.md` — Dream description and test count updated
+
+### Verification
+- `./mvnw compile` → BUILD SUCCESS
+- `./mvnw test -Dtest=DreamServiceTest` → 37 tests, 0 failures, 0 errors
+
 ## 🔧 PR Review Remediation — 8 Findings Resolved (2026-05-14)
 
+
 **Repo:** EDDI (`feature/agentic-improvements`)
 **What changed:** Addressed all PR review findings from Copilot (7) and CodeRabbit (1), round 2.
 

docs/scheduling.md

@@ -154,7 +154,7 @@ Dream Consolidation is a specialized schedule that performs **background memory
 ### What It Does
 
 1. **Stale entry pruning** — Removes outdated facts that are no longer relevant
-2. **Contradiction detection** — Identifies and resolves conflicting memories (e.g., "user likes coffee" vs "user hates coffee")
+2. **Contradiction detection** — Identifies conflicting memories (e.g., "user likes coffee" vs "user hates coffee") and logs them for review. Resolution is planned for a future version.
 3. **Fact summarization** — Consolidates verbose entries into concise summaries
 
 ### Configuration
@@ -172,10 +172,15 @@ Dream consolidation is configured in the agent configuration:
         "detectContradictions": true,
         "contradictionResolution": "keep_newest",
         "pruneStaleAfterDays": 90,
-        "summarizeInteractions": false,
+        "summarizeInteractions": true,
+        "summarizeMinEntries": 5,
+        "summarizeTargetEntries": 2,
+        "summarizeGroupBy": "category",
+        "preserveAgentProvenance": false,
+        "maxSummarizationCalls": 10,
         "llmProvider": "anthropic",
         "llmModel": "claude-sonnet-4-6",
-        "maxCostPerRun": 5.00,
+        "maxCostPerRun": 0.50,
         "batchSize": 50,
         "maxUsersPerRun": 1000
       }

docs/user-memory.md

@@ -12,7 +12,7 @@ Persistent User Memory enables EDDI agents to remember facts, preferences, and c
 | **Visibility** | `self`, `group`, `global` scoping |
 | **Guardrails** | Configurable key/value limits, write-rate limits, capacity caps |
 | **GDPR** | Full right-to-erasure support via REST API and MCP tools |
-| **Maintenance** | Background "Dream" consolidation (stale pruning, contradiction detection) |
+| **Maintenance** | Background "Dream" consolidation (stale pruning, contradiction detection, LLM summarization) |
 
 ## Architecture
 
@@ -51,11 +51,16 @@ Enable advanced memory features (LLM tools, Dream consolidation, guardrails, rec
       "maxWritesPerTurn": 10,
       "allowedCategories": ["preference", "fact", "context"]
     },
-    "dreamConfig": {
+    "dream": {
       "enabled": true,
       "pruneStaleAfterDays": 90,
       "detectContradictions": true,
-      "summarizeInteractions": false,
+      "summarizeInteractions": true,
+      "summarizeMinEntries": 5,
+      "summarizeTargetEntries": 2,
+      "summarizeGroupBy": "category",
+      "preserveAgentProvenance": false,
+      "maxSummarizationCalls": 10,
       "maxCostPerRun": 0.50
     }
   },
@@ -90,8 +95,16 @@ Enable advanced memory features (LLM tools, Dream consolidation, guardrails, rec
 | `enabled` | `boolean` | `false` | Enable background consolidation |
 | `pruneStaleAfterDays` | `int` | `90` | Remove entries not accessed in N days. Set to 0 to disable. |
 | `detectContradictions` | `boolean` | `true` | Flag entries with same key but different values |
-| `summarizeInteractions` | `boolean` | `false` | V2 feature: LLM-driven fact compression |
+| `summarizeInteractions` | `boolean` | `false` | Enable LLM-driven memory consolidation |
+| `summarizeMinEntries` | `int` | `5` | Minimum entries in a group before summarization triggers |
+| `summarizeTargetEntries` | `int` | `2` | Target number of entries per group after consolidation |
+| `summarizeGroupBy` | `String` | `"category"` | Grouping strategy: `"category"` or `"all"` |
+| `preserveAgentProvenance` | `boolean` | `false` | Sub-group by `sourceAgentId` (preserves per-agent provenance) |
+| `maxSummarizationCalls` | `int` | `10` | Maximum LLM calls per dream cycle per user (bounds cost) |
+| `summarizationPrompt` | `String` | *(built-in)* | Custom LLM instructions for consolidation |
 | `maxCostPerRun` | `double` | `0.50` | Maximum dollar cost per dream cycle |
+| `llmProvider` | `String` | `"anthropic"` | LLM provider for dream operations |
+| `llmModel` | `String` | `"claude-sonnet-4-6"` | Model for dream operations |
 
 ## LLM Tools
 
@@ -222,7 +235,7 @@ The Dream service performs background maintenance on user memories:
 
 2. **Contradiction Detection** — Identifies entries with the same key but different values (e.g., `language=English` from Agent A vs `language=German` from Agent B). V1 uses key-based matching; future versions will use LLM-driven semantic analysis.
 
-3. **Interaction Summarization** — (V2, not yet active) Compresses multiple related facts into consolidated summaries using the LLM.
+3. **Interaction Summarization** — When `summarizeInteractions=true`, compresses multiple related facts into consolidated summaries using the configured LLM. Entries are grouped by the `summarizeGroupBy` strategy (per-category or all together), and each group above `summarizeMinEntries` is distilled into `summarizeTargetEntries` entries. Safety guarantees: new entries are inserted before originals are deleted; LLM failures or invalid responses preserve the original entries.
 
 ### Metrics
 
@@ -233,6 +246,7 @@ The Dream service exposes Micrometer metrics:
 | `dream.users.processed` | Counter | Users processed across all dream cycles |
 | `dream.entries.pruned` | Counter | Total entries pruned |
 | `dream.contradictions.found` | Counter | Contradictions detected |
+| `dream.entries.summarized` | Counter | Entries reduced by LLM consolidation |
 | `dream.duration` | Timer | Duration of dream cycles |
 
 ## Migration from Legacy Properties

src/main/java/ai/labs/eddi/configs/agents/model/AgentConfiguration.java

@@ -517,15 +517,52 @@ public static class DreamConfig {
         private boolean enabled = false;
         private String schedule = "0 3 * * *";
         private boolean detectContradictions = true;
+        /**
+         * Contradiction resolution strategy. Reserved for future use — the current V1
+         * detector only counts and logs contradictions without resolving them.
+         */
         private String contradictionResolution = "keep_newest";
         private int pruneStaleAfterDays = 90;
         private boolean summarizeInteractions = false;
         private String llmProvider = "anthropic";
         private String llmModel = "claude-sonnet-4-6";
-        private double maxCostPerRun = 5.00;
+        private double maxCostPerRun = 0.50;
         private int batchSize = 50;
         private int maxUsersPerRun = 1000;
 
+        /** Minimum entries in a group before summarization triggers. */
+        private int summarizeMinEntries = 5;
+
+        /** Target number of entries per group after consolidation. */
+        private int summarizeTargetEntries = 2;
+
+        /**
+         * Grouping strategy: "category" (group by fact/preference/context) or "all"
+         * (single group).
+         */
+        private String summarizeGroupBy = "category";
+
+        /**
+         * Whether to sub-group by sourceAgentId before consolidating. true = entries
+         * from different agents stay separate (preserves provenance). false = entries
+         * from all agents consolidated together (better compression).
+         */
+        private boolean preserveAgentProvenance = false;
+
+        /** Maximum LLM calls per dream cycle per user. Bounds cost. */
+        private int maxSummarizationCalls = 10;
+
+        /**
+         * LLM instructions for memory consolidation. Customizable by the agent
+         * designer. Entries are appended as JSON after this prompt.
+         */
+        private String summarizationPrompt = "You are a memory consolidation assistant. Given a list of remembered facts "
+                + "about a user, distill them into fewer, non-redundant entries. Preserve all "
+                + "important details. Remove duplicates and merge related facts. Each entry "
+                + "should be a single, clear statement.\n\n"
+                + "Respond ONLY with a JSON array: [{\"key\": \"...\", \"value\": \"...\"}]\n"
+                + "Do not add any text outside the JSON array.";
+
         public boolean isEnabled() {
             return enabled;
         }
@@ -613,6 +650,57 @@ public int getMaxUsersPerRun() {
         public void setMaxUsersPerRun(int maxUsersPerRun) {
             this.maxUsersPerRun = maxUsersPerRun;
         }
+
+        public int getSummarizeMinEntries() {
+            return summarizeMinEntries;
+        }
+
+        public void setSummarizeMinEntries(int summarizeMinEntries) {
+            this.summarizeMinEntries = summarizeMinEntries;
+        }
+
+        public int getSummarizeTargetEntries() {
+            return summarizeTargetEntries;
+        }
+
+        public void setSummarizeTargetEntries(int summarizeTargetEntries) {
+            if (summarizeTargetEntries < 1) {
+                throw new IllegalArgumentException("summarizeTargetEntries must be >= 1");
+            }
+            this.summarizeTargetEntries = summarizeTargetEntries;
+        }
+
+        public String getSummarizeGroupBy() {
+            return summarizeGroupBy;
+        }
+
+        public void setSummarizeGroupBy(String summarizeGroupBy) {
+            this.summarizeGroupBy = summarizeGroupBy;
+        }
+
+        public boolean isPreserveAgentProvenance() {
+            return preserveAgentProvenance;
+        }
+
+        public void setPreserveAgentProvenance(boolean preserveAgentProvenance) {
+            this.preserveAgentProvenance = preserveAgentProvenance;
+        }
+
+        public int getMaxSummarizationCalls() {
+            return maxSummarizationCalls;
+        }
+
+        public void setMaxSummarizationCalls(int maxSummarizationCalls) {
+            this.maxSummarizationCalls = maxSummarizationCalls;
+        }
+
+        public String getSummarizationPrompt() {
+            return summarizationPrompt;
+        }
+
+        public void setSummarizationPrompt(String summarizationPrompt) {
+            this.summarizationPrompt = summarizationPrompt;
+        }
     }
 
     /**