From d508403265c8db5ac7447f5065dc9d5d60776daf Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Tue, 28 Jul 2026 22:14:33 +0200 Subject: [PATCH 01/11] =?UTF-8?q?fix(runtime):=20code-review=20findings=20?= =?UTF-8?q?wave=203=20=E2=80=94=20concurrency,=20lifecycle,=20cancellation?= =?UTF-8?q?,=20graceful=20shutdown,=20Dream=20wiring?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 16 fixed, 1 partial. The wave where the obvious fix is often the wrong one. C1 (critical): cancellation cancelled nothing. CompletableFuture.cancel(true) does not interrupt a runAsync/supplyAsync body — the JDK documents mayInterruptIfRunning as having no effect there. Five call sites relied on it, so "cancelled" agent threads kept mutating the task list, transcript and errors after the orchestrator had already persisted the group document. Replaced with a cooperative cancellation token checked at the turn's own await points. This is also the root cause of C7, where resetStrandedInProgressTasks could strand the very task it exists to rescue. C4: a ~100-turn scalability cliff. The inner pipeline was submitted through the same bounded pool as the outer coordinator callable and then blocked on get(). At the 200-thread default, ~100 concurrent turns leaves every thread a waiter, no inner task can be scheduled, and every turn fails at the watchdog. Nested submissions now route to a virtual-thread executor via a ThreadLocal marker scoped to the callable body. Chosen over CompletableFuture composition because the coordinator's contract is "callable returns => turn is done"; making the outer non-blocking would need an SPI change and would let the next turn of the same conversation start while the previous one still ran. C9/C13: a completed turn could be re-executed. onComplete sat inside the try whose catch(Throwable) called onFailure, which resubmitted the already-executed callable — LLM calls, tool side effects and cost running twice. Completion moved outside the guarded region and both callbacks gated behind a one-shot flag; the coordinator's blind 3x retry removed. C3: a timed-out turn still persisted over a newer one, because the stale-completion guard used the interrupt flag and the pipeline cleared it. Replaced with a per-submission abandonment token set before delegating cancel(). C10: a throwing submit left the callable queued with nothing scheduled to run it, wedging that conversation permanently and leaking the map entry. C5: /rerun destroyed output and regenerated nothing — the component-cache key was built from a sublist-relative index while the cache stores under the absolute index, so the output task ran with a null component after the prior output was already deleted. It survived because every existing LifecycleManagerTest stubs the component map empty, which is exactly the condition that hides it. B3: no graceful shutdown existed anywhere — no ShutdownEvent observer in src/main at all, so a rolling deploy dropped every queued and in-flight turn. B2: the finding's premise was inverted. It claimed 18 of 20 handlers swallow the interrupt; auditing all 28 sites found 14 already restore correctly and 9 rethrow, with only 4 genuine swallowers. Fixed those, plus a site hidden behind a broad catch(Exception) in ScheduleFireExecutor (the poller kept firing after being interrupted for shutdown) and the mirror bug in NatsConversationCoordinator, which restored the flag unconditionally and would abort its own @PreDestroy. I1/G8: DreamService wired to ScheduleFireExecutor per the owner's decision, with a dollar-based maxCostPerRun ceiling. G8 matters much more now it actually runs: consolidation upgraded self visibility to global whenever a group spanned multiple agents, and cross-agent grouping was the default path. F6 partial: the REST/pipeline half (ConnectionCallback sets cancelled on client disconnect) is done; the in-modules/llm half is deferred to that workstream. Clean compile passed first attempt despite three cross-workstream signature changes. 12,633 tests, 0 non-environmental failures. C1, C5, C9 and B2 all mutation-checked against whole test classes. --- docs/architecture.md | 2 +- docs/changelog.md | 58 +++ .../agents/model/AgentConfiguration.java | 59 ++- .../engine/internal/ConversationService.java | 305 +++++++---- .../internal/GroupConversationService.java | 361 +++++++++++-- .../eddi/engine/internal/RestAgentEngine.java | 10 +- .../internal/RestAgentEngineStreaming.java | 186 +++++-- .../lifecycle/internal/LifecycleManager.java | 48 +- .../labs/eddi/engine/runtime/BaseRuntime.java | 181 ++++++- .../workflows/WorkflowStoreClientLibrary.java | 8 + .../engine/runtime/internal/Conversation.java | 20 +- .../engine/runtime/internal/DreamService.java | 280 ++++++++-- .../internal/GracefulShutdownService.java | 188 +++++++ .../InMemoryConversationCoordinator.java | 101 +++- .../internal/NatsConversationCoordinator.java | 7 +- .../internal/ScheduleFireExecutor.java | 80 +++ .../ShutdownReadinessHealthCheck.java | 40 ++ .../eddi/modules/nlp/InputParserTask.java | 6 + .../modules/nlp/impl/RestSemanticParser.java | 91 +++- .../rules/impl/RulesEvaluationTask.java | 6 + .../agents/model/AgentConfigurationTest.java | 16 + .../ConversationServiceHitlCoverage2Test.java | 48 ++ ...onversationServiceProcessingGaugeTest.java | 279 ++++++++++ .../ConversationServiceStaleTurnTest.java | 217 ++++++++ ...oupConversationServiceConcurrencyTest.java | 486 ++++++++++++++++++ .../RestAgentEngineStreamingTest.java | 125 ++++- .../internal/LifecycleManagerTest.java | 244 +++++++++ .../runtime/BaseRuntimeConcurrencyTest.java | 382 ++++++++++++++ .../ConversationCancelPersistenceTest.java | 138 +++++ .../internal/DreamServiceExtendedTest.java | 79 ++- .../runtime/internal/DreamServiceTest.java | 340 ++++++++++-- .../internal/GracefulShutdownServiceTest.java | 177 +++++++ .../InMemoryConversationCoordinatorTest.java | 90 +++- .../internal/ScheduleFireExecutorTest.java | 154 ++++++ .../nlp/InputParserTaskInterruptTest.java | 132 +++++ .../nlp/impl/RestSemanticParserCacheTest.java | 203 ++++++++ .../RulesEvaluationTaskInterruptTest.java | 138 +++++ 37 files changed, 4897 insertions(+), 388 deletions(-) create mode 100644 src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java create mode 100644 src/main/java/ai/labs/eddi/engine/runtime/internal/ShutdownReadinessHealthCheck.java create mode 100644 src/test/java/ai/labs/eddi/engine/internal/ConversationServiceProcessingGaugeTest.java create mode 100644 src/test/java/ai/labs/eddi/engine/internal/ConversationServiceStaleTurnTest.java create mode 100644 src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java create mode 100644 src/test/java/ai/labs/eddi/engine/runtime/BaseRuntimeConcurrencyTest.java create mode 100644 src/test/java/ai/labs/eddi/engine/runtime/internal/ConversationCancelPersistenceTest.java create mode 100644 src/test/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownServiceTest.java create mode 100644 src/test/java/ai/labs/eddi/modules/nlp/InputParserTaskInterruptTest.java create mode 100644 src/test/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParserCacheTest.java create mode 100644 src/test/java/ai/labs/eddi/modules/rules/impl/RulesEvaluationTaskInterruptTest.java diff --git a/docs/architecture.md b/docs/architecture.md index c091172a4..1c7b82b83 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -899,7 +899,7 @@ EDDI's memory model extends beyond single conversations. The `IUserMemoryStore` - At **conversation init**, visible user memories are loaded as `longTerm` properties and made available in all templates via `{{properties.key}}` - During the pipeline, the LLM can autonomously store and recall facts using built-in memory tools (when enabled) - At **conversation teardown**, `longTerm` properties are persisted back to the user memory store -- **Background consolidation** (the "Dream" service) performs scheduled maintenance: stale pruning, contradiction detection, and optional LLM-driven summarization +- **Background consolidation** (the "Dream" service) performs stale pruning, contradiction detection, and optional LLM-driven summarization. It runs on the same cluster-aware schedule machinery as every other background job — a `ScheduleConfiguration` whose metadata carries `{"dreamType": "dream_consolidation"}` together with the target `agentId` and `userId` is claimed by `SchedulePollerService` and dispatched by `ScheduleFireExecutor` to `DreamService`, which reads the agent's `userMemoryConfig.dream` block and runs one cycle. Create such a schedule through the normal schedule surface (`POST /schedules` or the `create_schedule` MCP tool) using the cron expression from `dream.schedule`. Spend is bounded per cycle by `dream.maxCostPerRun` (US dollars), and because Dream has no parent LLM task to inherit credentials from, its model credentials come from `dream.parameters` (which resolves `${vault:…}` and `${vars:…}` like any LLM task's parameters). A cycle that cannot run — no `userId`, dream disabled on the agent, or a failing LLM call — is logged at ERROR and marked FAILED on the fire log, so it retries with backoff and dead-letters rather than silently doing nothing Memory visibility is enforced at the storage level — agents can only see memories matching their visibility scope, preventing cross-tenant memory leaks. diff --git a/docs/changelog.md b/docs/changelog.md index 93f83ce27..5a400d5c6 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -3,6 +3,64 @@ > **Purpose:** Living document tracking all changes, decisions, and reasoning during implementation. Updated as work progresses for easy reference and review. +--- + +## ⚙️ fix(runtime): code-review findings wave 3 — concurrency, lifecycle, cancellation, graceful shutdown, Dream wiring (2026-07-28) + +**Repo:** EDDI (`fix/code-review-concurrency`) + +Third wave of the 124-finding external review. **16 fixed, 1 partial.** This is the wave where the findings were hardest to fix correctly, because the bugs are non-deterministic and several of the obvious fixes are wrong. + +### Cancellation that cancelled nothing (C1) + +`CompletableFuture.cancel(true)` does **not** interrupt a `runAsync`/`supplyAsync` body — the JDK documents `mayInterruptIfRunning` as having no effect there. Five call sites in `GroupConversationService` relied on it, so "cancelled" agent threads **kept mutating `gc.getTaskList()`, `gc.getTranscript()` and the errors list after the orchestrator had already persisted the document**. Replaced with a cooperative `MemberTurnCancellation` token checked at the agent turn's own await points, plus a bounded drain. This is also the root cause of C7 (`resetStrandedInProgressTasks` could strand the very task it exists to rescue, because a falsely-"cancelled" thread flips state between the snapshot and the mutate). + +### The ~100-turn scalability cliff (C4) + +`ConversationService` submitted the inner pipeline through the **same** bounded pool as the outer coordinator callable and then blocked on `future.get()`. With no `quarkus.thread-pool.*` overrides that is the 200-thread default: at ~100 concurrent turns every thread is a waiter, no inner task can ever be scheduled, and **every turn fails at the 60s watchdog**. A cliff, not a gradual degradation. + +Fixed by routing *nested* submissions to a virtual-thread executor via a `ThreadLocal` marker scoped to the callable body. Chosen over `CompletableFuture` composition deliberately: the coordinator's ordering contract is "the callable returns ⇒ the turn is done", so making the outer non-blocking would need an `IEventBus`/`IConversationCoordinator` SPI change **and** would let the next turn of the same conversation start while the previous one still ran. Virtual threads are safe here — there is not a single `@RequestScoped` bean in `src/main`, and three existing callers already drive the pipeline with no request context on virtual-thread executors. The marker is cleared before callbacks run, so `submitNext` still schedules on the managed executor exactly as before; watchdog and timeout semantics are unchanged. + +### Re-execution and lost turns (C3, C9, C10, C13) + +- **C9** — `onComplete` sat inside the `try` whose `catch (Throwable)` called `onFailure`, so **any unchecked throw on the completion path resubmitted the already-executed callable as a retry** — LLM calls, tool side effects and cost all running twice. Completion dispatch moved outside the guarded region *and* both callbacks gated behind a one-shot `AtomicBoolean`. +- **C13** — The coordinator retried failed turns 3×. Because `onFailure` can only be raised from inside the executor task, every retry re-ran a turn that may already have called an LLM and spent money. Retry removed entirely; genuinely pre-execution failures surface as a synchronous throw and are handled by C10's rollback. +- **C3** — A timed-out turn still persisted over a newer one, because the stale-completion guard used the interrupt flag and the pipeline cleared it via `Thread.interrupted()`. Replaced with a per-submission abandonment token set *before* delegating `cancel()`, so nothing the work itself does can clear it. +- **C10** — A throwing `submit` left the callable queued with nothing scheduled to run it, wedging that conversation **permanently** and leaking the map entry for the JVM's lifetime. + +### No graceful shutdown existed at all (B3) + +`grep -rn ShutdownEvent src/main` matched nothing. A rolling deploy dropped every queued and in-flight turn with no drain and no readiness flip. Added `GracefulShutdownService` + `ShutdownReadinessHealthCheck`. + +### /rerun destroyed output and regenerated nothing (C5) + +Selective execution passes a sublist with `startIndex=0`, so the loop index is sublist-relative — but the component-cache **key** was built from that relative index while the cache **stores** under the absolute index. The output task then ran with `component == null` and no-op'd, *after* the prior output had already been deleted. `indexOffset` was already threaded in and used only for HITL bookkeeping. + +This one survived because **every existing `LifecycleManagerTest` stubs the component map empty** — the exact condition that hides it. The new test populates it at absolute indices and fails if the offset is removed. + +### B2: the finding's premise was inverted + +The review claimed "interrupt flags swallowed in 18 of 20 handlers". Auditing all 28 sites individually (27 explicit `catch (InterruptedException)` plus one hiding behind a broad `catch (Exception)`) found the opposite: **14 already restored the flag correctly and 9 rethrew; only 4 genuinely swallowed it.** Fixed those 4, plus: + +- **`ScheduleFireExecutor`** — a broad `catch (Exception)` swallowing `InterruptedException` from `latch.await(5, MINUTES)`, so the poller kept firing schedules after being interrupted for shutdown. +- **`NatsConversationCoordinator`** — the *mirror* bug, not in the finding: it called `interrupt()` **unconditionally** on `catch (InterruptedException | TimeoutException)`, so a drain timeout left the shutdown thread flagged and would abort the `@PreDestroy` steps after it. + +One restore is deliberately placed in a `finally` after the store round trips rather than at the top of the catch: the sync Mongo driver aborts with `MongoInterruptedException` when the calling thread is flagged, so an early restore would skip the very `EXECUTION_INTERRUPTED` write that branch exists to perform. + +### Dream wired up (I1, G8) + +Per the repo owner's decision, `DreamService` is now registered with `ScheduleFireExecutor` rather than deleted, with its ceiling switched from `maxSummarizationCalls` to the dollar-based `maxCostPerRun` the project's own guidance prescribes. `docs/architecture.md`'s claim that it performs scheduled maintenance is now true. + +**G8 mattered much more once Dream actually runs**: consolidation upgraded `self` visibility to `global` whenever a group spanned multiple agents — and with `summarizeGroupBy` defaulting to `category` and `preserveAgentProvenance` defaulting to false, cross-agent grouping was the *default* path. Two agents' private memories became one entry every agent could read. + +### Partial + +**F6** — the REST/pipeline half is done (a `ConnectionCallback` now sets cancelled on client disconnect, and cancellation is checked at more points). The in-`modules/llm` half — cancellation checks inside the tool loop and the cascade — is deferred, since that module is owned by another workstream. + +### Verification + +Clean compile passed **first attempt**, with no repairs needed despite three cross-workstream signature changes. Full suite: 12,633 tests, **0 non-environmental failures** (308 listed failures/errors all carry a loopback/selector/event-loop signature; this machine cannot bind sockets). All four mutation checks bite — C1, C5, C9 and B2 each fail a test when reverted, verified against whole test classes and with surefire reports checked to confirm the new classes actually executed rather than being silently skipped. + --- ## 🧠 fix(llm): code-review findings wave 2b — LLM core, persistent memory, migration, import/export (2026-07-28) diff --git a/src/main/java/ai/labs/eddi/configs/agents/model/AgentConfiguration.java b/src/main/java/ai/labs/eddi/configs/agents/model/AgentConfiguration.java index 5e3dd3da0..a39dcd89c 100644 --- a/src/main/java/ai/labs/eddi/configs/agents/model/AgentConfiguration.java +++ b/src/main/java/ai/labs/eddi/configs/agents/model/AgentConfiguration.java @@ -533,8 +533,15 @@ public void setAllowedCategories(List allowedCategories) { } /** - * Background Dream consolidation configuration. Uses - * {@code ScheduleFireExecutor} with SERVICE trigger type. + * Background Dream consolidation configuration. + *

+ * Dream runs through the regular cluster-aware schedule machinery: a + * {@code ScheduleConfiguration} carrying the metadata marker + * {@code {"dreamType": "dream_consolidation"}} plus the target {@code agentId} + * and {@code userId} is dispatched by {@code ScheduleFireExecutor} to + * {@code DreamService}, which resolves this block off the agent and runs the + * cycle. {@link #getSchedule()} is the cron expression such a schedule should + * use. */ public static class DreamConfig { private boolean enabled = false; @@ -569,10 +576,38 @@ public static class DreamConfig { * Whether to sub-group by sourceAgentId before consolidating. true = entries * from different agents stay separate (preserves provenance). false = entries * from all agents consolidated together (better compression). + *

+ * Note: this switch never applies to {@code self}-scoped memories. Those + * are always sub-grouped by {@code sourceAgentId}, because merging them across + * agents would produce a single entry readable by agents that never had access + * to the originals. */ private boolean preserveAgentProvenance = false; - /** Maximum LLM calls per dream cycle per user. Bounds cost. */ + /** + * Model parameters for the consolidation LLM — {@code apiKey}, {@code baseUrl}, + * {@code temperature}, … — passed through to {@code ChatModelRegistry} exactly + * like an LLM task's {@code parameters} block, so {@code ${vault:...}} and + * {@code ${vars:...}} references resolve the same way. + *

+ * Dream is a background job with no parent LLM task, so unlike the rolling + * conversation summary it has nothing to inherit credentials from — they must + * be configured here. Example: + * + * 

+         * "parameters": { "apiKey": "${vault:anthropic-api-key}" }
+         *
+ */ + private Map parameters = new HashMap<>(); + + /** + * @deprecated Since 6.1.0. Superseded by {@link #getMaxCostPerRun()} and no + * longer enforced. A call count is a meaningless ceiling because + * different consolidations cost vastly different amounts; Dream is + * bounded by the dollar budget instead. Retained only so existing + * stored/imported configurations keep deserializing. + */ + @Deprecated(since = "6.1.0", forRemoval = true) private int maxSummarizationCalls = 10; /** @@ -709,10 +744,28 @@ public void setPreserveAgentProvenance(boolean preserveAgentProvenance) { this.preserveAgentProvenance = preserveAgentProvenance; } + public Map getParameters() { + return parameters; + } + + public void setParameters(Map parameters) { + this.parameters = parameters != null ? parameters : new HashMap<>(); + } + + /** + * @deprecated Since 6.1.0. No longer enforced — see + * {@link #getMaxCostPerRun()}. + */ + @Deprecated(since = "6.1.0", forRemoval = true) public int getMaxSummarizationCalls() { return maxSummarizationCalls; } + /** + * @deprecated Since 6.1.0. No longer enforced — see + * {@link #setMaxCostPerRun(double)}. + */ + @Deprecated(since = "6.1.0", forRemoval = true) public void setMaxSummarizationCalls(int maxSummarizationCalls) { this.maxSummarizationCalls = maxSummarizationCalls; } diff --git a/src/main/java/ai/labs/eddi/engine/internal/ConversationService.java b/src/main/java/ai/labs/eddi/engine/internal/ConversationService.java index 977a563e6..95faee2fa 100644 --- a/src/main/java/ai/labs/eddi/engine/internal/ConversationService.java +++ b/src/main/java/ai/labs/eddi/engine/internal/ConversationService.java @@ -48,6 +48,7 @@ import ai.labs.eddi.engine.runtime.IRuntime; import ai.labs.eddi.engine.runtime.service.ServiceException; import ai.labs.eddi.engine.runtime.IConversationSetup; +import ai.labs.eddi.engine.runtime.internal.GracefulShutdownService; import ai.labs.eddi.engine.schedule.IScheduleStore; import ai.labs.eddi.engine.schedule.model.ScheduleConfiguration; import ai.labs.eddi.engine.security.ConversationAccessGuard; @@ -66,6 +67,8 @@ import java.time.Instant; import java.util.*; import java.util.concurrent.*; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Consumer; import static ai.labs.eddi.engine.memory.ConversationMemoryUtilities.*; @@ -133,6 +136,17 @@ public class ConversationService implements IConversationService { @Inject ConversationAccessGuard conversationAccessGuard; + /** + * Graceful-shutdown gate (B3). New turns are refused once a + * {@code ShutdownEvent} has been observed, so a rolling deploy drains what is + * already in flight instead of racing fresh work against the JVM exit. + *

+ * Field-injected for the same reason as {@link #attachmentStore}: the numerous + * direct-construction unit tests need no change. + */ + @Inject + GracefulShutdownService gracefulShutdownService; + /** * Fires {@link HitlResumeCompletedEvent} when a resume settles to a non-paused * state. Async so a slow channel observer never blocks the engine; observer @@ -160,7 +174,20 @@ public class ConversationService implements IConversationService { // existing). private final MeterRegistry meterRegistry; - private final List processingConversationReferences; + /** + * Number of turns currently being processed on this pod — the backing value of + * the {@code eddi_processing_conversation_count} gauge. + *

+ * C11: this used to be a {@code CopyOnWriteArrayList} of + * {@code agentId:conversationId} strings, mutated twice per turn purely to feed + * the gauge. That reference string is IDENTICAL for two concurrent turns on the + * same conversation, so a failing turn deleted a HEALTHY concurrent turn's + * entry; and a turn that never reached its completion consumer (watchdog + * timeout, inner future cancelled before starting) leaked its entry for the + * JVM's lifetime. A counter released exactly once per turn from a + * {@code finally} can do neither. + */ + private final AtomicInteger processingConversationCount = new AtomicInteger(); /** * Live memories of conversations currently executing on THIS pod, keyed by @@ -199,7 +226,6 @@ public ConversationService(IAgentFactory agentFactory, IConversationMemoryStore this.tenantQuotaService = tenantQuotaService; this.agentTimeout = agentTimeout; this.hitlResumeCompletedEvent = hitlResumeCompletedEvent; - this.processingConversationReferences = new CopyOnWriteArrayList<>(); this.timerConversationStart = meterRegistry.timer("eddi_conversation_start_duration"); this.timerConversationEnd = meterRegistry.timer("eddi_conversation_end_duration"); @@ -221,7 +247,46 @@ public ConversationService(IAgentFactory agentFactory, IConversationMemoryStore // vary per emission: verdict on resume, guard name on guard activation). this.meterRegistry = meterRegistry; - meterRegistry.gaugeCollectionSize("eddi_processing_conversation_count", Tags.empty(), processingConversationReferences); + meterRegistry.gauge("eddi_processing_conversation_count", Tags.empty(), processingConversationCount, AtomicInteger::doubleValue); + } + + /** + * One-shot release token for the in-flight-turn gauge (C11). Created when a + * turn is admitted, released exactly once no matter which of the many exit + * paths the turn takes — completion, skip, watchdog timeout, pipeline error or + * a pre-submission throw. Idempotent, so the turn callable's {@code finally} + * can act as a safety net behind the normal completion path. + */ + private static final class ProcessingTurn { + private final AtomicInteger counter; + private final AtomicBoolean released = new AtomicBoolean(false); + + private ProcessingTurn(AtomicInteger counter) { + this.counter = counter; + counter.incrementAndGet(); + } + + private void release() { + if (released.compareAndSet(false, true)) { + counter.decrementAndGet(); + } + } + } + + /** + * Rejects new work once {@link GracefulShutdownService} has observed a + * {@code ShutdownEvent} (B3). Turns already queued or in flight are drained by + * the shutdown observer; admitting new ones during the drain would either be + * dropped by the JVM exit or extend the drain indefinitely. + *

+ * A {@code null} gate means the bean was constructed outside CDI (only the + * direct-construction unit tests do that) and never rejects. + */ + private void rejectIfShuttingDown() { + if (gracefulShutdownService != null && gracefulShutdownService.isShuttingDown()) { + throw new RejectedExecutionException( + "This node is shutting down and no longer accepts new conversation turns — retry against another node"); + } } @Override @@ -231,6 +296,7 @@ public ConversationResult startConversation(Environment environment, String agen long startTime = System.nanoTime(); checkNotNull(environment, "environment"); checkNotNull(agentId, "agentId"); + rejectIfShuttingDown(); if (context == null) { context = new LinkedHashMap<>(); } @@ -411,6 +477,10 @@ public void say(Environment environment, String agentId, String conversationId, throws Exception { long startTime = System.nanoTime(); + rejectIfShuttingDown(); + // Assigned inside the try; the catch blocks need it, and the lambdas below + // need an effectively-final alias (processingTurn). + ProcessingTurn admittedTurn = null; try { final IConversationMemory conversationMemory = loadConversationMemory(conversationId); checkConversationMemoryNotNull(conversationMemory, conversationId); @@ -458,7 +528,8 @@ public void say(Environment environment, String agentId, String conversationId, throw new QuotaExceededException(quotaCheck.reason()); } - processingConversationReferences.add(createReferenceForMetrics(agentId, conversationId)); + admittedTurn = new ProcessingTurn(processingConversationCount); + final ProcessingTurn processingTurn = admittedTurn; // Set the audit collector on memory (if auditing is enabled) if (auditLedgerService.isEnabled()) { @@ -474,7 +545,7 @@ public void say(Environment environment, String agentId, String conversationId, cacheConversationState(conversationId, memorySnapshot.getConversationState()); conversationDescriptorStore.updateTimeStamp(conversationId); recordMetrics(timerConversationProcessing, counterConversationProcessing, startTime); - processingConversationReferences.remove(createReferenceForMetrics(agentId, conversationId)); + processingTurn.release(); responseHandler.onComplete(memorySnapshot); }); @@ -486,7 +557,7 @@ public void say(Environment environment, String agentId, String conversationId, returnDetailed, returnCurrentStepOnly, returningFields); memorySnapshot.setEnvironment(environment); recordMetrics(timerConversationProcessing, counterConversationProcessing, startTime); - processingConversationReferences.remove(createReferenceForMetrics(agentId, conversationId)); + processingTurn.release(); responseHandler.onSkipped(memorySnapshot); }; @@ -518,27 +589,37 @@ public void say(Environment environment, String agentId, String conversationId, } Callable processUserInput = processConversationStep(environment, conversationMemory, conversationId, loggingContext, - executeConversation, notifySkipped); + executeConversation, notifySkipped, processingTurn); conversationCoordinator.submitInOrder(conversationId, processUserInput); } catch (ProcessingRestrictedException | QuotaExceededException | ConversationAwaitingApprovalException e) { - throw e; // thrown before processingConversationReferences.add() + releaseTurn(admittedTurn); // all three are thrown before the turn is admitted + throw e; } catch (AgentMismatchException | AgentNotReadyException | ConversationEndedException e) { - processingConversationReferences.remove(createReferenceForMetrics(agentId, conversationId)); + releaseTurn(admittedTurn); throw e; } catch (Exception e) { LOGGER.error(e.getLocalizedMessage(), e); - processingConversationReferences.remove(createReferenceForMetrics(agentId, conversationId)); + releaseTurn(admittedTurn); throw e; } } + private static void releaseTurn(ProcessingTurn turn) { + if (turn != null) { + turn.release(); + } + } + @Override public void sayStreaming(Environment environment, String agentId, String conversationId, Boolean returnDetailed, Boolean returnCurrentStepOnly, List returningFields, InputData inputData, StreamingResponseHandler streamingHandler) throws Exception { long startTime = System.nanoTime(); + rejectIfShuttingDown(); + // See say(): assigned inside the try, aliased for the lambdas below. + ProcessingTurn admittedTurn = null; try { final IConversationMemory conversationMemory = loadConversationMemory(conversationId); checkConversationMemoryNotNull(conversationMemory, conversationId); @@ -584,7 +665,8 @@ public void sayStreaming(Environment environment, String agentId, String convers throw new QuotaExceededException(quotaCheck.reason()); } - processingConversationReferences.add(createReferenceForMetrics(agentId, conversationId)); + admittedTurn = new ProcessingTurn(processingConversationCount); + final ProcessingTurn processingTurn = admittedTurn; // Create event sink that delegates to the streaming handler var eventSink = new ConversationEventSink() { @@ -647,7 +729,7 @@ public void onTaskFailed(TaskId taskId, String taskType, long durationMs, cacheConversationState(conversationId, memorySnapshot.getConversationState()); conversationDescriptorStore.updateTimeStamp(conversationId); recordMetrics(timerConversationProcessing, counterConversationProcessing, startTime); - processingConversationReferences.remove(createReferenceForMetrics(agentId, conversationId)); + processingTurn.release(); streamingHandler.onComplete(memorySnapshot); }); @@ -673,22 +755,23 @@ public void onTaskFailed(TaskId taskId, String taskType, long durationMs, returnDetailed, returnCurrentStepOnly, returningFields); memorySnapshot.setEnvironment(environment); recordMetrics(timerConversationProcessing, counterConversationProcessing, startTime); - processingConversationReferences.remove(createReferenceForMetrics(agentId, conversationId)); + processingTurn.release(); streamingHandler.onSkipped(memorySnapshot); }; Callable processUserInput = processConversationStep(environment, conversationMemory, conversationId, loggingContext, - executeConversation, notifySkipped); + executeConversation, notifySkipped, processingTurn); conversationCoordinator.submitInOrder(conversationId, processUserInput); } catch (ProcessingRestrictedException | QuotaExceededException | ConversationAwaitingApprovalException e) { - throw e; // thrown before processingConversationReferences.add() + releaseTurn(admittedTurn); // all three are thrown before the turn is admitted + throw e; } catch (AgentMismatchException | AgentNotReadyException | ConversationEndedException e) { - processingConversationReferences.remove(createReferenceForMetrics(agentId, conversationId)); + releaseTurn(admittedTurn); throw e; } catch (Exception e) { LOGGER.error(e.getLocalizedMessage(), e); - processingConversationReferences.remove(createReferenceForMetrics(agentId, conversationId)); + releaseTurn(admittedTurn); throw e; } } @@ -963,70 +1046,86 @@ private IAgent getAgent(Environment environment, String agentId, Integer agentVe private Callable processConversationStep(Environment environment, IConversationMemory conversationMemory, String conversationId, Map loggingContext, Callable executeConversation, - Consumer skipNotifier) { + Consumer skipNotifier, ProcessingTurn processingTurn) { return () -> { - // Queued-say guard: this memory copy was loaded at REST-request time; - // a previously queued turn may have committed a pause (or a resume may - // be executing), or the conversation may have been terminally resolved - // (ENDED via endConversation) in the meantime. Skip the turn entirely — - // executing it against the stale snapshot would end with a full-document - // store that silently overwrites the pause (destroying the pending - // approval and orphaning its timeout schedule) or RESURRECTS a terminated - // conversation to READY with post-termination side effects. The skip - // notifier completes the caller's response handler with the persisted - // state, so the client gets a prompt, honest answer instead of a watchdog - // timeout. - // - // EXECUTION_INTERRUPTED is deliberately NOT skipped: unlike ENDED it is a - // RECOVERABLE marker meaning "the previous turn did not finish" (an - // agentTimeout watchdog expiry, or HitlCrashRecoveryObserver parking a - // stuck IN_PROGRESS conversation with the explicit intent to "unlock - // say()"). A fresh say must run a new turn to self-heal the conversation - // back to READY — mirroring the pre-HITL behavior where a retry after an - // interrupt executed normally. Skipping it would strand the conversation's - // input forever, since nothing else transitions EXECUTION_INTERRUPTED back - // to READY. - ConversationState persistedState = conversationMemoryStore.getConversationState(conversationId); - if (persistedState == ConversationState.AWAITING_HUMAN || persistedState == ConversationState.IN_PROGRESS - || persistedState == ConversationState.ENDED) { - conversationMemory.setConversationState(persistedState); - contextLogger.setLoggingContext(loggingContext); - LOGGER.warnf("Skipping queued turn for conversation %s: persisted state is %s (turn arrived before the state change)", - conversationId, persistedState); - if (skipNotifier != null) { - skipNotifier.accept(conversationMemory); - } - return null; - } - - // Zombie-pause guard: the state loaded WITH the snapshot at request - // time — on a backend whose snapshot state diverged from the CAS'd - // state column, this can still claim AWAITING_HUMAN even though the - // pause was terminally resolved (persistedState above says otherwise). - // Never execute against, persist, or re-arm a pause this turn did not - // produce. - final ConversationState memoryStateAtSubmit = conversationMemory.getConversationState(); - - // #2: register the live memory so cancelConversation can signal the - // running pipeline via setCancelled (checked at task boundaries). - inFlightConversations.put(conversationId, conversationMemory); - // Carry the agent-level tool-approval config onto memory BEFORE the - // pipeline (LlmTask) runs, so the tool-approval gate can resolve its - // effective config. Transient — never persisted; re-resolved each turn. - populateToolApprovalsConfig(conversationMemory); try { - runGuardedConversationStep(loggingContext, conversationId, environment, conversationMemory, - executeConversation, memoryStateAtSubmit, persistedState); + return runConversationStep(environment, conversationMemory, conversationId, loggingContext, + executeConversation, skipNotifier); } finally { - // value-conditional: only the leg that registered this memory may - // unregister — a plain remove(key) could evict a NEWER execution's - // entry and defeat its cooperative cancel. - inFlightConversations.remove(conversationId, conversationMemory); + // C11: the single guaranteed exit point of a turn. The completion + // consumer releases first on the happy path, but a watchdog timeout, + // a pipeline error or a cancelled inner future never reaches it — and + // an entry that is never released leaks into the gauge forever. + // release() is one-shot, so releasing twice is a no-op. + processingTurn.release(); } - return null; }; } + private Void runConversationStep(Environment environment, IConversationMemory conversationMemory, String conversationId, + Map loggingContext, Callable executeConversation, + Consumer skipNotifier) { + // Queued-say guard: this memory copy was loaded at REST-request time; + // a previously queued turn may have committed a pause (or a resume may + // be executing), or the conversation may have been terminally resolved + // (ENDED via endConversation) in the meantime. Skip the turn entirely — + // executing it against the stale snapshot would end with a full-document + // store that silently overwrites the pause (destroying the pending + // approval and orphaning its timeout schedule) or RESURRECTS a terminated + // conversation to READY with post-termination side effects. The skip + // notifier completes the caller's response handler with the persisted + // state, so the client gets a prompt, honest answer instead of a watchdog + // timeout. + // + // EXECUTION_INTERRUPTED is deliberately NOT skipped: unlike ENDED it is a + // RECOVERABLE marker meaning "the previous turn did not finish" (an + // agentTimeout watchdog expiry, or HitlCrashRecoveryObserver parking a + // stuck IN_PROGRESS conversation with the explicit intent to "unlock + // say()"). A fresh say must run a new turn to self-heal the conversation + // back to READY — mirroring the pre-HITL behavior where a retry after an + // interrupt executed normally. Skipping it would strand the conversation's + // input forever, since nothing else transitions EXECUTION_INTERRUPTED back + // to READY. + ConversationState persistedState = conversationMemoryStore.getConversationState(conversationId); + if (persistedState == ConversationState.AWAITING_HUMAN || persistedState == ConversationState.IN_PROGRESS + || persistedState == ConversationState.ENDED) { + conversationMemory.setConversationState(persistedState); + contextLogger.setLoggingContext(loggingContext); + LOGGER.warnf("Skipping queued turn for conversation %s: persisted state is %s (turn arrived before the state change)", + conversationId, persistedState); + if (skipNotifier != null) { + skipNotifier.accept(conversationMemory); + } + return null; + } + + // Zombie-pause guard: the state loaded WITH the snapshot at request + // time — on a backend whose snapshot state diverged from the CAS'd + // state column, this can still claim AWAITING_HUMAN even though the + // pause was terminally resolved (persistedState above says otherwise). + // Never execute against, persist, or re-arm a pause this turn did not + // produce. + final ConversationState memoryStateAtSubmit = conversationMemory.getConversationState(); + + // #2: register the live memory so cancelConversation can signal the + // running pipeline via setCancelled (checked at task boundaries). + inFlightConversations.put(conversationId, conversationMemory); + // Carry the agent-level tool-approval config onto memory BEFORE the + // pipeline (LlmTask) runs, so the tool-approval gate can resolve its + // effective config. Transient — never persisted; re-resolved each turn. + populateToolApprovalsConfig(conversationMemory); + try { + runGuardedConversationStep(loggingContext, conversationId, environment, conversationMemory, + executeConversation, memoryStateAtSubmit, persistedState); + } finally { + // value-conditional: only the leg that registered this memory may + // unregister — a plain remove(key) could evict a NEWER execution's + // entry and defeat its cooperative cancel. + inFlightConversations.remove(conversationId, conversationMemory); + } + return null; + } + private void runGuardedConversationStep(Map loggingContext, String conversationId, Environment environment, IConversationMemory conversationMemory, Callable executeConversation, ConversationState memoryStateAtSubmit, @@ -1151,7 +1250,14 @@ public void onComplete(Void result) { @Override public void onFailure(Throwable t) { - if (t instanceof LifecycleException.LifecycleInterruptedException) { + // C3: an abandoned turn that completed anyway is routed here by + // BaseRuntime's abandonment token. It must NOT be flipped to + // ERROR — the watchdog already persisted the accurate + // EXECUTION_INTERRUPTED (or deliberately left an AWAITING_HUMAN + // pause alone), and a late ERROR write from the zombie turn is + // exactly the stale overwrite the token exists to prevent. + // Mirrors the resume path's onFailure. + if (t instanceof InterruptedException || t instanceof LifecycleException.LifecycleInterruptedException) { String errorMessage = "Conversation processing got interrupted! (conversationId=%s)"; errorMessage = String.format(errorMessage, conversationId); contextLogger.setLoggingContext(loggingContext); @@ -1172,17 +1278,42 @@ private void waitForExecutionFinishOrTimeout(Map loggingContext, try { future.get(agentTimeout, TimeUnit.SECONDS); } catch (TimeoutException | InterruptedException e) { - // Guard: do not overwrite AWAITING_HUMAN with EXECUTION_INTERRUPTED (Invariant - // 10) - ConversationState currentState = conversationMemoryStore.getConversationState(conversationId); - if (currentState == ConversationState.AWAITING_HUMAN) { - return; + // C3: abandon the turn FIRST — before any further store round trip. The + // cancel marks BaseRuntime's per-submission abandonment token, which is + // what suppresses a late onComplete (and therefore its full-snapshot + // persist). The interrupt flag alone is NOT a safe completion guard: the + // pipeline consumes it via Thread.interrupted(), which CLEARS it, so a + // timed-out turn would still be reported complete and would overwrite a + // newer turn's state. Doing this before reading the persisted state keeps + // the "already abandoned but not yet flagged" window at ~0 instead of a + // full DB round trip. + // + // Cancelling on the AWAITING_HUMAN path too is deliberate: the watchdog + // has expired either way, so this turn's outcome must be discarded. Only + // the STATE write below is skipped there, to avoid overwriting a pause + // written by another writer with EXECUTION_INTERRUPTED (Invariant 10). + // + // B2: Future.get CLEARS the interrupt flag when it throws + // InterruptedException. The flag is restored in the finally below — + // deliberately AFTER the store round trips, not before them: a set flag + // makes the sync Mongo driver abort with MongoInterruptedException, which + // would skip the very EXECUTION_INTERRUPTED write this branch exists to + // perform. The finally also covers the AWAITING_HUMAN early return. + try { + future.cancel(true); + ConversationState currentState = conversationMemoryStore.getConversationState(conversationId); + if (currentState == ConversationState.AWAITING_HUMAN) { + return; + } + setConversationState(conversationId, ConversationState.EXECUTION_INTERRUPTED); + String errorMessage = "Execution of Workflows interrupted or timed out."; + contextLogger.setLoggingContext(loggingContext); + LOGGER.error(errorMessage, e); + } finally { + if (e instanceof InterruptedException) { + Thread.currentThread().interrupt(); + } } - setConversationState(conversationId, ConversationState.EXECUTION_INTERRUPTED); - String errorMessage = "Execution of Workflows interrupted or timed out."; - contextLogger.setLoggingContext(loggingContext); - LOGGER.error(errorMessage, e); - future.cancel(true); } catch (ExecutionException e) { logConversationError(loggingContext, conversationId, e); } @@ -1266,10 +1397,6 @@ private void recordMetrics(Timer timer, Counter counter, long startTime) { timer.record(System.nanoTime() - startTime, TimeUnit.NANOSECONDS); } - private static String createReferenceForMetrics(String agentId, String conversationId) { - return agentId.concat(":").concat(conversationId); - } - // --- HITL lifecycle --- @Override diff --git a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java index 1921b70c0..3952423c6 100644 --- a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java +++ b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java @@ -66,6 +66,8 @@ import java.time.Instant; import java.util.*; import java.util.concurrent.*; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; import java.util.stream.Collectors; /** @@ -94,6 +96,14 @@ public class GroupConversationService implements IGroupConversationService { */ private static final int DEFAULT_AGENT_TIMEOUT_SECONDS = 180; + /** + * How long an aborting orchestrator waits for cooperatively cancelled member + * turns to unwind before reclaiming their tasks. Cancellation releases the + * turns at their await points immediately, so this is only a safety bound for a + * turn that is between two await points. + */ + private static final int MEMBER_TURN_CANCEL_DRAIN_SECONDS = 5; + private final IAgentGroupStore groupStore; private final IGroupConversationStore conversationStore; private final IConversationService conversationService; @@ -1138,7 +1148,14 @@ public GroupConversation followUpWithMember(String groupConversationId, String t InputData inputData = new InputData(); inputData.setInput(question); Map context = new LinkedHashMap<>(); - context.put("groupTranscript", new Context(Context.ContextType.object, gc.getTranscript())); + // Snapshot, never the live list: the member conversation serialises this + // context on its own thread while this one keeps appending to the + // transcript (see the same hand-off in executeAgentTurn). + List followUpTranscript; + synchronized (gc.getTranscript()) { + followUpTranscript = List.copyOf(gc.getTranscript()); + } + context.put("groupTranscript", new Context(Context.ContextType.object, followUpTranscript)); context.put("groupId", new Context(Context.ContextType.string, gc.getGroupId())); context.put("groupConversationId", new Context(Context.ContextType.string, gc.getId())); inputData.setContext(context); @@ -1529,6 +1546,98 @@ private List resolveParticipants(DiscussionPhase phase, List m.speakingOrder() != null ? m.speakingOrder() : Integer.MAX_VALUE)).toList(); } + // ================================================================= + // Cooperative cancellation of in-flight member turns + // ================================================================= + + /** + * Cooperative cancellation handle shared by the member turns of a single + * parallel batch (a debate phase batch or a task-execution wave). + *

+ * {@link CompletableFuture#cancel(boolean)} does not interrupt the + * body of a {@code runAsync}/{@code supplyAsync} task — the JDK documents + * {@code mayInterruptIfRunning} as having no effect there. A "cancelled" member + * thread would therefore keep running and keep mutating the group document + * (transcript, task list, error list) long after the orchestrator gave up on it + * and persisted the document. Cancellation must be cooperative instead: the + * turn checks this token at its own await points and before every write. + *

+ * The lever is the response future the member turn blocks on — completing it + * exceptionally releases the turn immediately, without waiting for the agent's + * own timeout. + */ + static final class MemberTurnCancellation { + + private final AtomicBoolean cancelled = new AtomicBoolean(false); + private final Set> awaited = ConcurrentHashMap.newKeySet(); + + boolean isCancelled() { + return cancelled.get(); + } + + /** + * Register a future a member turn is about to block on. If cancellation already + * happened, the future is released right away — closing the race between + * {@link #cancel()} and a turn reaching its await point. + */ + void register(CompletableFuture future) { + awaited.add(future); + if (cancelled.get()) { + future.completeExceptionally(new MemberTurnCancelledException()); + } + } + + void unregister(CompletableFuture future) { + awaited.remove(future); + } + + /** Signal cancellation and release every member turn currently waiting. */ + void cancel() { + cancelled.set(true); + for (var future : awaited) { + future.completeExceptionally(new MemberTurnCancelledException()); + } + } + } + + /** + * Thrown out of a member turn that was cooperatively cancelled. It is never + * retried and never converted into a transcript entry by the member thread — + * the orchestrator owns the group document from the moment it cancels. + */ + static final class MemberTurnCancelledException extends RuntimeException { + + MemberTurnCancelledException() { + super("Member turn cancelled by the group orchestrator"); + } + } + + /** + * Atomically reserve one turn from the shared budget. + *

+ * A check-then-act ({@code turnCounter.get() >= maxTurns} followed by + * {@code incrementAndGet()}) lets all N member threads of a parallel wave pass + * the check on the last remaining turn and overshoot {@code maxTurns} by up to + * N-1 LLM calls. The CAS loop below hands out at most {@code maxTurns} turns in + * total, no matter how many threads race for them. The budget test itself is + * the one the callers used before ({@code counter >= maxTurns}), just fused + * with the increment. + * + * @return {@code true} if a turn was reserved, {@code false} if the budget is + * exhausted + */ + private static boolean reserveTurn(AtomicInteger turnCounter, int maxTurns) { + while (true) { + int current = turnCounter.get(); + if (current >= maxTurns) { + return false; + } + if (turnCounter.compareAndSet(current, current + 1)) { + return true; + } + } + } + // ================================================================= // Task-oriented phase execution (TASK_FORCE style) // ================================================================= @@ -1738,6 +1847,7 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura List errors = Collections.synchronizedList(new ArrayList<>()); int timeout = protocol.agentTimeoutSeconds() > 0 ? protocol.agentTimeoutSeconds() : DEFAULT_AGENT_TIMEOUT_SECONDS; int maxWaves = 100; // safety cap to prevent infinite loops + final SharedTaskList taskList = gc.getTaskList(); // Wave loop: re-query executable tasks after each wave completes. // Tasks that become executable when their dependencies finish are picked up @@ -1765,6 +1875,11 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura wave + 1, tasksByAgent.size(), tasksByAgent.values().stream().mapToInt(List::size).sum()); + // Cooperative cancellation for this wave's member turns: cancel(true) on + // the futures below does NOT interrupt their bodies, so aborting the wave + // has to signal through this token instead. + var cancellation = new MemberTurnCancellation(); + // Execute agents in parallel, tasks per agent sequentially List> futures = new ArrayList<>(); @@ -1780,12 +1895,19 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura CompletableFuture future = CompletableFuture.runAsync(() -> { for (TaskItem task : agentTasks) { - if (turnCounter.get() >= maxTurns) { - break; - } try { - turnCounter.incrementAndGet(); - gc.getTaskList().startTask(task.id()); + // Claim the turn budget and the task itself under the task-list + // monitor, together with the cancellation check. Atomically, + // because (a) N agent threads racing a check-then-act on the + // counter would overshoot maxTurns by up to N-1 LLM calls and + // (b) a task must never flip to IN_PROGRESS after an aborting + // orchestrator swept the list — that would strand it forever. + synchronized (taskList) { + if (cancellation.isCancelled() || !reserveTurn(turnCounter, maxTurns)) { + break; + } + taskList.startTask(task.id()); + } if (listener != null) { listener.onSpeakerStart(new GroupConversationEventSink.SpeakerStartEvent( @@ -1794,20 +1916,30 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura // Build task-specific input String taskInput = buildTaskExecutionInput(task, question, phase, gc); - TranscriptEntry entry = executeAgentTurn(member, gc, taskInput, protocol, phaseIdx, phase, null, listener); - - synchronized (gc.getTranscript()) { - gc.getTranscript().add(entry); - } + TranscriptEntry entry = executeAgentTurn(member, gc, taskInput, protocol, phaseIdx, phase, null, listener, cancellation); + + // The orchestrator owns the group document from the moment it + // cancels this wave: publish the result only if the wave is + // still live, again under the task-list monitor so the reset + // sweep cannot interleave. Lock order is always taskList → + // transcript. + synchronized (taskList) { + if (cancellation.isCancelled()) { + break; + } + synchronized (gc.getTranscript()) { + gc.getTranscript().add(entry); + } - // HITL TASK-level: submit for approval only when BOTH - // taskLevelHitl AND this phase requires approval. Otherwise - // auto-complete. Without this check, TASK_FORCE phases - // (requiresApproval=false) strand tasks in AWAITING_APPROVAL. - if (taskLevelHitl && phase.requiresApproval()) { - gc.getTaskList().submitForApproval(task.id(), entry.content()); - } else { - gc.getTaskList().completeTask(task.id(), entry.content()); + // HITL TASK-level: submit for approval only when BOTH + // taskLevelHitl AND this phase requires approval. Otherwise + // auto-complete. Without this check, TASK_FORCE phases + // (requiresApproval=false) strand tasks in AWAITING_APPROVAL. + if (taskLevelHitl && phase.requiresApproval()) { + taskList.submitForApproval(task.id(), entry.content()); + } else { + taskList.completeTask(task.id(), entry.content()); + } } if (listener != null) { @@ -1815,7 +1947,14 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura member.agentId(), member.displayName(), entry.content(), phaseIdx, phase.name())); } + } catch (MemberTurnCancelledException e) { + // Wave aborted while this turn was waiting — leave the group + // document alone; the reset sweep reclaims the task. + break; } catch (GroupDiscussionException e) { + if (cancellation.isCancelled()) { + break; // no writes after the orchestrator gave up on this wave + } // Quota errors are non-retryable — abort all tasks immediately if (e.getCause() instanceof QuotaExceededException) { errors.add(e); @@ -1826,6 +1965,9 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura break; } } catch (IllegalStateException e) { + if (cancellation.isCancelled()) { + break; // see above + } // H5 fix: catch status transition errors (e.g., double completion) LOGGER.warnf("Task state error for '%s': %s", task.subject(), e.getMessage()); handleTaskFailure(gc, task, member, e.getMessage(), phaseIdx, phase, listener, errors, @@ -1855,16 +1997,15 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura } catch (TimeoutException e) { LOGGER.warnf("Task execution timed out for group %s (wave %d)", LogSanitizer.sanitize(gc.getGroupId()), wave + 1); - futures.forEach(f -> f.cancel(true)); - resetStrandedInProgressTasks(gc, "wave timeout"); + abortWave(gc, futures, cancellation, "wave timeout"); break; } catch (java.util.concurrent.CancellationException e) { // R2: CANCEL_IMMEDIATE fires allOf.cancel(true) → CancellationException. - // Forward-cancel all source agent futures (allOf.cancel doesn't propagate). + // allOf.cancel does not propagate to the source futures — and cancelling + // those would not stop their bodies either — so abort cooperatively. LOGGER.infof("Wave cancelled via CANCEL_IMMEDIATE for group %s (wave %d)", LogSanitizer.sanitize(gc.getGroupId()), wave + 1); - futures.forEach(f -> f.cancel(true)); - resetStrandedInProgressTasks(gc, "wave cancellation"); + abortWave(gc, futures, cancellation, "wave cancellation"); break; } catch (ExecutionException | InterruptedException e) { LOGGER.warnf("Task execution error for group %s: %s", @@ -1872,9 +2013,7 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura if (e instanceof InterruptedException) { Thread.currentThread().interrupt(); } - // R2: Forward-cancel remaining source futures on any error - futures.forEach(f -> f.cancel(true)); - resetStrandedInProgressTasks(gc, "wave error"); + abortWave(gc, futures, cancellation, "wave error"); break; } @@ -1901,26 +2040,62 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura } } + /** + * Aborts a wave of member turns and reclaims what they left behind. + *

+ * Order matters: signal cooperative cancellation first (the futures' own + * {@code cancel(true)} would not stop their bodies), then give the member + * threads a bounded moment to unwind, and only then sweep the task list. + * Sweeping while a member thread is still running is what strands a task + * permanently IN_PROGRESS — the thread flips it after the sweep has passed it. + */ + private void abortWave(GroupConversation gc, List> futures, + MemberTurnCancellation cancellation, String cause) { + cancellation.cancel(); + try { + CompletableFuture.allOf(futures.toArray(CompletableFuture[]::new)) + .get(MEMBER_TURN_CANCEL_DRAIN_SECONDS, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } catch (TimeoutException | ExecutionException | CancellationException e) { + // A turn that has not reached its next await point yet still cannot write: + // every write is gated on the cancellation token under the task-list monitor. + LOGGER.debugf("Member turns of group %s did not unwind within %ds after %s", + LogSanitizer.sanitize(gc.getId()), MEMBER_TURN_CANCEL_DRAIN_SECONDS, cause); + } + resetStrandedInProgressTasks(gc, cause); + } + /** * Resets tasks stranded IN_PROGRESS by an aborted wave back to ASSIGNED. * Without this, a TASK-level pause committed after the abort persists tasks * that {@code findExecutableTasks} can never pick up again — they and their * dependents would silently never execute after resume (F11). + *

+ * The scan and the resets run under the task list's own monitor (the same one + * {@link SharedTaskList}'s synchronized methods use), so the sweep is a + * compare-and-set on each task's live state rather than on a stale snapshot: a + * member turn can neither start a task in the middle of the sweep nor complete + * one between the scan and the reset. */ private void resetStrandedInProgressTasks(GroupConversation gc, String cause) { - if (gc.getTaskList() == null) { + final SharedTaskList taskList = gc.getTaskList(); + if (taskList == null) { return; } - gc.getTaskList().all().stream() - .filter(t -> t.status() == SharedTaskList.TaskStatus.IN_PROGRESS) - .forEach(t -> { - try { - gc.getTaskList().resetToAssigned(t.id()); - LOGGER.infof("Reset stranded task '%s' to ASSIGNED after %s", t.id(), cause); - } catch (Exception ex) { - LOGGER.warnf("Failed to reset task '%s': %s", t.id(), ex.getMessage()); - } - }); + synchronized (taskList) { + for (TaskItem task : taskList.all()) { + if (task.status() != SharedTaskList.TaskStatus.IN_PROGRESS) { + continue; + } + try { + taskList.resetToAssigned(task.id()); + LOGGER.infof("Reset stranded task '%s' to ASSIGNED after %s", task.id(), cause); + } catch (Exception ex) { + LOGGER.warnf("Failed to reset task '%s': %s", task.id(), ex.getMessage()); + } + } + } } /** @@ -2353,7 +2528,15 @@ private void executeParallelPhase(GroupConversation gc, AgentGroupConfiguration : speakers; // SAFETY: Snapshot the transcript so parallel tasks each see a consistent view. - List snapshotTranscript = List.copyOf(gc.getTranscript()); + // Iterating a Collections.synchronizedList requires holding its monitor. + List snapshotTranscript; + synchronized (gc.getTranscript()) { + snapshotTranscript = List.copyOf(gc.getTranscript()); + } + + // Cooperative cancellation for this batch — cancel(true) does not stop a + // supplyAsync body, so a "cancelled" speaker would otherwise keep running. + var cancellation = new MemberTurnCancellation(); // Notify all speakers starting (parallel) if (listener != null) { @@ -2366,7 +2549,11 @@ private void executeParallelPhase(GroupConversation gc, AgentGroupConfiguration List> futures = batchSpeakers.stream().map(speaker -> CompletableFuture.supplyAsync(() -> { try { String input = buildPhaseInput(phase, speaker, question, snapshotTranscript, phaseIdx, null); - return executeAgentTurn(speaker, gc, input, protocol, phaseIdx, phase, null, listener); + return executeAgentTurn(speaker, gc, input, protocol, phaseIdx, phase, null, listener, cancellation); + } catch (MemberTurnCancelledException e) { + // The orchestrator stopped waiting for this batch — surface the + // cancellation instead of fabricating a contribution for it. + throw new java.util.concurrent.CompletionException(e); } catch (GroupDiscussionException e) { if (e.getCause() instanceof QuotaExceededException) { throw new java.util.concurrent.CompletionException(e); @@ -2380,16 +2567,23 @@ private void executeParallelPhase(GroupConversation gc, AgentGroupConfiguration }, executorService)).toList(); int timeout = protocol.agentTimeoutSeconds() > 0 ? protocol.agentTimeoutSeconds() : DEFAULT_AGENT_TIMEOUT_SECONDS; + // ONE deadline for the whole batch: these turns run concurrently, so giving + // every get() the full budget in turn made the worst case N × timeout + // (10 members × 180s = 30 minutes) instead of the configured timeout. + long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(timeout); for (int i = 0; i < futures.size(); i++) { try { - TranscriptEntry entry = futures.get(i).get(timeout, TimeUnit.SECONDS); + long remainingNanos = Math.max(0, deadlineNanos - System.nanoTime()); + TranscriptEntry entry = futures.get(i).get(remainingNanos, TimeUnit.NANOSECONDS); gc.getTranscript().add(entry); if (listener != null) { listener.onSpeakerComplete(new GroupConversationEventSink.SpeakerCompleteEvent(entry.speakerAgentId(), entry.speakerDisplayName(), entry.content(), phaseIdx, phase.name())); } } catch (TimeoutException e) { - futures.get(i).cancel(true); + // The batch deadline passed — release every speaker still waiting on a + // response, not just this one. + cancellation.cancel(); gc.getTranscript().add(new TranscriptEntry("unknown", "Unknown", null, phaseIdx, phase.name(), TranscriptEntryType.SKIPPED, Instant.now(), "Timeout", null)); } catch (ExecutionException e) { @@ -2399,12 +2593,17 @@ private void executeParallelPhase(GroupConversation gc, AgentGroupConfiguration if (cause instanceof java.util.concurrent.CompletionException ce) { cause = ce.getCause(); } + if (cause instanceof MemberTurnCancelledException) { + // Already released by the batch deadline above — same outcome as a + // speaker whose own get() timed out. + gc.getTranscript().add(new TranscriptEntry("unknown", "Unknown", null, phaseIdx, phase.name(), TranscriptEntryType.SKIPPED, + Instant.now(), "Timeout", null)); + continue; + } if (cause instanceof GroupDiscussionException gde && gde.getCause() instanceof QuotaExceededException) { - // Cancel remaining futures and propagate - for (int j = i + 1; j < futures.size(); j++) { - futures.get(j).cancel(true); - } + // Release the remaining speakers and propagate + cancellation.cancel(); throw gde; } gc.getTranscript().add(errorEntry(null, phaseIdx, phase, e.getMessage())); @@ -2457,9 +2656,32 @@ private void executePeerTargetedPhase(GroupConversation gc, AgentGroupConfigurat // Agent turn execution // ================================================================= + /** + * Runs a member turn that cannot be cancelled — sequential phases, where the + * orchestrator thread is the member turn. + */ private TranscriptEntry executeAgentTurn(GroupMember member, GroupConversation gc, String input, ProtocolConfig protocol, int phaseIdx, DiscussionPhase phase, String targetAgentId, GroupDiscussionEventListener listener) throws GroupDiscussionException { + return executeAgentTurn(member, gc, input, protocol, phaseIdx, phase, targetAgentId, listener, null); + } + + /** + * @param cancellation + * cooperative cancellation token for turns that run on a worker + * thread, or {@code null} for turns the orchestrator runs itself. + * When it is signalled the turn is released from its response wait + * and throws {@link MemberTurnCancelledException} instead of + * returning an entry — see {@link MemberTurnCancellation}. + */ + private TranscriptEntry executeAgentTurn(GroupMember member, GroupConversation gc, String input, ProtocolConfig protocol, int phaseIdx, + DiscussionPhase phase, String targetAgentId, GroupDiscussionEventListener listener, + MemberTurnCancellation cancellation) + throws GroupDiscussionException { + + if (cancellation != null && cancellation.isCancelled()) { + throw new MemberTurnCancelledException(); + } TranscriptEntryType entryType = mapPhaseToEntryType(phase.type()); @@ -2511,7 +2733,16 @@ private TranscriptEntry executeAgentTurn(GroupMember member, GroupConversation g InputData inputData = new InputData(); inputData.setInput(input); Map context = new LinkedHashMap<>(); - context.put("groupTranscript", new Context(Context.ContextType.object, gc.getTranscript())); + // Snapshot instead of handing out the live list: this context is serialised + // on the member conversation's own thread while the orchestrator keeps + // appending entries. Collections.synchronizedList makes add() safe but NOT + // iteration — publishing it by reference produced intermittent + // ConcurrentModificationExceptions that failed a member turn at random. + List transcriptSnapshot; + synchronized (gc.getTranscript()) { + transcriptSnapshot = List.copyOf(gc.getTranscript()); + } + context.put("groupTranscript", new Context(Context.ContextType.object, transcriptSnapshot)); context.put("groupId", new Context(Context.ContextType.string, gc.getGroupId())); context.put("groupConversationId", new Context(Context.ContextType.string, gc.getId())); context.put("groupDepth", new Context(Context.ContextType.string, String.valueOf(gc.getDepth()))); @@ -2541,6 +2772,9 @@ private TranscriptEntry executeAgentTurn(GroupMember member, GroupConversation g int timeout = protocol.agentTimeoutSeconds() > 0 ? protocol.agentTimeoutSeconds() : DEFAULT_AGENT_TIMEOUT_SECONDS; while (true) { + if (cancellation != null && cancellation.isCancelled()) { + throw new MemberTurnCancelledException(); + } try { CompletableFuture responseFuture = new CompletableFuture<>(); final String convId = privateConvId; @@ -2577,7 +2811,27 @@ private TranscriptEntry executeAgentTurn(GroupMember member, GroupConversation g responseFuture.complete(response); }); - String response = responseFuture.get(timeout, TimeUnit.SECONDS); + // The only await point of a member turn — and therefore the lever for + // cancelling it. Registering the future means an aborting orchestrator + // completes it exceptionally and releases this turn at once, instead of + // the turn running to completion and writing into a group document the + // orchestrator has already persisted. + String response; + try { + if (cancellation != null) { + cancellation.register(responseFuture); + } + response = responseFuture.get(timeout, TimeUnit.SECONDS); + } catch (ExecutionException ee) { + if (cancellation != null && cancellation.isCancelled()) { + throw new MemberTurnCancelledException(); + } + throw ee; + } finally { + if (cancellation != null) { + cancellation.unregister(responseFuture); + } + } // #3 / Task 13: member requested human approval mid-turn. A TOOL_CALL // pause is auto-resolved gracefully — the group rejects the gated @@ -2688,7 +2942,15 @@ private TranscriptEntry executeAgentTurn(GroupMember member, GroupConversation g signatureNonce, signatureTimestampMs, signatureKeyVersion); return entry; + } catch (MemberTurnCancelledException e) { + // Cooperative cancellation is not a member failure: never retried, never + // turned into a transcript entry by this thread. + throw e; + } catch (TimeoutException e) { + if (cancellation != null && cancellation.isCancelled()) { + throw new MemberTurnCancelledException(); + } if (protocol.onAgentFailure() == ProtocolConfig.MemberFailurePolicy.RETRY && retries < maxRetries) { retries++; LOGGER.warnf("Agent %s timed out (attempt %d/%d), retrying...", member.agentId(), retries, maxRetries); @@ -2710,6 +2972,9 @@ private TranscriptEntry executeAgentTurn(GroupMember member, GroupConversation g // convId is try-scoped, so pass the method-level privateConvId (same value). return handleMemberPause(member, gc, privateConvId, phaseIdx, phase, targetAgentId, listener); } catch (Exception e) { + if (cancellation != null && cancellation.isCancelled()) { + throw new MemberTurnCancelledException(); + } Throwable cause = e instanceof ExecutionException ? e.getCause() : e; // Quota errors are non-retryable and affect all agents — abort immediately if (cause instanceof QuotaExceededException) { diff --git a/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngine.java b/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngine.java index 3686790a7..726eb90e9 100644 --- a/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngine.java +++ b/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngine.java @@ -12,6 +12,7 @@ import ai.labs.eddi.engine.hitl.HitlAccessGuard; import ai.labs.eddi.engine.hitl.tools.IHitlToolJournalStore; import ai.labs.eddi.engine.api.IRestAgentEngine; +import ai.labs.eddi.engine.lifecycle.model.ControlSignal; import ai.labs.eddi.engine.lifecycle.model.HitlDecision; import ai.labs.eddi.engine.memory.IConversationMemoryStore; import ai.labs.eddi.engine.model.PendingApprovalSummary; @@ -312,8 +313,15 @@ public Response cancelConversation(String conversationId) { validateConversationOwnership(conversationId, true); try { String cancelledBy = identity.getPrincipal() != null ? identity.getPrincipal().getName() : null; + // GRACEFUL is deliberate and is the ONLY mode this endpoint offers: the + // REST contract (IRestAgentEngine#cancelConversation) takes no mode + // parameter and documents a graceful cancel, so there is no caller + // intent to downgrade here. On the regular surface CANCEL_IMMEDIATE has + // no implementation at all — ConversationService silently degrades it to + // graceful — which is why it is not, and must not be, reachable from + // this API until it either does something or is deleted from the enum. var outcome = conversationService.cancelConversation(conversationId, - ai.labs.eddi.engine.lifecycle.model.ControlSignal.CANCEL_GRACEFUL, cancelledBy); + ControlSignal.CANCEL_GRACEFUL, cancelledBy); // Plain-text, curated bodies: never reflect the raw conversationId (it is // a caller-supplied path param — echoing it is a reflected-XSS vector) and // never leak internal exception detail to the client. diff --git a/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngineStreaming.java b/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngineStreaming.java index 57d11f49d..a04479798 100644 --- a/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngineStreaming.java +++ b/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngineStreaming.java @@ -9,6 +9,7 @@ import ai.labs.eddi.engine.memory.model.SimpleConversationMemorySnapshot; import ai.labs.eddi.engine.lifecycle.TaskId; +import ai.labs.eddi.engine.lifecycle.model.ControlSignal; import ai.labs.eddi.engine.model.InputData; import ai.labs.eddi.engine.security.ConversationAccessGuard; import jakarta.enterprise.context.ApplicationScoped; @@ -21,6 +22,7 @@ import java.util.List; import java.util.Map; +import java.util.concurrent.atomic.AtomicBoolean; /** * SSE streaming implementation — maps ConversationService streaming events to @@ -47,6 +49,12 @@ public class RestAgentEngineStreaming implements IRestAgentEngineStreaming { private static final Logger LOGGER = Logger.getLogger(RestAgentEngineStreaming.class); private static final com.fasterxml.jackson.databind.ObjectMapper MAPPER = new com.fasterxml.jackson.databind.ObjectMapper(); + /** + * Audit actor recorded when a turn is cancelled because the SSE client went + * away. + */ + static final String CANCELLED_BY_CLIENT_DISCONNECT = "system:client-disconnect"; + private final IConversationService conversationService; private final ConversationAccessGuard conversationAccessGuard; @@ -70,12 +78,16 @@ public void sayStreaming(String conversationId, Boolean returnDetailed, Boolean // ConversationService re-checks: this layer is defence in depth. conversationAccessGuard.requireConversationOwner(conversationId); + // Every outbound frame goes through this stream, which doubles as the + // client-disconnect detector — see SseStream. + final SseStream stream = new SseStream(conversationId, safeConversationId, eventSink, sse); + try { conversationService.sayStreaming(conversationId, returnDetailed, returnCurrentStepOnly, returningFields, inputData, new IConversationService.StreamingResponseHandler() { @Override public void onTaskStart(TaskId taskId, String taskType, int index) { - sendEvent(eventSink, sse, "task_start", + stream.send("task_start", String.format("{\"taskId\":\"%s\",\"taskType\":\"%s\",\"index\":%d}", taskId.getIdentifier(), taskType, index)); } @@ -99,90 +111,184 @@ public void onTaskComplete(TaskId taskId, String taskType, long durationMs, Map< sb.append(",\"confidence\":").append(summary.get("confidence")); } sb.append("}"); - sendEvent(eventSink, sse, "task_complete", sb.toString()); + stream.send("task_complete", sb.toString()); } @Override public void onToken(String token) { - sendEvent(eventSink, sse, "token", token); + stream.send("token", token); } @Override public void onCascadeStepStart(int stepIndex, String modelType, String modelName, int totalSteps) { - sendJsonEvent(eventSink, sse, "cascade_step_start", + stream.sendJson("cascade_step_start", new CascadeStepStartEvent(stepIndex, modelType, modelName, totalSteps)); } @Override public void onCascadeEscalation(int fromStep, int toStep, double confidence, double threshold, String reason, long durationMs) { - sendJsonEvent(eventSink, sse, "cascade_escalation", + stream.sendJson("cascade_escalation", new CascadeEscalationEvent(fromStep, toStep, finite(confidence), finite(threshold), reason, durationMs)); } @Override public void onComplete(SimpleConversationMemorySnapshot snapshot) { + stream.markTerminal(); try { // Send the final snapshot as JSON - sendEvent(eventSink, sse, "done", toJson(snapshot)); + stream.send("done", toJson(snapshot)); } finally { - closeQuietly(eventSink); + stream.close(); } } @Override public void onError(Throwable error) { + stream.markTerminal(); try { LOGGER.errorf("Streaming error for conversation %s: %s", safeConversationId, error.getMessage()); - sendEvent(eventSink, sse, "error", String.format("{\"message\":\"%s\"}", escapeJson(error.getMessage()))); + stream.send("error", String.format("{\"message\":\"%s\"}", escapeJson(error.getMessage()))); } finally { - closeQuietly(eventSink); + stream.close(); } } @Override public void onTaskFailed(TaskId taskId, String taskType, long durationMs, String errorType, String errorSummary) { - sendEvent(eventSink, sse, "task_failed", + stream.send("task_failed", String.format("{\"taskId\":\"%s\",\"taskType\":\"%s\",\"durationMs\":%d,\"errorType\":\"%s\",\"error\":\"%s\"}", escapeJson(taskId.getIdentifier()), escapeJson(taskType), durationMs, escapeJson(errorType), escapeJson(errorSummary))); } }); } catch (Exception e) { + stream.markTerminal(); LOGGER.errorf("Failed to start streaming for conversation %s: %s", safeConversationId, e.getMessage()); - sendEvent(eventSink, sse, "error", String.format("{\"message\":\"%s\"}", escapeJson(e.getMessage()))); - closeQuietly(eventSink); + stream.send("error", String.format("{\"message\":\"%s\"}", escapeJson(e.getMessage()))); + stream.close(); } } - private void sendEvent(SseEventSink eventSink, Sse sse, String eventName, String data) { - if (eventSink.isClosed()) { - LOGGER.debugf("SSE sink closed, dropping event: %s", eventName); - return; + /** + * Per-request SSE stream: it owns the sink and, crucially, notices when the + * client is gone. + *

+ * Why the send path is the disconnect detector. The endpoint + * receives only {@link SseEventSink} and {@link Sse}. A JAX-RS + * {@code ConnectionCallback} is not an option here: RESTEasy Reactive's + * {@code AsyncResponseImpl.register} stores connection callbacks in a request + * property that nothing in the server ever reads, so registering one + * would be dead code that silently cancels nothing. What IS reliable is + * {@code SseEventSink.isClosed()} — RESTEasy Reactive implements it as + * {@code serverResponse().closed()}, which Vert.x flips as soon as the client + * drops the connection. Every outbound frame therefore re-checks it, which + * makes a disconnect observable at the very next token/task boundary. + *

+ * On the first such observation before the terminal frame, the in-flight turn + * is cancelled through {@code IConversationService.cancelConversation}, which + * sets the cooperative cancel flag on the live conversation memory. Without it, + * closing the tab at token 5 of 4000 still ran — and billed — the whole + * completion, and could escalate through every cascade model on the way. + *

+ * The cancel is issued synchronously on the calling (pipeline worker) thread + * and at most once, so it never touches the Vert.x event loop and never + * repeats. + */ + private final class SseStream { + private final String conversationId; + private final String safeConversationId; + private final SseEventSink eventSink; + private final Sse sse; + /** + * Set once the terminal ({@code done}/{@code error}) frame is being emitted. + */ + private final AtomicBoolean terminal = new AtomicBoolean(); + /** Guarantees the disconnect cancel is signalled at most once per stream. */ + private final AtomicBoolean cancelSignalled = new AtomicBoolean(); + + private SseStream(String conversationId, String safeConversationId, SseEventSink eventSink, Sse sse) { + this.conversationId = conversationId; + this.safeConversationId = safeConversationId; + this.eventSink = eventSink; + this.sse = sse; } - try { - eventSink.send(sse.newEventBuilder().name(eventName).data(String.class, data).build()); - } catch (Exception e) { - LOGGER.warnf("Failed to send SSE event '%s': %s", eventName, e.getMessage()); + + /** + * Marks the stream as finishing normally, so the sink closing from here on is + * our own doing and must not be mistaken for a client disconnect. + */ + void markTerminal() { + terminal.set(true); } - } - /** - * Serialize a typed event payload to JSON via Jackson and send it. Preferred - * over hand-built JSON strings — the mapper handles string escaping and number - * formatting. Falls back to an empty object on the (unexpected) serialization - * failure so a single bad payload cannot break the stream. - */ - private void sendJsonEvent(SseEventSink eventSink, Sse sse, String eventName, Object payload) { - String data; - try { - data = MAPPER.writeValueAsString(payload); - } catch (Exception e) { - LOGGER.warnf("Failed to serialize '%s' event payload: %s", eventName, e.getMessage()); - data = "{}"; + void send(String eventName, String data) { + if (eventSink.isClosed()) { + LOGGER.debugf("SSE sink closed, dropping event: %s", eventName); + onClientGone(); + return; + } + try { + eventSink.send(sse.newEventBuilder().name(eventName).data(String.class, data).build()); + } catch (Exception e) { + LOGGER.warnf("Failed to send SSE event '%s': %s", eventName, e.getMessage()); + // RESTEasy Reactive throws IllegalStateException synchronously when the + // sink closed between the check above and the write. Re-checking the + // sink (rather than treating every send failure as a disconnect) keeps + // an unrelated failure — a broken event payload, say — from cancelling + // a turn whose client is still connected and waiting. + if (eventSink.isClosed()) { + onClientGone(); + } + } + } + + /** + * Serialize a typed event payload to JSON via Jackson and send it. Preferred + * over hand-built JSON strings — the mapper handles string escaping and number + * formatting. Falls back to an empty object on the (unexpected) serialization + * failure so a single bad payload cannot break the stream. + */ + void sendJson(String eventName, Object payload) { + String data; + try { + data = MAPPER.writeValueAsString(payload); + } catch (Exception e) { + LOGGER.warnf("Failed to serialize '%s' event payload: %s", eventName, e.getMessage()); + data = "{}"; + } + send(eventName, data); + } + + void close() { + try { + if (!eventSink.isClosed()) { + eventSink.close(); + } + } catch (Exception e) { + LOGGER.debugf("Error closing SSE sink: %s", e.getMessage()); + } + } + + /** + * The client is no longer reading this stream. Cancel the turn it was waiting + * on — once, and never for a stream that already delivered its terminal frame. + */ + private void onClientGone() { + if (terminal.get() || !cancelSignalled.compareAndSet(false, true)) { + return; + } + LOGGER.infof("SSE client disconnected from conversation %s — cancelling the in-flight turn", + safeConversationId); + try { + conversationService.cancelConversation(conversationId, ControlSignal.CANCEL_GRACEFUL, + CANCELLED_BY_CLIENT_DISCONNECT); + } catch (Exception e) { + LOGGER.warnf("Failed to cancel conversation %s after client disconnect: %s", + safeConversationId, e.getMessage()); + } } - sendEvent(eventSink, sse, eventName, data); } /** Typed payload for the {@code cascade_step_start} SSE event. */ @@ -193,16 +299,6 @@ private record CascadeStepStartEvent(int stepIndex, String modelType, String mod private record CascadeEscalationEvent(int fromStep, int toStep, double confidence, double threshold, String reason, long durationMs) { } - private void closeQuietly(SseEventSink eventSink) { - try { - if (!eventSink.isClosed()) { - eventSink.close(); - } - } catch (Exception e) { - LOGGER.debugf("Error closing SSE sink: %s", e.getMessage()); - } - } - /** * Coerce a non-finite double (NaN/Infinity) to 0.0 so it serializes as valid * JSON. diff --git a/src/main/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManager.java b/src/main/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManager.java index 472e0c929..9e43caffc 100644 --- a/src/main/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManager.java +++ b/src/main/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManager.java @@ -212,10 +212,11 @@ public void executeLifecycle(final IConversationMemory conversationMemory, List< executeTaskRange(conversationMemory, this.lifecycleTasks, 0, 0); } else { // Selective execution: run the suffix of the pipeline starting at the - // first task whose type matches. The sublist is passed (preserving the - // component-cache/telemetry index base), but the absolute offset is - // threaded through so a HITL pause records an ABSOLUTE task index that - // resume can re-enter against the full task list. + // first task whose type matches. A SUBLIST is passed, so the loop index + // inside executeTaskRange is sublist-relative; the absolute offset is + // threaded through so every index-keyed lookup there (component cache, + // telemetry, audit, HITL bookmark) still resolves against the FULL task + // list. int startAbsolute = getLifecycleStartIndex(lifecycleTaskTypes); if (startAbsolute < 0) { return; // no task matches the requested types — nothing to execute @@ -269,6 +270,15 @@ private void executeTaskRange(IConversationMemory conversationMemory, for (int index = startIndex; index < tasks.size(); index++) { ILifecycleTask task = tasks.get(index); + // Position of this task in the workflow's FULL task list. On a selective + // (sublist) execution the loop index is sublist-relative, but every + // index-keyed lookup below is ABSOLUTE: WorkflowStoreClientLibrary caches + // each task's component under the workflow-step index, and the HITL + // bookmark / telemetry / audit rows are read back against the full list. + // Using the relative index made a rerun look up a component key that was + // never written, so the task ran with component == null and no-opped. + final int absoluteIndex = indexOffset + index; + // Cancel check (Wave 0) if (conversationMemory.isCancelled()) { throw new ConversationStopException(); @@ -304,7 +314,7 @@ private void executeTaskRange(IConversationMemory conversationMemory, Span taskSpan = getTracer().spanBuilder("eddi.pipeline.task") .setAttribute("eddi.task.id", task.getId().name()) .setAttribute("eddi.task.type", Objects.requireNonNullElse(task.getType(), "unknown")) - .setAttribute("eddi.task.index", (long) index) + .setAttribute("eddi.task.index", (long) absoluteIndex) .setAttribute("eddi.conversation.id", Objects.requireNonNullElse(conversationMemory.getConversationId(), "unknown")) .setAttribute("eddi.agent.id", @@ -318,12 +328,12 @@ private void executeTaskRange(IConversationMemory conversationMemory, // Component contains task-specific configuration loaded during agent // initialization var components = componentCache.getComponentMap(task.getId().name()); - var componentKey = createComponentKey(workflowId.getId(), workflowId.getVersion(), index); + var componentKey = createComponentKey(workflowId.getId(), workflowId.getVersion(), absoluteIndex); var component = components.getOrDefault(componentKey, null); // Emit task_start event if streaming if (eventSink != null) { - eventSink.onTaskStart(task.getId(), task.getType(), index); + eventSink.onTaskStart(task.getId(), task.getType(), absoluteIndex); } // Execute the task, transforming the conversation memory @@ -340,16 +350,16 @@ private void executeTaskRange(IConversationMemory conversationMemory, // Emit audit entry if audit collector is set var auditCollector = conversationMemory.getAuditCollector(); if (auditCollector != null) { - AuditEntry auditEntry = buildAuditEntry(conversationMemory, task, index, durationMs, summary); + AuditEntry auditEntry = buildAuditEntry(conversationMemory, task, absoluteIndex, durationMs, summary); auditCollector.collect(auditEntry); } // Check if task triggered a STOP_CONVERSATION action checkIfStopConversationAction(conversationMemory); - // The pause bookmark must be ABSOLUTE (offset + loop index) so resume - // re-enters the full task list at the right place, even when this is a - // selective (sublist) execution where the loop index is offset-relative. - checkIfPauseConversationAction(conversationMemory, indexOffset + index, actionsBefore); + // The pause bookmark must be ABSOLUTE so resume re-enters the full task + // list at the right place, even when this is a selective (sublist) + // execution where the loop index is offset-relative. + checkIfPauseConversationAction(conversationMemory, absoluteIndex, actionsBefore); } catch (LifecycleException | RuntimeException e) { // HITL tool pause: a gated LLM tool call is NOT a task failure. Convert @@ -359,7 +369,7 @@ private void executeTaskRange(IConversationMemory conversationMemory, // snapshot, exactly like the rule-based PAUSE_CONVERSATION path. if (e instanceof ai.labs.eddi.engine.hitl.tools.ToolApprovalRequiredException tare) { taskSpan.setAttribute("eddi.hitl.pause", "tool_call"); - throw new ConversationPauseException(workflowId.getId(), indexOffset + index, + throw new ConversationPauseException(workflowId.getId(), absoluteIndex, tare.getPauseReason(), ConversationPauseException.PauseOrigin.TOOL_CALL); } @@ -418,7 +428,7 @@ private void executeTaskRange(IConversationMemory conversationMemory, UUID.randomUUID().toString(), conversationMemory.getConversationId(), conversationMemory.getAgentId(), conversationMemory.getAgentVersion(), conversationMemory.getUserId(), null, conversationMemory.size() - 1, - errTaskId, errTaskType, index, failDurationMs, + errTaskId, errTaskType, absoluteIndex, failDurationMs, null, failureOutput, null, null, null, 0.0, Instant.now(), null, null); auditCollector.collect(failureEntry); @@ -455,6 +465,16 @@ private void executeTaskRange(IConversationMemory conversationMemory, taskSpan.end(); } } + + // Exit cancel check. The in-loop check only guards the transition INTO a task, + // so a cancel that lands while the LAST task runs was never observed: the loop + // simply ran out, the turn returned normally, and Conversation went on to + // commit the turn's side effects (long-term property upserts) for work the + // caller was already told is cancelled. Re-checking here closes that window + // for the last task of every workflow, and for an empty/exhausted range. + if (conversationMemory.isCancelled()) { + throw new ConversationStopException(); + } } /** diff --git a/src/main/java/ai/labs/eddi/engine/runtime/BaseRuntime.java b/src/main/java/ai/labs/eddi/engine/runtime/BaseRuntime.java index 71ca6de3e..c4b16f765 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/BaseRuntime.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/BaseRuntime.java @@ -13,6 +13,7 @@ import jakarta.inject.Inject; import java.util.Map; import java.util.concurrent.*; +import java.util.concurrent.atomic.AtomicBoolean; /** * @author ginccc @@ -26,6 +27,41 @@ public class BaseRuntime implements IRuntime { private final String projectName; private final ScheduledExecutorService scheduledExecutorService; + /** + * Executor for NESTED submissions — work submitted from a thread that is itself + * running a callable submitted through {@link #submitCallable}. + *

+ * A conversation turn consumes TWO threads: the coordinator's callable (the + * "outer" task) submits the pipeline execution (the "inner" task) and then + * BLOCKS on its {@link Future} for up to the agent timeout. When both come from + * the same bounded {@link ManagedExecutor}, concurrency collapses at HALF the + * pool size: every thread becomes a waiter, no inner task can ever be + * scheduled, and every turn fails at the watchdog. That is a hard cliff, not a + * slow degradation. + *

+ * Routing the nested (inner) task to a virtual-thread executor removes the + * cliff: the inner task can always be scheduled, so waiters always make + * progress. Virtual threads are already the established pattern for + * conversation-adjacent work in this codebase (SchedulePollerService, + * GroupConversationService, the Slack handlers), and the pipeline has no + * dependency on the ManagedExecutor's context propagation — there is not a + * single {@code @RequestScoped} bean in the engine, and the Slack/group entry + * points already drive full turns with no request context active. + *

+ * Only the work BODY is marked as nested — completion callbacks run with the + * marker cleared, so the coordinator's {@code submitNext} still schedules the + * next turn's outer task on the ManagedExecutor exactly as before. + */ + private final ExecutorService nestedExecutorService; + + /** + * Set on a thread for the duration of a submitted callable's BODY (not its + * callbacks). Read on the submitting thread to decide whether a submission is + * nested. Always cleared in a {@code finally} so pooled platform threads never + * carry the marker into unrelated work. + */ + private static final ThreadLocal EXECUTING_SUBMITTED_CALLABLE = new ThreadLocal<>(); + private boolean isInit = false; private final Logger log = Logger.getLogger(BaseRuntime.class); @@ -41,6 +77,8 @@ public BaseRuntime(@ConfigProperty(name = "systemRuntime.projectName") String pr t.setDaemon(true); return t; }); + this.nestedExecutorService = Executors.newThreadPerTaskExecutor( + Thread.ofVirtual().name("eddi-nested-", 0).factory()); init(); } @@ -48,6 +86,9 @@ public BaseRuntime(@ConfigProperty(name = "systemRuntime.projectName") String pr @PreDestroy void shutdown() { scheduledExecutorService.shutdownNow(); + // shutdown() (not shutdownNow()) so in-flight pipeline executions are given + // the chance to finish — the graceful-shutdown drain runs before this. + nestedExecutorService.shutdown(); } public void init() { @@ -102,35 +143,143 @@ public Future submitCallable(final Callable callable, final Map Future submitCallable(final Callable callable, final IFinishedExecution callback, final Map threadBindings) { - return getExecutorService().submit(() -> { + IFinishedExecution resolvedCallback = callback; + if (resolvedCallback == null) { + resolvedCallback = new IgnoredCallableResult<>(); + } + final IFinishedExecution completion = resolvedCallback; + + /* + * Per-submission abandonment token. The interrupt flag CANNOT be used as a + * completion guard: any intermediate catch block (or a bare + * Thread.interrupted() call) clears it, and the JDK does not guarantee a + * running body is interrupted at all. The token lives on the returned Future + * and can never be cleared by the work itself, so a turn the watchdog has + * already abandoned can never report success and persist its stale snapshot + * over a newer turn. + */ + final AtomicBoolean abandoned = new AtomicBoolean(false); + + /* + * One-shot callback gate: at most ONE of onComplete/onFailure may ever fire for + * a submission. Without it, an unchecked throw on the COMPLETION path + * re-entered onFailure — and callers that read onFailure as "the work did not + * run" (the coordinator's retry) then re-executed an already-executed turn, + * duplicating LLM calls, tool side effects and cost. + */ + final AtomicBoolean callbackFired = new AtomicBoolean(false); + + final ExecutorService target = Boolean.TRUE.equals(EXECUTING_SUBMITTED_CALLABLE.get()) + ? nestedExecutorService + : getExecutorService(); + + Future submitted = target.submit(() -> { try { - if (threadBindings != null) { - ThreadContext.setResources(threadBindings); + final T result; + try { + if (threadBindings != null) { + ThreadContext.setResources(threadBindings); + } + + EXECUTING_SUBMITTED_CALLABLE.set(Boolean.TRUE); + try { + result = callable.call(); + } finally { + // Cleared BEFORE the callbacks run: a callback that submits + // follow-up work (the coordinator scheduling the next turn) is + // not nested work and must keep using the ManagedExecutor. + EXECUTING_SUBMITTED_CALLABLE.remove(); + } + } catch (Throwable t) { + log.error(t.getLocalizedMessage(), t); + fireFailure(callbackFired, completion, t); + return null; } - final T result = callable.call(); - if (Thread.currentThread().isInterrupted()) { - // Execution was cancelled (e.g., agent timeout) but the callable - // completed anyway (non-interruptible I/O). Route to onFailure - // to skip stale persistence that would overwrite newer state. - // Return null to prevent leaking the stale result via the Future. + // Completion dispatch sits OUTSIDE the guarded region above, so a + // throwing onComplete can never be routed to onFailure. + if (abandoned.get() || Thread.currentThread().isInterrupted()) { + // Execution was abandoned (e.g. the agent-timeout watchdog cancelled + // it) but the callable completed anyway (non-interruptible I/O, or a + // swallowed interrupt). Route to onFailure to skip stale persistence + // that would overwrite newer state, and return null so the stale + // result cannot leak through the Future either. log.warnf("Execution completed after cancellation — discarding result to prevent stale persistence (thread=%s)", Thread.currentThread().getName()); - callback.onFailure(new InterruptedException( + fireFailure(callbackFired, completion, new InterruptedException( "Execution completed after cancellation — result discarded")); return null; - } else { - callback.onComplete(result); + } + + if (callbackFired.compareAndSet(false, true)) { + try { + completion.onComplete(result); + } catch (Throwable t) { + // Deliberately NOT routed to onFailure: the callable already ran. + log.error("Completion callback failed after the work had already executed — " + + "not reporting it as a failure to avoid re-execution", t); + } } return result; - } catch (Throwable t) { - log.error(t.getLocalizedMessage(), t); - callback.onFailure(t); - return null; } finally { ThreadContext.remove(); } }); + + return new AbandonableFuture<>(submitted, abandoned); + } + + private void fireFailure(AtomicBoolean callbackFired, IFinishedExecution completion, Throwable cause) { + if (callbackFired.compareAndSet(false, true)) { + try { + completion.onFailure(cause); + } catch (Throwable t) { + log.error("Failure callback threw — swallowing to keep the one-shot callback contract", t); + } + } + } + + /** + * Wraps the executor's Future so {@link #cancel(boolean)} marks the execution + * abandoned BEFORE delegating. Cancellation of an already-running body is + * advisory at best (the JDK's {@code mayInterruptIfRunning} is not honoured by + * every task, and pipelines routinely swallow interrupts), so the token — not + * the interrupt flag — is what suppresses the stale completion callback. + */ + private static final class AbandonableFuture implements Future { + private final Future delegate; + private final AtomicBoolean abandoned; + + private AbandonableFuture(Future delegate, AtomicBoolean abandoned) { + this.delegate = delegate; + this.abandoned = abandoned; + } + + @Override + public boolean cancel(boolean mayInterruptIfRunning) { + abandoned.set(true); + return delegate.cancel(mayInterruptIfRunning); + } + + @Override + public boolean isCancelled() { + return delegate.isCancelled(); + } + + @Override + public boolean isDone() { + return delegate.isDone(); + } + + @Override + public T get() throws InterruptedException, ExecutionException { + return delegate.get(); + } + + @Override + public T get(long timeout, TimeUnit unit) throws InterruptedException, ExecutionException, TimeoutException { + return delegate.get(timeout, unit); + } } private static class IgnoredCallableResult implements IFinishedExecution { diff --git a/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java b/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java index 84a13c70e..0e59db930 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java @@ -79,6 +79,14 @@ private IExecutableWorkflow createExecutableWorkflow(final DocumentDescriptor do throw new UnrecognizedExtensionException(String.format("Extension '%s' not found", type)); } + // The component key is the task's ABSOLUTE position in the + // workflow. LifecycleManager rebuilds the identical key when it + // looks the component up, including on a selective (sublist) + // execution, where it must add the sublist offset back on. + // Invariant this relies on: every workflow step uses the `eddi` + // scheme, so indexInWorkflow and the position in the + // lifecycleManager task list stay in lockstep. A non-eddi step + // would be skipped below and desynchronize the two. var componentKey = createComponentKey(workflowId.getId(), workflowId.getVersion(), indexInWorkflow); var lifecycleTask = lifecycleExtensionsProvider.get(type).get(); var component = lifecycleTask.configure(workflowStep.getConfig(), workflowStep.getExtensions()); diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/Conversation.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/Conversation.java index 34d1545c2..6ae745be6 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/Conversation.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/Conversation.java @@ -395,7 +395,13 @@ private void executeConversationStep(List> lifecycleData, List paused = true; } } - if (!paused) { + // A cancelled turn must not commit its side effects. ConversationService + // discards the snapshot of a cancelled turn, but storePropertiesPermanently() + // writes straight to the user-memory store, so without this guard a turn the + // caller was told is CANCELLED still upserts whatever longTerm properties it + // managed to set before stopping — the same "a failed turn must not persist + // partial state" rule the ERROR path already follows. + if (!paused && !conversationMemory.isCancelled()) { try { postConversationLifecycleTasks(); } catch (IResourceStore.ResourceStoreException e) { @@ -835,11 +841,13 @@ public void resume(HitlDecision decision) clearToolPauseState(); } // Persist long-term properties only on a clean outcome. Skip on a - // re-pause (AWAITING_HUMAN — the pause is not the end of the turn) and - // on ERROR — mirroring the say path (executeConversationStep only runs - // post-tasks when execution did not throw), so a failed resume does not - // upsert partial/inconsistent property state into the user memory store. - if (finalState != ConversationState.AWAITING_HUMAN && finalState != ConversationState.ERROR) { + // re-pause (AWAITING_HUMAN — the pause is not the end of the turn), on + // ERROR, and on a cancel — mirroring the say path (executeConversationStep + // only runs post-tasks when execution did not throw and was not + // cancelled), so a failed or cancelled resume does not upsert + // partial/inconsistent property state into the user memory store. + if (finalState != ConversationState.AWAITING_HUMAN && finalState != ConversationState.ERROR + && !conversationMemory.isCancelled()) { try { postConversationLifecycleTasks(); } catch (IResourceStore.ResourceStoreException ex) { diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java index 0a4e9682f..237b3eedf 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java @@ -4,6 +4,7 @@ */ package ai.labs.eddi.engine.runtime.internal; +import ai.labs.eddi.configs.agents.IAgentStore; import ai.labs.eddi.configs.agents.model.AgentConfiguration; import ai.labs.eddi.configs.properties.IUserMemoryStore; import ai.labs.eddi.configs.properties.model.Property.Visibility; @@ -41,11 +42,19 @@ *

* This service operates per-user and is invoked by the schedule system when an * agent has {@code dream.enabled=true} in its - * {@link AgentConfiguration.UserMemoryConfig}. + * {@link AgentConfiguration.UserMemoryConfig}. Wiring is the cluster-aware + * schedule machinery AGENTS.md prescribes — never a private scheduler: a + * {@code ScheduleConfiguration} carrying {@link #METADATA_TYPE_KEY} = + * {@link #METADATA_TYPE_CONSOLIDATION} in its metadata, plus the target + * {@code agentId} and {@code userId}, is claimed by + * {@code SchedulePollerService} and dispatched by {@code ScheduleFireExecutor} + * to {@link #processScheduledFire}. * *

- * Cost ceiling: {@code maxSummarizationCalls} bounds LLM calls per user per - * cycle. Round-robin: processes users ordered by oldest {@code updatedAt}. + * Cost ceiling: {@code maxCostPerRun} (US dollars, estimated from token usage) + * bounds the spend per user per cycle. The former {@code maxSummarizationCalls} + * count is deprecated and no longer enforced — different consolidations cost + * vastly different amounts, so a call count is not a budget. * * @author ginccc * @since 6.0.0 @@ -55,6 +64,22 @@ public class DreamService { private static final Logger LOGGER = Logger.getLogger(DreamService.class); + /** + * Metadata key marking a schedule as Dream-managed. Single source of truth for + * the contract between schedule authors (REST/MCP {@code create_schedule}) and + * the dispatcher ({@code ScheduleFireExecutor}). + */ + public static final String METADATA_TYPE_KEY = "dreamType"; + /** Metadata value for memory-consolidation schedules. */ + public static final String METADATA_TYPE_CONSOLIDATION = "dream_consolidation"; + + /** + * Placeholder identity the schedule surface assigns when no {@code userId} is + * supplied. Dream must never run under it: it is not a real user, so every + * cycle would silently consolidate an empty memory set. + */ + static final String SCHEDULER_PLACEHOLDER_USER_ID = "system:scheduler"; + /** * Max key length for consolidated entries (matches UserMemoryConfig.Guardrails * default). @@ -66,6 +91,7 @@ public class DreamService { */ static final int MAX_VALUE_LENGTH = 1000; private final IUserMemoryStore userMemoryStore; + private final IAgentStore agentStore; private final SummarizationService summarizationService; private final MeterRegistry meterRegistry; private final ObjectMapper objectMapper; @@ -74,14 +100,18 @@ public class DreamService { private Counter entriesPrunedCounter; private Counter contradictionsFoundCounter; private Counter entriesSummarizedCounter; + private Counter cyclesFailedCounter; + private Counter summarizationFailedCounter; private Timer dreamDurationTimer; @Inject public DreamService(IUserMemoryStore userMemoryStore, + IAgentStore agentStore, SummarizationService summarizationService, MeterRegistry meterRegistry, ObjectMapper objectMapper) { this.userMemoryStore = userMemoryStore; + this.agentStore = agentStore; this.summarizationService = summarizationService; this.meterRegistry = meterRegistry; this.objectMapper = objectMapper; @@ -93,12 +123,81 @@ void initMetrics() { entriesPrunedCounter = meterRegistry.counter("dream.entries.pruned"); contradictionsFoundCounter = meterRegistry.counter("dream.contradictions.found"); entriesSummarizedCounter = meterRegistry.counter("dream.entries.summarized"); + cyclesFailedCounter = meterRegistry.counter("dream.cycles.failed"); + summarizationFailedCounter = meterRegistry.counter("dream.summarization.failed"); dreamDurationTimer = meterRegistry.timer("dream.duration"); } /** - * Process dream consolidation for a specific user's memories. Called by the - * schedule system when a SERVICE-type schedule fires. + * True if the given schedule metadata marks a Dream consolidation schedule. + */ + public static boolean isDreamSchedule(Map metadata) { + return metadata != null && METADATA_TYPE_CONSOLIDATION.equals(metadata.get(METADATA_TYPE_KEY)); + } + + /** + * Entry point for a fired Dream schedule. Resolves the agent's + * {@link AgentConfiguration.DreamConfig} and runs one consolidation cycle for + * the scheduled user. + *

+ * Every rejection here is logged at ERROR and returned as a failed + * {@link DreamResult} so the caller marks the fire FAILED — a misconfigured + * Dream schedule must be visible in the fire log and dead-letter after retries, + * never degrade into a silent no-op. + * + * @param agentId + * the agent whose {@code userMemoryConfig.dream} block configures + * this cycle + * @param agentVersion + * the pinned agent version, or {@code null}/{@code <= 0} for latest + * @param userId + * the user whose memories to consolidate — required + */ + public DreamResult processScheduledFire(String agentId, Integer agentVersion, String userId) { + Instant start = Instant.now(); + + if (agentId == null || agentId.isBlank()) { + return rejected(userId, start, "Dream schedule has no agentId — cannot resolve a dream configuration."); + } + if (userId == null || userId.isBlank() || SCHEDULER_PLACEHOLDER_USER_ID.equals(userId)) { + return rejected(userId, start, "Dream schedule for agent '" + agentId + "' has no real userId (got '" + userId + + "'). Set 'userId' on the schedule to the user whose memories should be consolidated."); + } + + AgentConfiguration agentConfiguration; + try { + int version = agentVersion != null && agentVersion > 0 + ? agentVersion + : agentStore.getCurrentResourceId(agentId).getVersion(); + agentConfiguration = agentStore.read(agentId, version); + } catch (Exception e) { + LOGGER.errorf(e, "[DREAM] Could not read agent '%s' (version=%s) for a scheduled dream cycle", agentId, agentVersion); + return rejected(userId, start, "Could not read agent '" + agentId + "': " + e.getMessage()); + } + + if (agentConfiguration == null) { + return rejected(userId, start, "Agent '" + agentId + "' not found — cannot run dream consolidation."); + } + + var memoryConfig = agentConfiguration.getUserMemoryConfig(); + var dreamConfig = memoryConfig != null ? memoryConfig.getDream() : null; + if (dreamConfig == null || !dreamConfig.isEnabled()) { + return rejected(userId, start, "Agent '" + agentId + "' has dream consolidation disabled " + + "(userMemoryConfig.dream.enabled=false or absent), but a dream schedule fired for it."); + } + + return process(userId, dreamConfig); + } + + private DreamResult rejected(String userId, Instant start, String reason) { + LOGGER.errorf("[DREAM] %s", reason); + cyclesFailedCounter.increment(); + return new DreamResult(userId, 0, 0, 0, Duration.between(start, Instant.now()).toMillis(), 0.0, reason); + } + + /** + * Process dream consolidation for a specific user's memories. Called by + * {@link #processScheduledFire} when a Dream schedule fires. * * @param userId * the user whose memories to consolidate @@ -111,6 +210,8 @@ public DreamResult process(String userId, AgentConfiguration.DreamConfig dreamCo int pruned = 0; int contradictions = 0; int summarized = 0; + double estimatedCost = 0.0; + String summarizationError = null; try { LOGGER.infof("[DREAM] Starting dream cycle for user='%s'", userId); @@ -136,21 +237,33 @@ public DreamResult process(String userId, AgentConfiguration.DreamConfig dreamCo // 3. Summarize interactions (LLM-driven consolidation) if (dreamConfig.isSummarizeInteractions()) { - summarized = summarizeInteractions(userId, currentEntries, dreamConfig); + var outcome = summarizeInteractions(userId, currentEntries, dreamConfig); + summarized = outcome.entriesReduced(); + estimatedCost = outcome.estimatedCostUsd(); + summarizationError = outcome.error(); } usersProcessedCounter.increment(); var duration = Duration.between(start, Instant.now()); dreamDurationTimer.record(duration); - LOGGER.infof("[DREAM] Completed for user='%s': pruned=%d, contradictions=%d, summarized=%d, duration=%dms", userId, pruned, - contradictions, summarized, duration.toMillis()); + if (summarizationError != null) { + cyclesFailedCounter.increment(); + LOGGER.errorf("[DREAM] Completed WITH ERRORS for user='%s': pruned=%d, contradictions=%d, summarized=%d, " + + "estimatedCost=$%.4f, duration=%dms, error=%s", userId, pruned, contradictions, summarized, estimatedCost, + duration.toMillis(), summarizationError); + } else { + LOGGER.infof("[DREAM] Completed for user='%s': pruned=%d, contradictions=%d, summarized=%d, " + + "estimatedCost=$%.4f, duration=%dms", userId, pruned, contradictions, summarized, estimatedCost, duration.toMillis()); + } - return new DreamResult(userId, pruned, contradictions, summarized, duration.toMillis(), null); + return new DreamResult(userId, pruned, contradictions, summarized, duration.toMillis(), estimatedCost, summarizationError); } catch (Exception e) { + cyclesFailedCounter.increment(); LOGGER.errorf(e, "[DREAM] Failed for user='%s'", userId); - return new DreamResult(userId, pruned, contradictions, summarized, Duration.between(start, Instant.now()).toMillis(), e.getMessage()); + return new DreamResult(userId, pruned, contradictions, summarized, Duration.between(start, Instant.now()).toMillis(), estimatedCost, + e.getMessage()); } } @@ -206,6 +319,21 @@ private int detectContradictions(String userId, List allEntries return contradictions; } + /** + * Outcome of the summarization phase. + * + * @param entriesReduced + * net number of entries removed by consolidation + * @param estimatedCostUsd + * estimated dollar spend of this phase + * @param error + * {@code null} on success; a human-readable cause when the phase was + * aborted, so the caller can fail the schedule fire instead of + * reporting a silent no-op + */ + record SummarizationOutcome(int entriesReduced, double estimatedCostUsd, String error) { + } + /** * Summarize related interactions using LLM-driven consolidation. Groups entries * by the configured strategy, calls the LLM to distill each group, and @@ -218,13 +346,13 @@ private int detectContradictions(String userId, List allEntries *

  • If insert fails, originals are preserved
    • *
  • If LLM returns empty/garbage, the group is skipped
    • *
  • If LLM returns more entries than input, the group is skipped
    • - *
  • Call count bounded by {@code maxSummarizationCalls}
    • *
  • Cost bounded by {@code maxCostPerRun} (estimated from token usage)
    • + *
  • An LLM failure aborts the phase and is reported, never swallowed
    • * */ - private int summarizeInteractions(String userId, - List entries, - AgentConfiguration.DreamConfig config) { + private SummarizationOutcome summarizeInteractions(String userId, + List entries, + AgentConfiguration.DreamConfig config) { int totalConsolidated = 0; int llmCallsMade = 0; double estimatedCostAccumulated = 0.0; @@ -240,16 +368,11 @@ private int summarizeInteractions(String userId, continue; } - // Respect call limit - if (llmCallsMade >= config.getMaxSummarizationCalls()) { - LOGGER.infof("[DREAM] Summarization call limit (%d) reached for user='%s'", - config.getMaxSummarizationCalls(), userId); - break; - } - // Respect cost ceiling (soft cap: checked before each call, so the // last call may push total slightly over — this is by design, since - // we cannot know output cost before the call) + // we cannot know output cost before the call). This dollar budget is + // the ONLY ceiling; the legacy maxSummarizationCalls count is not + // enforced because a call count says nothing about spend. if (estimatedCostAccumulated >= config.getMaxCostPerRun()) { LOGGER.infof("[DREAM] Cost ceiling ($%.4f >= $%.2f) reached for user='%s' " + "after %d calls", estimatedCostAccumulated, config.getMaxCostPerRun(), userId, llmCallsMade); @@ -259,17 +382,30 @@ private int summarizeInteractions(String userId, // 2. Build content: JSON array of entries String content = buildEntriesJson(groupEntries); - // 3. Call LLM (isolated — failure skips this group only) + // 3. Call LLM. Dream is a background job with no parent LLM task to + // inherit credentials from, so the model parameters come from the + // agent's dream config (finding I1/F13). A failure here is almost + // always a configuration fault that would repeat for every remaining + // group — abort the phase, log at ERROR and report it upward so the + // schedule fire is marked FAILED, instead of leaving Dream to look + // like it ran and simply found nothing to do. SummarizationService.SummarizationResult llmResult; try { llmResult = summarizationService.summarizeWithUsage( content, config.getSummarizationPrompt(), - config.getLlmProvider(), config.getLlmModel()); + config.getLlmProvider(), config.getLlmModel(), + config.getParameters()); } catch (Exception e) { - LOGGER.warnf("[DREAM] LLM call failed for user='%s', group='%s': %s. " + - "Preserving original entries.", userId, group.getKey(), e.getMessage()); - llmCallsMade++; - continue; + summarizationFailedCounter.increment(); + LOGGER.errorf(e, "[DREAM] Memory consolidation LLM call failed for user='%s', group='%s' " + + "(provider=%s, model=%s, configured parameter keys=%s). Original entries are preserved and " + + "consolidation is ABORTED for this cycle. If this is an authentication or endpoint error, set the " + + "credentials on the agent under userMemoryConfig.dream.parameters (e.g. \"apiKey\": \"${vault:my-key}\") " + + "— a background dream cycle has no parent LLM task to inherit them from.", + userId, group.getKey(), config.getLlmProvider(), config.getLlmModel(), parameterKeys(config)); + return new SummarizationOutcome(totalConsolidated, estimatedCostAccumulated, + "Memory consolidation LLM call failed (" + config.getLlmProvider() + "/" + config.getLlmModel() + "): " + + e.getMessage()); } llmCallsMade++; estimatedCostAccumulated += estimateCost(llmResult, content.length()); @@ -297,9 +433,11 @@ private int summarizeInteractions(String userId, } // 7. SAFETY: Insert new entries FIRST - // Derive provenance from the group — when entries span multiple - // agents, upgrade self-scoped visibility to global so the - // consolidated entry remains reachable by all contributing agents. + // Derive provenance from the group. Visibility is the most restrictive + // of the originals and is NEVER widened (finding G8): a self-scoped + // memory belongs to exactly one agent, so it may only ever be merged + // with entries of that same agent — buildGroups guarantees that by + // splitting self-scoped groups per sourceAgentId. List insertedIds = new ArrayList<>(); try { Visibility mergedVisibility = mostRestrictiveVisibility(groupEntries); @@ -310,10 +448,14 @@ private int summarizeInteractions(String userId, String sourceAgent = distinctAgents.size() == 1 ? distinctAgents.iterator().next() : groupEntries.getFirst().sourceAgentId(); - // If entries from multiple agents are merged AND visibility is - // self-scoped, upgrade to global so no agent loses its memories + // Defence in depth: should a future grouping change ever hand us a + // self-scoped group spanning several agents, skip it rather than + // widen it — a privacy boundary must fail closed. if (distinctAgents.size() > 1 && mergedVisibility == Visibility.self) { - mergedVisibility = Visibility.global; + LOGGER.errorf("[DREAM] Refusing to merge self-scoped entries from %d agents for user='%s', " + + "group='%s' — that would expose one agent's private memories to the others.", + distinctAgents.size(), userId, group.getKey()); + continue; } Instant earliestCreated = groupEntries.stream() .map(UserMemoryEntry::createdAt) @@ -373,7 +515,16 @@ private int summarizeInteractions(String userId, } } - return totalConsolidated; + return new SummarizationOutcome(totalConsolidated, estimatedCostAccumulated, null); + } + + /** + * The configured model parameter keys — never the values, which hold + * credentials. Used to make a failure diagnosable without leaking secrets. + */ + private static String parameterKeys(AgentConfiguration.DreamConfig config) { + var parameters = config.getParameters(); + return parameters == null || parameters.isEmpty() ? "" : parameters.keySet().toString(); } /** @@ -404,6 +555,10 @@ static double estimateCost(SummarizationService.SummarizationResult result, /** * Build entry groups according to the configured grouping strategy. + *

    + * Whatever the strategy, the result is post-processed by + * {@link #splitSelfScopedGroupsByAgent} so a {@code self}-scoped memory is + * never merged with another agent's memories. */ private Map> buildGroups( List entries, @@ -411,7 +566,7 @@ private Map> buildGroups( if ("all".equals(config.getSummarizeGroupBy())) { // Single group - return Map.of("all", new ArrayList<>(entries)); + return splitSelfScopedGroupsByAgent(Map.of("all", new ArrayList<>(entries))); } // Default: group by category (null-safe — legacy entries may lack category) @@ -420,7 +575,7 @@ private Map> buildGroups( e -> e.category() != null ? e.category() : "fact")); if (!config.isPreserveAgentProvenance()) { - return byCategory; + return splitSelfScopedGroupsByAgent(byCategory); } // Sub-group by agent within each category @@ -430,6 +585,47 @@ private Map> buildGroups( .collect(Collectors.groupingBy(e -> e.sourceAgentId() != null ? e.sourceAgentId() : "unknown")) .forEach((agentId, agentEntries) -> result.put(catGroup.getKey() + ":" + agentId, agentEntries)); } + return splitSelfScopedGroupsByAgent(result); + } + + /** + * Privacy boundary (finding G8): a {@code self}-scoped memory is readable only + * by the agent that wrote it. Consolidating one into a shared entry would + * either lose it for its owner or — as the previous implementation did by + * upgrading the merged visibility to {@code global} — expose it to every other + * agent. Since {@code summarizeGroupBy} defaults to {@code "category"} and + * {@code preserveAgentProvenance} defaults to {@code false}, cross-agent + * grouping was the default path, so that widening was the default behaviour. + *

    + * Any group whose most restrictive visibility is {@code self} is therefore + * split per {@code sourceAgentId}, producing one consolidated entry per + * contributing agent, each keeping {@code self} and its own provenance. Groups + * that are already {@code group}- or {@code global}-scoped are left untouched — + * they were shared to begin with. Entries without a {@code sourceAgentId} are + * kept in their own bucket rather than folded into an arbitrary agent's. + */ + private static Map> splitSelfScopedGroupsByAgent( + Map> groups) { + Map> result = new LinkedHashMap<>(); + for (var group : groups.entrySet()) { + List groupEntries = group.getValue(); + if (groupEntries.isEmpty() || mostRestrictiveVisibility(groupEntries) != Visibility.self) { + result.put(group.getKey(), groupEntries); + continue; + } + + Map> byAgent = groupEntries.stream() + .collect(Collectors.groupingBy(e -> e.sourceAgentId() != null ? e.sourceAgentId() : "unknown", + LinkedHashMap::new, Collectors.toList())); + if (byAgent.size() <= 1) { + result.put(group.getKey(), groupEntries); + continue; + } + + LOGGER.infof("[DREAM] Group '%s' holds self-scoped memories from %d agents — consolidating each agent " + + "separately so no private memory is widened.", group.getKey(), byAgent.size()); + byAgent.forEach((agentId, agentEntries) -> result.put(group.getKey() + ":" + agentId, agentEntries)); + } return result; } @@ -544,8 +740,16 @@ static String truncate(String text, int maxLength) { /** * Result of a dream consolidation cycle. + * + * @param estimatedCostUsd + * estimated LLM spend of this cycle — reported on the schedule fire + * log so the dollar budget is observable + * @param error + * {@code null} on success; otherwise the cause, which the schedule + * dispatcher turns into a FAILED fire */ - public record DreamResult(String userId, int entriesPruned, int contradictionsFound, int entriesSummarized, long durationMs, String error) { + public record DreamResult(String userId, int entriesPruned, int contradictionsFound, int entriesSummarized, long durationMs, + double estimatedCostUsd, String error) { public boolean isSuccess() { return error == null; diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java new file mode 100644 index 000000000..f609794c4 --- /dev/null +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java @@ -0,0 +1,188 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.engine.runtime.internal; + +import ai.labs.eddi.engine.runtime.IConversationCoordinator; +import io.quarkus.runtime.ShutdownEvent; +import org.eclipse.microprofile.config.inject.ConfigProperty; +import org.jboss.logging.Logger; + +import jakarta.enterprise.context.ApplicationScoped; +import jakarta.enterprise.event.Observes; +import jakarta.inject.Inject; +import java.util.Map; + +/** + * Graceful shutdown for conversation processing (B3). + * + *

    + * Before this existed there was not a single {@code ShutdownEvent} observer in + * the engine: a rolling deploy simply killed the JVM, dropping every queued and + * in-flight turn with no drain and no readiness flip. Callers saw truncated + * responses and conversations were left in {@code IN_PROGRESS} for the crash + * recovery observer to clean up on the next boot. + *

    + * + *

    + * On {@link ShutdownEvent} this observer, in order: + *

    + *
      + *
    1. flips readiness — {@link ShutdownReadinessHealthCheck} is a + * MicroProfile {@code @Readiness} check, so {@code /q/health/ready} reports + * DOWN and the load balancer / Kubernetes endpoint controller stops routing new + * traffic to this pod. It joins the existing readiness set (alongside the agent + * readiness check) rather than inventing a parallel mechanism;
      2. + *
    2. stops accepting new turns — {@link #isShuttingDown()} is consulted + * by {@code ConversationService} on the start/say/sayStreaming entry points, + * which reject with {@code RejectedExecutionException} once it flips;
      3. + *
    3. drains the coordinator queues with a BOUNDED wait, so turns + * already queued or executing get the chance to finish and persist.
      4. + *
    + * + *

    + * The drain observes {@link IConversationCoordinator#getQueueDepths()}, which + * counts both the queued turns and the head that is currently executing, so an + * empty depth map means "nothing left in flight on this node". The wait is hard + * bounded: a pipeline that hangs must never prevent the process from exiting + * (the orchestrator's SIGKILL grace period is finite anyway). + *

    + * + * @author ginccc + */ +@ApplicationScoped +public class GracefulShutdownService { + + private static final Logger LOGGER = Logger.getLogger(GracefulShutdownService.class); + + private final IConversationCoordinator conversationCoordinator; + + /** + * How long to keep draining before giving up. Should be comfortably below the + * orchestrator's termination grace period. + */ + private final long drainTimeoutMillis; + + /** + * Quiet period between flipping readiness and starting the drain, giving the + * load balancer time to observe the DOWN state and stop sending new requests + * that would otherwise arrive during the drain and be rejected. + */ + private final long readinessGraceMillis; + + /** Poll interval for the drain loop. */ + private final long pollIntervalMillis; + + /** + * Volatile, not atomic: single writer (the shutdown observer), many readers. + */ + private volatile boolean shuttingDown; + + @Inject + public GracefulShutdownService(IConversationCoordinator conversationCoordinator, + @ConfigProperty(name = "eddi.shutdown.drain-timeout-seconds", defaultValue = "30") int drainTimeoutSeconds, + @ConfigProperty(name = "eddi.shutdown.readiness-grace-seconds", defaultValue = "3") int readinessGraceSeconds, + @ConfigProperty(name = "eddi.shutdown.drain-poll-millis", defaultValue = "100") long pollIntervalMillis) { + this(conversationCoordinator, drainTimeoutSeconds * 1000L, readinessGraceSeconds * 1000L, pollIntervalMillis); + } + + /** Millisecond-precision constructor — used by the tests. */ + GracefulShutdownService(IConversationCoordinator conversationCoordinator, long drainTimeoutMillis, + long readinessGraceMillis, long pollIntervalMillis) { + this.conversationCoordinator = conversationCoordinator; + this.drainTimeoutMillis = Math.max(0, drainTimeoutMillis); + this.readinessGraceMillis = Math.max(0, readinessGraceMillis); + this.pollIntervalMillis = Math.max(1, pollIntervalMillis); + } + + /** + * @return {@code true} once a {@link ShutdownEvent} has been observed. New + * conversation turns must be refused from this point on. + */ + public boolean isShuttingDown() { + return shuttingDown; + } + + void onShutdown(@Observes ShutdownEvent shutdownEvent) { + drain(); + } + + /** + * Flips readiness + the accept gate, then waits (bounded) for the coordinator + * to drain. Package-private so the tests can drive it without a CDI container. + * + * @return {@code true} if everything drained within the timeout + */ + boolean drain() { + // Steps 1 and 2 are the SAME flag flip: readiness reports DOWN and the + // conversation entry points start rejecting. Do it first and unconditionally + // — even if the drain below fails, no new work is admitted. + shuttingDown = true; + LOGGER.info("Shutdown signalled — readiness is now DOWN and new conversation turns are rejected"); + + sleepQuietly(readinessGraceMillis); + + long deadline = System.nanoTime() + drainTimeoutMillis * 1_000_000L; + int inFlight = countInFlight(); + if (inFlight == 0) { + LOGGER.info("Nothing in flight — shutdown drain completed immediately"); + return true; + } + + LOGGER.infof("Draining %d in-flight conversation task(s), waiting up to %d ms", inFlight, drainTimeoutMillis); + while (inFlight > 0 && System.nanoTime() < deadline) { + if (!sleepQuietly(pollIntervalMillis)) { + break; + } + inFlight = countInFlight(); + } + + if (inFlight > 0) { + LOGGER.warnf("Shutdown drain timed out after %d ms with %d conversation task(s) still in flight — " + + "they will be abandoned; crash recovery reconciles their state on the next boot", + drainTimeoutMillis, inFlight); + return false; + } + + LOGGER.info("Shutdown drain completed — all conversation tasks finished"); + return true; + } + + private int countInFlight() { + try { + Map depths = conversationCoordinator.getQueueDepths(); + if (depths == null || depths.isEmpty()) { + return 0; + } + int total = 0; + for (Integer depth : depths.values()) { + if (depth != null) { + total += depth; + } + } + return total; + } catch (RuntimeException e) { + // A coordinator that cannot report its depth must not block the exit. + LOGGER.warnf("Could not read coordinator queue depths during shutdown drain: %s", e.getMessage()); + return 0; + } + } + + /** + * @return {@code false} if the wait was interrupted (the caller should stop + * waiting — the interrupt flag is restored) + */ + private static boolean sleepQuietly(long millis) { + if (millis <= 0) { + return true; + } + try { + Thread.sleep(millis); + return true; + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + return false; + } + } +} diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinator.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinator.java index 9abfed6b7..8a3e3758a 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinator.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinator.java @@ -56,6 +56,17 @@ * {@code eddi.coordinator.total_processed}. * * + *

    Failure handling

    + *
      + *
    • No retry after execution starts: a task that reports failure has + * already run — possibly calling an LLM, executing tools and spending money. It + * is dead-lettered once, never re-executed.
      • + *
    • Submission rejection rolls back: if handing the task to the + * runtime throws, the task is taken back off the queue (and the map entry + * dropped when it was the head), so a rejected submission cannot wedge the + * conversation.
      • + *
    + * * @author ginccc * @see ai.labs.eddi.engine.runtime.IEventBus */ @@ -63,8 +74,6 @@ @DefaultBean public class InMemoryConversationCoordinator implements IConversationCoordinator { - private static final int MAX_RETRIES = 3; - private final Map>> conversationQueues = new ConcurrentHashMap<>(); private final ConcurrentLinkedDeque deadLetters = new ConcurrentLinkedDeque<>(); /** Serializes dead-letter add+trim so the cap is enforced deterministically. */ @@ -159,14 +168,45 @@ public void submitInOrder(String conversationId, Callable callable) { } if (wasEmpty) { - executeWithRetry(conversationId, queue, callable, 0); + try { + execute(conversationId, queue, callable); + } catch (RuntimeException | Error e) { + // C10: the submission failed, so NOTHING is scheduled to run + // the head of this queue — and submitNext() only ever runs + // from a completion callback. Leaving the callable queued + // would wedge this conversation permanently (every later turn + // sees a non-empty queue and just waits) and leak the map + // entry for the JVM's lifetime. Undo the enqueue and drop the + // now-empty queue so the next turn starts a fresh one. + // + // We still hold the queue monitor, so nothing can have been + // offered in between: our callable is the only element. + queue.remove(callable); + if (queue.isEmpty()) { + conversationQueues.remove(conversationId, queue); + } + log.warnf("Submission failed for conversationId=%s — rolled the task back off the queue " + + "so the conversation stays usable", safeConversationId); + throw e; + } } return; // success } } } - private void executeWithRetry(String conversationId, BlockingQueue> queue, Callable callable, int attempt) { + /** + * Hands a task to the runtime. Throws (synchronously) if the SUBMISSION itself + * is rejected — the only genuinely pre-execution failure mode; callers must + * un-queue the task in that case (C10). + *

    + * C13: there is deliberately NO retry on {@code onFailure}. That callback is + * only ever raised from INSIDE the executor task, i.e. after the turn has + * already started running — it may have called an LLM, executed tools, written + * memory and spent money. Re-running the very same callable repeats all of it. + * A failed turn is dead-lettered once and the queue moves on. + */ + private void execute(String conversationId, BlockingQueue> queue, Callable callable) { runtime.submitCallable(callable, new IRuntime.IFinishedExecution<>() { @Override public void onComplete(Void result) { @@ -176,18 +216,12 @@ public void onComplete(Void result) { @Override public void onFailure(Throwable t) { - int nextAttempt = attempt + 1; - if (nextAttempt < MAX_RETRIES) { - log.warnf(t, "In-memory task failed (conversationId=%s, attempt=%d/%d), retrying...", sanitize(conversationId), nextAttempt, - MAX_RETRIES); - executeWithRetry(conversationId, queue, callable, nextAttempt); - } else { - log.errorf(t, "In-memory task exhausted retries (conversationId=%s, attempts=%d), dead-lettering", sanitize(conversationId), - nextAttempt); - routeToDeadLetter(conversationId, t); - totalProcessed.incrementAndGet(); - submitNext(conversationId, queue); - } + log.errorf(t, "In-memory task failed after it had already started (conversationId=%s) — dead-lettering " + + "without retry; re-running it would repeat any side effects it already performed", + sanitize(conversationId)); + routeToDeadLetter(conversationId, t); + totalProcessed.incrementAndGet(); + submitNext(conversationId, queue); } }, null); } @@ -220,18 +254,33 @@ private void routeToDeadLetter(String conversationId, Throwable failure) { private void submitNext(String conversationId, BlockingQueue> queue) { synchronized (queue) { - if (!queue.isEmpty()) { - queue.remove(); - - if (!queue.isEmpty()) { - executeWithRetry(conversationId, queue, queue.element(), 0); - } else { - // Eager cleanup: remove empty queue to prevent memory leaks. - // Uses remove(key, value) to avoid removing a new queue that was - // just created by a concurrent submitInOrder call. - conversationQueues.remove(conversationId, queue); + if (queue.isEmpty()) { + return; + } + queue.remove(); // drop the task that just finished + + while (!queue.isEmpty()) { + try { + execute(conversationId, queue, queue.element()); + return; + } catch (RuntimeException | Error e) { + // C10 (submitNext side): there is no caller to propagate to here — + // this runs from a completion callback. Dropping out would leave + // the queue non-empty with nothing scheduled to drain it, wedging + // the conversation forever. Dead-letter the task we could not + // schedule and try the next one. + log.errorf(e, "Failed to schedule the next queued task (conversationId=%s) — dead-lettering it " + + "so the conversation queue keeps draining", sanitize(conversationId)); + routeToDeadLetter(conversationId, e); + totalProcessed.incrementAndGet(); + queue.remove(); } } + + // Eager cleanup: remove empty queue to prevent memory leaks. + // Uses remove(key, value) to avoid removing a new queue that was + // just created by a concurrent submitInOrder call. + conversationQueues.remove(conversationId, queue); } } diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java index 2498c9d47..8a84e081e 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java @@ -370,7 +370,12 @@ public void shutdown() { log.info("NATS connection closed"); } catch (InterruptedException | TimeoutException e) { log.warnf(e, "Error during NATS shutdown"); - Thread.currentThread().interrupt(); + // B2: only an actual interrupt may set the flag. A drain TimeoutException + // is not an interrupt — flagging the shutdown thread there would abort the + // remaining @PreDestroy steps that run after this one. + if (e instanceof InterruptedException) { + Thread.currentThread().interrupt(); + } } } } diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java index f9118e75b..8d8d003b7 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java @@ -25,6 +25,17 @@ * Executes a scheduled fire by resolving the conversation strategy and calling * {@link IConversationService#say}. *

    + * Two schedule kinds are recognised by their metadata and bypass the + * conversation path entirely, because neither is a conversation turn: + *

      + *
    • {@code hitlType=hitl_timeout} — HITL approval deadlines + * ({@code HitlTimeoutHandler})
      • + *
    • {@code dreamType=dream_consolidation} — background user-memory + * maintenance ({@link DreamService})
      • + *
    + * Both still run under the poller's cluster-wide claim, lease, retry/backoff + * and fire-log machinery. + *

    * All existing guards apply automatically: *

      *
    • {@code TenantQuotaService} — API call and cost quotas
      • @@ -49,6 +60,9 @@ public class ScheduleFireExecutor { @Inject ai.labs.eddi.engine.internal.HitlTimeoutHandler hitlTimeoutHandler; + @Inject + DreamService dreamService; + /** * Execute a schedule fire. Returns the fire log entry. * @@ -89,6 +103,15 @@ public ScheduleFireLog fire(ScheduleConfiguration schedule, String instanceId, i return hitlFireLog; } + if (DreamService.isDreamSchedule(md)) { + // Dream consolidation fast-path — a maintenance job over the user's + // persistent memories, not a conversation turn, so it never goes + // through say(). Everything else the schedule machinery provides + // (cluster-wide CAS claim, lease, retry/backoff, dead-lettering, fire + // log) applies unchanged. + return fireDreamConsolidation(schedule, instanceId, attemptNumber); + } + Instant startedAt = Instant.now(); String fireLogId = UUID.randomUUID().toString(); String conversationId = null; @@ -125,6 +148,14 @@ public ScheduleFireLog fire(ScheduleConfiguration schedule, String instanceId, i schedule.getTriggerType(), schedule.getAgentId(), conversationId); } catch (Exception e) { + // B2: latch.await() above CLEARS the interrupt flag when it throws + // InterruptedException, and this broad catch would otherwise swallow the + // poller thread's shutdown signal — it would keep firing further schedules + // while the executor is shutting down. Restore it before continuing; the + // fire is still logged FAILED below so the attempt stays visible. + if (e instanceof InterruptedException) { + Thread.currentThread().interrupt(); + } status = ScheduleConfiguration.FireStatus.FAILED.name(); errorMessage = e.getClass().getSimpleName() + ": " + e.getMessage(); LOGGER.warnf(e, "[SCHEDULE] Fire failed for schedule '%s' (id=%s): %s", schedule.getName(), schedule.getId(), errorMessage); @@ -143,6 +174,55 @@ public ScheduleFireLog fire(ScheduleConfiguration schedule, String instanceId, i return fireLog; } + /** + * Run one Dream memory-consolidation cycle for the schedule's user and record + * the fire. + *

      + * A rejected or failed cycle is logged FAILED rather than swallowed, so a + * misconfigured dream schedule surfaces in the fire log, retries with backoff + * and eventually dead-letters instead of appearing to run while doing nothing. + */ + private ScheduleFireLog fireDreamConsolidation(ScheduleConfiguration schedule, String instanceId, int attemptNumber) { + Instant startedAt = Instant.now(); + String status; + String errorMessage = null; + double cost = 0.0; + + try { + // userId is passed through unchanged — DreamService rejects a missing or + // placeholder identity loudly; defaulting it here would silently + // consolidate an empty memory set. + DreamService.DreamResult result = dreamService.processScheduledFire( + schedule.getAgentId(), schedule.getAgentVersion(), schedule.getUserId()); + cost = result.estimatedCostUsd(); + if (result.isSuccess()) { + status = ScheduleConfiguration.FireStatus.COMPLETED.name(); + LOGGER.infof("[SCHEDULE] Dream consolidation for schedule '%s' (id=%s, agent=%s, user=%s): " + + "pruned=%d, contradictions=%d, summarized=%d, estimatedCost=$%.4f", schedule.getName(), schedule.getId(), + schedule.getAgentId(), schedule.getUserId(), result.entriesPruned(), result.contradictionsFound(), + result.entriesSummarized(), cost); + } else { + status = ScheduleConfiguration.FireStatus.FAILED.name(); + errorMessage = result.error(); + LOGGER.errorf("[SCHEDULE] Dream consolidation failed for schedule '%s' (id=%s): %s", schedule.getName(), schedule.getId(), + errorMessage); + } + } catch (Exception e) { + status = ScheduleConfiguration.FireStatus.FAILED.name(); + errorMessage = e.getClass().getSimpleName() + ": " + e.getMessage(); + LOGGER.errorf(e, "[SCHEDULE] Dream consolidation threw for schedule '%s' (id=%s)", schedule.getName(), schedule.getId()); + } + + var fireLog = new ScheduleFireLog(UUID.randomUUID().toString(), schedule.getId(), schedule.getFireId(), schedule.getNextFire(), startedAt, + Instant.now(), status, instanceId, null, errorMessage, attemptNumber, cost); + try { + scheduleStore.logFire(fireLog); + } catch (Exception e) { + LOGGER.errorf(e, "[SCHEDULE] Failed to log dream fire for schedule %s", schedule.getId()); + } + return fireLog; + } + private String resolveConversation(ScheduleConfiguration schedule, Environment env) throws Exception { String strategy = schedule.getConversationStrategy(); if (strategy == null) { diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/ShutdownReadinessHealthCheck.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/ShutdownReadinessHealthCheck.java new file mode 100644 index 000000000..e12c76040 --- /dev/null +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/ShutdownReadinessHealthCheck.java @@ -0,0 +1,40 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.engine.runtime.internal; + +import org.eclipse.microprofile.health.HealthCheck; +import org.eclipse.microprofile.health.HealthCheckResponse; +import org.eclipse.microprofile.health.Readiness; + +import jakarta.enterprise.context.ApplicationScoped; +import jakarta.inject.Inject; + +/** + * Reports readiness DOWN as soon as {@link GracefulShutdownService} observes a + * shutdown signal (B3), so the load balancer takes this pod out of rotation + * BEFORE the drain starts. + *

      + * MicroProfile Health aggregates every {@code @Readiness} check, so this simply + * joins the existing readiness set (see {@code AgentsReadinessHealthCheck}) + * rather than introducing a parallel readiness mechanism: one DOWN check makes + * {@code /q/health/ready} DOWN. + */ +@ApplicationScoped +@Readiness +public class ShutdownReadinessHealthCheck implements HealthCheck { + + private final GracefulShutdownService gracefulShutdownService; + + @Inject + public ShutdownReadinessHealthCheck(GracefulShutdownService gracefulShutdownService) { + this.gracefulShutdownService = gracefulShutdownService; + } + + @Override + public HealthCheckResponse call() { + var responseBuilder = HealthCheckResponse.named("Graceful shutdown readiness check"); + return gracefulShutdownService.isShuttingDown() ? responseBuilder.down().build() : responseBuilder.up().build(); + } +} diff --git a/src/main/java/ai/labs/eddi/modules/nlp/InputParserTask.java b/src/main/java/ai/labs/eddi/modules/nlp/InputParserTask.java index 4277c65b7..7dff107f2 100644 --- a/src/main/java/ai/labs/eddi/modules/nlp/InputParserTask.java +++ b/src/main/java/ai/labs/eddi/modules/nlp/InputParserTask.java @@ -131,6 +131,12 @@ public void execute(IConversationMemory memory, Object component) { storeNormalizedResultInMemory(memory.getCurrentStep(), normalizedUserInput); parsedSolutions = parser.parse(normalizedUserInput, userLanguage, temporaryDictionaries); } catch (InterruptedException e) { + // B2: the pipeline's graceful-stop signal IS the thread's interrupt flag — + // LifecycleManager re-checks Thread.currentThread().isInterrupted() before + // every task. Catching the exception consumes the signal, so restore the + // flag before returning normally; otherwise the remaining tasks of an + // interrupted turn keep running. + Thread.currentThread().interrupt(); log.warn(e.getLocalizedMessage(), e); return; } diff --git a/src/main/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParser.java b/src/main/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParser.java index 1f5c535e9..67b92e880 100644 --- a/src/main/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParser.java +++ b/src/main/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParser.java @@ -16,6 +16,8 @@ import ai.labs.eddi.modules.nlp.Solution; import ai.labs.eddi.modules.nlp.expressions.Expressions; import ai.labs.eddi.modules.nlp.internal.matches.RawSolution; +import com.github.benmanes.caffeine.cache.Cache; +import com.github.benmanes.caffeine.cache.Caffeine; import org.jboss.logging.Logger; import jakarta.enterprise.context.ApplicationScoped; @@ -25,6 +27,7 @@ import jakarta.ws.rs.InternalServerErrorException; import jakarta.ws.rs.container.AsyncResponse; import java.net.URI; +import java.time.Duration; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -39,10 +42,33 @@ @ApplicationScoped public class RestSemanticParser implements IRestSemanticParser { + + /** + * Upper bound on the number of distinct parser configurations kept in memory. + * The cache key is derived from a caller-supplied config id, so the cache must + * be bounded — otherwise a careless or malicious caller could grow it without + * limit. + */ + static final int MAX_CACHED_PARSERS = 100; + + /** + * Bounds how long an edited parser configuration keeps being served from the + * cache. Without a TTL, a config change would only take effect after a restart. + */ + private static final Duration PARSER_CACHE_TTL = Duration.ofMinutes(5); + private final IRuntime runtime; private final IResourceClientLibrary resourceClientLibrary; private final Provider parserProvider; - private final Map cache; + + /** + * Bounded, thread-safe parser cache. This bean is an {@code @ApplicationScoped} + * singleton and parsers are created on runtime pool threads, so the cache is + * mutated concurrently. Caffeine applies the loader at most once per key, which + * guarantees that concurrent requests for the same, not-yet-cached parser + * configuration create exactly one parser instance. + */ + private final Cache parserCache; private final Logger log = Logger.getLogger(RestSemanticParser.class); @@ -53,7 +79,10 @@ public RestSemanticParser(IRuntime runtime, IResourceClientLibrary resourceClien this.resourceClientLibrary = resourceClientLibrary; this.parserProvider = lifecycleTasks.get("ai.labs.parser"); - cache = new HashMap<>(); + this.parserCache = Caffeine.newBuilder() + .maximumSize(MAX_CACHED_PARSERS) + .expireAfterWrite(PARSER_CACHE_TTL) + .build(); } @Override @@ -79,29 +108,69 @@ public void parse(String configId, Integer version, String sentence, AsyncRespon } private IInputParser getParser(URI resourceUri) throws Exception { - return createParserIfAbsent(resourceUri); + try { + return parserCache.get(resourceUri, this::createParser); + } catch (ParserCreationException e) { + if (e.getCause() instanceof Exception cause) { + throw cause; + } + throw e; + } } - private IInputParser createParserIfAbsent(URI resourceUri) throws Exception { - if (!cache.containsKey(resourceUri)) { + /** + * Cache loader. Runs at most once per key, even when several pool threads + * request the same parser configuration simultaneously. + */ + private IInputParser createParser(URI resourceUri) { + try { ILifecycleTask parserTask = parserProvider.get(); var parserConfiguration = fetchParserConfiguration(resourceUri); var config = parserConfiguration.getConfig(); var extensions = parserConfiguration.getExtensions(); - var inputParser = (IInputParser) parserTask.configure(config != null ? config : new HashMap<>(), + return (IInputParser) parserTask.configure(config != null ? config : new HashMap<>(), extensions != null ? extensions : new HashMap<>()); - - cache.put(resourceUri, inputParser); - return inputParser; - } else { - return cache.get(resourceUri); + } catch (RuntimeException e) { + throw e; + } catch (Exception e) { + throw new ParserCreationException(e); } } + /** + * Drops all cached parsers so that the next request re-reads the parser + * configuration from the store. Intended as the explicit hook for parser + * configuration updates; until a store wires it up, the TTL is what bounds + * staleness. + */ + public void invalidateCache() { + parserCache.invalidateAll(); + } + + /** + * Number of parsers currently cached, after running pending cache maintenance. + * Never exceeds {@link #MAX_CACHED_PARSERS}. + */ + long cachedParserCount() { + parserCache.cleanUp(); + return parserCache.estimatedSize(); + } + private ParserConfiguration fetchParserConfiguration(URI resourceUri) throws ServiceException { return resourceClientLibrary.getResource(resourceUri, ParserConfiguration.class); } + /** + * Carries a checked exception out of the cache loader, which cannot declare + * checked exceptions. Unwrapped again in {@link #getParser(URI)} so callers + * keep seeing the original exception type. + */ + private static final class ParserCreationException extends RuntimeException { + private ParserCreationException(Exception cause) { + super(cause); + } + } + public static class ResponseSolution { private String expressions; diff --git a/src/main/java/ai/labs/eddi/modules/rules/impl/RulesEvaluationTask.java b/src/main/java/ai/labs/eddi/modules/rules/impl/RulesEvaluationTask.java index 6180ce7ee..1dfb4bb3c 100644 --- a/src/main/java/ai/labs/eddi/modules/rules/impl/RulesEvaluationTask.java +++ b/src/main/java/ai/labs/eddi/modules/rules/impl/RulesEvaluationTask.java @@ -92,6 +92,12 @@ public void execute(IConversationMemory memory, Object component) throws Lifecyc log.error(msg, e); throw new LifecycleException(msg, e); } catch (InterruptedException e) { + // B2: the pipeline's graceful-stop signal IS the thread's interrupt flag — + // LifecycleManager re-checks Thread.currentThread().isInterrupted() before + // every task. Catching the exception consumes the signal, so restore the + // flag before returning normally; otherwise the remaining tasks of an + // interrupted turn keep running. + Thread.currentThread().interrupt(); log.warn(e.getLocalizedMessage(), e); } } diff --git a/src/test/java/ai/labs/eddi/configs/agents/model/AgentConfigurationTest.java b/src/test/java/ai/labs/eddi/configs/agents/model/AgentConfigurationTest.java index 98e08e7ef..211faeffd 100644 --- a/src/test/java/ai/labs/eddi/configs/agents/model/AgentConfigurationTest.java +++ b/src/test/java/ai/labs/eddi/configs/agents/model/AgentConfigurationTest.java @@ -292,6 +292,22 @@ void setters() { assertTrue(dc.isEnabled()); assertEquals(10.0, dc.getMaxCostPerRun()); } + + @Test + void parametersDefaultToEmptyAndSetterIsNullSafe() { + var dc = new AgentConfiguration.DreamConfig(); + assertNotNull(dc.getParameters()); + assertTrue(dc.getParameters().isEmpty()); + + dc.setParameters(Map.of("apiKey", "${vault:dream-key}")); + assertEquals("${vault:dream-key}", dc.getParameters().get("apiKey")); + + // A config that omits the block must never leave a null map behind for + // the summarizer to trip over + dc.setParameters(null); + assertNotNull(dc.getParameters()); + assertTrue(dc.getParameters().isEmpty()); + } } @Nested diff --git a/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceHitlCoverage2Test.java b/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceHitlCoverage2Test.java index 98e90809f..f0758d6e6 100644 --- a/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceHitlCoverage2Test.java +++ b/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceHitlCoverage2Test.java @@ -45,6 +45,7 @@ import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; import org.mockito.ArgumentCaptor; import org.mockito.Mock; import org.mockito.MockitoAnnotations; @@ -408,6 +409,35 @@ void timeout_awaitingHuman_suppressed() throws Exception { CONVERSATION_ID, ConversationState.EXECUTION_INTERRUPTED); } + @Test + @Timeout(10) + @DisplayName("B2: an interrupted wait restores the interrupt flag — after the state write, not before") + void interrupted_restoresFlagAfterStateWrite() throws Exception { + var queued = acceptTurnAndCaptureCallable(); + doReturn(ConversationState.READY) + .when(conversationMemoryStore).getConversationState(CONVERSATION_ID); + stubRuntimeInterrupted(); + + assertFalse(Thread.currentThread().isInterrupted(), "precondition: flag starts clear"); + try { + queued.call(); + + // Future.get CLEARS the interrupt status when it throws + // InterruptedException, so this handler has to re-assert it — otherwise + // the executor thread's shutdown signal dies here. + assertTrue(Thread.currentThread().isInterrupted(), + "waitForExecutionFinishOrTimeout must restore the interrupt flag that Future.get consumed"); + // ...and it must be restored only AFTER the store round trips: a set flag + // aborts the sync Mongo driver, which would skip the very + // EXECUTION_INTERRUPTED write this branch exists to perform. + verify(conversationMemoryStore).setConversationState( + CONVERSATION_ID, ConversationState.EXECUTION_INTERRUPTED); + } finally { + // Never let the flag leak into the next test on this thread. + Thread.interrupted(); + } + } + /** * Wires the happy say() path up to submit and returns the captured coordinator * callable. The runtime is NOT stubbed here — the caller stubs it (timeout). @@ -490,6 +520,24 @@ private void stubRuntimeTimeout() throws Exception { }).when(runtime).submitCallable(any(Callable.class), any(IRuntime.IFinishedExecution.class), any()); } + /** + * Runtime whose returned Future throws InterruptedException on get() — drives + * the B2 interrupt branch of waitForExecutionFinishOrTimeout. Mirrors the JDK + * contract: {@code Future.get} CLEARS the thread's interrupt status before it + * throws, so the handler is the only thing that can put it back. + */ + @SuppressWarnings("unchecked") + private void stubRuntimeInterrupted() throws Exception { + doAnswer(invocation -> { + Callable callable = invocation.getArgument(0); + callable.call(); + Future future = mock(Future.class); + doThrow(new InterruptedException("watchdog interrupted")) + .when(future).get(anyLong(), any(TimeUnit.class)); + return future; + }).when(runtime).submitCallable(any(Callable.class), any(IRuntime.IFinishedExecution.class), any()); + } + /** * Minimal snapshot loadable into a live ConversationMemory in the given state. */ diff --git a/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceProcessingGaugeTest.java b/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceProcessingGaugeTest.java new file mode 100644 index 000000000..351231540 --- /dev/null +++ b/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceProcessingGaugeTest.java @@ -0,0 +1,279 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.engine.internal; + +import ai.labs.eddi.configs.agents.IAgentStore; +import ai.labs.eddi.configs.properties.IUserMemoryStore; +import ai.labs.eddi.datastore.serialization.IJsonSerialization; +import ai.labs.eddi.engine.api.IConversationService.AgentNotReadyException; +import ai.labs.eddi.engine.api.IConversationService.ConversationResponseHandler; +import ai.labs.eddi.engine.audit.AuditLedgerService; +import ai.labs.eddi.engine.caching.ICache; +import ai.labs.eddi.engine.caching.ICacheFactory; +import ai.labs.eddi.engine.gdpr.GdprComplianceService; +import ai.labs.eddi.engine.lifecycle.IConversation; +import ai.labs.eddi.engine.memory.IConversationMemoryStore; +import ai.labs.eddi.engine.memory.descriptor.IConversationDescriptorStore; +import ai.labs.eddi.engine.memory.model.ConversationMemorySnapshot; +import ai.labs.eddi.engine.memory.model.ConversationMemorySnapshot.ConversationStepSnapshot; +import ai.labs.eddi.engine.memory.model.ConversationMemorySnapshot.ResultSnapshot; +import ai.labs.eddi.engine.memory.model.ConversationMemorySnapshot.WorkflowRunSnapshot; +import ai.labs.eddi.engine.memory.model.ConversationOutput; +import ai.labs.eddi.engine.memory.model.ConversationState; +import ai.labs.eddi.engine.model.Deployment.Environment; +import ai.labs.eddi.engine.model.InputData; +import ai.labs.eddi.engine.runtime.IAgent; +import ai.labs.eddi.engine.runtime.IAgentFactory; +import ai.labs.eddi.engine.runtime.IConversationCoordinator; +import ai.labs.eddi.engine.runtime.IConversationSetup; +import ai.labs.eddi.engine.runtime.IRuntime; +import ai.labs.eddi.engine.runtime.internal.GracefulShutdownService; +import ai.labs.eddi.engine.schedule.IScheduleStore; +import ai.labs.eddi.engine.tenancy.TenantQuotaService; +import ai.labs.eddi.engine.tenancy.model.QuotaCheckResult; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.mockito.ArgumentCaptor; + +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.Callable; +import java.util.concurrent.Future; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicBoolean; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +/** + * C11 — the {@code eddi_processing_conversation_count} gauge, and B3's accept + * gate on the conversation entry points. + * + *

      + * The gauge used to be backed by a {@code CopyOnWriteArrayList} of + * {@code agentId:conversationId} strings. Two defects followed from that: a + * turn that never reached its completion consumer (watchdog timeout, cancelled + * inner future) leaked its entry forever, and because the string is IDENTICAL + * for two concurrent turns on the same conversation, a failing turn deleted a + * healthy concurrent turn's entry. + *

      + */ +class ConversationServiceProcessingGaugeTest { + + private static final Environment ENV = Environment.production; + private static final String AGENT_ID = "gauge-agent-id"; + private static final String CONVERSATION_ID = "gauge-conversation-id"; + private static final String USER_ID = "gauge-user-id"; + private static final int AGENT_TIMEOUT = 30; + private static final String GAUGE = "eddi_processing_conversation_count"; + + private IAgentFactory agentFactory; + private IConversationMemoryStore conversationMemoryStore; + private IConversationCoordinator conversationCoordinator; + private IRuntime runtime; + private TenantQuotaService tenantQuotaService; + private SimpleMeterRegistry meterRegistry; + + private ConversationService conversationService; + + @SuppressWarnings("unchecked") + @BeforeEach + void setUp() { + agentFactory = mock(IAgentFactory.class); + conversationMemoryStore = mock(IConversationMemoryStore.class); + conversationCoordinator = mock(IConversationCoordinator.class); + runtime = mock(IRuntime.class); + tenantQuotaService = mock(TenantQuotaService.class); + meterRegistry = new SimpleMeterRegistry(); + + var conversationDescriptorStore = mock(IConversationDescriptorStore.class); + var conversationSetup = mock(IConversationSetup.class); + var cacheFactory = mock(ICacheFactory.class); + var conversationStateCache = (ICache) mock(ICache.class); + var contextLogger = mock(IContextLogger.class); + var auditLedgerService = mock(AuditLedgerService.class); + var gdprComplianceService = mock(GdprComplianceService.class); + var scheduleStore = mock(IScheduleStore.class); + var agentStore = mock(IAgentStore.class); + var userMemoryStore = mock(IUserMemoryStore.class); + var jsonSerialization = mock(IJsonSerialization.class); + + doReturn(conversationStateCache).when(cacheFactory).getCache("conversationState"); + when(contextLogger.createLoggingContext(any(), any(), any(), any())).thenReturn(new HashMap<>()); + when(tenantQuotaService.acquireApiCallSlot()).thenReturn(QuotaCheckResult.OK); + when(auditLedgerService.isEnabled()).thenReturn(false); + + conversationService = new ConversationService(agentFactory, conversationMemoryStore, + conversationDescriptorStore, userMemoryStore, conversationCoordinator, + conversationSetup, cacheFactory, runtime, contextLogger, auditLedgerService, + gdprComplianceService, tenantQuotaService, scheduleStore, agentStore, + jsonSerialization, meterRegistry, + ConversationServiceTestFixtures.hitlResumeEvent(), AGENT_TIMEOUT); + } + + private double gauge() { + return meterRegistry.get(GAUGE).gauge().value(); + } + + private ConversationMemorySnapshot snapshot() { + var snapshot = new ConversationMemorySnapshot(); + snapshot.setConversationId(CONVERSATION_ID); + snapshot.setAgentId(AGENT_ID); + snapshot.setUserId(USER_ID); + snapshot.setAgentVersion(1); + snapshot.setEnvironment(ENV); + snapshot.setConversationState(ConversationState.READY); + + var stepSnapshot = new ConversationStepSnapshot(); + var workflowRun = new WorkflowRunSnapshot(); + workflowRun.getLifecycleTasks().add(new ResultSnapshot("input:initial", "hello", null, new Date(), null, true)); + stepSnapshot.getWorkflows().add(workflowRun); + snapshot.getConversationSteps().add(stepSnapshot); + var output = new ConversationOutput(); + output.put("input", "hello"); + snapshot.getConversationOutputs().add(output); + return snapshot; + } + + private void stubHealthySay() throws Exception { + IAgent agent = mock(IAgent.class); + IConversation conversation = mock(IConversation.class); + + when(conversationMemoryStore.loadConversationMemorySnapshot(CONVERSATION_ID)).thenReturn(snapshot()); + when(conversationMemoryStore.getConversationState(CONVERSATION_ID)).thenReturn(ConversationState.READY); + when(agentFactory.getAgent(ENV, AGENT_ID, 1)).thenReturn(agent); + when(agent.continueConversation(any(), any(), any())).thenReturn(conversation); + when(conversation.isEnded()).thenReturn(false); + } + + private void say() throws Exception { + conversationService.say(ENV, AGENT_ID, CONVERSATION_ID, false, false, List.of(), + new InputData("hello", Map.of()), false, mock(ConversationResponseHandler.class)); + } + + @SuppressWarnings("unchecked") + private Callable captureQueuedTurn() { + ArgumentCaptor> captor = ArgumentCaptor.forClass(Callable.class); + verify(conversationCoordinator, atLeastOnce()).submitInOrder(eq(CONVERSATION_ID), captor.capture()); + return captor.getValue(); + } + + /** + * C11 acceptance — the watchdog cancels the inner execution, so the completion + * consumer (which used to be the only place the entry was removed) is never + * invoked. The gauge must still return to zero. + */ + @Test + @Timeout(30) + @DisplayName("gauge returns to zero after a turn the watchdog cancelled") + @SuppressWarnings("unchecked") + void gaugeReturnsToZeroAfterACancelledTurn() throws Exception { + stubHealthySay(); + + // The inner execution never finishes — future.get() times out, which is what + // the agent-timeout watchdog reacts to. + Future hungExecution = mock(Future.class); + when(hungExecution.get(anyLong(), any(TimeUnit.class))).thenThrow(new TimeoutException("pipeline hung")); + doReturn(hungExecution).when(runtime).submitCallable(any(Callable.class), any(IRuntime.IFinishedExecution.class), isNull()); + + say(); + + assertEquals(1.0, gauge(), "an admitted turn must be counted while it is in flight"); + + // Run the queued turn on this thread; it hits the watchdog and is abandoned. + captureQueuedTurn().call(); + + verify(hungExecution).cancel(true); + assertEquals(0.0, gauge(), + "the gauge must return to zero after a cancelled turn — the completion consumer never runs for one"); + } + + /** + * C11 (cross-delete) — the old removal keyed on {@code agentId:conversationId}, + * which is identical for two concurrent turns of the same conversation. A turn + * rejected BEFORE it was ever admitted therefore deleted a healthy in-flight + * turn's entry and under-reported the gauge. + */ + @Test + @Timeout(30) + @DisplayName("a rejected turn does not decrement a healthy concurrent turn's entry") + void rejectedTurnDoesNotCancelOutAHealthyConcurrentTurn() throws Exception { + stubHealthySay(); + + say(); + assertEquals(1.0, gauge()); + + // A second turn on the SAME conversation is rejected before admission. + when(agentFactory.getAgent(ENV, AGENT_ID, 1)).thenReturn(null); + assertThrows(AgentNotReadyException.class, this::say); + + assertEquals(1.0, gauge(), + "the still-running first turn must remain counted — a rejected turn shares its metrics key"); + } + + @Test + @Timeout(30) + @DisplayName("gauge returns to zero after a turn skipped by the queued-say guard") + void gaugeReturnsToZeroAfterASkippedTurn() throws Exception { + stubHealthySay(); + say(); + assertEquals(1.0, gauge()); + + // By the time the queued turn runs, the conversation has ended → skipped. + when(conversationMemoryStore.getConversationState(CONVERSATION_ID)).thenReturn(ConversationState.ENDED); + captureQueuedTurn().call(); + + assertEquals(0.0, gauge()); + } + + /** + * B3 — once the shutdown gate is closed, the conversation entry points must + * refuse new turns rather than admitting work the JVM is about to drop. + */ + @Test + @Timeout(30) + @DisplayName("B3: new turns are rejected once shutdown has been signalled") + void newTurnsAreRejectedDuringShutdown() throws Exception { + stubHealthySay(); + + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + when(coordinator.getQueueDepths()).thenReturn(Map.of()); + // The event→flag→drain wiring is covered by GracefulShutdownServiceTest; here + // we only assert that the service consults the gate. (drainTimeoutSeconds, + // readinessGraceSeconds, drainPollMillis) + AtomicBoolean signalled = new AtomicBoolean(false); + conversationService.gracefulShutdownService = new GracefulShutdownService(coordinator, 1, 0, 1L) { + @Override + public boolean isShuttingDown() { + return signalled.get(); + } + }; + + // Before the signal, turns are admitted normally. + say(); + verify(conversationCoordinator).submitInOrder(eq(CONVERSATION_ID), any()); + + signalled.set(true); + + assertThrows(RejectedExecutionException.class, this::say); + assertThrows(RejectedExecutionException.class, + () -> conversationService.sayStreaming(ENV, AGENT_ID, CONVERSATION_ID, false, false, List.of(), + new InputData("hello", Map.of()), null)); + assertThrows(RejectedExecutionException.class, + () -> conversationService.startConversation(ENV, AGENT_ID, USER_ID, Map.of())); + + // A rejected turn must not be counted, and must not have been queued. + verify(conversationCoordinator, times(1)).submitInOrder(eq(CONVERSATION_ID), any()); + assertEquals(1.0, gauge(), "the rejected turns must not touch the in-flight gauge"); + } +} diff --git a/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceStaleTurnTest.java b/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceStaleTurnTest.java new file mode 100644 index 000000000..e35004928 --- /dev/null +++ b/src/test/java/ai/labs/eddi/engine/internal/ConversationServiceStaleTurnTest.java @@ -0,0 +1,217 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.engine.internal; + +import ai.labs.eddi.configs.agents.IAgentStore; +import ai.labs.eddi.configs.properties.IUserMemoryStore; +import ai.labs.eddi.datastore.serialization.IJsonSerialization; +import ai.labs.eddi.engine.api.IConversationService.ConversationResponseHandler; +import ai.labs.eddi.engine.audit.AuditLedgerService; +import ai.labs.eddi.engine.caching.ICache; +import ai.labs.eddi.engine.caching.ICacheFactory; +import ai.labs.eddi.engine.gdpr.GdprComplianceService; +import ai.labs.eddi.engine.lifecycle.IConversation; +import ai.labs.eddi.engine.memory.IConversationMemoryStore; +import ai.labs.eddi.engine.memory.descriptor.IConversationDescriptorStore; +import ai.labs.eddi.engine.memory.model.ConversationMemorySnapshot; +import ai.labs.eddi.engine.memory.model.ConversationMemorySnapshot.ConversationStepSnapshot; +import ai.labs.eddi.engine.memory.model.ConversationMemorySnapshot.ResultSnapshot; +import ai.labs.eddi.engine.memory.model.ConversationMemorySnapshot.WorkflowRunSnapshot; +import ai.labs.eddi.engine.memory.model.ConversationOutput; +import ai.labs.eddi.engine.memory.model.ConversationState; +import ai.labs.eddi.engine.model.Deployment.Environment; +import ai.labs.eddi.engine.model.InputData; +import ai.labs.eddi.engine.runtime.BaseRuntime; +import ai.labs.eddi.engine.runtime.IAgent; +import ai.labs.eddi.engine.runtime.IAgentFactory; +import ai.labs.eddi.engine.runtime.IConversationCoordinator; +import ai.labs.eddi.engine.runtime.IConversationSetup; +import ai.labs.eddi.engine.schedule.IScheduleStore; +import ai.labs.eddi.engine.tenancy.TenantQuotaService; +import ai.labs.eddi.engine.tenancy.model.QuotaCheckResult; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import org.eclipse.microprofile.context.ManagedExecutor; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.mockito.ArgumentCaptor; + +import java.util.Date; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.Callable; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.*; +import static org.mockito.Mockito.*; + +/** + * C3 — a turn that exceeds the agent-timeout watchdog must never persist its + * outcome, because by then the conversation may already have moved on (a newer + * turn, a cancel, an end). + * + *

      + * The guard used to be the thread's interrupt flag. That is not a safe + * completion guard: the pipeline consumes the interrupt via + * {@code Thread.interrupted()}, which CLEARS the flag, so the abandoned turn + * reported success and wrote its full stale snapshot over whatever came after + * it. This test wires the REAL {@link BaseRuntime} into the service so the + * abandonment token is exercised end to end. + *

      + */ +class ConversationServiceStaleTurnTest { + + private static final Environment ENV = Environment.production; + private static final String AGENT_ID = "stale-agent-id"; + private static final String CONVERSATION_ID = "stale-conversation-id"; + private static final String USER_ID = "stale-user-id"; + /** Seconds — the smallest watchdog the service accepts. */ + private static final int AGENT_TIMEOUT = 1; + + private IAgentFactory agentFactory; + private IConversationMemoryStore conversationMemoryStore; + private IConversationCoordinator conversationCoordinator; + private IConversation conversation; + + private BaseRuntime runtime; + private ExecutorService pool; + private ConversationService conversationService; + + @SuppressWarnings("unchecked") + @BeforeEach + void setUp() throws Exception { + agentFactory = mock(IAgentFactory.class); + conversationMemoryStore = mock(IConversationMemoryStore.class); + conversationCoordinator = mock(IConversationCoordinator.class); + conversation = mock(IConversation.class); + + runtime = new BaseRuntime("TestProject", "1.0.0"); + pool = Executors.newFixedThreadPool(4); + ManagedExecutor managedExecutor = mock(ManagedExecutor.class); + when(managedExecutor.submit(any(Callable.class))).thenAnswer(inv -> pool.submit((Callable) inv.getArgument(0))); + var executorField = BaseRuntime.class.getDeclaredField("executorService"); + executorField.setAccessible(true); + executorField.set(runtime, managedExecutor); + + var conversationDescriptorStore = mock(IConversationDescriptorStore.class); + var conversationSetup = mock(IConversationSetup.class); + var cacheFactory = mock(ICacheFactory.class); + var conversationStateCache = (ICache) mock(ICache.class); + var contextLogger = mock(IContextLogger.class); + var auditLedgerService = mock(AuditLedgerService.class); + var gdprComplianceService = mock(GdprComplianceService.class); + var tenantQuotaService = mock(TenantQuotaService.class); + var scheduleStore = mock(IScheduleStore.class); + var agentStore = mock(IAgentStore.class); + var userMemoryStore = mock(IUserMemoryStore.class); + var jsonSerialization = mock(IJsonSerialization.class); + + doReturn(conversationStateCache).when(cacheFactory).getCache("conversationState"); + when(contextLogger.createLoggingContext(any(), any(), any(), any())).thenReturn(new HashMap<>()); + when(tenantQuotaService.acquireApiCallSlot()).thenReturn(QuotaCheckResult.OK); + when(auditLedgerService.isEnabled()).thenReturn(false); + + conversationService = new ConversationService(agentFactory, conversationMemoryStore, + conversationDescriptorStore, userMemoryStore, conversationCoordinator, + conversationSetup, cacheFactory, runtime, contextLogger, auditLedgerService, + gdprComplianceService, tenantQuotaService, scheduleStore, agentStore, + jsonSerialization, new SimpleMeterRegistry(), + ConversationServiceTestFixtures.hitlResumeEvent(), AGENT_TIMEOUT); + } + + @AfterEach + void tearDown() { + runtime.getScheduledExecutorService().shutdownNow(); + pool.shutdownNow(); + } + + @Test + @Timeout(60) + @DisplayName("C3: a turn that outlives the watchdog never persists its snapshot") + @SuppressWarnings("unchecked") + void abandonedTurnDoesNotPersistOverANewerTurn() throws Exception { + var snapshot = new ConversationMemorySnapshot(); + snapshot.setConversationId(CONVERSATION_ID); + snapshot.setAgentId(AGENT_ID); + snapshot.setUserId(USER_ID); + snapshot.setAgentVersion(1); + snapshot.setEnvironment(ENV); + snapshot.setConversationState(ConversationState.READY); + + var stepSnapshot = new ConversationStepSnapshot(); + var workflowRun = new WorkflowRunSnapshot(); + workflowRun.getLifecycleTasks().add(new ResultSnapshot("input:initial", "hello", null, new Date(), null, true)); + stepSnapshot.getWorkflows().add(workflowRun); + snapshot.getConversationSteps().add(stepSnapshot); + var output = new ConversationOutput(); + output.put("input", "hello"); + snapshot.getConversationOutputs().add(output); + + IAgent agent = mock(IAgent.class); + when(conversationMemoryStore.loadConversationMemorySnapshot(CONVERSATION_ID)).thenReturn(snapshot); + when(conversationMemoryStore.getConversationState(CONVERSATION_ID)).thenReturn(ConversationState.READY); + when(agentFactory.getAgent(ENV, AGENT_ID, 1)).thenReturn(agent); + when(agent.continueConversation(any(), any(), any())).thenReturn(conversation); + when(conversation.isEnded()).thenReturn(false); + + CountDownLatch pipelineStarted = new CountDownLatch(1); + CountDownLatch releasePipeline = new CountDownLatch(1); + CountDownLatch pipelineFinished = new CountDownLatch(1); + // Any persistence attempt by the abandoned turn trips this latch. + CountDownLatch stalePersistAttempted = new CountDownLatch(1); + + when(conversationMemoryStore.storeConversationMemorySnapshot(any())).thenAnswer(inv -> { + stalePersistAttempted.countDown(); + return CONVERSATION_ID; + }); + + doAnswer(inv -> { + pipelineStarted.countDown(); + // A pipeline that swallows the watchdog's interrupt — exactly what + // Thread.interrupted() in the lifecycle code does — and then completes. + while (releasePipeline.getCount() > 0) { + try { + releasePipeline.await(); + } catch (InterruptedException e) { + Thread.interrupted(); + } + } + pipelineFinished.countDown(); + return null; + }).when(conversation).say(anyString(), anyMap()); + + conversationService.say(ENV, AGENT_ID, CONVERSATION_ID, false, false, List.of(), + new InputData("hello", Map.of()), false, mock(ConversationResponseHandler.class)); + + ArgumentCaptor> captor = ArgumentCaptor.forClass(Callable.class); + verify(conversationCoordinator).submitInOrder(eq(CONVERSATION_ID), captor.capture()); + + // Runs the turn: it blocks on the hung pipeline until the watchdog expires. + captor.getValue().call(); + + assertTrue(pipelineStarted.await(10, TimeUnit.SECONDS), "the pipeline should have started"); + // The watchdog parked the conversation — this is the state the abandoned turn + // must not overwrite. + verify(conversationMemoryStore).setConversationState(CONVERSATION_ID, ConversationState.EXECUTION_INTERRUPTED); + verify(conversationMemoryStore, never()).storeConversationMemorySnapshot(any()); + + // Now the abandoned pipeline finishes anyway. + releasePipeline.countDown(); + assertTrue(pipelineFinished.await(10, TimeUnit.SECONDS), "the pipeline should have completed"); + + assertFalse(stalePersistAttempted.await(2, TimeUnit.SECONDS), + "a turn abandoned by the watchdog must never persist its stale snapshot — " + + "that is what overwrites a newer turn"); + // ...and it must not downgrade the watchdog's EXECUTION_INTERRUPTED to ERROR. + verify(conversationMemoryStore, never()).setConversationState(CONVERSATION_ID, ConversationState.ERROR); + } +} diff --git a/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java new file mode 100644 index 000000000..c42150745 --- /dev/null +++ b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java @@ -0,0 +1,486 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.engine.internal; + +import ai.labs.eddi.configs.groups.IAgentGroupStore; +import ai.labs.eddi.configs.groups.IGroupConversationStore; +import ai.labs.eddi.configs.groups.model.AgentGroupConfiguration; +import ai.labs.eddi.configs.groups.model.AgentGroupConfiguration.ContextScope; +import ai.labs.eddi.configs.groups.model.AgentGroupConfiguration.DiscussionPhase; +import ai.labs.eddi.configs.groups.model.AgentGroupConfiguration.DiscussionStyle; +import ai.labs.eddi.configs.groups.model.AgentGroupConfiguration.GroupMember; +import ai.labs.eddi.configs.groups.model.AgentGroupConfiguration.PhaseType; +import ai.labs.eddi.configs.groups.model.AgentGroupConfiguration.ProtocolConfig; +import ai.labs.eddi.configs.groups.model.AgentGroupConfiguration.TurnOrder; +import ai.labs.eddi.configs.groups.model.GroupConversation; +import ai.labs.eddi.configs.groups.model.GroupConversation.GroupConversationState; +import ai.labs.eddi.configs.groups.model.GroupConversation.TranscriptEntry; +import ai.labs.eddi.configs.groups.model.GroupConversation.TranscriptEntryType; +import ai.labs.eddi.configs.groups.model.SharedTaskList; +import ai.labs.eddi.configs.groups.model.SharedTaskList.TaskItem; +import ai.labs.eddi.configs.groups.model.SharedTaskList.TaskStatus; +import ai.labs.eddi.datastore.serialization.IJsonSerialization; +import ai.labs.eddi.engine.api.IConversationService; +import ai.labs.eddi.engine.api.IGroupConversationService.GroupDiscussionEventListener; +import ai.labs.eddi.engine.lifecycle.model.ControlSignal; +import ai.labs.eddi.engine.lifecycle.model.DiscussionControlToken; +import ai.labs.eddi.engine.memory.model.ConversationOutput; +import ai.labs.eddi.engine.memory.model.ConversationState; +import ai.labs.eddi.engine.memory.model.SimpleConversationMemorySnapshot; +import ai.labs.eddi.engine.model.InputData; +import ai.labs.eddi.engine.runtime.IAgent; +import ai.labs.eddi.engine.runtime.IAgentFactory; +import ai.labs.eddi.modules.templating.ITemplatingEngine; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.mockito.Mock; +import org.mockito.MockitoAnnotations; + +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.net.URI; +import java.time.Instant; +import java.util.ArrayList; +import java.util.ConcurrentModificationException; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; + +/** + * Concurrency regression tests for {@link GroupConversationService}. Each test + * forces the interleaving it needs with latches/barriers instead of sleeping. + *
        + *
      • C1/C7 — {@code CompletableFuture.cancel(true)} does not interrupt + * a {@code runAsync} body, so a "cancelled" member turn used to keep writing to + * the group document after the orchestrator had persisted it — and could flip a + * task back to IN_PROGRESS behind the reset sweep, stranding it forever.
        • + *
      • C2 — the live {@code Collections.synchronizedList} transcript was + * published by reference into every member conversation's context and then + * iterated on the member's own thread.
        • + *
      • C6 — the {@code maxTurns} budget was enforced with a + * check-then-act on an {@code AtomicInteger} shared by N parallel agent + * threads.
        • + *
      • C8 — the parallel-phase timeout was applied serially, so N hanging + * members cost N × timeout instead of one timeout.
        • + *
      + */ +@DisplayName("GroupConversationService — concurrency regressions") +class GroupConversationServiceConcurrencyTest { + + @Mock + private IAgentGroupStore groupStore; + @Mock + private IGroupConversationStore conversationStore; + @Mock + private IConversationService conversationService; + @Mock + private IAgentFactory agentFactory; + @Mock + private ITemplatingEngine templatingEngine; + @Mock + private IJsonSerialization jsonSerialization; + @Mock + private IAgent agent; + + private GroupConversationService service; + + private static final String GROUP_ID = "group-concurrency"; + private static final String USER_ID = "user-concurrency"; + private static final String QUESTION = "What should we do?"; + + @BeforeEach + void setUp() throws Exception { + MockitoAnnotations.openMocks(this); + service = new GroupConversationService( + groupStore, conversationStore, conversationService, + agentFactory, templatingEngine, jsonSerialization, + new SimpleMeterRegistry(), null, null, null, null, null, "default", 3); + + doReturn(agent).when(agentFactory).getLatestReadyAgent(any(), any()); + var convCounter = new AtomicInteger(); + doAnswer(inv -> new IConversationService.ConversationResult( + "conv-" + convCounter.incrementAndGet(), URI.create("eddi://conv"))) + .when(conversationService).startConversation(any(), any(), any(), any()); + } + + // ================================================================= + // C1 + C7 — cooperative cancellation and the stranded-task sweep + // ================================================================= + + @Test + @Timeout(90) + @DisplayName("C1/C7: a cancelled wave stops member writes and reclaims the task it stranded") + void cancelledWave_stopsMemberWrites_andReclaimsStrandedTask() throws Exception { + var gc = groupConversation("gc-cancel-wave"); + var taskList = new SharedTaskList(); + var task = taskList.addTask(new TaskItem("Task A", "do A", 0)); + taskList.assignTask(task.id(), "agent-0", "Agent 0"); + gc.setTaskList(taskList); + + var sayEntered = new CountDownLatch(1); + var memberThread = new AtomicReference(); + // The response never arrives: the member turn parks on its own await point, + // which is exactly where cooperative cancellation has to release it. + doAnswer(inv -> { + memberThread.set(Thread.currentThread()); + sayEntered.countDown(); + return null; + }).when(conversationService).say(any(), any(), any(), any(), any(), any(), any(), anyBoolean(), any()); + + controlTokens().put(gc.getId(), new DiscussionControlToken()); + var cancelFailure = new AtomicReference(); + var canceller = new Thread(() -> { + try { + if (sayEntered.await(30, TimeUnit.SECONDS)) { + service.cancelDiscussion(gc.getId(), ControlSignal.CANCEL_IMMEDIATE); + } + } catch (Throwable t) { + cancelFailure.compareAndSet(null, t); + } + }, "group-canceller"); + canceller.start(); + + invoke(executionPhaseMethod(), gc, config(List.of(member(0))), List.of(member(0)), + phase(PhaseType.EXECUTE, TurnOrder.PARALLEL), protocol(2), QUESTION, 0, null, + new AtomicInteger(0), 50); + + canceller.join(TimeUnit.SECONDS.toMillis(30)); + assertNull(cancelFailure.get(), () -> "cancel failed: " + cancelFailure.get()); + + Thread worker = memberThread.get(); + assertNotNull(worker, "the member turn must have reached the agent call"); + // Wait for the cancelled turn to finish: everything it could still write to the + // group document happens after this point, so the assertions below are not a + // race. + worker.join(TimeUnit.SECONDS.toMillis(30)); + assertFalse(worker.isAlive(), "the cancelled member turn must unwind instead of running to completion"); + + assertEquals(TaskStatus.ASSIGNED, gc.getTaskList().all().getFirst().status(), + "the task the cancelled turn started must be reclaimed as ASSIGNED — neither left " + + "IN_PROGRESS nor failed by a write that landed after the abort"); + assertTrue(gc.getTranscript().isEmpty(), + "a cancelled member turn must not append to the transcript after the orchestrator gave up on it"); + } + + // ================================================================= + // C6 — turn budget under contention + // ================================================================= + + @Test + @Timeout(60) + @DisplayName("C6: reserveTurn hands out at most maxTurns when 8 threads race for the last turn") + void reserveTurn_underFullContention_neverOvershootsTheBudget() throws Exception { + Method reserveTurn = GroupConversationService.class.getDeclaredMethod( + "reserveTurn", AtomicInteger.class, int.class); + reserveTurn.setAccessible(true); + + int threads = 8; + int maxTurns = 3; + var turnCounter = new AtomicInteger(0); + var granted = new AtomicInteger(0); + var failure = new AtomicReference(); + var barrier = new CyclicBarrier(threads); + var done = new CountDownLatch(threads); + + for (int i = 0; i < threads; i++) { + new Thread(() -> { + try { + barrier.await(30, TimeUnit.SECONDS); + if (Boolean.TRUE.equals(reserveTurn.invoke(null, turnCounter, maxTurns))) { + granted.incrementAndGet(); + } + } catch (Throwable t) { + failure.compareAndSet(null, t); + } finally { + done.countDown(); + } + }, "reserve-turn-" + i).start(); + } + + assertTrue(done.await(45, TimeUnit.SECONDS), "reservation threads did not finish"); + assertNull(failure.get(), () -> "reservation thread failed: " + failure.get()); + assertEquals(maxTurns, granted.get(), + "exactly maxTurns turns may be granted, no matter how many threads pass the check together"); + assertEquals(maxTurns, turnCounter.get(), "the shared budget must not be inflated past maxTurns"); + } + + @Test + @Timeout(90) + @DisplayName("C6: an execution wave with 8 members racing for the last turn never exceeds maxTurns") + void executionWave_withEightMembers_neverExceedsMaxTurns() throws Exception { + int memberCount = 8; + int maxTurns = memberCount + 1; // the 8 first tasks fit; all 8 then race for turn 9 + + var gc = groupConversation("gc-turn-budget"); + var taskList = new SharedTaskList(); + List members = new ArrayList<>(); + for (int i = 0; i < memberCount; i++) { + var m = member(i); + members.add(m); + for (int t = 0; t < 2; t++) { + var task = taskList.addTask(new TaskItem("Task " + i + "." + t, "do " + i + "." + t, 0)); + taskList.assignTask(task.id(), m.agentId(), m.displayName()); + } + } + gc.setTaskList(taskList); + + // Rendezvous inside the first turn of every member, so all 8 agent threads + // reach the budget check for their second task at the same moment. Once all + // 8 have arrived the latch stays open, so later turns pass straight through. + var rendezvous = new CountDownLatch(memberCount); + doAnswer(inv -> { + rendezvous.countDown(); + rendezvous.await(30, TimeUnit.SECONDS); + handlerOf(inv.getArgument(8)).onComplete(snapshot("contribution")); + return null; + }).when(conversationService).say(any(), any(), any(), any(), any(), any(), any(), anyBoolean(), any()); + + var turnCounter = new AtomicInteger(0); + invoke(executionPhaseMethod(), gc, config(members), members, + phase(PhaseType.EXECUTE, TurnOrder.PARALLEL), protocol(30), QUESTION, 0, null, + turnCounter, maxTurns); + + long completed = gc.getTaskList().all().stream() + .filter(t -> t.status() == TaskStatus.COMPLETED).count(); + assertEquals((long) maxTurns, completed, "only the turns the budget allows may run"); + assertEquals((long) (2 * memberCount - maxTurns), gc.getTaskList().all().stream() + .filter(t -> t.status() == TaskStatus.ASSIGNED).count(), + "the remaining tasks stay ASSIGNED — untouched, not half-executed"); + assertEquals(maxTurns, turnCounter.get(), "the turn counter must land exactly on the budget"); + verify(conversationService, times(maxTurns)) + .say(any(), any(), any(), any(), any(), any(), any(), anyBoolean(), any()); + } + + // ================================================================= + // C2 — the transcript handed to a member must be a snapshot + // ================================================================= + + @Test + @Timeout(90) + @DisplayName("C2: a parallel phase with 8 members publishes a transcript snapshot, never the live list") + void parallelPhase_publishesTranscriptSnapshot_notTheLiveList() throws Exception { + var gc = groupConversation("gc-transcript-handoff"); + for (int i = 0; i < 20; i++) { + gc.getTranscript().add(transcriptEntry("seed-" + i)); + } + + List members = new ArrayList<>(); + for (int i = 0; i < 8; i++) { + members.add(member(i)); + } + + var liveListPublished = new AtomicInteger(0); + var concurrentModifications = new AtomicInteger(0); + var entriesVisited = new AtomicInteger(0); + var firstMemberStarted = new CountDownLatch(1); + + doAnswer(inv -> { + InputData inputData = inv.getArgument(6); + Object published = inputData.getContext().get("groupTranscript").getValue(); + if (published == gc.getTranscript()) { + liveListPublished.incrementAndGet(); + } + firstMemberStarted.countDown(); + + // Serialising this context is what a member conversation really does with + // it — on this thread, while the orchestrator appends to the transcript. + @SuppressWarnings("unchecked") + List handedOver = (List) published; + long until = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(300); + try { + while (System.nanoTime() < until) { + for (TranscriptEntry e : handedOver) { + if (e != null) { + entriesVisited.incrementAndGet(); + } + } + } + } catch (ConcurrentModificationException e) { + concurrentModifications.incrementAndGet(); + } + + handlerOf(inv.getArgument(8)).onComplete(snapshot("contribution")); + return null; + }).when(conversationService).say(any(), any(), any(), any(), any(), any(), any(), anyBoolean(), any()); + + var appender = new Thread(() -> { + try { + if (!firstMemberStarted.await(30, TimeUnit.SECONDS)) { + return; + } + long until = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(280); + int i = 0; + while (System.nanoTime() < until) { + gc.getTranscript().add(transcriptEntry("appended-" + i++)); + Thread.sleep(1); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }, "transcript-appender"); + appender.start(); + + invoke(parallelPhaseMethod(), gc, config(members), members, + phase(PhaseType.OPINION, TurnOrder.PARALLEL), protocol(60), QUESTION, 0, null, + new AtomicInteger(0), 50); + appender.join(TimeUnit.SECONDS.toMillis(30)); + + assertEquals(0, liveListPublished.get(), + "the live synchronized transcript must never be handed to a member conversation by reference"); + assertTrue(entriesVisited.get() > 0, "the members must actually have iterated their transcript copy"); + assertEquals(0, concurrentModifications.get(), + "iterating the transcript handed to a member must not race the orchestrator's appends"); + } + + // ================================================================= + // C8 — one deadline for the whole parallel batch + // ================================================================= + + @Test + @Timeout(120) + @DisplayName("C8: 5 hanging members cost one timeout, not five") + void parallelPhase_appliesOneDeadlineAcrossAllMembers() throws Exception { + var gc = groupConversation("gc-batch-deadline"); + List members = new ArrayList<>(); + for (int i = 0; i < 5; i++) { + members.add(member(i)); + } + + var release = new CountDownLatch(1); + var entered = new CountDownLatch(members.size()); + // Every member hangs inside the agent call for longer than the batch deadline. + doAnswer(inv -> { + entered.countDown(); + release.await(60, TimeUnit.SECONDS); + return null; + }).when(conversationService).say(any(), any(), any(), any(), any(), any(), any(), anyBoolean(), any()); + + long startNanos = System.nanoTime(); + try { + invoke(parallelPhaseMethod(), gc, config(members), members, + phase(PhaseType.OPINION, TurnOrder.PARALLEL), protocol(2), QUESTION, 0, null, + new AtomicInteger(0), 50); + } finally { + release.countDown(); + } + long elapsedMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos); + + assertTrue(entered.await(30, TimeUnit.SECONDS), "all members should have been dispatched in parallel"); + assertTrue(elapsedMs >= 1500, + () -> "the batch must still honour its 2s deadline, took " + elapsedMs + "ms"); + assertTrue(elapsedMs < 6000, + () -> "5 hanging members must not each restart the 2s budget (10s serial), took " + elapsedMs + "ms"); + assertEquals(5L, gc.getTranscript().stream() + .filter(e -> e.type() == TranscriptEntryType.SKIPPED).count(), + "every hanging member is recorded as SKIPPED exactly once"); + } + + // ================================================================= + // Helpers + // ================================================================= + + private GroupConversation groupConversation(String id) { + var gc = new GroupConversation(); + gc.setId(id); + gc.setGroupId(GROUP_ID); + gc.setUserId(USER_ID); + gc.setState(GroupConversationState.IN_PROGRESS); + gc.setOriginalQuestion(QUESTION); + return gc; + } + + private GroupMember member(int index) { + return new GroupMember("agent-" + index, "Agent " + index, index, null); + } + + private AgentGroupConfiguration config(List members) { + var config = new AgentGroupConfiguration(); + config.setName("Concurrency Group"); + config.setStyle(DiscussionStyle.CUSTOM); + config.setMembers(members); + return config; + } + + private DiscussionPhase phase(PhaseType type, TurnOrder turnOrder) { + return new DiscussionPhase("P-" + type, type, "ALL", turnOrder, ContextScope.FULL, false, null, 1, false); + } + + private ProtocolConfig protocol(int agentTimeoutSeconds) { + return new ProtocolConfig(agentTimeoutSeconds, ProtocolConfig.MemberFailurePolicy.SKIP, 0, + ProtocolConfig.MemberUnavailablePolicy.SKIP); + } + + private TranscriptEntry transcriptEntry(String content) { + return new TranscriptEntry("agent-0", "Agent 0", content, 0, "P", TranscriptEntryType.OPINION, + Instant.now(), null, null); + } + + private SimpleConversationMemorySnapshot snapshot(String text) { + var snapshot = new SimpleConversationMemorySnapshot(); + snapshot.setConversationState(ConversationState.READY); + var output = new ConversationOutput(); + output.put("output", List.of(text)); + snapshot.setConversationOutputs(new ArrayList<>(List.of(output))); + return snapshot; + } + + private IConversationService.ConversationResponseHandler handlerOf(Object argument) { + return (IConversationService.ConversationResponseHandler) argument; + } + + @SuppressWarnings("unchecked") + private Map controlTokens() throws Exception { + var field = GroupConversationService.class.getDeclaredField("activeTokens"); + field.setAccessible(true); + return (Map) field.get(service); + } + + private Method executionPhaseMethod() throws NoSuchMethodException { + return phaseMethod("executeTaskExecutionPhase"); + } + + private Method parallelPhaseMethod() throws NoSuchMethodException { + return phaseMethod("executeParallelPhase"); + } + + private Method phaseMethod(String name) throws NoSuchMethodException { + Method m = GroupConversationService.class.getDeclaredMethod(name, + GroupConversation.class, AgentGroupConfiguration.class, List.class, DiscussionPhase.class, + ProtocolConfig.class, String.class, int.class, GroupDiscussionEventListener.class, + AtomicInteger.class, int.class); + m.setAccessible(true); + return m; + } + + private void invoke(Method m, Object... args) throws Exception { + try { + m.invoke(service, args); + } catch (InvocationTargetException e) { + if (e.getCause() instanceof Exception ex) { + throw ex; + } + throw e; + } + } +} diff --git a/src/test/java/ai/labs/eddi/engine/internal/RestAgentEngineStreamingTest.java b/src/test/java/ai/labs/eddi/engine/internal/RestAgentEngineStreamingTest.java index cf10e8b36..b0f2ab76f 100644 --- a/src/test/java/ai/labs/eddi/engine/internal/RestAgentEngineStreamingTest.java +++ b/src/test/java/ai/labs/eddi/engine/internal/RestAgentEngineStreamingTest.java @@ -5,6 +5,9 @@ package ai.labs.eddi.engine.internal; import ai.labs.eddi.engine.api.IConversationService; +import ai.labs.eddi.engine.lifecycle.TaskId; +import ai.labs.eddi.engine.lifecycle.model.ControlSignal; +import ai.labs.eddi.engine.memory.model.ConversationState; import ai.labs.eddi.engine.memory.model.SimpleConversationMemorySnapshot; import ai.labs.eddi.engine.model.InputData; import ai.labs.eddi.engine.security.ConversationAccessGuard; @@ -16,9 +19,11 @@ import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; import java.lang.reflect.Method; import java.util.List; +import java.util.Map; import static org.junit.jupiter.api.Assertions.*; import static org.mockito.Mockito.*; @@ -265,24 +270,124 @@ void handleClosedSink() throws Exception { } } + /** + * F6 — the streaming endpoint receives only {@link SseEventSink}/{@link Sse} + * and never signalled cancellation, so closing the tab at token 5 of 4000 still + * streamed (and billed) the whole completion, possibly escalating through the + * whole model cascade. + *

      + * A JAX-RS {@code ConnectionCallback} is not the mechanism here: RESTEasy + * Reactive registers connection callbacks into a request property that nothing + * ever reads, so one would cancel nothing. The observable signal is + * {@code SseEventSink.isClosed()} (backed by + * {@code serverResponse().closed()}), which is why the disconnect is detected + * on the next outbound frame — i.e. within one token boundary. + */ @Nested - @DisplayName("sendEvent helper") - class SendEvent { + @DisplayName("F6 — client disconnect cancels the in-flight turn") + class ClientDisconnect { - @Test - @DisplayName("should skip sending when sink is closed") - void skipsClosedSink() throws Exception { - Method method = RestAgentEngineStreaming.class.getDeclaredMethod("sendEvent", - SseEventSink.class, Sse.class, String.class, String.class); - method.setAccessible(true); + private SseEventSink eventSink; - var eventSink = mock(SseEventSink.class); + /** Starts a stream and returns the handler the endpoint wired up. */ + private IConversationService.StreamingResponseHandler start(String conversationId) throws Exception { + eventSink = mock(SseEventSink.class); var sse = mock(Sse.class); + var eventBuilder = mock(OutboundSseEvent.Builder.class); + var sseEvent = mock(OutboundSseEvent.class); + when(sse.newEventBuilder()).thenReturn(eventBuilder); + when(eventBuilder.name(anyString())).thenReturn(eventBuilder); + when(eventBuilder.data(any(Class.class), anyString())).thenReturn(eventBuilder); + when(eventBuilder.build()).thenReturn(sseEvent); + when(eventSink.isClosed()).thenReturn(false); + + var inputData = new InputData(); + inputData.setInput("Hello"); + streaming.sayStreaming(conversationId, false, false, List.of(), inputData, eventSink, sse); + + var captor = ArgumentCaptor.forClass(IConversationService.StreamingResponseHandler.class); + verify(conversationService).sayStreaming(eq(conversationId), any(), any(), any(), any(), captor.capture()); + return captor.getValue(); + } + + private SimpleConversationMemorySnapshot readySnapshot() { + var snapshot = new SimpleConversationMemorySnapshot(); + snapshot.setConversationState(ConversationState.READY); + return snapshot; + } + + @Test + @DisplayName("a token emitted after the client vanished cancels the turn, exactly once") + void tokenAfterDisconnectCancelsOnce() throws Exception { + var handler = start("conv-1"); + + // The client closed the tab: Vert.x closes the response, so the sink + // reports closed from the next frame onwards. when(eventSink.isClosed()).thenReturn(true); - method.invoke(streaming, eventSink, sse, "test", "data"); + handler.onToken("tok-5"); + handler.onToken("tok-6"); + handler.onTaskComplete(new TaskId("ai.labs.llm"), "langchain", 5L, Map.of()); + verify(conversationService, times(1)).cancelConversation("conv-1", + ControlSignal.CANCEL_GRACEFUL, RestAgentEngineStreaming.CANCELLED_BY_CLIENT_DISCONNECT); + // …and nothing is written to a sink the client is no longer reading. verify(eventSink, never()).send(any(OutboundSseEvent.class)); } + + @Test + @DisplayName("a send that fails because the sink closed mid-write also cancels") + void sendFailureOnClosedSinkCancels() throws Exception { + var handler = start("conv-4"); + + // The sink closes between the isClosed() pre-check and the write, which is + // when RESTEasy Reactive throws IllegalStateException("Already closed"). + when(eventSink.isClosed()).thenReturn(false, true); + doThrow(new IllegalStateException("Already closed")) + .when(eventSink).send(any(OutboundSseEvent.class)); + + handler.onToken("tok-5"); + + verify(conversationService, times(1)).cancelConversation("conv-4", + ControlSignal.CANCEL_GRACEFUL, RestAgentEngineStreaming.CANCELLED_BY_CLIENT_DISCONNECT); + } + + @Test + @DisplayName("a send failure on a still-open sink is NOT a disconnect and does not cancel") + void sendFailureOnOpenSinkDoesNotCancel() throws Exception { + var handler = start("conv-5"); + + // A payload/serialization fault, not a vanished client. + doThrow(new RuntimeException("bad payload")) + .when(eventSink).send(any(OutboundSseEvent.class)); + + handler.onToken("tok-5"); + + verify(conversationService, never()).cancelConversation(anyString(), any(), anyString()); + } + + @Test + @DisplayName("a stream that completes normally is never cancelled") + void normalCompletionDoesNotCancel() throws Exception { + var handler = start("conv-2"); + + handler.onToken("hi"); + handler.onComplete(readySnapshot()); + + verify(conversationService, never()).cancelConversation(anyString(), any(), anyString()); + } + + @Test + @DisplayName("a sink already closed when the terminal frame is emitted is not a disconnect") + void closedAtTerminalFrameDoesNotCancel() throws Exception { + var handler = start("conv-3"); + + // Client went away right as the turn finished — the answer is already + // produced, so there is nothing left to cancel. + when(eventSink.isClosed()).thenReturn(true); + handler.onComplete(readySnapshot()); + + verify(conversationService, never()).cancelConversation(anyString(), any(), anyString()); + } } } diff --git a/src/test/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManagerTest.java b/src/test/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManagerTest.java index 4f17451c3..c14107a7a 100644 --- a/src/test/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManagerTest.java +++ b/src/test/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManagerTest.java @@ -24,12 +24,16 @@ import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; import org.mockito.ArgumentCaptor; import java.util.Arrays; import java.util.HashMap; import java.util.LinkedList; import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Consumer; import java.util.Map; @@ -1663,4 +1667,244 @@ void newKeyUncommitted() throws Exception { verify(newData).setCommitted(false); } } + + /** + * C5 — the component cache is keyed by the task's ABSOLUTE index in the + * workflow ({@code WorkflowStoreClientLibrary} writes + * {@code createComponentKey(id, version, indexInWorkflow)}), but a selective + * execution hands the loop a SUBLIST, so the loop index is sublist-relative. + * Building the lookup key from that relative index resolved a key that was + * never written, the task ran with {@code component == null} and no-opped — + * which is why {@code /rerun} deleted the previous output and regenerated + * nothing. + *

      + * Every other test in this class stubs the component map EMPTY, which is + * exactly why the bug survived: with an empty map both the right and the wrong + * key resolve to {@code null}. These cases populate it at ABSOLUTE indices and + * assert on the component the task actually receives. + */ + @Nested + @DisplayName("C5 — Component Cache Keying (absolute vs. sublist-relative index)") + class ComponentCacheKeyingTests { + + private static final String PARSER_COMPONENT = "parser-config"; + private static final String BEHAVIOR_COMPONENT = "behavior-config"; + private static final String OUTPUT_COMPONENT = "output-config"; + + private ILifecycleTask parser; + private ILifecycleTask behavior; + private ILifecycleTask output; + private IConversationMemory memory; + + /** + * parser@0, behavior@1, output@2 — each with its component cached under the + * ABSOLUTE workflow-step key, exactly as WorkflowStoreClientLibrary writes it. + */ + @BeforeEach + void wireWorkflow() { + parser = mock(ILifecycleTask.class); + when(parser.getId()).thenReturn(new TaskId("ai.labs.parser")); + when(parser.getType()).thenReturn("expressions"); + + behavior = mock(ILifecycleTask.class); + when(behavior.getId()).thenReturn(new TaskId("ai.labs.behavior")); + when(behavior.getType()).thenReturn("behavior_rules"); + + output = mock(ILifecycleTask.class); + when(output.getId()).thenReturn(new TaskId("ai.labs.output")); + when(output.getType()).thenReturn("output"); + + lifecycleManager.addLifecycleTask(parser); + lifecycleManager.addLifecycleTask(behavior); + lifecycleManager.addLifecycleTask(output); + + when(componentCache.getComponentMap("ai.labs.parser")) + .thenReturn(new HashMap<>(Map.of("wf1:1:0", PARSER_COMPONENT))); + when(componentCache.getComponentMap("ai.labs.behavior")) + .thenReturn(new HashMap<>(Map.of("wf1:1:1", BEHAVIOR_COMPONENT))); + when(componentCache.getComponentMap("ai.labs.output")) + .thenReturn(new HashMap<>(Map.of("wf1:1:2", OUTPUT_COMPONENT))); + + memory = mock(IConversationMemory.class); + var currentStep = mock(IConversationMemory.IWritableConversationStep.class); + when(memory.getCurrentStep()).thenReturn(currentStep); + when(memory.getConversationId()).thenReturn("conv1"); + when(memory.getAgentId()).thenReturn("agent1"); + } + + @Test + @DisplayName("full execution resolves each task's component (regression guard)") + void fullExecutionResolvesComponents() throws Exception { + lifecycleManager.executeLifecycle(memory, null); + + verify(parser).execute(memory, PARSER_COMPONENT); + verify(behavior).execute(memory, BEHAVIOR_COMPONENT); + verify(output).execute(memory, OUTPUT_COMPONENT); + } + + @Test + @DisplayName("selective execution (the /rerun path) still resolves the output task's component") + void selectiveExecutionResolvesComponentAtAbsoluteIndex() throws Exception { + // Exactly what Conversation#rerun does: run the suffix from "output". + lifecycleManager.executeLifecycle(memory, List.of("output", "quickReplies")); + + verify(parser, never()).execute(any(), any()); + verify(behavior, never()).execute(any(), any()); + // Before the fix the key was "wf1:1:0" (sublist-relative), which is not in + // the output task's component map → null → the task no-opped. + verify(output).execute(memory, OUTPUT_COMPONENT); + } + + @Test + @DisplayName("selective execution reports the ABSOLUTE task index on the streaming event") + void selectiveExecutionReportsAbsoluteTaskIndex() throws Exception { + var eventSink = mock(ConversationEventSink.class); + when(memory.getEventSink()).thenReturn(eventSink); + + lifecycleManager.executeLifecycle(memory, List.of("output")); + + verify(eventSink).onTaskStart(new TaskId("ai.labs.output"), "output", 2); + } + + @Test + @DisplayName("selective execution from the middle resolves BOTH remaining components") + void selectiveExecutionResolvesEveryRemainingComponent() throws Exception { + lifecycleManager.executeLifecycle(memory, List.of("behavior_rules")); + + verify(parser, never()).execute(any(), any()); + verify(behavior).execute(memory, BEHAVIOR_COMPONENT); + verify(output).execute(memory, OUTPUT_COMPONENT); + } + + @Test + @DisplayName("HITL resume from an absolute index resolves the component at that index") + void resumeFromIndexResolvesComponent() throws Exception { + lifecycleManager.executeLifecycleFromIndex(memory, 2); + + verify(parser, never()).execute(any(), any()); + verify(behavior, never()).execute(any(), any()); + verify(output).execute(memory, OUTPUT_COMPONENT); + } + } + + /** + * F6 — cancellation was checked at exactly ONE point: the transition INTO a + * task. A cancel that landed while the LAST task of a workflow was running was + * therefore never observed, and the turn returned as if nothing happened — + * letting {@code Conversation} commit the side effects of work the caller had + * already been told was cancelled. + */ + @Nested + @DisplayName("F6 — Cancellation Checks") + class CancellationTests { + + @Test + @Timeout(15) + @DisplayName("a cancel that lands WHILE the last task runs stops the turn") + void cancelDuringLastTaskIsObserved() throws Exception { + var task = mock(ILifecycleTask.class); + when(task.getId()).thenReturn(new TaskId("ai.labs.llm")); + when(task.getType()).thenReturn("langchain"); + lifecycleManager.addLifecycleTask(task); + + var memory = mock(IConversationMemory.class); + var currentStep = mock(IConversationMemory.IWritableConversationStep.class); + when(memory.getCurrentStep()).thenReturn(currentStep); + when(memory.getConversationId()).thenReturn("conv1"); + when(memory.getAgentId()).thenReturn("agent1"); + when(componentCache.getComponentMap(anyString())).thenReturn(new HashMap<>()); + + // Real cancel flag, read exactly as ConversationMemory exposes it. + var cancelled = new AtomicBoolean(false); + when(memory.isCancelled()).thenAnswer(invocation -> cancelled.get()); + + // Force the interleaving: the cancel lands after the task has started and + // before it returns — i.e. strictly INSIDE the only task of the pipeline. + var taskEntered = new CountDownLatch(1); + var cancelApplied = new CountDownLatch(1); + doAnswer(invocation -> { + taskEntered.countDown(); + assertTrue(cancelApplied.await(10, TimeUnit.SECONDS), "canceller thread did not run"); + return null; + }).when(task).execute(any(), any()); + + var canceller = new Thread(() -> { + try { + assertTrue(taskEntered.await(10, TimeUnit.SECONDS), "task never started"); + cancelled.set(true); + cancelApplied.countDown(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }, "cancel-signal"); + canceller.start(); + + try { + assertThrows(ConversationStopException.class, + () -> lifecycleManager.executeLifecycle(memory, null)); + } finally { + canceller.join(10_000); + } + + // The task did run to completion — this is cooperative cancellation, not + // an interrupt — but the pipeline must not report a clean turn. + verify(task).execute(any(), any()); + } + + @Test + @Timeout(15) + @DisplayName("a cancel between tasks still stops before the next task (unchanged)") + void cancelBetweenTasksStopsPipeline() throws Exception { + var first = mock(ILifecycleTask.class); + when(first.getId()).thenReturn(new TaskId("ai.labs.behavior")); + when(first.getType()).thenReturn("behavior_rules"); + var second = mock(ILifecycleTask.class); + when(second.getId()).thenReturn(new TaskId("ai.labs.output")); + when(second.getType()).thenReturn("output"); + lifecycleManager.addLifecycleTask(first); + lifecycleManager.addLifecycleTask(second); + + var memory = mock(IConversationMemory.class); + var currentStep = mock(IConversationMemory.IWritableConversationStep.class); + when(memory.getCurrentStep()).thenReturn(currentStep); + when(memory.getConversationId()).thenReturn("conv1"); + when(memory.getAgentId()).thenReturn("agent1"); + when(componentCache.getComponentMap(anyString())).thenReturn(new HashMap<>()); + + var cancelled = new AtomicBoolean(false); + when(memory.isCancelled()).thenAnswer(invocation -> cancelled.get()); + doAnswer(invocation -> { + cancelled.set(true); + return null; + }).when(first).execute(any(), any()); + + assertThrows(ConversationStopException.class, + () -> lifecycleManager.executeLifecycle(memory, null)); + + verify(first).execute(any(), any()); + verify(second, never()).execute(any(), any()); + } + + @Test + @Timeout(15) + @DisplayName("an uncancelled turn completes normally — the exit check is not a blanket throw") + void uncancelledTurnCompletes() throws Exception { + var task = mock(ILifecycleTask.class); + when(task.getId()).thenReturn(new TaskId("ai.labs.output")); + when(task.getType()).thenReturn("output"); + lifecycleManager.addLifecycleTask(task); + + var memory = mock(IConversationMemory.class); + var currentStep = mock(IConversationMemory.IWritableConversationStep.class); + when(memory.getCurrentStep()).thenReturn(currentStep); + when(memory.getConversationId()).thenReturn("conv1"); + when(memory.getAgentId()).thenReturn("agent1"); + when(componentCache.getComponentMap(anyString())).thenReturn(new HashMap<>()); + when(memory.isCancelled()).thenReturn(false); + + lifecycleManager.executeLifecycle(memory, null); + + verify(task).execute(any(), any()); + } + } } diff --git a/src/test/java/ai/labs/eddi/engine/runtime/BaseRuntimeConcurrencyTest.java b/src/test/java/ai/labs/eddi/engine/runtime/BaseRuntimeConcurrencyTest.java new file mode 100644 index 000000000..26f216f7f --- /dev/null +++ b/src/test/java/ai/labs/eddi/engine/runtime/BaseRuntimeConcurrencyTest.java @@ -0,0 +1,382 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.engine.runtime; + +import org.eclipse.microprofile.context.ManagedExecutor; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.mockito.Mockito; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.Callable; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.TimeoutException; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.when; + +/** + * Concurrency contracts of {@link BaseRuntime} — the three findings that make + * conversation turns either deadlock, resurrect stale state, or run twice. + * + *

      + * Every test forces its interleaving with latches/barriers rather than + * sleeping, and carries a {@link Timeout} so a regression fails fast instead of + * hanging the fork. + *

      + */ +class BaseRuntimeConcurrencyTest { + + /** + * Deliberately small stand-in for the bounded {@code ManagedExecutor}: the + * production pool is 200 threads and the cliff sits at half of it. + */ + private static final int POOL_SIZE = 4; + + private BaseRuntime runtime; + private ExecutorService boundedPool; + + @BeforeEach + void setUp() throws Exception { + runtime = new BaseRuntime("TestProject", "1.0.0"); + + boundedPool = Executors.newFixedThreadPool(POOL_SIZE); + ManagedExecutor mockExecutor = Mockito.mock(ManagedExecutor.class); + when(mockExecutor.submit(any(Callable.class))).thenAnswer(inv -> boundedPool.submit((Callable) inv.getArgument(0))); + + var executorField = BaseRuntime.class.getDeclaredField("executorService"); + executorField.setAccessible(true); + executorField.set(runtime, mockExecutor); + } + + @AfterEach + void tearDown() { + runtime.shutdown(); + boundedPool.shutdownNow(); + } + + /** + * C4 — a conversation turn consumes TWO threads: the coordinator's callable + * submits the pipeline execution and then BLOCKS on its Future. When both come + * from the same bounded pool, concurrency collapses at HALF the pool size: + * every thread is a waiter and no inner task can ever be scheduled. + * + *

      + * The barrier makes that precondition deterministic — all {@value #POOL_SIZE} + * pool threads are parked in the outer task before any inner task is submitted, + * so with the shared-pool behaviour NOTHING can make progress. + *

      + */ + @Test + @Timeout(60) + @DisplayName("C4: nested submissions do not starve the bounded pool — N turns at pool size all complete") + void nestedSubmissionsDoNotStarveTheBoundedPool() throws Exception { + CyclicBarrier allOuterTasksRunning = new CyclicBarrier(POOL_SIZE); + List> outerFutures = new ArrayList<>(); + + for (int i = 0; i < POOL_SIZE; i++) { + final int index = i; + outerFutures.add(runtime.submitCallable(() -> { + // Occupy every thread of the bounded pool BEFORE submitting inner work. + allOuterTasksRunning.await(20, TimeUnit.SECONDS); + + Future inner = runtime.submitCallable(() -> "inner-" + index, null); + return inner.get(20, TimeUnit.SECONDS); + }, null)); + } + + for (int i = 0; i < POOL_SIZE; i++) { + assertEquals("inner-" + i, outerFutures.get(i).get(30, TimeUnit.SECONDS), + "every turn must complete; a null result means the inner task was never scheduled (pool deadlock)"); + } + } + + /** + * C3 — the watchdog abandons a turn, but the pipeline swallows the interrupt + * (any {@code Thread.interrupted()} call CLEARS the flag) and runs to + * completion. If completion is still reported, the abandoned turn persists its + * stale full snapshot over whatever ran after it. + */ + @Test + @Timeout(30) + @DisplayName("C3: a watchdog-abandoned turn reports failure even when the pipeline swallows the interrupt") + void abandonedExecutionNeverReportsCompletion() throws Exception { + CountDownLatch executionStarted = new CountDownLatch(1); + CountDownLatch releaseExecution = new CountDownLatch(1); + CountDownLatch onCompleteCalled = new CountDownLatch(1); + CountDownLatch onFailureCalled = new CountDownLatch(1); + AtomicInteger failureCount = new AtomicInteger(); + List failures = new ArrayList<>(); + + Future future = runtime.submitCallable( + () -> { + executionStarted.countDown(); + // Pipeline-style interrupt swallowing: the flag is cleared and never + // restored, so it cannot serve as a completion guard. + while (releaseExecution.getCount() > 0) { + try { + releaseExecution.await(); + } catch (InterruptedException e) { + Thread.interrupted(); + } + } + return "stale-result"; + }, + new IRuntime.IFinishedExecution() { + @Override + public void onComplete(String result) { + onCompleteCalled.countDown(); + } + + @Override + public void onFailure(Throwable t) { + synchronized (failures) { + failures.add(t); + } + failureCount.incrementAndGet(); + onFailureCalled.countDown(); + } + }, null); + + assertTrue(executionStarted.await(10, TimeUnit.SECONDS), "execution should have started"); + + // The watchdog gives up on the turn... + future.cancel(true); + // ...and only then does the pipeline finish anyway. + releaseExecution.countDown(); + + assertTrue(onFailureCalled.await(10, TimeUnit.SECONDS), + "a turn abandoned by the watchdog must be routed to onFailure, never onComplete"); + assertEquals(1, failureCount.get()); + synchronized (failures) { + assertInstanceOf(InterruptedException.class, failures.get(0)); + } + assertEquals(1, onCompleteCalled.getCount(), + "onComplete must not fire for an abandoned turn — it would persist stale state over a newer turn"); + } + + /** + * C9 — {@code onComplete} used to sit inside the {@code catch (Throwable)} that + * routes to {@code onFailure}. Any unchecked throw on the completion path + * therefore reported the turn as failed, and the coordinator's retry then + * re-executed a callable that had ALREADY run (duplicate LLM calls, duplicate + * tool side effects, duplicate cost). + */ + @Test + @Timeout(30) + @DisplayName("C9: a throwing completion callback is not reported as a failure") + void throwingCompletionCallbackIsNotRoutedToOnFailure() throws Exception { + AtomicInteger executions = new AtomicInteger(); + CountDownLatch onCompleteCalled = new CountDownLatch(1); + CountDownLatch onFailureCalled = new CountDownLatch(1); + + Future future = runtime.submitCallable( + () -> { + executions.incrementAndGet(); + return "ok"; + }, + new IRuntime.IFinishedExecution() { + @Override + public void onComplete(String result) { + onCompleteCalled.countDown(); + throw new IllegalStateException("completion callback blew up"); + } + + @Override + public void onFailure(Throwable t) { + onFailureCalled.countDown(); + } + }, null); + + assertTrue(onCompleteCalled.await(10, TimeUnit.SECONDS), "onComplete should have been invoked"); + // The Future settles only after the callback dispatch, so once it is done the + // decision about onFailure has already been made — no polling window needed. + assertEquals("ok", future.get(10, TimeUnit.SECONDS)); + assertEquals(1, onFailureCalled.getCount(), + "onFailure must not fire after the work already executed — callers read it as 'never ran' and re-execute"); + assertEquals(1, executions.get(), "the callable must be executed exactly once"); + } + + /** + * Pair test for C9: the one-shot gate must not swallow a genuine failure — a + * callable that throws still reaches onFailure exactly once. + */ + @Test + @Timeout(30) + @DisplayName("C9: a genuinely failing callable still reaches onFailure exactly once") + void failingCallableStillReportsFailureOnce() throws Exception { + CountDownLatch onFailureCalled = new CountDownLatch(1); + AtomicInteger failureCount = new AtomicInteger(); + + Future future = runtime.submitCallable( + () -> { + throw new IllegalStateException("boom"); + }, + new IRuntime.IFinishedExecution() { + @Override + public void onComplete(String result) { + fail("onComplete must not fire for a failing callable"); + } + + @Override + public void onFailure(Throwable t) { + failureCount.incrementAndGet(); + onFailureCalled.countDown(); + } + }, null); + + assertTrue(onFailureCalled.await(10, TimeUnit.SECONDS)); + assertNull(future.get(10, TimeUnit.SECONDS)); + assertEquals(1, failureCount.get()); + } + + /** + * The abandonment token must not fire for a turn that finished before the + * watchdog gave up — cancelling a completed Future is a no-op. + */ + @Test + @Timeout(30) + @DisplayName("cancel() after completion does not retroactively invalidate the completed turn") + void cancelAfterCompletionDoesNotAffectAlreadyDispatchedCallback() throws Exception { + CountDownLatch onCompleteCalled = new CountDownLatch(1); + CountDownLatch onFailureCalled = new CountDownLatch(1); + + Future future = runtime.submitCallable( + () -> "done", + new IRuntime.IFinishedExecution() { + @Override + public void onComplete(String result) { + onCompleteCalled.countDown(); + } + + @Override + public void onFailure(Throwable t) { + onFailureCalled.countDown(); + } + }, null); + + assertEquals("done", future.get(10, TimeUnit.SECONDS)); + assertTrue(onCompleteCalled.await(10, TimeUnit.SECONDS)); + + assertFalse(future.cancel(true), "cancelling a completed Future must report false"); + assertEquals(1, onFailureCalled.getCount(), "no callback may fire a second time"); + } + + /** + * Sanity check that the nested routing does not change the observable contract + * for nested work: it still completes, still reports through the callback, and + * still propagates its failures. + */ + @Test + @Timeout(30) + @DisplayName("nested submissions keep the normal callback contract") + void nestedSubmissionKeepsCallbackContract() throws Exception { + CountDownLatch nestedCompleted = new CountDownLatch(1); + + Future outer = runtime.submitCallable(() -> { + Future inner = runtime.submitCallable( + () -> "nested", + new IRuntime.IFinishedExecution() { + @Override + public void onComplete(String result) { + nestedCompleted.countDown(); + } + + @Override + public void onFailure(Throwable t) { + // not expected + } + }, null); + return inner.get(20, TimeUnit.SECONDS); + }, null); + + assertEquals("nested", outer.get(30, TimeUnit.SECONDS)); + assertTrue(nestedCompleted.await(10, TimeUnit.SECONDS)); + } + + /** + * Guards the marker's lifetime: it is scoped to the work BODY, so a completion + * callback that submits follow-up work (the coordinator scheduling the next + * turn) is treated as a top-level submission and keeps using the bounded pool. + */ + @Test + @Timeout(30) + @DisplayName("work submitted from a completion callback goes back to the bounded pool") + void submissionFromCompletionCallbackIsNotTreatedAsNested() throws Exception { + AtomicInteger poolSubmissions = new AtomicInteger(); + ManagedExecutor countingExecutor = Mockito.mock(ManagedExecutor.class); + when(countingExecutor.submit(any(Callable.class))).thenAnswer(inv -> { + poolSubmissions.incrementAndGet(); + return boundedPool.submit((Callable) inv.getArgument(0)); + }); + var executorField = BaseRuntime.class.getDeclaredField("executorService"); + executorField.setAccessible(true); + executorField.set(runtime, countingExecutor); + + CountDownLatch followUpDone = new CountDownLatch(1); + + runtime.submitCallable( + () -> "first", + new IRuntime.IFinishedExecution() { + @Override + public void onComplete(String result) { + runtime.submitCallable(() -> { + followUpDone.countDown(); + return null; + }, null); + } + + @Override + public void onFailure(Throwable t) { + // not expected + } + }, null); + + assertTrue(followUpDone.await(20, TimeUnit.SECONDS)); + assertEquals(2, poolSubmissions.get(), + "both the original submission and the one made from its completion callback belong on the bounded pool"); + } + + /** + * Documents the pre-fix failure mode explicitly: with a fully saturated pool, + * an inner task submitted onto that SAME pool can never be scheduled. The + * nested executor is what makes + * {@link #nestedSubmissionsDoNotStarveTheBoundedPool} pass, and this test + * proves the saturation is real rather than incidental. + */ + @Test + @Timeout(30) + @DisplayName("baseline: the bounded pool really is saturated by the waiting outer tasks") + void boundedPoolIsSaturatedByWaitingOuterTasks() throws Exception { + CyclicBarrier allRunning = new CyclicBarrier(POOL_SIZE); + CountDownLatch release = new CountDownLatch(1); + + for (int i = 0; i < POOL_SIZE; i++) { + boundedPool.submit(() -> { + allRunning.await(20, TimeUnit.SECONDS); + release.await(); + return null; + }); + } + + // With every pool thread parked, a direct pool submission cannot run. + Future starved = boundedPool.submit(() -> "never"); + assertThrows(TimeoutException.class, () -> starved.get(200, TimeUnit.MILLISECONDS)); + + // The runtime's nested executor is unaffected by that saturation. + release.countDown(); + } +} diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/ConversationCancelPersistenceTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/ConversationCancelPersistenceTest.java new file mode 100644 index 000000000..9fc7cd0bc --- /dev/null +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/ConversationCancelPersistenceTest.java @@ -0,0 +1,138 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.engine.runtime.internal; + +import ai.labs.eddi.configs.properties.IUserMemoryStore; +import ai.labs.eddi.configs.properties.model.Property; +import ai.labs.eddi.configs.properties.model.Property.Scope; +import ai.labs.eddi.configs.properties.model.UserMemoryEntry; +import ai.labs.eddi.engine.lifecycle.IConversation; +import ai.labs.eddi.engine.lifecycle.ILifecycleManager; +import ai.labs.eddi.engine.lifecycle.exceptions.ConversationStopException; +import ai.labs.eddi.engine.memory.ConversationMemory; +import ai.labs.eddi.engine.memory.IPropertiesHandler; +import ai.labs.eddi.engine.runtime.IExecutableWorkflow; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; + +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.*; + +/** + * F6 — a turn that is cancelled mid-flight must not commit its side effects. + *

      + * {@code ConversationService} discards the SNAPSHOT of a cancelled turn, but + * {@code Conversation#storePropertiesPermanently} writes straight to the + * user-memory store, outside that snapshot. So a client that disconnected (or a + * reviewer who hit {@code /cancel}) was told the turn was cancelled while the + * partial {@code longTerm} properties it had set were still upserted into + * persistent user memory — the exact inconsistency the ERROR path already + * guards against. + *

      + * The interleaving is forced with latches: the cancel flag is flipped by + * another thread strictly WHILE the pipeline is running, which is when a real + * cancel arrives (a REST/admin thread that is not serialized by the + * per-conversation coordinator). + */ +class ConversationCancelPersistenceTest { + + private ConversationMemory memory; + private IUserMemoryStore userMemoryStore; + private IPropertiesHandler propertiesHandler; + private ILifecycleManager lifecycleManager; + private IExecutableWorkflow workflow; + + @BeforeEach + void setUp() { + memory = new ConversationMemory("aabbccddeeff112233445566", "agent-1", 1, "user-1"); + userMemoryStore = mock(IUserMemoryStore.class); + propertiesHandler = mock(IPropertiesHandler.class); + when(propertiesHandler.getUserMemoryStore()).thenReturn(userMemoryStore); + + lifecycleManager = mock(ILifecycleManager.class); + workflow = mock(IExecutableWorkflow.class); + when(workflow.getWorkflowId()).thenReturn("wf-1"); + when(workflow.getLifecycleManager()).thenReturn(lifecycleManager); + } + + /** Mirrors {@code Agent#continueConversation}: a NEW Conversation per turn. */ + private Conversation nextTurn() { + return new Conversation(List.of(workflow), memory, propertiesHandler, + (IConversation.IConversationOutputRenderer) null); + } + + /** + * Wires the pipeline to behave exactly like the fixed {@code LifecycleManager} + * under a cancel: it runs, observes the flag, and aborts with + * {@link ConversationStopException}. The flag itself is set by a second thread + * while the pipeline is inside the workflow. + */ + private Thread cancelWhilePipelineRuns() throws Exception { + var pipelineEntered = new CountDownLatch(1); + var cancelApplied = new CountDownLatch(1); + + doAnswer(invocation -> { + pipelineEntered.countDown(); + assertTrue(cancelApplied.await(10, TimeUnit.SECONDS), "canceller thread did not run"); + throw new ConversationStopException(); + }).when(lifecycleManager).executeLifecycle(any(), any()); + + var canceller = new Thread(() -> { + try { + assertTrue(pipelineEntered.await(10, TimeUnit.SECONDS), "pipeline never started"); + memory.setCancelled(true); + cancelApplied.countDown(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }, "cancel-signal"); + canceller.start(); + return canceller; + } + + @Test + @Timeout(30) + @DisplayName("a turn cancelled mid-pipeline does not upsert the longTerm properties it set") + void cancelledTurnDoesNotPersistProperties() throws Exception { + Conversation conversation = nextTurn(); + // Set during the turn — not part of the constructor baseline, so an + // unguarded storePropertiesPermanently() would definitely write it. + memory.getConversationProperties() + .put("dietary_restriction", new Property("dietary_restriction", "vegan", Scope.longTerm)); + + var canceller = cancelWhilePipelineRuns(); + try { + conversation.say("I am vegan", new LinkedHashMap<>()); + } finally { + canceller.join(10_000); + } + + verify(userMemoryStore, never()).upsert(any(UserMemoryEntry.class)); + } + + @Test + @Timeout(30) + @DisplayName("control: the SAME stop path without a cancel still persists — the guard is the cancel, not the stop") + void stoppedButNotCancelledTurnStillPersists() throws Exception { + Conversation conversation = nextTurn(); + memory.getConversationProperties() + .put("dietary_restriction", new Property("dietary_restriction", "vegan", Scope.longTerm)); + + // Same abort as above (STOP_CONVERSATION), but nobody cancelled the turn. + doThrow(new ConversationStopException()).when(lifecycleManager).executeLifecycle(any(), any()); + + conversation.say("I am vegan", new LinkedHashMap<>()); + + verify(userMemoryStore, times(1)).upsert(any(UserMemoryEntry.class)); + } +} diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceExtendedTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceExtendedTest.java index 204c46296..bd5bd7b18 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceExtendedTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceExtendedTest.java @@ -4,6 +4,7 @@ */ package ai.labs.eddi.engine.runtime.internal; +import ai.labs.eddi.configs.agents.IAgentStore; import ai.labs.eddi.configs.agents.model.AgentConfiguration; import ai.labs.eddi.configs.properties.IUserMemoryStore; import ai.labs.eddi.configs.properties.model.Property.Visibility; @@ -40,6 +41,8 @@ class DreamServiceExtendedTest { @Mock private IUserMemoryStore store; @Mock + private IAgentStore agentStore; + @Mock private SummarizationService summarizationService; private DreamService dreamService; @@ -50,7 +53,7 @@ class DreamServiceExtendedTest { void setUp() { MockitoAnnotations.openMocks(this); meterRegistry = new SimpleMeterRegistry(); - dreamService = new DreamService(store, summarizationService, meterRegistry, new ObjectMapper()); + dreamService = new DreamService(store, agentStore, summarizationService, meterRegistry, new ObjectMapper()); dreamService.initMetrics(); dreamConfig = new AgentConfiguration.DreamConfig(); @@ -261,7 +264,7 @@ void costCeilingStopsProcessing() throws Exception { // Return a result with high token count to exceed cost ceiling String llmResponse = "[{\"key\": \"s\", \"value\": \"v\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 5000, 5000)); var result = dreamService.process("user-1", dreamConfig); @@ -289,7 +292,7 @@ void consolidatedCapped() throws Exception { // LLM returns 3 entries but target is 1 → should cap to 1 String llmResponse = "[{\"key\": \"s1\", \"value\": \"v1\"}, {\"key\": \"s2\", \"value\": \"v2\"}, {\"key\": \"s3\", \"value\": \"v3\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 100, 50)); var result = dreamService.process("user-1", dreamConfig); @@ -318,7 +321,7 @@ void tooManyEntriesSkipped() throws Exception { // LLM returns 3 entries = same as original → skip String llmResponse = "[{\"key\": \"s1\", \"value\": \"v1\"}, {\"key\": \"s2\", \"value\": \"v2\"}, {\"key\": \"s3\", \"value\": \"v3\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 100, 50)); var result = dreamService.process("user-1", dreamConfig); @@ -332,36 +335,71 @@ void tooManyEntriesSkipped() throws Exception { // ==================== @Nested - @DisplayName("process — multi-agent visibility upgrade") + @DisplayName("process — multi-agent visibility (finding G8)") class MultiAgentVisibilityTests { @Test - @DisplayName("self-scoped entries from multiple agents upgrade to global") - void selfUpgradedToGlobal() throws Exception { + @DisplayName("self-scoped entries from multiple agents are never merged into a global entry") + void selfNeverUpgradedToGlobal() throws Exception { dreamConfig.setSummarizeGroupBy("all"); dreamConfig.setPreserveAgentProvenance(false); dreamConfig.setSummarizeMinEntries(2); - // Entries from 2 different agents, both self-scoped + // Two self-scoped entries per agent, from 2 different agents var entries = new ArrayList<>(List.of( new UserMemoryEntry("id1", "user-1", "k1", "v1", "fact", Visibility.self, "agent-1", null, "source", false, 0, Instant.now().minusSeconds(100), Instant.now()), new UserMemoryEntry("id2", "user-1", "k2", "v2", "fact", + Visibility.self, "agent-1", null, "source", false, 0, + Instant.now().minusSeconds(90), Instant.now()), + new UserMemoryEntry("id3", "user-1", "k3", "v3", "fact", Visibility.self, "agent-2", null, "source", false, 0, - Instant.now().minusSeconds(50), Instant.now()))); + Instant.now().minusSeconds(50), Instant.now()), + new UserMemoryEntry("id4", "user-1", "k4", "v4", "fact", + Visibility.self, "agent-2", null, "source", false, 0, + Instant.now().minusSeconds(40), Instant.now()))); when(store.getAllEntries("user-1")).thenReturn(entries); when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id"); String llmResponse = "[{\"key\": \"consolidated\", \"value\": \"merged\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 0, 0)); var result = dreamService.process("user-1", dreamConfig); assertTrue(result.isSuccess()); - // Verify upserted entry has global visibility - verify(store).upsert(argThat(entry -> entry.visibility() == Visibility.global)); + // One self-scoped entry per contributing agent; nothing widened + verify(store, never()).upsert(argThat(entry -> entry.visibility() != Visibility.self)); + verify(store).upsert(argThat(entry -> "agent-1".equals(entry.sourceAgentId()))); + verify(store).upsert(argThat(entry -> "agent-2".equals(entry.sourceAgentId()))); + } + + @Test + @DisplayName("group-scoped entries from multiple agents may still be merged") + void groupScopedStillMerged() throws Exception { + dreamConfig.setSummarizeGroupBy("all"); + dreamConfig.setPreserveAgentProvenance(false); + dreamConfig.setSummarizeMinEntries(2); + + var entries = new ArrayList<>(List.of( + new UserMemoryEntry("id1", "user-1", "k1", "v1", "fact", + Visibility.group, "agent-1", List.of("team-a"), "source", false, 0, + Instant.now().minusSeconds(100), Instant.now()), + new UserMemoryEntry("id2", "user-1", "k2", "v2", "fact", + Visibility.group, "agent-2", List.of("team-a"), "source", false, 0, + Instant.now().minusSeconds(50), Instant.now()))); + when(store.getAllEntries("user-1")).thenReturn(entries); + when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id"); + + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenReturn(new SummarizationResult("[{\"key\": \"c\", \"value\": \"m\"}]", 0, 0)); + + var result = dreamService.process("user-1", dreamConfig); + assertTrue(result.isSuccess()); + + // Already shared → a single merged entry, still group-scoped + verify(store, times(1)).upsert(argThat(entry -> entry.visibility() == Visibility.group)); } } @@ -383,14 +421,14 @@ void groupByAll() throws Exception { when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id"); String llmResponse = "[{\"key\": \"s\", \"value\": \"v\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 0, 0)); var result = dreamService.process("user-1", dreamConfig); assertTrue(result.isSuccess()); // One LLM call for the "all" group verify(summarizationService, times(1)) - .summarizeWithUsage(anyString(), anyString(), anyString(), anyString()); + .summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); } @Test @@ -417,14 +455,14 @@ void groupByCategoryWithProvenance() throws Exception { when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id"); String llmResponse = "[{\"key\": \"s\", \"value\": \"v\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 0, 0)); var result = dreamService.process("user-1", dreamConfig); assertTrue(result.isSuccess()); // Two sub-groups: fact:agent-1 and fact:agent-2 verify(summarizationService, times(2)) - .summarizeWithUsage(anyString(), anyString(), anyString(), anyString()); + .summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); } @Test @@ -445,13 +483,13 @@ void nullCategoryDefaultsFact() throws Exception { when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id"); String llmResponse = "[{\"key\": \"s\", \"value\": \"v\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 0, 0)); var result = dreamService.process("user-1", dreamConfig); assertTrue(result.isSuccess()); verify(summarizationService, times(1)) - .summarizeWithUsage(anyString(), anyString(), anyString(), anyString()); + .summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); } } @@ -482,14 +520,15 @@ class DreamResultTests { @Test @DisplayName("isSuccess returns true when error is null") void isSuccessTrue() { - var result = new DreamService.DreamResult("user1", 5, 2, 3, 100L, null); + var result = new DreamService.DreamResult("user1", 5, 2, 3, 100L, 0.25, null); assertTrue(result.isSuccess()); + assertEquals(0.25, result.estimatedCostUsd(), 1e-9); } @Test @DisplayName("isSuccess returns false when error is present") void isSuccessFalse() { - var result = new DreamService.DreamResult("user1", 0, 0, 0, 50L, "failed"); + var result = new DreamService.DreamResult("user1", 0, 0, 0, 50L, 0.0, "failed"); assertFalse(result.isSuccess()); } } diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java index c6dd8dc21..6b07482e1 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java @@ -4,10 +4,12 @@ */ package ai.labs.eddi.engine.runtime.internal; +import ai.labs.eddi.configs.agents.IAgentStore; import ai.labs.eddi.configs.agents.model.AgentConfiguration; import ai.labs.eddi.configs.properties.IUserMemoryStore; import ai.labs.eddi.configs.properties.model.Property.Visibility; import ai.labs.eddi.configs.properties.model.UserMemoryEntry; +import ai.labs.eddi.datastore.IResourceStore; import ai.labs.eddi.modules.llm.impl.SummarizationService; import io.micrometer.core.instrument.simple.SimpleMeterRegistry; import org.junit.jupiter.api.BeforeEach; @@ -16,6 +18,8 @@ import java.time.Duration; import java.time.Instant; import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; import static org.junit.jupiter.api.Assertions.*; import static org.mockito.Mockito.*; @@ -26,6 +30,7 @@ class DreamServiceTest { private IUserMemoryStore store; + private IAgentStore agentStore; private SummarizationService summarizationService; private DreamService dreamService; private AgentConfiguration.DreamConfig dreamConfig; @@ -33,8 +38,9 @@ class DreamServiceTest { @BeforeEach void setUp() { store = mock(IUserMemoryStore.class); + agentStore = mock(IAgentStore.class); summarizationService = mock(SummarizationService.class); - dreamService = new DreamService(store, summarizationService, new SimpleMeterRegistry(), + dreamService = new DreamService(store, agentStore, summarizationService, new SimpleMeterRegistry(), new com.fasterxml.jackson.databind.ObjectMapper()); dreamService.initMetrics(); @@ -221,7 +227,7 @@ void summarize_aboveThreshold_consolidates() throws Exception { String llmResponse = "[{\"key\": \"summary-1\", \"value\": \"combined fact 1\"}, " + "{\"key\": \"summary-2\", \"value\": \"combined fact 2\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); var result = dreamService.process("user-1", dreamConfig); @@ -237,7 +243,7 @@ void summarize_llmReturnsEmpty_preservesEntries() throws Exception { enableSummarization(); var entries = makeEntries(6, "fact", "agent-1"); when(store.getAllEntries("user-1")).thenReturn(entries); - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult("")); var result = dreamService.process("user-1", dreamConfig); @@ -253,7 +259,7 @@ void summarize_llmReturnsGarbage_preservesEntries() throws Exception { enableSummarization(); var entries = makeEntries(6, "fact", "agent-1"); when(store.getAllEntries("user-1")).thenReturn(entries); - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult("I can't do that, sorry!")); var result = dreamService.process("user-1", dreamConfig); @@ -270,7 +276,7 @@ void summarize_llmReturnsMarkdownFences_parsesCorrectly() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); String llmResponse = "```json\n[{\"key\": \"s1\", \"value\": \"v1\"}, {\"key\": \"s2\", \"value\": \"v2\"}]\n```"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); var result = dreamService.process("user-1", dreamConfig); @@ -290,7 +296,7 @@ void summarize_llmReturnsMoreThanOriginals_skips() throws Exception { "{\"key\":\"a\",\"value\":\"1\"},{\"key\":\"b\",\"value\":\"2\"}," + "{\"key\":\"c\",\"value\":\"3\"},{\"key\":\"d\",\"value\":\"4\"}," + "{\"key\":\"e\",\"value\":\"5\"},{\"key\":\"f\",\"value\":\"6\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); var result = dreamService.process("user-1", dreamConfig); @@ -307,7 +313,7 @@ void summarize_insertFails_preservesEntries() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); String llmResponse = "[{\"key\": \"s1\", \"value\": \"v1\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); doThrow(new RuntimeException("DB write failed")).when(store).upsert(any(UserMemoryEntry.class)); @@ -318,10 +324,16 @@ void summarize_insertFails_preservesEntries() throws Exception { verify(store, never()).deleteEntry(anyString()); // originals preserved } + /** + * Finding I1: the legacy {@code maxSummarizationCalls} count is no longer a + * ceiling — the dollar budget {@code maxCostPerRun} is. A config still setting + * the call count must not cap a run that is well inside its budget. + */ @Test - void summarize_callLimitReached_stopsEarly() throws Exception { + void summarize_legacyCallCount_isNotACeiling() throws Exception { enableSummarization(); - dreamConfig.setMaxSummarizationCalls(1); + dreamConfig.setMaxSummarizationCalls(1); // legacy, ignored + dreamConfig.setMaxCostPerRun(1.00); // generous dollar budget dreamConfig.setSummarizeGroupBy("category"); Instant now = Instant.now(); @@ -338,14 +350,36 @@ void summarize_callLimitReached_stopsEarly() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); String llmResponse = "[{\"key\": \"s1\", \"value\": \"v1\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) - .thenReturn(llmResult(llmResponse)); + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenReturn(llmResult(llmResponse, 10, 5)); // ~$0.00015 per call var result = dreamService.process("user-1", dreamConfig); assertTrue(result.isSuccess()); - // Only 1 LLM call should have been made (limit=1) - verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString()); + // Both category groups are consolidated despite maxSummarizationCalls=1 + verify(summarizationService, times(2)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); + assertEquals(10, result.entriesSummarized()); // (6-1) + (6-1) + } + + /** + * Finding I1: Dream has no parent LLM task to inherit credentials from, so the + * agent's {@code dream.parameters} block must reach the summarizer — otherwise + * it authenticates with nothing and every cycle fails. + */ + @Test + void summarize_passesConfiguredModelParametersToSummarizer() throws Exception { + enableSummarization(); + var parameters = Map.of("apiKey", "${vault:dream-key}", "baseUrl", "https://llm.example/v1"); + dreamConfig.setParameters(parameters); + + when(store.getAllEntries("user-1")).thenReturn(makeEntries(6, "fact", "agent-1")); + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenReturn(llmResult("[{\"key\": \"s\", \"value\": \"v\"}]")); + + dreamService.process("user-1", dreamConfig); + + verify(summarizationService).summarizeWithUsage(anyString(), anyString(), + eq(dreamConfig.getLlmProvider()), eq(dreamConfig.getLlmModel()), eq(parameters)); } @Test @@ -366,7 +400,7 @@ void summarize_groupByAll_singleGroup() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); String llmResponse = "[{\"key\": \"s1\", \"value\": \"v1\"}, {\"key\": \"s2\", \"value\": \"v2\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); var result = dreamService.process("user-1", dreamConfig); @@ -374,7 +408,7 @@ void summarize_groupByAll_singleGroup() throws Exception { assertTrue(result.isSuccess()); // All 6 entries in one group → 2 consolidated → 4 reduced assertEquals(4, result.entriesSummarized()); - verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString()); + verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); } @Test @@ -398,7 +432,7 @@ void summarize_preserveAgentProvenance_subGroups() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); String llmResponse = "[{\"key\": \"s\", \"value\": \"v\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); var result = dreamService.process("user-1", dreamConfig); @@ -406,7 +440,7 @@ void summarize_preserveAgentProvenance_subGroups() throws Exception { assertTrue(result.isSuccess()); // Two separate groups, each 3→1 = 2 reduced per group = 4 total assertEquals(4, result.entriesSummarized()); - verify(summarizationService, times(2)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString()); + verify(summarizationService, times(2)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); } @Test @@ -417,12 +451,12 @@ void summarize_customPrompt_passedToService() throws Exception { var entries = makeEntries(6, "fact", "agent-1"); when(store.getAllEntries("user-1")).thenReturn(entries); - when(summarizationService.summarizeWithUsage(anyString(), eq(customPrompt), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), eq(customPrompt), anyString(), anyString(), any())) .thenReturn(llmResult("[{\"key\": \"s\", \"value\": \"v\"}]")); dreamService.process("user-1", dreamConfig); - verify(summarizationService).summarizeWithUsage(anyString(), eq(customPrompt), anyString(), anyString()); + verify(summarizationService).summarizeWithUsage(anyString(), eq(customPrompt), anyString(), anyString(), any()); } @Test @@ -443,7 +477,7 @@ void summarize_mostRestrictiveVisibility_applied() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); String llmResponse = "[{\"key\": \"s1\", \"value\": \"v1\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); dreamService.process("user-1", dreamConfig); @@ -569,7 +603,7 @@ void summarize_llmReturnsTooMany_cappedToTarget() throws Exception { // LLM returns 4 (< 8 originals, but > 2 target) → capped to 2 String llmResponse = "[{\"key\":\"a\",\"value\":\"1\"},{\"key\":\"b\",\"value\":\"2\"}," + "{\"key\":\"c\",\"value\":\"3\"},{\"key\":\"d\",\"value\":\"4\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); var result = dreamService.process("user-1", dreamConfig); @@ -587,7 +621,7 @@ void summarize_deletePartiallyFails_logsAndContinues() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); String llmResponse = "[{\"key\": \"s1\", \"value\": \"v1\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); // First 3 deletes succeed, last 3 fail doNothing().doNothing().doNothing() @@ -602,22 +636,59 @@ void summarize_deletePartiallyFails_logsAndContinues() throws Exception { verify(store, times(6)).deleteEntry(anyString()); } + /** + * Finding I1: an LLM failure used to be a swallowed WARN, so a Dream cycle that + * could never consolidate anything still reported success — the same defect F13 + * fixed for the conversation summarizer. The failure must now surface on the + * result so the schedule fire is marked FAILED (and retries/dead-letters). + */ @Test - void summarize_llmThrows_preservesEntriesAndContinues() throws Exception { + void summarize_llmThrows_preservesEntriesAndFailsTheCycle() throws Exception { enableSummarization(); var entries = makeEntries(6, "fact", "agent-1"); when(store.getAllEntries("user-1")).thenReturn(entries); - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) - .thenThrow(new RuntimeException("LLM provider down")); + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenThrow(new RuntimeException("401 Unauthorized")); var result = dreamService.process("user-1", dreamConfig); - assertTrue(result.isSuccess()); // dream cycle succeeds even though LLM failed + assertFalse(result.isSuccess()); + assertNotNull(result.error()); + assertTrue(result.error().contains("401 Unauthorized"), "cause must be reported, got: " + result.error()); assertEquals(0, result.entriesSummarized()); verify(store, never()).upsert(any(UserMemoryEntry.class)); verify(store, never()).deleteEntry(anyString()); } + /** + * A failing LLM is a configuration fault that would repeat for every remaining + * group — the phase aborts rather than burning the budget group by group. + */ + @Test + void summarize_llmThrows_abortsRemainingGroups() throws Exception { + enableSummarization(); + dreamConfig.setSummarizeGroupBy("category"); + + Instant now = Instant.now(); + var entries = new java.util.ArrayList(); + for (int i = 0; i < 6; i++) { + entries.add(new UserMemoryEntry("f-" + i, "user-1", "fk-" + i, "fv-" + i, + "fact", Visibility.self, "agent-1", List.of(), "conv-1", false, 0, now, now)); + } + for (int i = 0; i < 6; i++) { + entries.add(new UserMemoryEntry("p-" + i, "user-1", "pk-" + i, "pv-" + i, + "preference", Visibility.self, "agent-1", List.of(), "conv-1", false, 0, now, now)); + } + when(store.getAllEntries("user-1")).thenReturn(entries); + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenThrow(new RuntimeException("connection refused")); + + var result = dreamService.process("user-1", dreamConfig); + + assertFalse(result.isSuccess()); + verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); + } + @Test void summarize_afterPruning_reloadsEntries() throws Exception { enableSummarization(); @@ -639,7 +710,7 @@ void summarize_afterPruning_reloadsEntries() throws Exception { when(store.getAllEntries("user-1")).thenReturn(initialEntries).thenReturn(afterPruneEntries); String llmResponse = "[{\"key\": \"s\", \"value\": \"v\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); var result = dreamService.process("user-1", dreamConfig); @@ -717,46 +788,98 @@ void summarize_costCeilingReached_stopsEarly() throws Exception { // Return result with high token usage (1000 tokens → $0.01 > $0.005 ceiling) String llmResponse = "[{\"key\": \"s1\", \"value\": \"v1\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse, 800, 200)); var result = dreamService.process("user-1", dreamConfig); assertTrue(result.isSuccess()); // Only 1 LLM call should have been made — cost ceiling stops second group - verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString()); + verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); } // === PR Review Fixes: Copilot + CodeRabbit findings === + /** + * Finding G8 — privacy boundary. Two agents' {@code self}-scoped memories used + * to be merged into ONE entry whose visibility was upgraded to {@code global}, + * i.e. readable by every agent. Since {@code summarizeGroupBy} defaults to + * "category" and {@code preserveAgentProvenance} to false, that was the default + * path. Consolidation must now produce one {@code self} entry per contributing + * agent, and never a {@code global} one. + */ @Test - void summarize_multiAgentSelfScope_upgradesVisibility() throws Exception { + void summarize_multiAgentSelfScope_neverWidensVisibility() throws Exception { enableSummarization(); dreamConfig.setPreserveAgentProvenance(false); - dreamConfig.setSummarizeMinEntries(4); + dreamConfig.setSummarizeGroupBy("category"); + dreamConfig.setSummarizeMinEntries(2); Instant now = Instant.now(); var entries = new java.util.ArrayList(); - for (int i = 0; i < 2; i++) { - entries.add(new UserMemoryEntry("a1-" + i, "user-1", "k1-" + i, "v", + for (int i = 0; i < 3; i++) { + entries.add(new UserMemoryEntry("a1-" + i, "user-1", "k1-" + i, "secret of agent-1", "fact", Visibility.self, "agent-1", List.of(), "conv-1", false, 0, now, now)); } - for (int i = 0; i < 2; i++) { - entries.add(new UserMemoryEntry("a2-" + i, "user-1", "k2-" + i, "v", + for (int i = 0; i < 3; i++) { + entries.add(new UserMemoryEntry("a2-" + i, "user-1", "k2-" + i, "secret of agent-2", "fact", Visibility.self, "agent-2", List.of(), "conv-1", false, 0, now, now)); } when(store.getAllEntries("user-1")).thenReturn(entries); when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id-1"); String llmResponse = "[{\"key\": \"s\", \"value\": \"v\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); dreamService.process("user-1", dreamConfig); + // One consolidated entry per contributing agent — never a merged one var captor = org.mockito.ArgumentCaptor.forClass(UserMemoryEntry.class); - verify(store).upsert(captor.capture()); - assertEquals(Visibility.global, captor.getValue().visibility()); + verify(store, times(2)).upsert(captor.capture()); + var written = captor.getAllValues(); + assertTrue(written.stream().allMatch(e -> e.visibility() == Visibility.self), + "self-scoped memories must never be widened, got: " + + written.stream().map(UserMemoryEntry::visibility).toList()); + assertEquals(List.of("agent-1", "agent-2"), + written.stream().map(UserMemoryEntry::sourceAgentId).sorted().toList()); + } + + /** + * Finding G8, mixed case: a group holding one agent's {@code self} memories and + * another agent's {@code global} ones must still keep the private half private. + */ + @Test + void summarize_selfAndGlobalAcrossAgents_selfHalfStaysSelf() throws Exception { + enableSummarization(); + dreamConfig.setPreserveAgentProvenance(false); + dreamConfig.setSummarizeGroupBy("all"); + dreamConfig.setSummarizeMinEntries(2); + + Instant now = Instant.now(); + var entries = new java.util.ArrayList(); + for (int i = 0; i < 2; i++) { + entries.add(new UserMemoryEntry("priv-" + i, "user-1", "pk-" + i, "private", + "fact", Visibility.self, "agent-1", List.of(), "conv-1", false, 0, now, now)); + } + for (int i = 0; i < 2; i++) { + entries.add(new UserMemoryEntry("pub-" + i, "user-1", "gk-" + i, "shared", + "fact", Visibility.global, "agent-2", List.of(), "conv-1", false, 0, now, now)); + } + when(store.getAllEntries("user-1")).thenReturn(entries); + when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id"); + + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenReturn(llmResult("[{\"key\": \"s\", \"value\": \"v\"}]")); + + dreamService.process("user-1", dreamConfig); + + var captor = org.mockito.ArgumentCaptor.forClass(UserMemoryEntry.class); + verify(store, times(2)).upsert(captor.capture()); + var byAgent = captor.getAllValues().stream() + .collect(Collectors.toMap(UserMemoryEntry::sourceAgentId, UserMemoryEntry::visibility)); + assertEquals(Visibility.self, byAgent.get("agent-1")); + assertEquals(Visibility.global, byAgent.get("agent-2")); } @Test @@ -774,7 +897,7 @@ void summarize_preservesGroupIds() throws Exception { when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id-1"); String llmResponse = "[{\"key\": \"s\", \"value\": \"v\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); dreamService.process("user-1", dreamConfig); @@ -800,13 +923,13 @@ void summarize_nullCategory_defaultsToFact() throws Exception { when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id"); String llmResponse = "[{\"key\": \"s\", \"value\": \"v\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); var result = dreamService.process("user-1", dreamConfig); assertTrue(result.isSuccess()); - verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString()); + verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); } @Test @@ -850,7 +973,7 @@ void summarize_partialInsertFails_rollsBack() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); String llmResponse = "[{\"key\": \"s1\", \"value\": \"v1\"}, {\"key\": \"s2\", \"value\": \"v2\"}]"; - when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString())) + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); // First upsert succeeds, second throws @@ -881,4 +1004,137 @@ void setSummarizeTargetEntries_rejectsNegative() { assertThrows(IllegalArgumentException.class, () -> config.setSummarizeTargetEntries(-1)); } + + // === Finding I1: schedule wiring (processScheduledFire) === + + @Test + void isDreamSchedule_recognisesOnlyTheDreamMarker() { + assertTrue(DreamService.isDreamSchedule( + Map.of(DreamService.METADATA_TYPE_KEY, DreamService.METADATA_TYPE_CONSOLIDATION))); + assertFalse(DreamService.isDreamSchedule(null)); + assertFalse(DreamService.isDreamSchedule(Map.of())); + assertFalse(DreamService.isDreamSchedule(Map.of("hitlType", "hitl_timeout"))); + assertFalse(DreamService.isDreamSchedule(Map.of(DreamService.METADATA_TYPE_KEY, "something_else"))); + } + + @Test + void processScheduledFire_resolvesDreamConfigFromAgentAndRunsCycle() throws Exception { + var dream = new AgentConfiguration.DreamConfig(); + dream.setEnabled(true); + dream.setPruneStaleAfterDays(30); + dream.setDetectContradictions(false); + dream.setSummarizeInteractions(false); + + var memoryConfig = new AgentConfiguration.UserMemoryConfig(); + memoryConfig.setDream(dream); + var agentConfiguration = new AgentConfiguration(); + agentConfiguration.setUserMemoryConfig(memoryConfig); + when(agentStore.read("agent-1", 7)).thenReturn(agentConfiguration); + + Instant stale = Instant.now().minus(Duration.ofDays(60)); + when(store.getAllEntries("user-1")).thenReturn(List.of( + new UserMemoryEntry("1", "user-1", "old", "v", "fact", Visibility.self, "agent-1", List.of(), "conv-1", false, 0, stale, stale))) + .thenReturn(List.of()); + + var result = dreamService.processScheduledFire("agent-1", 7, "user-1"); + + assertTrue(result.isSuccess(), "expected success, got: " + result.error()); + assertEquals(1, result.entriesPruned()); + verify(store).deleteEntry("1"); + } + + @Test + void processScheduledFire_versionZero_resolvesLatestAgentVersion() throws Exception { + var dream = new AgentConfiguration.DreamConfig(); + dream.setEnabled(true); + dream.setPruneStaleAfterDays(0); + dream.setDetectContradictions(false); + dream.setSummarizeInteractions(false); + + var memoryConfig = new AgentConfiguration.UserMemoryConfig(); + memoryConfig.setDream(dream); + var agentConfiguration = new AgentConfiguration(); + agentConfiguration.setUserMemoryConfig(memoryConfig); + + when(agentStore.getCurrentResourceId("agent-1")).thenReturn(resourceId("agent-1", 4)); + when(agentStore.read("agent-1", 4)).thenReturn(agentConfiguration); + when(store.getAllEntries("user-1")).thenReturn(List.of()); + + var result = dreamService.processScheduledFire("agent-1", 0, "user-1"); + + assertTrue(result.isSuccess(), "expected success, got: " + result.error()); + verify(agentStore).read("agent-1", 4); + } + + @Test + void processScheduledFire_missingUserId_failsLoudlyWithoutTouchingMemory() throws Exception { + var result = dreamService.processScheduledFire("agent-1", 1, null); + + assertFalse(result.isSuccess()); + assertTrue(result.error().contains("userId"), "error must name the missing field, got: " + result.error()); + verifyNoInteractions(store); + verifyNoInteractions(agentStore); + } + + @Test + void processScheduledFire_schedulerPlaceholderUserId_failsLoudly() throws Exception { + // The schedule REST surface defaults userId to "system:scheduler"; running + // Dream under it would consolidate an empty memory set and look successful. + var result = dreamService.processScheduledFire("agent-1", 1, "system:scheduler"); + + assertFalse(result.isSuccess()); + assertTrue(result.error().contains("system:scheduler"), "got: " + result.error()); + // Rejected before any lookup — not merely failing later for another reason + verifyNoInteractions(store); + verifyNoInteractions(agentStore); + } + + @Test + void processScheduledFire_dreamDisabledOnAgent_failsLoudly() throws Exception { + var memoryConfig = new AgentConfiguration.UserMemoryConfig(); + memoryConfig.getDream().setEnabled(false); + var agentConfiguration = new AgentConfiguration(); + agentConfiguration.setUserMemoryConfig(memoryConfig); + when(agentStore.read("agent-1", 1)).thenReturn(agentConfiguration); + + var result = dreamService.processScheduledFire("agent-1", 1, "user-1"); + + assertFalse(result.isSuccess()); + assertTrue(result.error().contains("dream consolidation disabled"), "got: " + result.error()); + verifyNoInteractions(store); + } + + @Test + void processScheduledFire_agentNotFound_failsLoudly() throws Exception { + when(agentStore.read("agent-1", 1)).thenThrow(new IResourceStore.ResourceNotFoundException("no such agent")); + + var result = dreamService.processScheduledFire("agent-1", 1, "user-1"); + + assertFalse(result.isSuccess()); + assertTrue(result.error().contains("no such agent"), "got: " + result.error()); + verifyNoInteractions(store); + } + + @Test + void processScheduledFire_missingAgentId_failsLoudly() throws Exception { + var result = dreamService.processScheduledFire(" ", 1, "user-1"); + + assertFalse(result.isSuccess()); + assertTrue(result.error().contains("agentId"), "got: " + result.error()); + verifyNoInteractions(store); + } + + private static IResourceStore.IResourceId resourceId(String id, int version) { + return new IResourceStore.IResourceId() { + @Override + public String getId() { + return id; + } + + @Override + public Integer getVersion() { + return version; + } + }; + } } diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownServiceTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownServiceTest.java new file mode 100644 index 000000000..a94faa826 --- /dev/null +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownServiceTest.java @@ -0,0 +1,177 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.engine.runtime.internal; + +import ai.labs.eddi.engine.runtime.IConversationCoordinator; +import io.quarkus.runtime.ShutdownEvent; +import org.eclipse.microprofile.health.HealthCheckResponse; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.jupiter.api.Assertions.*; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +/** + * B3 — graceful shutdown: readiness flip, accept gate, bounded drain. + * + *

      + * The drain is driven by a fake coordinator whose reported depth is controlled + * by latches, so "SIGTERM arrives while a turn is running" is deterministic + * rather than timing-dependent. + *

      + */ +class GracefulShutdownServiceTest { + + private static GracefulShutdownService service(IConversationCoordinator coordinator, long drainTimeoutMillis) { + // no readiness grace + a tight poll interval: the tests control progress with + // latches, not with wall-clock waits. + return new GracefulShutdownService(coordinator, drainTimeoutMillis, 0L, 1L); + } + + @Test + @DisplayName("starts accepting work — isShuttingDown is false before any shutdown signal") + void notShuttingDownInitially() { + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + assertFalse(service(coordinator, 1_000).isShuttingDown()); + } + + /** + * The ACCEPT gate must flip before the drain starts — otherwise the drain keeps + * chasing turns that are still being admitted. + */ + @Test + @Timeout(30) + @DisplayName("SIGTERM during an active turn: the turn completes, and new turns are rejected while it drains") + void drainWaitsForTheActiveTurnAndRejectsNewOnesMeanwhile() throws Exception { + // "one turn in flight" until the worker below finishes it. + AtomicInteger inFlight = new AtomicInteger(1); + CountDownLatch drainObservedTheActiveTurn = new CountDownLatch(1); + AtomicBoolean gateWasClosedWhileTurnRan = new AtomicBoolean(false); + CountDownLatch turnFinished = new CountDownLatch(1); + + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + GracefulShutdownService shutdownService = service(coordinator, 20_000); + + when(coordinator.getQueueDepths()).thenAnswer(inv -> { + int remaining = inFlight.get(); + if (remaining > 0) { + // The gate MUST already be closed the very first time the drain looks. + gateWasClosedWhileTurnRan.set(shutdownService.isShuttingDown()); + drainObservedTheActiveTurn.countDown(); + } + return remaining > 0 ? Map.of("conv-active", remaining) : Map.of(); + }); + + // The "active turn": finishes only after the drain has observed it. + Thread activeTurn = new Thread(() -> { + try { + assertTrue(drainObservedTheActiveTurn.await(20, TimeUnit.SECONDS)); + // countDown FIRST: the drain may observe the depth drop the instant + // inFlight flips, and the assertion below must not race it. + turnFinished.countDown(); + inFlight.set(0); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }, "active-turn"); + activeTurn.start(); + + boolean drained = shutdownService.drain(); + + assertTrue(drained, "the drain must wait for the in-flight turn instead of dropping it"); + assertEquals(0, turnFinished.getCount(), "the drain must not return before the active turn finished"); + assertTrue(gateWasClosedWhileTurnRan.get(), + "new turns must already be rejected while the drain is still waiting for the in-flight one"); + assertTrue(shutdownService.isShuttingDown()); + + activeTurn.join(TimeUnit.SECONDS.toMillis(10)); + } + + /** + * The wait must be hard bounded — a hung pipeline cannot be allowed to prevent + * the process from exiting. + */ + @Test + @Timeout(30) + @DisplayName("a turn that never finishes does not block shutdown forever") + void drainIsBounded() { + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + when(coordinator.getQueueDepths()).thenReturn(Map.of("conv-hung", 1)); + + GracefulShutdownService shutdownService = service(coordinator, 150); + + long start = System.nanoTime(); + boolean drained = shutdownService.drain(); + long elapsedMillis = (System.nanoTime() - start) / 1_000_000L; + + assertFalse(drained, "a drain that times out must report failure"); + assertTrue(shutdownService.isShuttingDown(), "the accept gate stays closed even when the drain times out"); + assertTrue(elapsedMillis < 20_000, "the drain must be bounded, took " + elapsedMillis + " ms"); + } + + @Test + @Timeout(30) + @DisplayName("nothing in flight — drain returns immediately and still closes the gate") + void drainWithEmptyQueuesCompletesImmediately() { + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + when(coordinator.getQueueDepths()).thenReturn(Map.of()); + + GracefulShutdownService shutdownService = service(coordinator, 20_000); + + assertTrue(shutdownService.drain()); + assertTrue(shutdownService.isShuttingDown()); + } + + @Test + @Timeout(30) + @DisplayName("a coordinator that cannot report its depth does not block the exit") + void drainToleratesCoordinatorFailure() { + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + when(coordinator.getQueueDepths()).thenThrow(new IllegalStateException("bus disconnected")); + + GracefulShutdownService shutdownService = service(coordinator, 20_000); + + assertTrue(shutdownService.drain()); + assertTrue(shutdownService.isShuttingDown()); + } + + @Test + @Timeout(30) + @DisplayName("the ShutdownEvent observer runs the drain") + void shutdownEventTriggersTheDrain() { + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + when(coordinator.getQueueDepths()).thenReturn(Map.of()); + + GracefulShutdownService shutdownService = service(coordinator, 20_000); + shutdownService.onShutdown(new ShutdownEvent()); + + assertTrue(shutdownService.isShuttingDown()); + } + + @Test + @DisplayName("readiness reports UP before shutdown and DOWN afterwards") + void readinessFlipsOnShutdown() { + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + when(coordinator.getQueueDepths()).thenReturn(Map.of()); + + GracefulShutdownService shutdownService = service(coordinator, 1_000); + var healthCheck = new ShutdownReadinessHealthCheck(shutdownService); + + assertEquals(HealthCheckResponse.Status.UP, healthCheck.call().getStatus()); + + shutdownService.drain(); + + assertEquals(HealthCheckResponse.Status.DOWN, healthCheck.call().getStatus(), + "readiness must go DOWN so the load balancer stops routing traffic to this node"); + } +} diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinatorTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinatorTest.java index b11bd912b..dfaaeff09 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinatorTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinatorTest.java @@ -14,6 +14,7 @@ import java.util.List; import java.util.Map; import java.util.concurrent.Callable; +import java.util.concurrent.RejectedExecutionException; import static org.junit.jupiter.api.Assertions.*; import static org.mockito.ArgumentMatchers.*; @@ -79,28 +80,85 @@ void shouldIncrementProcessedOnComplete() { assertEquals(1, coordinator.getTotalProcessed()); } + /** + * C13 — a post-execution failure is NOT retried. {@code onFailure} is only ever + * raised once the callable has started running, so the turn may already have + * called an LLM, executed tools and spent money; re-running it repeats those + * side effects. The task is dead-lettered on the first failure instead. + */ @Test @SuppressWarnings("unchecked") - void shouldDeadLetterAfterMaxRetries() { + void shouldDeadLetterOnFirstFailureWithoutReExecutingTheTask() { Callable task = mock(Callable.class); coordinator.submitInOrder("conv-fail", task); - // Simulate 3 failures (MAX_RETRIES = 3) - for (int i = 0; i < 3; i++) { - ArgumentCaptor> callbackCaptor = ArgumentCaptor.forClass(IRuntime.IFinishedExecution.class); - verify(runtime, times(i + 1)).submitCallable(eq(task), callbackCaptor.capture(), isNull()); + ArgumentCaptor> callbackCaptor = ArgumentCaptor.forClass(IRuntime.IFinishedExecution.class); + verify(runtime).submitCallable(eq(task), callbackCaptor.capture(), isNull()); + + callbackCaptor.getValue().onFailure(new RuntimeException("Test failure")); - List> callbacks = callbackCaptor.getAllValues(); - callbacks.get(i).onFailure(new RuntimeException("Test failure " + (i + 1))); - } + // The already-executed callable must NEVER be handed to the runtime again. + verify(runtime, times(1)).submitCallable(eq(task), any(), isNull()); - // Should be dead-lettered after 3 retries assertEquals(1, coordinator.getTotalDeadLettered()); assertEquals(1, coordinator.getDeadLetters().size()); DeadLetterEntry entry = coordinator.getDeadLetters().get(0); assertEquals("conv-fail", entry.conversationId()); - assertTrue(entry.error().contains("Test failure 3")); + assertTrue(entry.error().contains("Test failure")); + } + + /** + * C10 — a submission the runtime rejects must not wedge the conversation. The + * offered task is rolled back off the queue (nothing is scheduled to run it), + * so the next turn on the same conversation is dispatched normally. + */ + @Test + @SuppressWarnings("unchecked") + void rejectedSubmissionLeavesTheConversationUsable() { + Callable rejected = mock(Callable.class); + Callable followUp = mock(Callable.class); + + doThrow(new RejectedExecutionException("pool saturated")) + .when(runtime).submitCallable(eq(rejected), any(), isNull()); + + assertThrows(RejectedExecutionException.class, () -> coordinator.submitInOrder("conv-wedge", rejected)); + + // The queue must not retain the task nobody is going to run, and the map + // entry must be gone (otherwise it leaks for the JVM's lifetime). + assertFalse(coordinator.getQueueDepths().containsKey("conv-wedge"), + "A rejected submission must not leave a task queued with nothing scheduled to run it"); + + // The conversation is still usable: the next turn is dispatched. + coordinator.submitInOrder("conv-wedge", followUp); + verify(runtime).submitCallable(eq(followUp), any(), isNull()); + } + + /** + * C10 (submitNext side) — a rejection while scheduling the NEXT queued task has + * no caller to propagate to. It must dead-letter and keep draining rather than + * leaving the queue populated with nothing scheduled to run it. + */ + @Test + @SuppressWarnings("unchecked") + void rejectedSubmissionOfQueuedTaskDrainsInsteadOfWedging() { + Callable first = mock(Callable.class); + Callable second = mock(Callable.class); + + coordinator.submitInOrder("conv-drain", first); + ArgumentCaptor> captor = ArgumentCaptor.forClass(IRuntime.IFinishedExecution.class); + verify(runtime).submitCallable(eq(first), captor.capture(), isNull()); + + coordinator.submitInOrder("conv-drain", second); + doThrow(new RejectedExecutionException("pool saturated")) + .when(runtime).submitCallable(eq(second), any(), isNull()); + + // first completes → submitNext tries (and fails) to schedule second + captor.getValue().onComplete(null); + + assertFalse(coordinator.getQueueDepths().containsKey("conv-drain"), + "The queue must drain even when the next task cannot be scheduled"); + assertEquals(1, coordinator.getTotalDeadLettered()); } // ==================== Dead-Letter CRUD ==================== @@ -334,13 +392,11 @@ private void causeDeadLetter(String conversationId, Callable task) { private void causeDeadLetter(InMemoryConversationCoordinator coord, IRuntime rt, String conversationId, Callable task) { coord.submitInOrder(conversationId, task); - // Simulate MAX_RETRIES (3) failures - for (int i = 0; i < 3; i++) { - ArgumentCaptor> captor = ArgumentCaptor.forClass(IRuntime.IFinishedExecution.class); - verify(rt, atLeast(1)).submitCallable(eq(task), captor.capture(), isNull()); + // A single post-execution failure dead-letters immediately (no retry — C13). + ArgumentCaptor> captor = ArgumentCaptor.forClass(IRuntime.IFinishedExecution.class); + verify(rt, atLeast(1)).submitCallable(eq(task), captor.capture(), isNull()); - List> callbacks = captor.getAllValues(); - callbacks.get(callbacks.size() - 1).onFailure(new RuntimeException("forced failure")); - } + List> callbacks = captor.getAllValues(); + callbacks.get(callbacks.size() - 1).onFailure(new RuntimeException("forced failure")); } } diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java index 3b1b473dd..3300edba0 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java @@ -14,6 +14,7 @@ import ai.labs.eddi.engine.model.InputData; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; import org.mockito.ArgumentCaptor; import java.time.Instant; @@ -30,6 +31,7 @@ class ScheduleFireExecutorTest { private IConversationService conversationService; private IScheduleStore scheduleStore; private ai.labs.eddi.engine.internal.HitlTimeoutHandler hitlTimeoutHandler; + private DreamService dreamService; private ScheduleFireExecutor executor; @BeforeEach @@ -37,12 +39,14 @@ void setUp() { conversationService = mock(IConversationService.class); scheduleStore = mock(IScheduleStore.class); hitlTimeoutHandler = mock(ai.labs.eddi.engine.internal.HitlTimeoutHandler.class); + dreamService = mock(DreamService.class); executor = new ScheduleFireExecutor(); // Inject mocks via reflection (field injection) setField(executor, "conversationService", conversationService); setField(executor, "scheduleStore", scheduleStore); setField(executor, "hitlTimeoutHandler", hitlTimeoutHandler); + setField(executor, "dreamService", dreamService); } @Test @@ -176,6 +180,48 @@ void fire_exceptionResultsInFailedStatus() throws Exception { assertEquals(3, result.attemptNumber()); } + /** + * Finding B2 — {@code fire()} waits on a + * {@link java.util.concurrent.CountDownLatch} inside a broad + * {@code catch (Exception)}. {@code latch.await} CLEARS the thread's interrupt + * status before it throws, so without an explicit restore the poller thread's + * shutdown signal is swallowed and it keeps firing further schedules while the + * executor is shutting down. + */ + @Test + @Timeout(10) + void fire_interruptedWhileWaiting_restoresInterruptFlagAndLogsFailed() throws Exception { + var schedule = makeCronSchedule("sched-interrupt", "new"); + when(conversationService.startConversation(any(), any(), any(), any())) + .thenReturn(new IConversationService.ConversationResult("conv-int", null)); + + // Never completes the response handler; interrupts the caller instead, so the + // latch.await() below throws InterruptedException immediately (and clears the + // flag) rather than blocking for its 5-minute budget. + doAnswer(inv -> { + Thread.currentThread().interrupt(); + return null; + }).when(conversationService).say(any(), any(), any(), anyBoolean(), anyBoolean(), any(), any(), anyBoolean(), any()); + + assertFalse(Thread.currentThread().isInterrupted(), "precondition: flag starts clear"); + try { + ScheduleFireLog result = executor.fire(schedule, "instance-1", 1); + + assertTrue(Thread.currentThread().isInterrupted(), + "fire() must re-assert the interrupt flag that latch.await consumed, otherwise the " + + "poller keeps firing schedules through shutdown"); + // The attempt still has to be recorded — restoring the flag must not + // short-circuit the fire log. + assertEquals(FireStatus.FAILED.name(), result.status()); + assertTrue(result.errorMessage().startsWith("InterruptedException"), + "expected the interrupt to be recorded, got: " + result.errorMessage()); + verify(scheduleStore).logFire(argThat(log -> log.status().equals(FireStatus.FAILED.name()))); + } finally { + // Never let the flag leak into the next test on this thread. + Thread.interrupted(); + } + } + @Test void fire_logsFireAttemptEvenOnFailure() throws Exception { var schedule = makeCronSchedule("sched-err2", "new"); @@ -259,8 +305,116 @@ void fire_hitlTimeout_fireLogFailure_isSwallowed() throws Exception { verify(hitlTimeoutHandler).handleTimeout(any()); } + // --- Dream consolidation dispatch (finding I1) --- + + /** + * The conversation path blocks on a 5-minute {@code CountDownLatch}; if the + * Dream fast-path ever stops short-circuiting, these tests would hang rather + * than fail, hence the timeouts. + */ + @Test + @Timeout(10) + void fire_dreamSchedule_dispatchesToDreamServiceInsteadOfSayingAnything() throws Exception { + var schedule = makeDreamSchedule("sched-dream-1", "user-42"); + schedule.setAgentVersion(3); + when(dreamService.processScheduledFire("agent-1", 3, "user-42")) + .thenReturn(new DreamService.DreamResult("user-42", 4, 1, 2, 120L, 0.0125, null)); + + ScheduleFireLog result = executor.fire(schedule, "instance-1", 1); + + assertEquals(FireStatus.COMPLETED.name(), result.status()); + assertNull(result.errorMessage()); + assertEquals(0.0125, result.cost(), 1e-9); + assertNull(result.conversationId()); + verify(dreamService).processScheduledFire("agent-1", 3, "user-42"); + // A dream cycle is maintenance, not a conversation turn + verifyNoInteractions(conversationService); + verify(scheduleStore).logFire(argThat(log -> log.status().equals(FireStatus.COMPLETED.name()))); + } + + @Test + @Timeout(10) + void fire_dreamSchedule_passesUserIdThroughUndefaulted() throws Exception { + // No userId on the schedule: it must reach DreamService as-is so the + // rejection is loud, rather than being defaulted to "system:scheduler". + var schedule = makeDreamSchedule("sched-dream-2", null); + when(dreamService.processScheduledFire(any(), any(), any())) + .thenReturn(new DreamService.DreamResult(null, 0, 0, 0, 1L, 0.0, "no userId")); + + ScheduleFireLog result = executor.fire(schedule, "instance-1", 1); + + assertEquals(FireStatus.FAILED.name(), result.status()); + assertEquals("no userId", result.errorMessage()); + verify(dreamService).processScheduledFire("agent-1", 0, null); + } + + @Test + @Timeout(10) + void fire_dreamSchedule_failedCycle_marksFireFailedSoItRetries() throws Exception { + var schedule = makeDreamSchedule("sched-dream-3", "user-7"); + when(dreamService.processScheduledFire(any(), any(), any())) + .thenReturn(new DreamService.DreamResult("user-7", 0, 0, 0, 5L, 0.0, + "Memory consolidation LLM call failed (anthropic/claude): 401 Unauthorized")); + + ScheduleFireLog result = executor.fire(schedule, "instance-1", 2); + + assertEquals(FireStatus.FAILED.name(), result.status()); + assertTrue(result.errorMessage().contains("401 Unauthorized"), + "the cause must reach the fire log, got: " + result.errorMessage()); + assertEquals(2, result.attemptNumber()); + verify(scheduleStore).logFire(argThat(log -> log.status().equals(FireStatus.FAILED.name()))); + } + + @Test + @Timeout(10) + void fire_dreamSchedule_serviceThrows_logsFailedWithoutPropagating() throws Exception { + var schedule = makeDreamSchedule("sched-dream-4", "user-7"); + when(dreamService.processScheduledFire(any(), any(), any())) + .thenThrow(new RuntimeException("store exploded")); + + ScheduleFireLog result = executor.fire(schedule, "instance-1", 1); + + assertEquals(FireStatus.FAILED.name(), result.status()); + assertTrue(result.errorMessage().contains("store exploded"), + "error must carry the cause, got: " + result.errorMessage()); + verify(scheduleStore).logFire(any()); + } + + @Test + @Timeout(10) + void fire_nonDreamSchedule_doesNotReachDreamService() throws Exception { + var schedule = makeCronSchedule("sched-plain", "new"); + when(conversationService.startConversation(any(), eq("agent-1"), eq("system:scheduler"), any())) + .thenReturn(new IConversationService.ConversationResult("conv-1", null)); + doAnswer(inv -> { + ((IConversationService.ConversationResponseHandler) inv.getArgument(8)).onComplete(null); + return null; + }).when(conversationService).say(any(), any(), any(), anyBoolean(), anyBoolean(), any(), any(), anyBoolean(), any()); + + executor.fire(schedule, "instance-1", 1); + + verifyNoInteractions(dreamService); + } + // --- Helpers --- + private static ScheduleConfiguration makeDreamSchedule(String id, String userId) { + var s = new ScheduleConfiguration(); + s.setId(id); + s.setName("dream-agent-1"); + s.setTriggerType(TriggerType.CRON); + s.setAgentId("agent-1"); + s.setCronExpression("0 3 * * *"); + s.setEnvironment("production"); + s.setTimeZone("UTC"); + s.setUserId(userId); + s.setFireStatus(FireStatus.CLAIMED); + s.setNextFire(Instant.now().minusSeconds(1)); + s.setMetadata(java.util.Map.of( + DreamService.METADATA_TYPE_KEY, DreamService.METADATA_TYPE_CONSOLIDATION)); + return s; + } + private static ScheduleConfiguration makeHitlTimeoutSchedule(String id, String conversationId, String policy) { var s = new ScheduleConfiguration(); s.setId(id); diff --git a/src/test/java/ai/labs/eddi/modules/nlp/InputParserTaskInterruptTest.java b/src/test/java/ai/labs/eddi/modules/nlp/InputParserTaskInterruptTest.java new file mode 100644 index 000000000..fe540041b --- /dev/null +++ b/src/test/java/ai/labs/eddi/modules/nlp/InputParserTaskInterruptTest.java @@ -0,0 +1,132 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.modules.nlp; + +import ai.labs.eddi.datastore.IResourceStore; +import ai.labs.eddi.engine.TestMemoryFactory; +import ai.labs.eddi.engine.lifecycle.IComponentCache; +import ai.labs.eddi.engine.lifecycle.ILifecycleTask; +import ai.labs.eddi.engine.lifecycle.TaskId; +import ai.labs.eddi.engine.lifecycle.exceptions.LifecycleException; +import ai.labs.eddi.engine.lifecycle.internal.LifecycleManager; +import ai.labs.eddi.modules.nlp.expressions.utilities.IExpressionProvider; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.util.HashMap; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.argThat; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +/** + * Finding B2 — {@link InputParserTask} must NOT swallow the thread's interrupt + * flag. + * + *

      + * {@code InputParserTask.execute} catches {@link InterruptedException} from the + * parser and returns normally. The engine's graceful-stop signal is the + * interrupt flag ({@code LifecycleManager} re-checks it before every task), so + * consuming the exception without re-asserting the flag lets the remaining + * tasks of an abandoned turn run anyway. + *

      + * + *

      + * The parser is stubbed to throw with the flag CLEAR — what every real blocking + * JDK call does before it throws. Remove the + * {@code Thread.currentThread().interrupt()} from the catch block and both + * tests fail. + *

      + */ +@DisplayName("InputParserTask — interrupt flag restoration (B2)") +class InputParserTaskInterruptTest { + + private InputParserTask task; + + @BeforeEach + void setUp() { + // Never inherit a stale flag from an earlier test on this JUnit thread. + Thread.interrupted(); + task = new InputParserTask(mock(IExpressionProvider.class), new HashMap<>(), new HashMap<>(), + new HashMap<>(), new ObjectMapper()); + } + + @AfterEach + void clearInterruptFlag() { + // These tests deliberately leave the thread interrupted — clear it so the + // flag cannot leak into unrelated tests sharing this thread. + Thread.interrupted(); + } + + @Test + @Timeout(10) + @DisplayName("catching InterruptedException re-asserts the flag and skips result storage") + void execute_interruptedParser_restoresFlag() throws Exception { + var ctx = TestMemoryFactory.createWithInput("hello"); + var parser = mock(IInputParser.class); + doThrow(new InterruptedException("Execution was interrupted!")) + .when(parser).normalize(anyString(), any()); + + task.execute(ctx.memory(), parser); + + assertTrue(Thread.currentThread().isInterrupted(), + "InputParserTask must re-assert the interrupt flag it consumed, otherwise " + + "LifecycleManager's graceful-stop check never fires"); + // The early return still holds: no parse result is written for the aborted + // turn. + verify(ctx.currentStep(), never()).storeData( + argThat(data -> "expressions:parsed".equals(data.getKey()))); + } + + @Test + @Timeout(10) + @DisplayName("an interrupted parser task aborts the pipeline instead of running the next task") + void pipeline_interruptedParserTask_stopsBeforeNextTask() throws Exception { + var componentCache = mock(IComponentCache.class); + var workflowId = mock(IResourceStore.IResourceId.class); + when(workflowId.getId()).thenReturn("wf1"); + when(workflowId.getVersion()).thenReturn(1); + + var ctx = TestMemoryFactory.createWithInput("hello"); + var parser = mock(IInputParser.class); + doThrow(new InterruptedException("Execution was interrupted!")) + .when(parser).normalize(anyString(), any()); + + // The parser task sits at absolute index 0 of workflow "wf1" version 1, so + // LifecycleManager resolves its component under the key "wf1:1:0". + Map parserComponents = new HashMap<>(); + parserComponents.put("wf1:1:0", parser); + when(componentCache.getComponentMap(InputParserTask.ID)).thenReturn(parserComponents); + when(componentCache.getComponentMap("ai.labs.behavior")).thenReturn(new HashMap<>()); + + var nextTask = mock(ILifecycleTask.class); + when(nextTask.getId()).thenReturn(new TaskId("ai.labs.behavior")); + when(nextTask.getType()).thenReturn("behavior_rules"); + + var lifecycleManager = new LifecycleManager(componentCache, workflowId); + lifecycleManager.addLifecycleTask(task); + lifecycleManager.addLifecycleTask(nextTask); + + assertThrows(LifecycleException.LifecycleInterruptedException.class, + () -> lifecycleManager.executeLifecycle(ctx.memory(), null), + "the restored flag must abort the turn at the next task boundary"); + + // The decisive assertion: with the flag swallowed the pipeline sails on and + // the abandoned turn keeps executing side-effectful tasks. + verify(nextTask, never()).execute(any(), any()); + } +} diff --git a/src/test/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParserCacheTest.java b/src/test/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParserCacheTest.java new file mode 100644 index 000000000..8c340ed22 --- /dev/null +++ b/src/test/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParserCacheTest.java @@ -0,0 +1,203 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.modules.nlp.impl; + +import ai.labs.eddi.configs.parser.model.ParserConfiguration; +import ai.labs.eddi.engine.lifecycle.ILifecycleTask; +import ai.labs.eddi.engine.runtime.IRuntime; +import ai.labs.eddi.engine.runtime.client.configuration.IResourceClientLibrary; +import ai.labs.eddi.modules.nlp.IInputParser; +import jakarta.inject.Provider; +import jakarta.ws.rs.container.AsyncResponse; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; +import org.mockito.ArgumentCaptor; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.Callable; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.ArgumentMatchers.isNull; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; + +/** + * Concurrency and bounding tests for the parser cache of + * {@link RestSemanticParser}. + *

      + * {@code RestSemanticParser} is an {@code @ApplicationScoped} singleton whose + * parsers are created on {@code IRuntime} pool threads, so its cache is mutated + * concurrently. Before the fix the cache was an unsynchronised {@link HashMap} + * with a {@code containsKey}/{@code put} sequence, unbounded, and keyed on a + * caller-supplied config id. + */ +@DisplayName("RestSemanticParser — parser cache") +@SuppressWarnings("unchecked") +class RestSemanticParserCacheTest { + + private static final String CONFIG_ID = "aabbccdd11223344eeff5566"; + + private IRuntime runtime; + private IResourceClientLibrary resourceClientLibrary; + private Provider parserProvider; + private ILifecycleTask parserTask; + private IInputParser inputParser; + private AsyncResponse asyncResponse; + private RestSemanticParser parser; + + @BeforeEach + void setUp() throws Exception { + runtime = mock(IRuntime.class); + resourceClientLibrary = mock(IResourceClientLibrary.class); + parserProvider = mock(Provider.class); + parserTask = mock(ILifecycleTask.class); + inputParser = mock(IInputParser.class); + asyncResponse = mock(AsyncResponse.class); + + Map> lifecycleTasks = new HashMap<>(); + lifecycleTasks.put("ai.labs.parser", parserProvider); + parser = new RestSemanticParser(runtime, resourceClientLibrary, lifecycleTasks); + + doReturn(parserTask).when(parserProvider).get(); + doReturn(inputParser).when(parserTask).configure(any(), any()); + doReturn(List.of()).when(inputParser).parse(anyString()); + doReturn(mock(Future.class)).when(runtime).submitCallable(any(Callable.class), any()); + } + + private List> captureCallables(int expectedInvocations) { + ArgumentCaptor> captor = ArgumentCaptor.forClass(Callable.class); + verify(runtime, times(expectedInvocations)).submitCallable(captor.capture(), isNull()); + return captor.getAllValues(); + } + + @Test + @Timeout(60) + @DisplayName("concurrent requests for the same uncached parser create exactly one parser instance") + void concurrentRequestsCreateExactlyOneParser() throws Exception { + AtomicInteger configurationLoads = new AtomicInteger(); + // Two parties: both threads only ever reach the loader if the cache lets + // them in concurrently, which is precisely the bug. + CountDownLatch insideLoader = new CountDownLatch(2); + + doAnswer(invocation -> { + configurationLoads.incrementAndGet(); + insideLoader.countDown(); + // Holds the creation open so a second thread has ample opportunity to + // slip into the old containsKey/put window. With a single-flight cache + // only one thread arrives here and this await simply times out. + insideLoader.await(1, TimeUnit.SECONDS); + return new ParserConfiguration(); + }).when(resourceClientLibrary).getResource(any(), eq(ParserConfiguration.class)); + + AsyncResponse secondResponse = mock(AsyncResponse.class); + parser.parse(CONFIG_ID, 1, "first", asyncResponse); + parser.parse(CONFIG_ID, 1, "second", secondResponse); + + List> callables = captureCallables(2); + assertEquals(2, callables.size()); + + CyclicBarrier startTogether = new CyclicBarrier(2); + ExecutorService pool = Executors.newFixedThreadPool(2); + try { + List> futures = new ArrayList<>(); + for (Callable callable : callables) { + futures.add(pool.submit(() -> { + startTogether.await(20, TimeUnit.SECONDS); + return callable.call(); + })); + } + for (Future future : futures) { + future.get(30, TimeUnit.SECONDS); + } + } finally { + pool.shutdownNow(); + } + + assertEquals(1, configurationLoads.get(), + "parser configuration must be fetched exactly once for concurrent requests"); + verify(parserProvider, times(1)).get(); + verify(parserTask, times(1)).configure(any(), any()); + + // Both callers still get a successful result, served by the single instance. + assertResumedWithSolutionList(asyncResponse); + assertResumedWithSolutionList(secondResponse); + verify(inputParser).parse("first"); + verify(inputParser).parse("second"); + } + + private void assertResumedWithSolutionList(AsyncResponse response) { + ArgumentCaptor captor = ArgumentCaptor.forClass(Object.class); + verify(response).resume(captor.capture()); + assertInstanceOf(List.class, captor.getValue()); + } + + @Test + @Timeout(60) + @DisplayName("cache stays bounded when callers request more parser ids than the maximum") + void cacheIsBoundedByMaximumSize() throws Exception { + doReturn(new ParserConfiguration()).when(resourceClientLibrary).getResource(any(), eq(ParserConfiguration.class)); + + int maxCachedParsers = RestSemanticParser.MAX_CACHED_PARSERS; + int distinctConfigIds = maxCachedParsers + 50; + for (int i = 0; i < distinctConfigIds; i++) { + parser.parse(String.format("%024x", i), 1, "hello", asyncResponse); + } + for (Callable callable : captureCallables(distinctConfigIds)) { + callable.call(); + } + + // Every distinct id really was resolved — i.e. we genuinely tried to cache + // more entries than the maximum. + verify(parserProvider, times(distinctConfigIds)).get(); + + long cached = parser.cachedParserCount(); + assertTrue(cached > 0, "cache should retain entries, but was empty"); + assertTrue(cached <= maxCachedParsers, + "cache must be bounded by " + maxCachedParsers + " entries, but held " + cached); + } + + @Test + @Timeout(60) + @DisplayName("invalidateCache drops cached parsers so an updated configuration is re-read") + void invalidateCacheForcesReload() throws Exception { + doReturn(new ParserConfiguration()).when(resourceClientLibrary).getResource(any(), eq(ParserConfiguration.class)); + + parser.parse(CONFIG_ID, 1, "first", asyncResponse); + captureCallables(1).getFirst().call(); + assertEquals(1, parser.cachedParserCount()); + + parser.invalidateCache(); + assertEquals(0, parser.cachedParserCount(), "invalidateCache must empty the cache"); + + reset(runtime); + doReturn(mock(Future.class)).when(runtime).submitCallable(any(Callable.class), any()); + parser.parse(CONFIG_ID, 1, "second", mock(AsyncResponse.class)); + captureCallables(1).getFirst().call(); + + verify(parserProvider, times(2)).get(); + verify(resourceClientLibrary, times(2)).getResource(any(), eq(ParserConfiguration.class)); + } +} diff --git a/src/test/java/ai/labs/eddi/modules/rules/impl/RulesEvaluationTaskInterruptTest.java b/src/test/java/ai/labs/eddi/modules/rules/impl/RulesEvaluationTaskInterruptTest.java new file mode 100644 index 000000000..c96a17711 --- /dev/null +++ b/src/test/java/ai/labs/eddi/modules/rules/impl/RulesEvaluationTaskInterruptTest.java @@ -0,0 +1,138 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.modules.rules.impl; + +import ai.labs.eddi.datastore.IResourceStore; +import ai.labs.eddi.datastore.serialization.IJsonSerialization; +import ai.labs.eddi.engine.lifecycle.IComponentCache; +import ai.labs.eddi.engine.lifecycle.ILifecycleTask; +import ai.labs.eddi.engine.lifecycle.TaskId; +import ai.labs.eddi.engine.lifecycle.exceptions.LifecycleException; +import ai.labs.eddi.engine.lifecycle.internal.LifecycleManager; +import ai.labs.eddi.engine.memory.IConversationMemory; +import ai.labs.eddi.engine.runtime.client.configuration.IResourceClientLibrary; +import ai.labs.eddi.modules.nlp.expressions.utilities.IExpressionProvider; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.Timeout; + +import java.util.HashMap; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +/** + * Finding B2 — {@link RulesEvaluationTask} must NOT swallow the thread's + * interrupt flag. + * + *

      + * The engine's graceful-stop signal is the interrupt flag: + * {@code LifecycleManager} re-checks + * {@code Thread.currentThread().isInterrupted()} before every task and aborts + * the turn with {@link LifecycleException.LifecycleInterruptedException}. A + * task that catches {@link InterruptedException} and then returns normally + * therefore has to restore the flag — otherwise the rest of an abandoned turn + * keeps running and (per finding C3) can persist its stale snapshot over a + * newer turn's state. + *

      + * + *

      + * Both tests model the flag as already cleared when the exception arrives, + * which is exactly what every blocking JDK call ({@code Future.get}, + * {@code latch.await}, {@code Thread.sleep}) does before it throws. Remove the + * {@code Thread.currentThread().interrupt()} from the catch block and both + * tests fail. + *

      + */ +@DisplayName("RulesEvaluationTask — interrupt flag restoration (B2)") +class RulesEvaluationTaskInterruptTest { + + private RulesEvaluationTask task; + + @BeforeEach + void setUp() { + // Never inherit a stale flag from an earlier test on this JUnit thread. + Thread.interrupted(); + task = new RulesEvaluationTask(mock(IResourceClientLibrary.class), mock(IJsonSerialization.class), + mock(IRuleDeserialization.class), mock(IExpressionProvider.class)); + } + + @AfterEach + void clearInterruptFlag() { + // These tests deliberately leave the thread interrupted — clear it so the + // flag cannot leak into unrelated tests sharing this thread. + Thread.interrupted(); + } + + @Test + @Timeout(10) + @DisplayName("catching InterruptedException re-asserts the interrupt flag") + void execute_interruptedEvaluator_restoresFlag() throws Exception { + var memory = mock(IConversationMemory.class); + var currentStep = mock(IConversationMemory.IWritableConversationStep.class); + when(memory.getCurrentStep()).thenReturn(currentStep); + + var evaluator = mock(RulesEvaluator.class); + // Thrown with the flag CLEAR, like any real blocking call would. + when(evaluator.evaluate(memory)).thenThrow(new InterruptedException("Execution was interrupted!")); + + task.execute(memory, evaluator); + + assertTrue(Thread.currentThread().isInterrupted(), + "RulesEvaluationTask must re-assert the interrupt flag it consumed, otherwise " + + "LifecycleManager's graceful-stop check never fires"); + } + + @Test + @Timeout(10) + @DisplayName("an interrupted rules task aborts the pipeline instead of running the next task") + void pipeline_interruptedRulesTask_stopsBeforeNextTask() throws Exception { + var componentCache = mock(IComponentCache.class); + var workflowId = mock(IResourceStore.IResourceId.class); + when(workflowId.getId()).thenReturn("wf1"); + when(workflowId.getVersion()).thenReturn(1); + + var memory = mock(IConversationMemory.class); + var currentStep = mock(IConversationMemory.IWritableConversationStep.class); + when(memory.getCurrentStep()).thenReturn(currentStep); + when(memory.getConversationId()).thenReturn("conv-b2"); + when(memory.getAgentId()).thenReturn("agent-b2"); + + var evaluator = mock(RulesEvaluator.class); + when(evaluator.evaluate(any(IConversationMemory.class))) + .thenThrow(new InterruptedException("Execution was interrupted!")); + + // The rules task sits at absolute index 0 of workflow "wf1" version 1, so + // LifecycleManager resolves its component under the key "wf1:1:0". + Map rulesComponents = new HashMap<>(); + rulesComponents.put("wf1:1:0", evaluator); + when(componentCache.getComponentMap(RulesEvaluationTask.ID)).thenReturn(rulesComponents); + when(componentCache.getComponentMap("ai.labs.output")).thenReturn(new HashMap<>()); + + var nextTask = mock(ILifecycleTask.class); + when(nextTask.getId()).thenReturn(new TaskId("ai.labs.output")); + when(nextTask.getType()).thenReturn("output"); + + var lifecycleManager = new LifecycleManager(componentCache, workflowId); + lifecycleManager.addLifecycleTask(task); + lifecycleManager.addLifecycleTask(nextTask); + + assertThrows(LifecycleException.LifecycleInterruptedException.class, + () -> lifecycleManager.executeLifecycle(memory, null), + "the restored flag must abort the turn at the next task boundary"); + + // The decisive assertion: with the flag swallowed the pipeline sails on and + // the abandoned turn keeps executing side-effectful tasks. + verify(nextTask, never()).execute(any(), any()); + } +} From 2421b7299359e80a7727736f65620f6f9bac7c12 Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Wed, 29 Jul 2026 20:34:20 +0200 Subject: [PATCH 02/11] fix(runtime): report an interrupted drain as an interrupt, not a timeout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two Copilot findings on #619. A third — "the parameter 'shutdownEvent' is never used" from the code-quality bot — is a false positive and is deliberately NOT actioned: onShutdown(@Observes ShutdownEvent) is a CDI observer, the parameter IS the subscription, and removing it would silently unregister the shutdown drain rather than tidy anything. - drain() discarded sleepQuietly's return for the readiness grace while the poll loop right below it honoured the same value, and sleepQuietly's own javadoc says the caller should stop waiting. Now honoured, and interrupt is reported separately from timeout: "timed out after 30000 ms" for a drain cut short after 50 ms sends whoever reads that log hunting the wrong problem. Being precise about the size of this: it is a DIAGNOSIS fix, not a behavioural one, and Copilot's "will still proceed to wait/drain" overstates it. Because sleepQuietly restores the interrupt flag, the old code's next Thread.sleep threw immediately and it broke out just as promptly with the same boolean. Verified by mutation, not assumed: reverting the fix leaves all 9 tests green. The two new tests are therefore labelled as pinning the interrupt CONTRACT (prompt return, honest boolean, gate stays shut) and explicitly say they do not distinguish the fix — a confident javadoc over a test that proves nothing is exactly the test theatre this review pass keeps finding. What the fix does buy: accurate logs, and an exit that no longer depends on that restored-flag side effect. - The F6 changelog entry claimed a ConnectionCallback sets cancelled on client disconnect. RestAgentEngineStreaming documents at the call site that ConnectionCallback is NOT invoked on this path and detects disconnects via SseEventSink.isClosed(). Corrected, and the correction says which approach was tried versus which shipped. Second time in this stack a changelog entry described intent rather than behaviour. --- docs/changelog.md | 4 +- .../internal/GracefulShutdownService.java | 31 +++++++-- .../internal/GracefulShutdownServiceTest.java | 65 +++++++++++++++++++ 3 files changed, 95 insertions(+), 5 deletions(-) diff --git a/docs/changelog.md b/docs/changelog.md index 5c3272e16..cb13ce1c1 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -55,7 +55,9 @@ Per the repo owner's decision, `DreamService` is now registered with `ScheduleFi ### Partial -**F6** — the REST/pipeline half is done (a `ConnectionCallback` now sets cancelled on client disconnect, and cancellation is checked at more points). The in-`modules/llm` half — cancellation checks inside the tool loop and the cascade — is deferred, since that module is owned by another workstream. +**F6** — the REST/pipeline half is done (client disconnect now sets cancelled, and cancellation is checked at more points). The in-`modules/llm` half — cancellation checks inside the tool loop and the cascade — is deferred, since that module is owned by another workstream. + +> Disconnect is detected by testing `SseEventSink.isClosed()` before and around each send — **not** by a `ConnectionCallback`, which RESTEasy Reactive does not invoke on this path, as `RestAgentEngineStreaming` documents at the call site. An earlier draft of this entry named `ConnectionCallback`: it described the approach that was tried, not the one that shipped. ### Verification diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java index f609794c4..8bad37192 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java @@ -121,7 +121,20 @@ boolean drain() { shuttingDown = true; LOGGER.info("Shutdown signalled — readiness is now DOWN and new conversation turns are rejected"); - sleepQuietly(readinessGraceMillis); + // The poll loop below honours this return value; the grace sleep must too, or + // an interrupt gets acknowledged (the flag is restored) and then ignored — we + // would keep waiting after being told to stop, and the first poll would throw + // immediately anyway, landing in the "timed out" branch and misreporting why. + if (!sleepQuietly(readinessGraceMillis)) { + int remaining = countInFlight(); + if (remaining == 0) { + LOGGER.info("Readiness grace interrupted, but nothing was in flight — shutdown drain completed"); + return true; + } + LOGGER.warnf("Shutdown drain interrupted during the readiness grace window with %d conversation task(s) in flight — " + + "not waiting further; crash recovery reconciles their state on the next boot", remaining); + return false; + } long deadline = System.nanoTime() + drainTimeoutMillis * 1_000_000L; int inFlight = countInFlight(); @@ -131,17 +144,27 @@ boolean drain() { } LOGGER.infof("Draining %d in-flight conversation task(s), waiting up to %d ms", inFlight, drainTimeoutMillis); + boolean interrupted = false; while (inFlight > 0 && System.nanoTime() < deadline) { if (!sleepQuietly(pollIntervalMillis)) { + interrupted = true; break; } inFlight = countInFlight(); } if (inFlight > 0) { - LOGGER.warnf("Shutdown drain timed out after %d ms with %d conversation task(s) still in flight — " - + "they will be abandoned; crash recovery reconciles their state on the next boot", - drainTimeoutMillis, inFlight); + // Interrupt and timeout are different failures and must not read alike: + // "timed out after 30000 ms" for a drain cut short after 50 ms sends + // whoever reads that log hunting the wrong problem. + if (interrupted) { + LOGGER.warnf("Shutdown drain interrupted with %d conversation task(s) still in flight — " + + "they will be abandoned; crash recovery reconciles their state on the next boot", inFlight); + } else { + LOGGER.warnf("Shutdown drain timed out after %d ms with %d conversation task(s) still in flight — " + + "they will be abandoned; crash recovery reconciles their state on the next boot", + drainTimeoutMillis, inFlight); + } return false; } diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownServiceTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownServiceTest.java index a94faa826..e6a506d13 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownServiceTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownServiceTest.java @@ -174,4 +174,69 @@ void readinessFlipsOnShutdown() { assertEquals(HealthCheckResponse.Status.DOWN, healthCheck.call().getStatus(), "readiness must go DOWN so the load balancer stops routing traffic to this node"); } + + /** + * Pins the interrupt CONTRACT: an interrupted drain returns promptly, returns + * the honest boolean, and leaves the accept gate shut. + *

      + * Be clear about what this does not do: it does not distinguish the + * readiness-grace fix from the code before it. Reverting that fix leaves these + * tests green, because {@code sleepQuietly} restores the interrupt flag — so + * the next {@code Thread.sleep} in the poll loop threw immediately and the old + * code broke out just as fast, with the same return value. The fix buys two + * things this test cannot see: the failure is logged as an interrupt rather + * than as a 30-second TIMEOUT that never happened, and the exit no longer + * depends on that restored-flag side effect to stop waiting. + */ + @Test + @Timeout(30) + @DisplayName("an interrupt during the readiness grace stops the drain instead of being ignored") + void interruptDuringReadinessGraceStopsTheDrain() throws Exception { + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + when(coordinator.getQueueDepths()).thenReturn(Map.of("conv-1", 1)); + + // A grace window long enough that the interrupt lands inside it. + var shutdownService = new GracefulShutdownService(coordinator, 30_000L, 10_000L, 1L); + + var drained = new AtomicBoolean(true); + var finished = new CountDownLatch(1); + Thread drainThread = new Thread(() -> { + drained.set(shutdownService.drain()); + finished.countDown(); + }); + drainThread.start(); + + // Let it reach the grace sleep, then interrupt. + Thread.sleep(150); + drainThread.interrupt(); + + assertTrue(finished.await(10, TimeUnit.SECONDS), + "an interrupted drain must return promptly, not sit out the full 30s grace window"); + assertFalse(drained.get(), "work was still in flight, so the drain did not complete"); + assertTrue(shutdownService.isShuttingDown(), "the accept gate stays closed regardless of how the drain ended"); + } + + @Test + @Timeout(30) + @DisplayName("an interrupt with nothing in flight still reports a completed drain") + void interruptWithNothingInFlightStillCompletes() throws Exception { + IConversationCoordinator coordinator = mock(IConversationCoordinator.class); + when(coordinator.getQueueDepths()).thenReturn(Map.of()); + + var shutdownService = new GracefulShutdownService(coordinator, 30_000L, 10_000L, 1L); + + var drained = new AtomicBoolean(false); + var finished = new CountDownLatch(1); + Thread drainThread = new Thread(() -> { + drained.set(shutdownService.drain()); + finished.countDown(); + }); + drainThread.start(); + + Thread.sleep(150); + drainThread.interrupt(); + + assertTrue(finished.await(10, TimeUnit.SECONDS)); + assertTrue(drained.get(), "nothing was in flight, so an interrupted wait is still a clean drain"); + } } From f22a52e69a0d21d44e587d49bdfb94b8aca22425 Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Wed, 29 Jul 2026 20:45:07 +0200 Subject: [PATCH 03/11] fix(runtime): re-check in-flight count on the interrupt path Copilot follow-up, and the same misreporting class as the interrupt/timeout split in the previous commit: breaking out of the poll loop on an interrupt used the inFlight value read BEFORE the sleep. If the last turn finished during that sleep and the interrupt landed immediately after, drain() returned false and logged "still in flight" for a drain that had actually completed. One re-check on the interrupt path. Narrow window, but the whole point of splitting interrupt from timeout was that shutdown logs should say what really happened. --- .../engine/runtime/internal/GracefulShutdownService.java | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java index 8bad37192..d22a35215 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/GracefulShutdownService.java @@ -148,6 +148,11 @@ boolean drain() { while (inFlight > 0 && System.nanoTime() < deadline) { if (!sleepQuietly(pollIntervalMillis)) { interrupted = true; + // Re-check once before deciding: the last turn may have finished DURING + // this sleep, with the interrupt arriving immediately after. Reporting + // "still in flight" for a drain that actually completed is the same + // misreporting the interrupt/timeout split above exists to prevent. + inFlight = countInFlight(); break; } inFlight = countInFlight(); From 7e4f00f91d721988ca524751aa77bb5c9b3a0abd Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Wed, 29 Jul 2026 20:56:49 +0200 Subject: [PATCH 04/11] fix(runtime): restore the interrupt flag on the Dream fast-path too (B2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fire() restores the interrupt flag inside its broad catch, with a comment naming B2 as the reason. The Dream fast-path added by this PR has an equally broad catch (Exception) in the same class and did not — so a blocking call inside consolidation that throws InterruptedException cleared the flag and the poller thread kept firing further schedules through a shutdown. Exactly the failure B2 was raised for, reachable through the second of two sibling catches. Same shape as the Mongo/Postgres divergences earlier in this stack: a fix applied to one copy of duplicated logic and not to its twin. That is now five instances, and scanners cannot see it — CodeQL flags a reachable path, not an inconsistency between two paths that look independently fine. Mutation-checked: removing the restore fails the new test on the flag assertion, not incidentally. --- .../internal/ScheduleFireExecutor.java | 9 +++++ .../internal/ScheduleFireExecutorTest.java | 39 +++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java index 8d8d003b7..b18e6a262 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java @@ -208,6 +208,15 @@ private ScheduleFireLog fireDreamConsolidation(ScheduleConfiguration schedule, S errorMessage); } } catch (Exception e) { + // Same B2 reasoning as fire() above, and it has to be repeated here because + // this catch is just as broad: a blocking call inside Dream consolidation + // CLEARS the interrupt flag when it throws InterruptedException, so + // swallowing it would leave the poller thread running further schedules + // through a shutdown. The fix landing on only one of two sibling catches in + // the same class is exactly how these gaps happen. + if (e instanceof InterruptedException) { + Thread.currentThread().interrupt(); + } status = ScheduleConfiguration.FireStatus.FAILED.name(); errorMessage = e.getClass().getSimpleName() + ": " + e.getMessage(); LOGGER.errorf(e, "[SCHEDULE] Dream consolidation threw for schedule '%s' (id=%s)", schedule.getName(), schedule.getId()); diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java index 3300edba0..0059c67ba 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java @@ -398,6 +398,45 @@ void fire_nonDreamSchedule_doesNotReachDreamService() throws Exception { // --- Helpers --- + /** + * B2 again, on the OTHER broad catch. {@code fire()} restores the interrupt + * flag with an explicit comment; the Dream fast-path added by this PR has an + * equally broad {@code catch (Exception)} and did not. A blocking call inside + * consolidation that throws InterruptedException CLEARS the flag, so swallowing + * it leaves the poller thread firing further schedules through a shutdown — the + * exact failure B2 was raised for, reachable by the second of two sibling + * catches in the same class. + */ + @Test + @Timeout(10) + void fire_dreamScheduleInterrupted_restoresInterruptFlagAndLogsFailed() throws Exception { + var schedule = makeDreamSchedule("sched-dream-interrupt", "user-9"); + schedule.setAgentVersion(1); + + // Mimic a blocking call inside consolidation being interrupted: the flag is + // consumed by the throw, exactly as latch.await() does on the conversation + // path. + when(dreamService.processScheduledFire(any(), any(), any())).thenAnswer(inv -> { + Thread.interrupted(); + throw new InterruptedException("consolidation interrupted"); + }); + + assertFalse(Thread.currentThread().isInterrupted(), "precondition: flag starts clear"); + try { + ScheduleFireLog result = executor.fire(schedule, "instance-1", 1); + + assertTrue(Thread.currentThread().isInterrupted(), + "the Dream fast-path must re-assert the interrupt flag its catch consumed, or the poller " + + "keeps firing schedules through shutdown"); + assertEquals(FireStatus.FAILED.name(), result.status()); + assertTrue(result.errorMessage().startsWith("InterruptedException"), + "expected the interrupt to be recorded, got: " + result.errorMessage()); + verify(scheduleStore).logFire(argThat(log -> log.status().equals(FireStatus.FAILED.name()))); + } finally { + Thread.interrupted(); + } + } + private static ScheduleConfiguration makeDreamSchedule(String id, String userId) { var s = new ScheduleConfiguration(); s.setId(id); From 45fad7cef70e8e4b0fa2e790b6482d40dc0181a9 Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Wed, 29 Jul 2026 23:02:08 +0200 Subject: [PATCH 05/11] =?UTF-8?q?fix(runtime):=20pre-merge=20review=20find?= =?UTF-8?q?ings=20=E2=80=94=20schedule=20ownership,=20Dream=20scoping,=20i?= =?UTF-8?q?nterrupt=20bookkeeping?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Like #618, this PR reached "approved" with no CI run and no bot review: a stacked base disables CodeRabbit, and a base retarget does not fire the CI trigger. A dedicated pass over the 37-file diff produced 6 findings that survived adversarial verification (12 of 18 refuted) plus 22 from critics. The one that matters most is an access-control escalation this PR created: - The schedule REST surface never checked schedule.userId. Inert on its own — but the new dreamType=dream_consolidation dispatch armed it, so any eddi-editor could create and fire a schedule that BULK-DELETES ANOTHER USER'S persistent memories. RestScheduleStore already injected OwnershipValidator and simply did not use it here. Now admin-or-self on create, update (the re-point path) and fireNow, refusing with 403 rather than silently rewriting userId — a rewrite would hand back a schedule that does something other than what was asked. system:scheduler and blank ids stay exempt so stored schedules and Manager round-trips keep working. - Dream consolidation crossed agent boundaries: process() read getAllEntries(userId) — agent-unscoped — while every knob came from ONE agent's config, so agent A's pruneStaleAfterDays deleted agent B's memories and A's model endpoint saw B's text. Now scoped to the firing agent's own writes, with crossAgentMaintenance as an explicit opt-in. Newly reachable here, because this PR gave process() its first scheduled caller. - The B2 interrupt fix destroyed the bookkeeping it protected. The restore in fire() ran BEFORE logFire(), and the sync Mongo driver throws MongoInterruptedException on connection checkout while the flag is set — so on exactly the interrupt it existed to handle, the FAILED fire log was lost and failCount never incremented. Parked and re-asserted in a finally after the store round trip. The residual half was in SchedulePollerService, which ran markFailed() on the same still-interrupted thread: the schedule stayed CLAIMED with nextFire in the past, was re-claimed every lease expiry, and could never reach maxRetries — an interrupt turned a failing schedule into an unbounded re-fire loop. - A draining node answered 500 instead of a retryable signal, because sayInternal's trailing catch (Exception) swallowed the shutdown RejectedExecutionException — defeating this PR's own graceful-shutdown work. - A single transient LLM failure aborted a whole Dream cycle and marked the fire FAILED, so three consecutive 429s permanently dead-lettered the schedule. Also: the parallel batch deadline was sized at one member ATTEMPT, so it always fired first and made the per-member RETRY/ABORT/attributed-SKIP branches unreachable; maxSummarizationCalls silently stopped being enforced for stored configs (now an explicit backstop, deprecated for maxCostPerRun); BaseRuntime swallowed onComplete failures with no identifying context; and WorkflowStoreClientLibrary documented an invariant nothing enforced — the component key no longer depends on it. Docs verified against the implementation rather than intent (third and fourth instance in this stack): architecture.md told operators to create Dream schedules with an MCP tool that has no metadata parameter and therefore cannot set the marker the dispatcher matches on, so the documented procedure produced a schedule that never consolidated. IEventBus and InMemoryConversationCoordinator both claimed runtime coordinator selection via eddi.messaging.type; it is @IfBuildProfile("nats"), build-time, and that property is read by no Java code. One disagreement adjudicated rather than settled by severity: the critic rated the NATS C13/C10 parity gap CRITICAL, two verifiers refuted it as unreachable in shipped builds. The defect was real and is fixed; the rating was not. Two tests relabelled rather than trusted: both GracefulShutdownService interrupt tests pass identically with and without the fix, so they now say so instead of implying coverage they lack. Full suite: 12,912 tests. The only failures are the 15 known network-dependent classes from the sandbox baseline — none from this change surface. --- docs/architecture.md | 25 +- docs/changelog.md | 17 + docs/scheduling.md | 5 +- docs/user-memory.md | 5 +- .../agents/model/AgentConfiguration.java | 82 +++- .../internal/GroupConversationService.java | 65 +++- .../eddi/engine/internal/RestAgentEngine.java | 16 + .../lifecycle/internal/LifecycleManager.java | 28 +- .../labs/eddi/engine/runtime/BaseRuntime.java | 22 +- .../labs/eddi/engine/runtime/IEventBus.java | 8 +- .../workflows/WorkflowStoreClientLibrary.java | 66 ++-- .../engine/runtime/internal/Conversation.java | 52 ++- .../engine/runtime/internal/DreamService.java | 156 +++++++- .../InMemoryConversationCoordinator.java | 15 +- .../internal/NatsConversationCoordinator.java | 132 +++++-- .../internal/ScheduleFireExecutor.java | 44 ++- .../internal/SchedulePollerService.java | 18 + .../schedule/rest/RestScheduleStore.java | 78 +++- .../modules/nlp/impl/RestSemanticParser.java | 37 +- .../agents/model/AgentConfigurationTest.java | 50 +++ ...oupConversationServiceConcurrencyTest.java | 165 ++++++++- .../engine/internal/RestAgentEngineTest.java | 70 ++++ .../internal/LifecycleManagerTest.java | 84 ++++- .../runtime/BaseRuntimeConcurrencyTest.java | 19 +- .../WorkflowStoreClientLibraryTest.java | 204 ++++++++++ .../ConversationCancelPersistenceTest.java | 200 +++++++++- .../internal/DreamServiceExtendedTest.java | 22 +- .../runtime/internal/DreamServiceTest.java | 349 +++++++++++++++--- ...NatsConversationCoordinatorBranchTest.java | 27 +- ...tsConversationCoordinatorExtendedTest.java | 39 +- .../NatsConversationCoordinatorIT.java | 11 +- .../NatsConversationCoordinatorTest.java | 100 ++++- .../internal/ScheduleFireExecutorTest.java | 92 +++++ .../internal/SchedulePollerServiceTest.java | 52 +++ .../schedule/rest/RestScheduleStoreTest.java | 170 ++++++++- .../nlp/impl/RestSemanticParserCacheTest.java | 54 ++- 36 files changed, 2317 insertions(+), 262 deletions(-) create mode 100644 src/test/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibraryTest.java diff --git a/docs/architecture.md b/docs/architecture.md index f2d5245ae..dbe1a8450 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -899,7 +899,30 @@ EDDI's memory model extends beyond single conversations. The `IUserMemoryStore` - At **conversation init**, visible user memories are loaded as `longTerm` properties and made available in all templates via `{properties.key}` - During the pipeline, the LLM can autonomously store and recall facts using built-in memory tools (when enabled) - At **conversation teardown**, `longTerm` properties are persisted back to the user memory store -- **Background consolidation** (the "Dream" service) performs stale pruning, contradiction detection, and optional LLM-driven summarization. It runs on the same cluster-aware schedule machinery as every other background job — a `ScheduleConfiguration` whose metadata carries `{"dreamType": "dream_consolidation"}` together with the target `agentId` and `userId` is claimed by `SchedulePollerService` and dispatched by `ScheduleFireExecutor` to `DreamService`, which reads the agent's `userMemoryConfig.dream` block and runs one cycle. Create such a schedule through the normal schedule surface (`POST /schedules` or the `create_schedule` MCP tool) using the cron expression from `dream.schedule`. Spend is bounded per cycle by `dream.maxCostPerRun` (US dollars), and because Dream has no parent LLM task to inherit credentials from, its model credentials come from `dream.parameters` (which resolves `${vault:…}` and `${vars:…}` like any LLM task's parameters). A cycle that cannot run — no `userId`, dream disabled on the agent, or a failing LLM call — is logged at ERROR and marked FAILED on the fire log, so it retries with backoff and dead-letters rather than silently doing nothing +- **Background consolidation** (the "Dream" service) performs stale pruning, contradiction detection, and optional LLM-driven summarization. It runs on the same cluster-aware schedule machinery as every other background job — a `ScheduleConfiguration` whose `metadata` carries `{"dreamType": "dream_consolidation"}` is claimed by `SchedulePollerService` and dispatched by `ScheduleFireExecutor` to `DreamService`, which reads the agent's `userMemoryConfig.dream` block and runs one cycle. The target agent and user come from the schedule's **top-level** `agentId` / `agentVersion` / `userId` fields — `metadata` carries only the `dreamType` marker. Spend is bounded per cycle by `dream.maxCostPerRun` (US dollars), and because Dream has no parent LLM task to inherit credentials from, its model credentials come from `dream.parameters` (which resolves `${vault:…}` and `${vars:…}` like any LLM task's parameters). A cycle that cannot run — no `userId`, dream disabled on the agent, or a failing LLM call — is logged at ERROR and marked FAILED on the fire log, so it retries with backoff and dead-letters rather than silently doing nothing + +**Creating a Dream schedule** — use the raw REST body, `POST /schedulestore/schedules`, with the cron expression from `dream.schedule`: + +```json +{ + "name": "nightly dream — alice", + "agentId": "5a8b1c2d3e4f5a6b7c8d9e0f", + "agentVersion": 0, + "triggerType": "CRON", + "cronExpression": "0 3 * * *", + "timeZone": "UTC", + "userId": "alice", + "message": "dream", + "metadata": { "dreamType": "dream_consolidation" }, + "enabled": true +} +``` + +Three things this body does that are easy to get wrong: + +- **The `create_schedule` MCP tool cannot do this.** Its arguments (`agentId`, `triggerType`, `cron`, `heartbeatIntervalSeconds`, `message`, `name`, `timeZone`, `conversationStrategy`, `userId`, `environment`) contain no `metadata`, so a schedule created that way has `metadata == null`, `DreamService.isDreamSchedule(…)` returns `false`, and the schedule fires an ordinary chat turn against the agent on the dream cron forever — logged COMPLETED, consolidating nothing. REST is the only route that produces a working Dream schedule today. +- **`message` is required even though Dream never reads it.** `RestScheduleStore.validateSchedule` rejects a CRON schedule without a non-blank `message`; the Dream fast-path bypasses `say()` entirely, so the value is inert — supply any placeholder. +- **`userId` must name the real user whose memories are consolidated.** Left unset it defaults to `system:scheduler`, which `DreamService` rejects (the cycle is marked FAILED rather than consolidating an empty memory set). Memory visibility is enforced at the storage level — agents can only see memories matching their visibility scope, preventing cross-tenant memory leaks. diff --git a/docs/changelog.md b/docs/changelog.md index 0da86d581..3ac937364 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -11,6 +11,23 @@ Third wave of the 124-finding external review. **16 fixed, 1 partial.** This is the wave where the findings were hardest to fix correctly, because the bugs are non-deterministic and several of the obvious fixes are wrong. +### Pre-merge review pass (2026-07-29) + +Like #618, this PR reached "approved" without CI or any review bot having seen it — a stacked base disables CodeRabbit, and a base retarget does not fire the CI trigger. A dedicated pass over the 37-file diff produced **6 findings that survived adversarial verification (12 of 18 were refuted) plus 22 from completeness/test critics**, and Copilot found four more once CI could finally run. The high refutation rate is the point: concurrency invites "this looks racy", and verifiers were required to name the interleaving or drop the claim. + +- **A destructive primitive behind a missing ownership check (HIGH).** The schedule REST surface never checked `schedule.userId`, which on its own was inert. This PR's `dreamType=dream_consolidation` dispatch armed it: any `eddi-editor` could create and fire a schedule that **bulk-deletes another user's persistent memories**. `RestScheduleStore` already injected `OwnershipValidator` and simply did not use it here. Now admin-or-self on create, update (the re-point path) and `fireNow` — refusing with 403 rather than silently rewriting `userId`, which would hand back a schedule that does something other than what was asked. `system:scheduler` and blank ids stay exempt so existing stored schedules and Manager round-trips keep working. +- **Dream consolidation crossed agent boundaries.** `process()` read `getAllEntries(userId)` — userId-only, agent-unscoped — while every knob it obeyed came from *one* agent's config, so agent A's `pruneStaleAfterDays` deleted agent B's memories and A's model endpoint saw B's text. Cycles are now scoped to the firing agent's own `sourceAgentId` writes, with `crossAgentMaintenance: true` as an explicit opt-in. Newly reachable in this PR, which gave `process()` its first scheduled caller. +- **A transient LLM blip permanently disabled a schedule.** A single failure aborted the whole cycle and marked the fire FAILED, so three consecutive 429s dead-lettered the user's dream schedule. Transient classes (429/timeout/5xx) now skip the group and continue. +- **The B2 interrupt fix destroyed the bookkeeping it was protecting.** The restore in `fire()` ran *before* `logFire()`, and the sync Mongo driver throws `MongoInterruptedException` on connection checkout while the flag is set — so on exactly the interrupt the restore existed to handle, the FAILED fire log was lost and `failCount` never incremented. The flag is now parked and re-asserted in a `finally` after the store round trip, in both `fire()` and the Dream fast-path. The residual half was in `SchedulePollerService`, which ran `markFailed()` on the same still-interrupted thread: the schedule stayed CLAIMED with `nextFire` in the past, was re-claimed every lease expiry, and could never reach `maxRetries` — **an interrupt turned a failing schedule into an unbounded re-fire loop.** +- **A draining node answered 500 instead of "retry elsewhere".** `RestAgentEngine.sayInternal`'s trailing `catch (Exception)` swallowed the `RejectedExecutionException` from the new shutdown gate and rethrew it as a generic 500 — defeating the point of the graceful-shutdown work in this same PR. +- Also: the parallel-phase batch deadline was sized at one member *attempt*, so it always fired first and made the per-member RETRY/ABORT/attributed-SKIP branches unreachable; `maxSummarizationCalls` silently stopped being enforced for stored configs (now honoured as an explicit backstop, deprecated in favour of `maxCostPerRun`); `BaseRuntime` swallowed `onComplete` failures with no identifying context; and `WorkflowStoreClientLibrary` documented an invariant the code neither enforced nor detected — now the component key no longer depends on it at all. + +**Docs corrected against the code, not against intent** — the third and fourth instances of that error in this stack, so every claim was re-read out of the implementation: `architecture.md` told operators to create Dream schedules with the `create_schedule` MCP tool, which has **no `metadata` parameter** and therefore cannot set the marker the dispatcher matches on, so the documented procedure produced a schedule that never consolidated (REST is the only working route today, now written out with the two gotchas that bite: a `message` is required for CRON triggers even though the Dream path ignores it, and an unset `userId` defaults to `system:scheduler`, which `DreamService` refuses). `IEventBus` and `InMemoryConversationCoordinator` both claimed the coordinator is selected at runtime via `eddi.messaging.type`; it is `@IfBuildProfile("nats")`, a **build-time** condition, and that property is read by no Java code at all. + +**One disagreement adjudicated rather than deferred to severity.** The completeness critic rated the NATS C13/C10 parity gap CRITICAL; two independent verifiers refuted it because `@IfBuildProfile("nats")` keeps that class out of shipped artifacts. Both cannot be right. The code defect is real and was fixed, but the CRITICAL rating was not — it is unreachable unless someone builds with that profile, and the class now records why the two coordinators differ. + +**Two tests were relabelled rather than trusted.** The critics caught that both new `GracefulShutdownService` interrupt tests pass identically with and without the fix — `sleepQuietly` restores the flag, so the old code's next `sleep` threw immediately and it exited just as fast. That fix buys accurate logs (an interrupt was being reported as a 30-second timeout), not changed behaviour, and the tests now say so instead of implying coverage they do not have. + ### Cancellation that cancelled nothing (C1) `CompletableFuture.cancel(true)` does **not** interrupt a `runAsync`/`supplyAsync` body — the JDK documents `mayInterruptIfRunning` as having no effect there. Five call sites in `GroupConversationService` relied on it, so "cancelled" agent threads **kept mutating `gc.getTaskList()`, `gc.getTranscript()` and the errors list after the orchestrator had already persisted the document**. Replaced with a cooperative `MemberTurnCancellation` token checked at the agent turn's own await points, plus a bounded drain. This is also the root cause of C7 (`resetStrandedInProgressTasks` could strand the very task it exists to rescue, because a falsely-"cancelled" thread flips state between the snapshot and the mutate). diff --git a/docs/scheduling.md b/docs/scheduling.md index b45019998..e339117e7 100644 --- a/docs/scheduling.md +++ b/docs/scheduling.md @@ -177,7 +177,6 @@ Dream consolidation is configured in the agent configuration: "summarizeTargetEntries": 2, "summarizeGroupBy": "category", "preserveAgentProvenance": false, - "maxSummarizationCalls": 10, "llmProvider": "anthropic", "llmModel": "claude-sonnet-4-6", "maxCostPerRun": 0.50, @@ -189,6 +188,10 @@ Dream consolidation is configured in the agent configuration: } ``` +> **Scope:** a dream cycle only touches memories the **firing agent** wrote (`sourceAgentId`). Set `crossAgentMaintenance: true` to maintain the user's whole memory set across agents — without it, agent A's `pruneStaleAfterDays` would delete agent B's memories and A's model endpoint would see B's private text. +> +> `maxSummarizationCalls` is **deprecated** in favour of `maxCostPerRun` (a call count is a poor budget — consolidations differ wildly in cost). It is still honoured as a secondary backstop if a stored config sets it explicitly, so existing configurations keep their bound. + ### Cost Control Dream cycles consume LLM tokens. Use `maxCostPerRun` (in the **Agent Configuration**) to set a dollar ceiling per run: diff --git a/docs/user-memory.md b/docs/user-memory.md index 3975c1e58..046778c8a 100644 --- a/docs/user-memory.md +++ b/docs/user-memory.md @@ -100,9 +100,10 @@ Enable advanced memory features (LLM tools, Dream consolidation, guardrails, rec | `summarizeTargetEntries` | `int` | `2` | Target number of entries per group after consolidation | | `summarizeGroupBy` | `String` | `"category"` | Grouping strategy: `"category"` or `"all"` | | `preserveAgentProvenance` | `boolean` | `false` | Sub-group by `sourceAgentId` (preserves per-agent provenance) | -| `maxSummarizationCalls` | `int` | `10` | Maximum LLM calls per dream cycle per user (bounds cost) | +| `maxSummarizationCalls` | `int` | `10` | **Deprecated** — prefer `maxCostPerRun`. Still honoured as a secondary backstop *if you set it explicitly*, because silently dropping a bound an operator wrote is worse than enforcing a redundant one. A call count is a poor budget: consolidations differ wildly in cost. | | `summarizationPrompt` | `String` | *(built-in)* | Custom LLM instructions for consolidation | -| `maxCostPerRun` | `double` | `0.50` | Maximum dollar cost per dream cycle | +| `maxCostPerRun` | `double` | `0.50` | Maximum dollar cost per dream cycle — the primary ceiling | +| `crossAgentMaintenance` | `boolean` | `false` | By default a dream cycle only touches memories the **firing agent** wrote (`sourceAgentId`). Set `true` to let it maintain the user's whole memory set across agents — otherwise agent A's retention setting would delete agent B's memories, and A's model endpoint would see B's private text. | | `llmProvider` | `String` | `"anthropic"` | LLM provider for dream operations | | `llmModel` | `String` | `"claude-sonnet-4-6"` | Model for dream operations | diff --git a/src/main/java/ai/labs/eddi/configs/agents/model/AgentConfiguration.java b/src/main/java/ai/labs/eddi/configs/agents/model/AgentConfiguration.java index 506b76717..46f728f85 100644 --- a/src/main/java/ai/labs/eddi/configs/agents/model/AgentConfiguration.java +++ b/src/main/java/ai/labs/eddi/configs/agents/model/AgentConfiguration.java @@ -6,6 +6,7 @@ import ai.labs.eddi.configs.hitl.HitlTimeoutPolicy; import com.fasterxml.jackson.annotation.JsonAlias; +import com.fasterxml.jackson.annotation.JsonIgnore; import java.net.URI; import java.util.ArrayList; @@ -584,6 +585,29 @@ public static class DreamConfig { */ private boolean preserveAgentProvenance = false; + /** + * Whether this agent's dream cycle may act on memories written by + * other agents. + *

      + * Every knob in this block — {@link #getPruneStaleAfterDays()}, the grouping + * strategy, the consolidation model and its endpoint — comes from exactly one + * agent. Letting that agent's cycle delete or rewrite another agent's memories + * means a retention value their owner never configured decides when their data + * disappears, and (with {@link #isSummarizeInteractions()} on) their text is + * sent to this agent's provider. So the default is {@code false}: the cycle + * only touches entries whose {@code sourceAgentId} is the firing agent, the + * same ownership rule {@code UserMemoryTool} applies before evicting. + *

      + * Set to {@code true} for a dedicated housekeeping agent that is meant to + * maintain the user's whole memory set across agents — the cross-agent + * consolidation {@link #isPreserveAgentProvenance()}{@code =false} describes. + * Entries without a {@code sourceAgentId} (legacy/migrated rows) are only in + * scope in this mode, since no agent owns them. + * + * @since 6.1.0 + */ + private boolean crossAgentMaintenance = false; + /** * Model parameters for the consolidation LLM — {@code apiKey}, {@code baseUrl}, * {@code temperature}, … — passed through to {@code ChatModelRegistry} exactly @@ -601,15 +625,29 @@ public static class DreamConfig { private Map parameters = new HashMap<>(); /** - * @deprecated Since 6.1.0. Superseded by {@link #getMaxCostPerRun()} and no - * longer enforced. A call count is a meaningless ceiling because - * different consolidations cost vastly different amounts; Dream is - * bounded by the dollar budget instead. Retained only so existing - * stored/imported configurations keep deserializing. + * @deprecated Since 6.1.0. Superseded by {@link #getMaxCostPerRun()}, which is + * the real budget: a call count says nothing about spend, because + * different consolidations cost vastly different amounts. It is + * still enforced as a secondary backstop whenever a stored + * configuration actually carries the field + * ({@link #isMaxSummarizationCallsSet()}) — silently discarding a + * ceiling an operator wrote would let a config that says "at most 3 + * calls" make hundreds. Configurations that never set it are + * bounded by the dollar budget alone, so this field's default value + * never caps anything on its own. */ @Deprecated(since = "6.1.0", forRemoval = true) private int maxSummarizationCalls = 10; + /** + * Whether {@link #maxSummarizationCalls} was explicitly configured, as opposed + * to sitting at its default. Set by the setter, which Jackson calls only when + * the property is present in the stored/imported JSON — that is what lets the + * deprecated ceiling stay honoured for the configs that declare it without + * imposing it on the ones that do not. + */ + private boolean maxSummarizationCallsSet = false; + /** * LLM instructions for memory consolidation. Customizable by the agent * designer. Entries are appended as JSON after this prompt. @@ -744,6 +782,14 @@ public void setPreserveAgentProvenance(boolean preserveAgentProvenance) { this.preserveAgentProvenance = preserveAgentProvenance; } + public boolean isCrossAgentMaintenance() { + return crossAgentMaintenance; + } + + public void setCrossAgentMaintenance(boolean crossAgentMaintenance) { + this.crossAgentMaintenance = crossAgentMaintenance; + } + public Map getParameters() { return parameters; } @@ -753,8 +799,9 @@ public void setParameters(Map parameters) { } /** - * @deprecated Since 6.1.0. No longer enforced — see - * {@link #getMaxCostPerRun()}. + * @deprecated Since 6.1.0. Secondary backstop only, and only when + * {@link #isMaxSummarizationCallsSet()} — see + * {@link #getMaxCostPerRun()} for the real budget. */ @Deprecated(since = "6.1.0", forRemoval = true) public int getMaxSummarizationCalls() { @@ -762,12 +809,29 @@ public int getMaxSummarizationCalls() { } /** - * @deprecated Since 6.1.0. No longer enforced — see - * {@link #setMaxCostPerRun(double)}. + * @deprecated Since 6.1.0. Prefer {@link #setMaxCostPerRun(double)}. Calling + * this marks the ceiling as explicitly configured, which keeps it + * enforced as a backstop until the field is removed. */ @Deprecated(since = "6.1.0", forRemoval = true) public void setMaxSummarizationCalls(int maxSummarizationCalls) { this.maxSummarizationCalls = maxSummarizationCalls; + this.maxSummarizationCallsSet = true; + } + + /** + * True when {@link #getMaxSummarizationCalls()} was explicitly configured + * (present in the stored/imported JSON, or set programmatically) rather than + * left at its default. Never serialized — it is derived from the presence of + * {@code maxSummarizationCalls}, so it survives an export/import round trip + * without adding a field to stored agent configurations. + * + * @deprecated Since 6.1.0, together with the ceiling it guards. + */ + @JsonIgnore + @Deprecated(since = "6.1.0", forRemoval = true) + public boolean isMaxSummarizationCallsSet() { + return maxSummarizationCallsSet; } public String getSummarizationPrompt() { diff --git a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java index 7a1279a33..4499b2234 100644 --- a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java +++ b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java @@ -98,6 +98,30 @@ public class GroupConversationService implements IGroupConversationService { */ private static final int DEFAULT_AGENT_TIMEOUT_SECONDS = 180; + /** + * Default number of retries per member turn when not configured via + * {@code protocol.maxRetries}. Shared by the retry loop in + * {@code executeAgentTurn} and the batch budget a parallel phase derives from + * it, so the two cannot drift apart. + */ + private static final int DEFAULT_MAX_RETRIES = 2; + + /** + * Slack added on top of a member's own budget when a parallel phase arms its + * batch deadline. A member reaches its {@code responseFuture.get(timeout)} only + * after agent lookup, conversation start and attachment sharing, so without a + * grace the orchestrator's deadline — armed the instant the batch is dispatched + * — expires while the member is still legitimately inside its own budget. + */ + private static final int PARALLEL_BATCH_GRACE_SECONDS = 1; + + /** + * Ceiling on the derived parallel-batch budget, so an absurd + * {@code agentTimeoutSeconds} × {@code maxRetries} combination cannot overflow + * the nanosecond deadline into the past. + */ + private static final long MAX_PARALLEL_BATCH_BUDGET_SECONDS = TimeUnit.HOURS.toSeconds(24); + /** * How long an aborting orchestrator waits for cooperatively cancelled member * turns to unwind before reclaiming their tasks. Cancellation releases the @@ -1669,6 +1693,30 @@ private static boolean reserveTurn(AtomicInteger turnCounter, int maxTurns) { } } + /** + * Wall-clock budget a parallel batch gets before the orchestrator gives up on + * the speakers still running. + *

      + * It is derived from what ONE member turn may legitimately consume — its + * per-attempt {@code agentTimeoutSeconds} multiplied by the number of attempts + * {@code onAgentFailure} allows (only {@code RETRY} retries, and it retries at + * most {@code maxRetries} times) — plus {@link #PARALLEL_BATCH_GRACE_SECONDS}. + * The normalisation of both protocol values is deliberately identical to + * {@code executeAgentTurn}'s: if the orchestrator's deadline is shorter than + * the member's own, the member's timeout handling (retry / abort / attributed + * SKIP) becomes unreachable. + * + * @return the batch budget in seconds, capped at + * {@link #MAX_PARALLEL_BATCH_BUDGET_SECONDS} + */ + static long parallelBatchBudgetSeconds(ProtocolConfig protocol) { + long timeout = protocol.agentTimeoutSeconds() > 0 ? protocol.agentTimeoutSeconds() : DEFAULT_AGENT_TIMEOUT_SECONDS; + long attempts = protocol.onAgentFailure() == ProtocolConfig.MemberFailurePolicy.RETRY + ? (protocol.maxRetries() > 0 ? protocol.maxRetries() : DEFAULT_MAX_RETRIES) + 1L + : 1L; + return Math.min(timeout * attempts + PARALLEL_BATCH_GRACE_SECONDS, MAX_PARALLEL_BATCH_BUDGET_SECONDS); + } + // ================================================================= // Task-oriented phase execution (TASK_FORCE style) // ================================================================= @@ -2608,11 +2656,18 @@ private void executeParallelPhase(GroupConversation gc, AgentGroupConfiguration } }), executorService)).toList(); - int timeout = protocol.agentTimeoutSeconds() > 0 ? protocol.agentTimeoutSeconds() : DEFAULT_AGENT_TIMEOUT_SECONDS; // ONE deadline for the whole batch: these turns run concurrently, so giving // every get() the full budget in turn made the worst case N × timeout - // (10 members × 180s = 30 minutes) instead of the configured timeout. - long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(timeout); + // (10 members × 180s = 30 minutes) instead of the configured timeout. The + // budget stays independent of the batch size — that is the point — but it has + // to cover what a SINGLE member is allowed to take: its per-attempt timeout + // times the attempts onAgentFailure grants it, plus a grace for the setup it + // does before reaching its own await point. Armed at exactly one attempt, the + // orchestrator won every race: it cancelled the batch while members were still + // inside their own budget, so executeAgentTurn's TimeoutException branch — + // which owns the RETRY and ABORT policies — was unreachable in parallel phases + // and every member timeout became an unattributed SKIPPED "unknown" entry. + long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(parallelBatchBudgetSeconds(protocol)); for (int i = 0; i < futures.size(); i++) { try { long remainingNanos = Math.max(0, deadlineNanos - System.nanoTime()); @@ -2810,7 +2865,9 @@ private TranscriptEntry executeAgentTurn(GroupMember member, GroupConversation g // Call through ConversationService with retry int retries = 0; - int maxRetries = protocol.maxRetries() > 0 ? protocol.maxRetries() : 2; + // Keep both normalisations in step with parallelBatchBudgetSeconds(), which + // sizes the orchestrator's batch deadline from exactly these two values. + int maxRetries = protocol.maxRetries() > 0 ? protocol.maxRetries() : DEFAULT_MAX_RETRIES; int timeout = protocol.agentTimeoutSeconds() > 0 ? protocol.agentTimeoutSeconds() : DEFAULT_AGENT_TIMEOUT_SECONDS; while (true) { diff --git a/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngine.java b/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngine.java index 726eb90e9..9bfafa502 100644 --- a/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngine.java +++ b/src/main/java/ai/labs/eddi/engine/internal/RestAgentEngine.java @@ -49,6 +49,7 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.concurrent.RejectedExecutionException; import java.util.concurrent.TimeUnit; import static ai.labs.eddi.engine.internal.RestAgentManagement.KEY_LANG; @@ -248,6 +249,21 @@ public void onSkipped(SimpleConversationMemorySnapshot snapshot) { response.resume(Response.status(TOO_MANY_REQUESTS) .entity(Map.of("error", "quota_exceeded", "message", e.getMessage())) .type(MediaType.APPLICATION_JSON).header("Retry-After", "60").build()); + } catch (RejectedExecutionException e) { + // Same reason as the quota branch above: say() is resumed through an + // AsyncResponse, so RejectedExecutionExceptionMapper never runs and the + // generic handler below turned backpressure into a 500. Both sources — + // coordinator saturation and the graceful-shutdown gate + // (ConversationService#rejectIfShuttingDown, which fires on the request + // thread once a SIGTERM drain starts) — are retryable elsewhere, so the + // status/body/Retry-After mirror the mapper verbatim. Without this, + // POST /agents/{agentId}/conversations answered a draining node with 503 + // while say()/rerun answered 500, and clients keyed on 503 never failed over. + LOGGER.warnf("Turn rejected for conversation %s: %s", sanitize(conversationId), e.getMessage()); + response.resume(Response.status(Response.Status.SERVICE_UNAVAILABLE) + .entity(Map.of("error", "capacity_exceeded", + "message", e.getMessage() != null ? e.getMessage() : "Service temporarily unavailable")) + .type(MediaType.APPLICATION_JSON).header("Retry-After", "5").build()); } catch (Exception e) { LOGGER.error(e.getLocalizedMessage(), e); throw new InternalServerErrorException("An internal error occurred"); diff --git a/src/main/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManager.java b/src/main/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManager.java index 9e43caffc..a0b97391c 100644 --- a/src/main/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManager.java +++ b/src/main/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManager.java @@ -273,8 +273,10 @@ private void executeTaskRange(IConversationMemory conversationMemory, // Position of this task in the workflow's FULL task list. On a selective // (sublist) execution the loop index is sublist-relative, but every // index-keyed lookup below is ABSOLUTE: WorkflowStoreClientLibrary caches - // each task's component under the workflow-step index, and the HITL - // bookmark / telemetry / audit rows are read back against the full list. + // each task's component under its position in THIS task list (which is why + // a workflow step that contributes no task can no longer shift the keys of + // later ones), and the HITL bookmark / telemetry / audit rows are read back + // against the full list. // Using the relative index made a rerun look up a component key that was // never written, so the task ran with component == null and no-opped. final int absoluteIndex = indexOffset + index; @@ -466,15 +468,27 @@ private void executeTaskRange(IConversationMemory conversationMemory, } } - // Exit cancel check. The in-loop check only guards the transition INTO a task, - // so a cancel that lands while the LAST task runs was never observed: the loop + // Exit checks. The in-loop checks only guard the transition INTO a task, so an + // abort signal that lands while the LAST task runs was never observed: the loop // simply ran out, the turn returned normally, and Conversation went on to - // commit the turn's side effects (long-term property upserts) for work the - // caller was already told is cancelled. Re-checking here closes that window - // for the last task of every workflow, and for an empty/exhausted range. + // commit the turn's side effects (long-term property upserts) for work whose + // outcome the runtime then discards. Re-checking here closes that window for + // the last task of every workflow, and for an empty/exhausted range. + // + // Both signals are re-checked, mirroring the in-loop pair, because the two + // abort paths are distinct: a cooperative cancel sets the memory flag, while + // the runtime watchdog abandons a timed-out turn by interrupting this thread + // ONLY (AbandonableFuture#cancel never touches the memory flag) and routes the + // late completion to onFailure, so the conversation document is thrown away. + // This is the earlier of two guards, not the durable one — an interrupt can + // still land after this point, which is why Conversation re-checks immediately + // before the post-conversation tasks. if (conversationMemory.isCancelled()) { throw new ConversationStopException(); } + if (Thread.currentThread().isInterrupted()) { + throw new LifecycleException.LifecycleInterruptedException("Execution was interrupted!"); + } } /** diff --git a/src/main/java/ai/labs/eddi/engine/runtime/BaseRuntime.java b/src/main/java/ai/labs/eddi/engine/runtime/BaseRuntime.java index c4b16f765..c0e2fe12e 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/BaseRuntime.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/BaseRuntime.java @@ -214,10 +214,24 @@ public Future submitCallable(final Callable callable, final IFinishedE if (callbackFired.compareAndSet(false, true)) { try { completion.onComplete(result); - } catch (Throwable t) { - // Deliberately NOT routed to onFailure: the callable already ran. - log.error("Completion callback failed after the work had already executed — " - + "not reporting it as a failure to avoid re-execution", t); + } catch (RuntimeException | Error t) { + // Deliberately NOT routed to onFailure: the callable already ran, + // and every caller reads onFailure as "the work never happened" + // (the coordinator dead-letters the turn, ConversationService + // writes ERROR for a turn that actually succeeded). + // + // It must not vanish either: name the callback that failed AND + // fail the Future, so the submitter — which knows the conversation + // — still sees it. ConversationService.waitForExecutionFinishOrTimeout + // turns the resulting ExecutionException into a context-carrying + // logConversationError (conversationId + logging context + ERROR + // state), which is exactly what the pre-token onFailure route did. + log.errorf(t, "Completion callback %s failed after the work had already executed " + + "(callable=%s, thread=%s) — not reported through onFailure (callers read that as " + + "'never ran' and would re-execute); surfaced through the Future instead", + completion.getClass().getName(), callable.getClass().getName(), + Thread.currentThread().getName()); + throw t; } } return result; diff --git a/src/main/java/ai/labs/eddi/engine/runtime/IEventBus.java b/src/main/java/ai/labs/eddi/engine/runtime/IEventBus.java index 576c97e46..f7dd58b4e 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/IEventBus.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/IEventBus.java @@ -23,8 +23,12 @@ * * *

      - * Selected via config property {@code eddi.messaging.type} (default: - * {@code in-memory}). + * Selected at build time, not at runtime: the in-memory + * coordinator is the {@code @DefaultBean} and the NATS one is gated on + * {@code @IfBuildProfile("nats")}, so only an artifact built with that profile + * contains it. The {@code eddi.messaging.type} property in + * {@code application.properties} is read by no Java code and does not switch + * implementations — this javadoc previously said it did. *

      * * @author ginccc diff --git a/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java b/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java index 0e59db930..3d4e6c6ce 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java @@ -18,6 +18,7 @@ import ai.labs.eddi.engine.runtime.service.ServiceException; import ai.labs.eddi.configs.descriptors.model.DocumentDescriptor; import ai.labs.eddi.utils.RestUtilities; +import org.jboss.logging.Logger; import jakarta.enterprise.context.ApplicationScoped; import jakarta.inject.Inject; @@ -34,6 +35,8 @@ */ @ApplicationScoped public class WorkflowStoreClientLibrary implements IWorkflowStoreClientLibrary { + private static final Logger LOGGER = Logger.getLogger(WorkflowStoreClientLibrary.class); + private final IWorkflowStoreService workflowStoreService; private final Map> lifecycleExtensionsProvider; private static final String URI_SCHEME_ID = "eddi"; @@ -70,36 +73,47 @@ private IExecutableWorkflow createExecutableWorkflow(final DocumentDescriptor do try { List workflowSteps = workflowConfiguration.getWorkflowSteps(); - for (int indexInWorkflow = 0; indexInWorkflow < workflowSteps.size(); indexInWorkflow++) { - WorkflowConfiguration.WorkflowStep workflowStep = workflowSteps.get(indexInWorkflow); + + // Position of the NEXT task in the lifecycle manager's task list. It is + // deliberately not the workflow-step index: a step whose type URI does not + // use the `eddi` scheme is skipped below without producing a task, so the + // two counters diverge from that step on. LifecycleManager looks each + // component up by the task's ABSOLUTE position in its own task list + // (sublist offset added back on a selective execution), so keying off the + // raw step index would shift every later component one slot past the task + // that needs it — and the task would then run with component == null. + int indexInTaskList = 0; + + for (WorkflowConfiguration.WorkflowStep workflowStep : workflowSteps) { URI extensionType = workflowStep.getType(); - if (URI_SCHEME_ID.equals(extensionType.getScheme())) { - String type = extensionType.getHost(); - if (!lifecycleExtensionsProvider.containsKey(type)) { - throw new UnrecognizedExtensionException(String.format("Extension '%s' not found", type)); - } + if (!URI_SCHEME_ID.equals(extensionType.getScheme())) { + // Not an error — historically these were skipped silently and a + // stored config that still contains one must keep loading. But it + // disables a pipeline step, so say so instead of no-opping quietly. + LOGGER.warnf("Workflow '%s' declares step '%s', which does not use the '%s' URI scheme — " + + "the step is skipped and will not run.", + documentDescriptor.getResource(), extensionType, URI_SCHEME_ID); + continue; + } + + String type = extensionType.getHost(); + if (!lifecycleExtensionsProvider.containsKey(type)) { + throw new UnrecognizedExtensionException(String.format("Extension '%s' not found", type)); + } + + var componentKey = createComponentKey(workflowId.getId(), workflowId.getVersion(), indexInTaskList); + var lifecycleTask = lifecycleExtensionsProvider.get(type).get(); + var component = lifecycleTask.configure(workflowStep.getConfig(), workflowStep.getExtensions()); - // The component key is the task's ABSOLUTE position in the - // workflow. LifecycleManager rebuilds the identical key when it - // looks the component up, including on a selective (sublist) - // execution, where it must add the sublist offset back on. - // Invariant this relies on: every workflow step uses the `eddi` - // scheme, so indexInWorkflow and the position in the - // lifecycleManager task list stay in lockstep. A non-eddi step - // would be skipped below and desynchronize the two. - var componentKey = createComponentKey(workflowId.getId(), workflowId.getVersion(), indexInWorkflow); - var lifecycleTask = lifecycleExtensionsProvider.get(type).get(); - var component = lifecycleTask.configure(workflowStep.getConfig(), workflowStep.getExtensions()); - - if (component != null) { - if (lifecycleTask.getId() == null) { - throw new WorkflowInitializationException( - "Lifecycle task returned null TaskId: " + lifecycleTask.getClass().getName(), null); - } - componentCache.put(lifecycleTask.getId().name(), componentKey, component); + if (component != null) { + if (lifecycleTask.getId() == null) { + throw new WorkflowInitializationException( + "Lifecycle task returned null TaskId: " + lifecycleTask.getClass().getName(), null); } - lifecycleManager.addLifecycleTask(lifecycleTask); + componentCache.put(lifecycleTask.getId().name(), componentKey, component); } + lifecycleManager.addLifecycleTask(lifecycleTask); + indexInTaskList++; } } catch (IllegalExtensionConfigurationException | UnrecognizedExtensionException e) { throw new WorkflowInitializationException(e.getMessage(), e); diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/Conversation.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/Conversation.java index c8fce9877..89518a34c 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/Conversation.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/Conversation.java @@ -439,13 +439,13 @@ private void executeConversationStep(List> lifecycleData, List paused = true; } } - // A cancelled turn must not commit its side effects. ConversationService - // discards the snapshot of a cancelled turn, but storePropertiesPermanently() - // writes straight to the user-memory store, so without this guard a turn the - // caller was told is CANCELLED still upserts whatever longTerm properties it - // managed to set before stopping — the same "a failed turn must not persist + // A turn whose outcome is discarded must not commit its side effects. + // ConversationService discards the snapshot of a cancelled or abandoned turn, + // but storePropertiesPermanently() writes straight to the user-memory store, + // so without this guard such a turn still upserts whatever longTerm properties + // it managed to set before stopping — the same "a failed turn must not persist // partial state" rule the ERROR path already follows. - if (!paused && !conversationMemory.isCancelled()) { + if (!paused && !isTurnDiscarded()) { try { postConversationLifecycleTasks(); } catch (IResourceStore.ResourceStoreException e) { @@ -454,6 +454,35 @@ private void executeConversationStep(List> lifecycleData, List } } + /** + * True when this turn's result will be thrown away, so its side effects outside + * the conversation snapshot must not be committed either. + *

      + * Two independent abort signals reach a running turn, and only one of them is a + * flag on the memory: + *

        + *
      • a cooperative cancel ({@code /cancel}, client disconnect) sets + * {@code conversationMemory.setCancelled(true)}; the pipeline observes it and + * the snapshot is discarded;
        • + *
      • the runtime watchdog abandons a turn that exceeded + * {@code agentTimeoutInSeconds} by interrupting the pipeline thread — + * {@code BaseRuntime.AbandonableFuture#cancel} only sets its own + * {@code abandoned} flag and interrupts, it never sets the memory's cancel flag + * — and routes the late completion to {@code onFailure}, so the conversation + * document is never stored.
        • + *
      + * Only the first was checked here, so an interrupt that landed during the last + * task of the last workflow (a short in-memory task such as output or + * templating; interruptibly-blocking tasks throw instead) left the pipeline + * returning normally and this turn's changed longTerm properties upserted into + * {@code usermemories} for a turn whose conversation document was then thrown + * away. Narrow race, pre-existing; the interrupt flag is only read here, never + * cleared, so the runtime still sees it. + */ + private boolean isTurnDiscarded() { + return conversationMemory.isCancelled() || Thread.currentThread().isInterrupted(); + } + private void checkActionsForConversationEnd() { IData> actionData = conversationMemory.getCurrentStep().getLatestData(ACTIONS); if (actionData != null) { @@ -925,12 +954,13 @@ public void resume(HitlDecision decision) } // Persist long-term properties only on a clean outcome. Skip on a // re-pause (AWAITING_HUMAN — the pause is not the end of the turn), on - // ERROR, and on a cancel — mirroring the say path (executeConversationStep - // only runs post-tasks when execution did not throw and was not - // cancelled), so a failed or cancelled resume does not upsert - // partial/inconsistent property state into the user memory store. + // ERROR, and on a discarded turn (cancelled, or abandoned by the watchdog + // that guards the resume exactly like the say path) — mirroring + // executeConversationStep, so a failed, cancelled or abandoned resume does + // not upsert partial/inconsistent property state into the user memory + // store. if (finalState != ConversationState.AWAITING_HUMAN && finalState != ConversationState.ERROR - && !conversationMemory.isCancelled()) { + && !isTurnDiscarded()) { try { postConversationLifecycleTasks(); } catch (IResourceStore.ResourceStoreException ex) { diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java index 237b3eedf..61ba110cd 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java @@ -21,10 +21,15 @@ import jakarta.inject.Inject; import org.jboss.logging.Logger; +import java.net.ConnectException; +import java.net.SocketTimeoutException; +import java.net.UnknownHostException; import java.time.Duration; import java.time.Instant; import java.util.*; import java.util.Collection; +import java.util.concurrent.TimeoutException; +import java.util.regex.Pattern; import java.util.stream.Collectors; /** @@ -53,8 +58,19 @@ *

      * Cost ceiling: {@code maxCostPerRun} (US dollars, estimated from token usage) * bounds the spend per user per cycle. The former {@code maxSummarizationCalls} - * count is deprecated and no longer enforced — different consolidations cost - * vastly different amounts, so a call count is not a budget. + * count is deprecated — different consolidations cost vastly different amounts, + * so a call count is not a budget — but it is still honoured as a secondary + * backstop for stored configurations that set it explicitly + * ({@link AgentConfiguration.DreamConfig#isMaxSummarizationCallsSet()}), + * because silently dropping a bound an operator wrote is worse than enforcing a + * redundant one. + * + *

      + * Ownership: a cycle is configured by exactly one agent, so by default it only + * acts on memories that agent wrote ({@code sourceAgentId}) — see + * {@link AgentConfiguration.DreamConfig#isCrossAgentMaintenance()}. Otherwise + * agent A's retention value would delete agent B's memories and A's model + * endpoint would see B's private text. * * @author ginccc * @since 6.0.0 @@ -186,7 +202,7 @@ public DreamResult processScheduledFire(String agentId, Integer agentVersion, St + "(userMemoryConfig.dream.enabled=false or absent), but a dream schedule fired for it."); } - return process(userId, dreamConfig); + return process(userId, agentId, dreamConfig); } private DreamResult rejected(String userId, Instant start, String reason) { @@ -195,17 +211,61 @@ private DreamResult rejected(String userId, Instant start, String reason) { return new DreamResult(userId, 0, 0, 0, Duration.between(start, Instant.now()).toMillis(), 0.0, reason); } + /** + * Restrict a user's memory set to what the firing agent is entitled to + * maintain. + *

      + * {@code getAllEntries(userId)} is deliberately agent-unscoped (its documented + * use case is admin/export), but every knob this cycle obeys — + * {@code pruneStaleAfterDays}, the grouping strategy, the consolidation model + * and its endpoint — comes from one agent's {@code userMemoryConfig.dream}. + * Acting on the whole set would let agent A delete agent B's memories under a + * retention value B's owner never configured, and hand B's {@code self}-scoped + * text to A's provider. So the default keeps the same ownership rule + * {@code UserMemoryTool.evictableEntries()} applies before it evicts: an entry + * belongs to the agent whose id is its {@code sourceAgentId}, and nothing else + * is touched. Entries without a {@code sourceAgentId} have no owner and are + * therefore left alone as well. + *

      + * {@code crossAgentMaintenance=true} opts back into whole-set maintenance for a + * dedicated housekeeping agent — the cross-agent consolidation + * {@code preserveAgentProvenance=false} describes. + */ + private static List scopeToOwningAgent(List entries, String agentId, + AgentConfiguration.DreamConfig dreamConfig) { + if (dreamConfig.isCrossAgentMaintenance()) { + return entries; + } + + List owned = entries.stream() + .filter(entry -> agentId != null && agentId.equals(entry.sourceAgentId())) + .toList(); + + int foreign = entries.size() - owned.size(); + if (foreign > 0) { + LOGGER.infof("[DREAM] Skipping %d of %d memory entries not owned by agent '%s' — set " + + "userMemoryConfig.dream.crossAgentMaintenance=true if this agent is meant to maintain " + + "the user's memories across agents.", foreign, entries.size(), agentId); + } + return owned; + } + /** * Process dream consolidation for a specific user's memories. Called by * {@link #processScheduledFire} when a Dream schedule fires. * * @param userId * the user whose memories to consolidate + * @param agentId + * the agent whose {@code dreamConfig} governs this cycle — also the + * ownership boundary: unless + * {@link AgentConfiguration.DreamConfig#isCrossAgentMaintenance()}, + * only memories this agent wrote are touched * @param dreamConfig * the dream configuration from the agent * @return a summary of what was done */ - public DreamResult process(String userId, AgentConfiguration.DreamConfig dreamConfig) { + public DreamResult process(String userId, String agentId, AgentConfiguration.DreamConfig dreamConfig) { Instant start = Instant.now(); int pruned = 0; int contradictions = 0; @@ -214,10 +274,10 @@ public DreamResult process(String userId, AgentConfiguration.DreamConfig dreamCo String summarizationError = null; try { - LOGGER.infof("[DREAM] Starting dream cycle for user='%s'", userId); + LOGGER.infof("[DREAM] Starting dream cycle for user='%s', agent='%s'", userId, agentId); // Load entries once — shared across pruning and contradiction detection - List allEntries = userMemoryStore.getAllEntries(userId); + List allEntries = scopeToOwningAgent(userMemoryStore.getAllEntries(userId), agentId, dreamConfig); // 1. Prune stale entries (deterministic, zero LLM cost) if (dreamConfig.getPruneStaleAfterDays() > 0) { @@ -227,7 +287,7 @@ public DreamResult process(String userId, AgentConfiguration.DreamConfig dreamCo // After pruning, reload once — shared by contradiction detection and // summarization List currentEntries = pruned > 0 - ? userMemoryStore.getAllEntries(userId) + ? scopeToOwningAgent(userMemoryStore.getAllEntries(userId), agentId, dreamConfig) : allEntries; // 2. Detect contradictions (read-only — does not modify entries) @@ -346,8 +406,13 @@ record SummarizationOutcome(int entriesReduced, double estimatedCostUsd, String *

    • If insert fails, originals are preserved
      • *
    • If LLM returns empty/garbage, the group is skipped
      • *
    • If LLM returns more entries than input, the group is skipped
      • - *
    • Cost bounded by {@code maxCostPerRun} (estimated from token usage)
      • - *
    • An LLM failure aborts the phase and is reported, never swallowed
      • + *
    • Cost bounded by {@code maxCostPerRun} (estimated from token usage), plus + * the deprecated {@code maxSummarizationCalls} when a config sets it
      • + *
    • A permanent LLM failure (auth, endpoint, unknown model) aborts + * the phase and is reported, never swallowed
      • + *
    • A transient LLM failure (rate limit, timeout, 5xx) skips its + * group and is logged, but does not fail the cycle — see + * {@link #isTransientLlmFailure}
      • * */ private SummarizationOutcome summarizeInteractions(String userId, @@ -355,6 +420,7 @@ private SummarizationOutcome summarizeInteractions(String userId, AgentConfiguration.DreamConfig config) { int totalConsolidated = 0; int llmCallsMade = 0; + int transientFailures = 0; double estimatedCostAccumulated = 0.0; // 1. Build groups @@ -371,24 +437,40 @@ private SummarizationOutcome summarizeInteractions(String userId, // Respect cost ceiling (soft cap: checked before each call, so the // last call may push total slightly over — this is by design, since // we cannot know output cost before the call). This dollar budget is - // the ONLY ceiling; the legacy maxSummarizationCalls count is not - // enforced because a call count says nothing about spend. + // the primary ceiling, because a call count says nothing about spend. if (estimatedCostAccumulated >= config.getMaxCostPerRun()) { LOGGER.infof("[DREAM] Cost ceiling ($%.4f >= $%.2f) reached for user='%s' " + "after %d calls", estimatedCostAccumulated, config.getMaxCostPerRun(), userId, llmCallsMade); break; } + // Deprecated secondary backstop: a stored configuration that explicitly + // sets maxSummarizationCalls asked for a hard call ceiling, and silently + // dropping it would let "at most 3 calls" turn into hundreds under the + // dollar budget alone. Configs that never set it are bounded by + // maxCostPerRun only — the field's default value caps nothing. + if (config.isMaxSummarizationCallsSet() && llmCallsMade >= config.getMaxSummarizationCalls()) { + LOGGER.warnf("[DREAM] Legacy call ceiling maxSummarizationCalls=%d reached for user='%s' " + + "after $%.4f of an allowed $%.2f. This field is deprecated — configure maxCostPerRun " + + "instead, which bounds actual spend.", + config.getMaxSummarizationCalls(), userId, estimatedCostAccumulated, config.getMaxCostPerRun()); + break; + } + // 2. Build content: JSON array of entries String content = buildEntriesJson(groupEntries); // 3. Call LLM. Dream is a background job with no parent LLM task to // inherit credentials from, so the model parameters come from the - // agent's dream config (finding I1/F13). A failure here is almost + // agent's dream config (finding I1/F13). A PERMANENT failure is almost // always a configuration fault that would repeat for every remaining // group — abort the phase, log at ERROR and report it upward so the // schedule fire is marked FAILED, instead of leaving Dream to look - // like it ran and simply found nothing to do. + // like it ran and simply found nothing to do. A TRANSIENT failure + // (rate limit, timeout, 5xx) is not a fault of the configuration and + // must not be reported as a failed fire: three of them in a row would + // exhaust the schedule's retry budget and dead-letter the user's dream + // schedule for good over what is typically a minutes-long provider blip. SummarizationService.SummarizationResult llmResult; try { llmResult = summarizationService.summarizeWithUsage( @@ -397,6 +479,14 @@ private SummarizationOutcome summarizeInteractions(String userId, config.getParameters()); } catch (Exception e) { summarizationFailedCounter.increment(); + if (isTransientLlmFailure(e)) { + transientFailures++; + LOGGER.warnf(e, "[DREAM] Memory consolidation LLM call failed transiently for user='%s', group='%s' " + + "(provider=%s, model=%s). Original entries are preserved; skipping this group and continuing. " + + "The cycle is NOT marked failed, so a provider blip cannot dead-letter the dream schedule.", + userId, group.getKey(), config.getLlmProvider(), config.getLlmModel()); + continue; + } LOGGER.errorf(e, "[DREAM] Memory consolidation LLM call failed for user='%s', group='%s' " + "(provider=%s, model=%s, configured parameter keys=%s). Original entries are preserved and " + "consolidation is ABORTED for this cycle. If this is an authentication or endpoint error, set the " @@ -515,9 +605,49 @@ private SummarizationOutcome summarizeInteractions(String userId, } } + if (transientFailures > 0) { + LOGGER.warnf("[DREAM] %d of %d groups were skipped for user='%s' after transient LLM failures — " + + "they are retried on the next dream cycle.", transientFailures, groups.size(), userId); + } + return new SummarizationOutcome(totalConsolidated, estimatedCostAccumulated, null); } + /** + * Transient-failure signatures in an exception message — throttling, timeouts + * and server-side 5xx, which resolve on their own. Mirrors + * {@code CascadingModelExecutor.isRetryableError} / + * {@code AgentExecutionHelper} (both private to their modules). + */ + private static final Pattern TRANSIENT_LLM_FAILURE = Pattern.compile( + "timeout|timed out|rate limit|too many requests|429|50[234]|529|overloaded|temporarily unavailable", + Pattern.CASE_INSENSITIVE); + + /** + * Whether an LLM failure is transient (retry later succeeds) rather than a + * permanent configuration fault (bad credentials, wrong endpoint, unknown + * model). Only the latter should fail the schedule fire, because a FAILED fire + * consumes the schedule's dead-letter budget. + */ + static boolean isTransientLlmFailure(Throwable throwable) { + Throwable current = throwable; + // Bounded walk — a self-referential cause chain must not spin forever. + for (int depth = 0; current != null && depth < 10; depth++, current = current.getCause()) { + if (current instanceof SocketTimeoutException || current instanceof TimeoutException + || current instanceof ConnectException || current instanceof UnknownHostException) { + return true; + } + String message = current.getMessage(); + if (message != null && TRANSIENT_LLM_FAILURE.matcher(message).find()) { + return true; + } + if (current.getCause() == current) { + break; + } + } + return false; + } + /** * The configured model parameter keys — never the values, which hold * credentials. Used to make a failure diagnosable without leaking secrets. diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinator.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinator.java index 8a3e3758a..6d22c97fe 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinator.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/InMemoryConversationCoordinator.java @@ -26,10 +26,17 @@ * In-memory implementation of {@link IConversationCoordinator}. * *

      - * This is the default event bus — uses in-process queues with no external - * dependencies. Suitable for single-instance deployments. For horizontal - * scaling, use {@code NatsConversationCoordinator} by setting - * {@code eddi.messaging.type=nats}. + * This is the default event bus ({@code @DefaultBean}) — in-process queues, no + * external dependencies, suitable for single-instance deployments. + *

      + * + *

      + * For horizontal scaling there is {@code NatsConversationCoordinator}, but it + * is gated on {@code @IfBuildProfile("nats")} — a build-time + * condition. It exists only in an artifact built with that profile, and setting + * {@code eddi.messaging.type=nats} at runtime does NOT switch coordinators: + * that property sits in {@code application.properties} but no Java code reads + * it. The previous wording here promised a runtime switch that does not exist. *

      * *

      diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java index 8a84e081e..1ae817283 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java @@ -34,8 +34,18 @@ * *

      * Uses NATS JetStream for durable, ordered message processing per conversation. - * Activated when {@code eddi.messaging.type=nats} is set in - * application.properties. + *

      + * + *

      + * Activation is BUILD-time, not runtime: this bean carries + * {@code @IfBuildProfile("nats")}, so it only exists in an artifact built with + * {@code -Dquarkus.profile=nats}. The stock distribution (and the published + * Docker image) is built without it and therefore always runs + * {@link InMemoryConversationCoordinator}, the {@code @DefaultBean}; setting + * {@code eddi.messaging.type=nats} at runtime does NOT switch coordinators. + * Nothing here is on the shipped path — which is exactly why the + * failure-handling rules below have to be maintained deliberately rather than + * being caught by production traffic. *

      * *

      @@ -47,12 +57,25 @@ *

      * *

      - * Dead-letter handling: When a task fails more than {@code maxRetries} - * times, the message is published to a dead-letter stream - * ({@code eddi.deadletter.}) with 30-day retention for operator - * inspection and replay. + * Dead-letter handling: A failed task is published to a dead-letter + * stream ({@code eddi.deadletter.}) with 30-day retention for + * operator inspection and replay. *

      * + *

      Failure handling (kept in lockstep with + * {@link InMemoryConversationCoordinator})

      + *
        + *
      • No retry after execution starts: a task that reports failure has + * already run — possibly calling an LLM, executing tools and spending money. It + * is dead-lettered once, never re-executed. {@code eddi.nats.max-retries} is + * retained as a configuration key (so existing deployments keep loading) but no + * longer governs re-execution of a conversation turn.
        • + *
      • Submission rejection rolls back: if handing the task to the + * runtime throws, the task is taken back off the queue (and the map entry + * dropped when it was the head), so a rejected submission cannot wedge the + * conversation.
        • + *
      + * *

      * For horizontal scaling, a future enhancement will serialize InputData instead * of Callable, allowing cross-instance message consumption. @@ -242,7 +265,8 @@ public void submitInOrder(String conversationId, Callable callable) { } boolean wasEmpty = queue.isEmpty(); - boolean enqueued = queue.offer(new RetryableCallable(callable)); + RetryableCallable retryable = new RetryableCallable(callable); + boolean enqueued = queue.offer(retryable); if (!enqueued) { log.warnf("Failed to enqueue task for conversationId=%s", safeConversationId); throw new java.util.concurrent.RejectedExecutionException( @@ -250,13 +274,48 @@ public void submitInOrder(String conversationId, Callable callable) { } if (wasEmpty) { - publishAndExecute(conversationId, queue, queue.element()); + try { + publishAndExecute(conversationId, queue, retryable); + } catch (RuntimeException | java.lang.Error e) { + // C10: the submission failed, so NOTHING is scheduled to run + // the head of this queue — and submitNext() only ever runs + // from a completion callback. Leaving the callable queued + // would wedge this conversation permanently (every later turn + // sees a non-empty queue and just waits) and leak the map + // entry for the JVM's lifetime. Undo the enqueue and drop the + // now-empty queue so the next turn starts a fresh one. + // + // We still hold the queue monitor, so nothing can have been + // offered in between: our callable is the only element. + queue.remove(retryable); + if (queue.isEmpty()) { + conversationQueues.remove(conversationId, queue); + } + log.warnf("Submission failed for conversationId=%s — rolled the task back off the queue " + + "so the conversation stays usable", safeConversationId); + throw e; + } } return; // success } } } + /** + * Publishes the ordering marker to NATS and hands the task to the runtime. + * Throws (synchronously) if the SUBMISSION itself is rejected — the only + * genuinely pre-execution failure mode; callers must un-queue the task in that + * case (C10). A failed NATS publish is not fatal: the callable is still + * executed locally, as before. + *

      + * C13: there is deliberately NO retry on {@code onFailure}. That callback is + * only ever raised from INSIDE the executor task, i.e. after the turn has + * already started running — it may have called an LLM, executed tools, written + * memory and spent money. Re-running the very same callable repeats all of it. + * A failed turn is dead-lettered once and the queue moves on. This mirrors + * {@link InMemoryConversationCoordinator} exactly; the two coordinators must + * not diverge on failure semantics. + */ private void publishAndExecute(String conversationId, BlockingQueue queue, RetryableCallable retryable) { String subject = SUBJECT_PREFIX + sanitizeSubject(conversationId); @@ -289,20 +348,15 @@ public void onComplete(Void result) { @Override public void onFailure(Throwable t) { recordConsumeMetrics(consumeStart); + // The attempt counter is still advanced so the dead-letter log records + // how often this conversation's head task was handed to the runtime, + // but it never triggers a re-execution (see the C13 note above). int attempt = retryable.incrementAndGetAttempt(); - - if (attempt < maxRetries) { - log.warnf(t, "Conversation task failed (conversationId=%s, attempt=%d/%d), retrying...", sanitize(conversationId), attempt, - maxRetries); - // Re-execute the same callable (retry) - publishAndExecute(conversationId, queue, retryable); - } else { - log.errorf(t, "Conversation task exhausted retries (conversationId=%s, attempts=%d), " + "routing to dead-letter", - sanitize(conversationId), - attempt); - routeToDeadLetter(conversationId, t); - submitNext(conversationId, queue); - } + log.errorf(t, "Conversation task failed after it had already started (conversationId=%s, attempts=%d) — " + + "dead-lettering without retry; re-running it would repeat any side effects it already performed", + sanitize(conversationId), attempt); + routeToDeadLetter(conversationId, t); + submitNext(conversationId, queue); } }, null); } @@ -338,16 +392,30 @@ private void routeToDeadLetter(String conversationId, Throwable failure) { private void submitNext(String conversationId, BlockingQueue queue) { synchronized (queue) { - if (!queue.isEmpty()) { - queue.remove(); + if (queue.isEmpty()) { + return; + } + queue.remove(); // drop the task that just finished - if (!queue.isEmpty()) { + while (!queue.isEmpty()) { + try { publishAndExecute(conversationId, queue, queue.element()); - } else { - // Eager cleanup: remove empty queue to prevent memory leaks. - conversationQueues.remove(conversationId, queue); + return; + } catch (RuntimeException | java.lang.Error e) { + // C10 (submitNext side): there is no caller to propagate to here — + // this runs from a completion callback. Dropping out would leave + // the queue non-empty with nothing scheduled to drain it, wedging + // the conversation forever. Dead-letter the task we could not + // schedule and try the next one. + log.errorf(e, "Failed to schedule the next queued task (conversationId=%s) — dead-lettering it " + + "so the conversation queue keeps draining", sanitize(conversationId)); + routeToDeadLetter(conversationId, e); + queue.remove(); } } + + // Eager cleanup: remove empty queue to prevent memory leaks. + conversationQueues.remove(conversationId, queue); } } @@ -497,7 +565,10 @@ private long extractTimestamp(String json) { } /** - * @return the max retries configuration value (for testing) + * @return the {@code eddi.nats.max-retries} configuration value. Retained so + * existing deployments keep starting; it no longer causes a + * conversation turn to be re-executed (see the class javadoc, "Failure + * handling"). */ int getMaxRetries() { return maxRetries; @@ -511,7 +582,10 @@ private Optional getMetrics() { } /** - * Wraps a Callable with a retry attempt counter. + * Wraps a Callable with an attempt counter. The counter is diagnostic only — it + * records how often the head task of a conversation was handed to the runtime + * and is reported in the dead-letter log. It never drives a re-execution: a + * turn that reported failure has already run. */ static class RetryableCallable { private final Callable callable; diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java index b18e6a262..bfde36bf1 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutor.java @@ -118,6 +118,7 @@ public ScheduleFireLog fire(ScheduleConfiguration schedule, String instanceId, i String errorMessage = null; String status; double cost = 0.0; + boolean interrupted = false; try { Environment env = resolveEnvironment(schedule.getEnvironment()); @@ -150,11 +151,10 @@ public ScheduleFireLog fire(ScheduleConfiguration schedule, String instanceId, i } catch (Exception e) { // B2: latch.await() above CLEARS the interrupt flag when it throws // InterruptedException, and this broad catch would otherwise swallow the - // poller thread's shutdown signal — it would keep firing further schedules - // while the executor is shutting down. Restore it before continuing; the - // fire is still logged FAILED below so the attempt stays visible. + // cancellation signal entirely. Only REMEMBER it here — re-asserting the + // flag now would break the fire log below (see the finally). if (e instanceof InterruptedException) { - Thread.currentThread().interrupt(); + interrupted = true; } status = ScheduleConfiguration.FireStatus.FAILED.name(); errorMessage = e.getClass().getSimpleName() + ": " + e.getMessage(); @@ -169,11 +169,31 @@ public ScheduleFireLog fire(ScheduleConfiguration schedule, String instanceId, i scheduleStore.logFire(fireLog); } catch (Exception e) { LOGGER.errorf(e, "[SCHEDULE] Failed to log fire for schedule %s", schedule.getId()); + } finally { + restoreInterrupt(interrupted); } return fireLog; } + /** + * Re-assert an interrupt that was consumed by a blocking call inside this fire, + * so the cancellation signal still reaches the caller. + *

      + * Ordering matters: this MUST run only after the store round trips this method + * owns. The synchronous MongoDB driver checks out a connection with + * {@code lockInterruptibly()} and aborts with {@code MongoInterruptedException} + * when the calling thread's flag is already set, so restoring the flag before + * {@code logFire} would destroy the FAILED fire log that the interrupt path + * exists to write — leaving the attempt invisible on exactly the path where it + * matters most. + */ + private static void restoreInterrupt(boolean interrupted) { + if (interrupted) { + Thread.currentThread().interrupt(); + } + } + /** * Run one Dream memory-consolidation cycle for the schedule's user and record * the fire. @@ -187,6 +207,7 @@ private ScheduleFireLog fireDreamConsolidation(ScheduleConfiguration schedule, S String status; String errorMessage = null; double cost = 0.0; + boolean interrupted = false; try { // userId is passed through unchanged — DreamService rejects a missing or @@ -208,14 +229,13 @@ private ScheduleFireLog fireDreamConsolidation(ScheduleConfiguration schedule, S errorMessage); } } catch (Exception e) { - // Same B2 reasoning as fire() above, and it has to be repeated here because - // this catch is just as broad: a blocking call inside Dream consolidation - // CLEARS the interrupt flag when it throws InterruptedException, so - // swallowing it would leave the poller thread running further schedules - // through a shutdown. The fix landing on only one of two sibling catches in - // the same class is exactly how these gaps happen. + // Same B2 reasoning — and the same ordering — as fire() above, repeated here + // because this catch is just as broad: a blocking call inside Dream + // consolidation CLEARS the interrupt flag when it throws + // InterruptedException. Remember it and re-assert it in the finally below, + // AFTER the fire log is written. if (e instanceof InterruptedException) { - Thread.currentThread().interrupt(); + interrupted = true; } status = ScheduleConfiguration.FireStatus.FAILED.name(); errorMessage = e.getClass().getSimpleName() + ": " + e.getMessage(); @@ -228,6 +248,8 @@ private ScheduleFireLog fireDreamConsolidation(ScheduleConfiguration schedule, S scheduleStore.logFire(fireLog); } catch (Exception e) { LOGGER.errorf(e, "[SCHEDULE] Failed to log dream fire for schedule %s", schedule.getId()); + } finally { + restoreInterrupt(interrupted); } return fireLog; } diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/SchedulePollerService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/SchedulePollerService.java index 388ab0e8d..3e8bf74ba 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/SchedulePollerService.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/SchedulePollerService.java @@ -256,6 +256,7 @@ private void dispatchClaimed(List claimed) { * error isolation for the concurrent dispatch depends on this. */ private void fireClaimedSchedule(ScheduleConfiguration schedule) { + boolean wasInterrupted = false; try { // Fix #4: compute correct attempt number from schedule state int attemptNumber = schedule.getFailCount() + 1; @@ -264,6 +265,17 @@ private void fireClaimedSchedule(ScheduleConfiguration schedule) { fireCounter.increment(); ScheduleFireLog fireLog = fireDurationTimer.record(() -> fireExecutor.fire(schedule, instanceId, attemptNumber)); + // fire() deliberately re-asserts the interrupt flag before returning, so the + // signal is not lost. Park it for the duration of the bookkeeping below and + // restore it in the finally: markFailed()/markCompleted() are Mongo writes, and + // the sync driver throws MongoInterruptedException on connection checkout while + // the flag is set. onFireFailed() swallows that, so failCount would never + // increment — leaving the schedule CLAIMED with nextFire in the past, + // re-claimed + // on every lease expiry, and unable to ever reach maxRetries or dead-letter. + // An interrupt must not turn a failing schedule into an unbounded re-fire loop. + wasInterrupted = Thread.interrupted(); + // Handle result if (FireStatus.COMPLETED.name().equals(fireLog.status())) { onFireCompleted(schedule); @@ -272,11 +284,17 @@ private void fireClaimedSchedule(ScheduleConfiguration schedule) { } } catch (Exception e) { LOGGER.errorf(e, "[SCHEDULE] Error processing schedule %s", schedule.getId()); + // Same reasoning as above: the bookkeeping write must not run under a set flag. + wasInterrupted |= Thread.interrupted(); try { onFireFailed(schedule); } catch (Exception nested) { LOGGER.errorf(nested, "[SCHEDULE] Could not mark schedule %s as failed", schedule.getId()); } + } finally { + if (wasInterrupted) { + Thread.currentThread().interrupt(); + } } } diff --git a/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java b/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java index 8af9218c9..4f46413b3 100644 --- a/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java +++ b/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java @@ -15,6 +15,7 @@ import ai.labs.eddi.engine.runtime.internal.CronParser; import ai.labs.eddi.engine.runtime.internal.ScheduleFireExecutor; import ai.labs.eddi.engine.runtime.internal.SchedulePollerService; +import ai.labs.eddi.engine.hitl.HitlSchedules; import ai.labs.eddi.engine.security.OwnershipValidator; import io.quarkus.security.identity.SecurityIdentity; import jakarta.enterprise.context.ApplicationScoped; @@ -51,8 +52,15 @@ public class RestScheduleStore implements IRestScheduleStore { * side-step the owner/admin/approver check on {@code /resume}) or disarm them * (defeating an ABORT/AUTO_REJECT deadline). */ - private static final String HITL_TYPE_KEY = ai.labs.eddi.engine.hitl.HitlSchedules.METADATA_TYPE_KEY; - private static final String HITL_TYPE_TIMEOUT = ai.labs.eddi.engine.hitl.HitlSchedules.METADATA_TYPE_TIMEOUT; + private static final String HITL_TYPE_KEY = HitlSchedules.METADATA_TYPE_KEY; + private static final String HITL_TYPE_TIMEOUT = HitlSchedules.METADATA_TYPE_TIMEOUT; + + /** + * Placeholder identity for schedules that act for the system rather than for a + * specific end user. It is not a real principal — {@code DreamService} refuses + * to consolidate memories for it — so ownership checks treat it as unowned. + */ + private static final String SCHEDULER_USER_ID = "system:scheduler"; @Inject IScheduleStore scheduleStore; @@ -126,6 +134,13 @@ public Response createSchedule(ScheduleConfiguration schedule) { return bodyGuard; } + // A schedule runs AS its userId — refuse to mint one that acts as + // somebody else (see requireOwnUserId). + Response ownerGuard = requireOwnUserId(schedule != null ? schedule.getUserId() : null, "create"); + if (ownerGuard != null) { + return ownerGuard; + } + // Validate validateSchedule(schedule); @@ -169,6 +184,13 @@ public Response updateSchedule(String scheduleId, ScheduleConfiguration schedule return guard; } + // Checked on the incoming BODY: this is the path that would re-point an + // otherwise harmless schedule at another user's identity. + Response ownerGuard = requireOwnUserId(schedule != null ? schedule.getUserId() : null, "update"); + if (ownerGuard != null) { + return ownerGuard; + } + validateSchedule(schedule); // Recompute nextFire @@ -244,6 +266,14 @@ public Response fireNow(String scheduleId) { + "or terminate via POST /agents/{conversationId}/cancel.") .build(); } + // Checked on the STORED schedule: firing one that acts as another user is + // the step that actually executes as them — including the + // dreamType=dream_consolidation fast-path, which prunes and rewrites that + // user's persistent memories. + Response ownerGuard = requireOwnUserId(schedule.getUserId(), "fire"); + if (ownerGuard != null) { + return ownerGuard; + } ScheduleFireLog fireLog = fireExecutor.fire(schedule, pollerService.getInstanceId(), 1); return Response.ok(fireLog).build(); } catch (IResourceStore.ResourceNotFoundException e) { @@ -352,6 +382,48 @@ private Response rejectHitlTimeoutBody(ScheduleConfiguration schedule, String op return null; } + /** + * A schedule's {@code userId} is the identity every fire ACTS AS: it becomes + * the owner of the conversation the fire starts, and for a + * {@code dreamType=dream_consolidation} schedule it is the user whose + * persistent memories {@code DreamService} prunes, rewrites and permanently + * deletes. The direct memory API ({@code IRestUserMemoryStore}) refuses a + * non-admin access to another user's memories on every method, so this surface + * must not become a back door around it: a non-admin may only create, re-point + * or manually fire a schedule that runs as themselves, or as the unowned + * {@value #SCHEDULER_USER_ID} placeholder (which Dream consolidation explicitly + * rejects). + *

      + * Deliberately refuses instead of silently rewriting {@code userId} to the + * caller: a rewrite would hand back a schedule that does something other than + * what was asked for, and would hide the attempt. Blank/absent and placeholder + * identities are left alone, so existing stored schedules and system schedules + * keep working unchanged. All checks are no-ops when + * {@code authorization.enabled=false}. + * + * @param userId + * the identity the schedule would run as (may be {@code null}) + * @param operation + * the operation name, for the log line and error message + * @return a 403 {@link Response} to short-circuit the caller, or {@code null} + * when the operation may proceed + */ + private Response requireOwnUserId(String userId, String operation) { + if (userId == null || userId.isBlank() || SCHEDULER_USER_ID.equals(userId)) { + return null; // no end-user identity to act as + } + if (ownershipValidator.isAdmin(identity) || ownershipValidator.isOwner(identity, userId)) { + return null; + } + LOGGER.warnf("Refused %s of a schedule running as another user by a non-admin caller", sanitize(operation)); + LOGGER.debugf("Schedule ownership detail: operation='%s', userId='%s'", sanitize(operation), sanitize(userId)); + return Response.status(Response.Status.FORBIDDEN) + .entity("A schedule may only run as yourself: set userId to your own identity, or leave it " + + "unset to run as the system scheduler. Only an administrator may " + operation + + " a schedule that runs as another user.") + .build(); + } + /** * For mutating operations on a HITL timeout schedule, require the eddi-admin * role. Reads the STORED schedule so a request body cannot hide the marker. The @@ -424,7 +496,7 @@ private void applyDefaults(ScheduleConfiguration schedule) { schedule.setEnvironment("production"); } if (schedule.getUserId() == null || schedule.getUserId().isBlank()) { - schedule.setUserId("system:scheduler"); + schedule.setUserId(SCHEDULER_USER_ID); } if (schedule.getConversationStrategy() == null || schedule.getConversationStrategy().isBlank()) { // Heartbeats default to persistent, cron to new diff --git a/src/main/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParser.java b/src/main/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParser.java index 67b92e880..4d2a48845 100644 --- a/src/main/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParser.java +++ b/src/main/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParser.java @@ -18,6 +18,7 @@ import ai.labs.eddi.modules.nlp.internal.matches.RawSolution; import com.github.benmanes.caffeine.cache.Cache; import com.github.benmanes.caffeine.cache.Caffeine; +import com.github.benmanes.caffeine.cache.Ticker; import org.jboss.logging.Logger; import jakarta.enterprise.context.ApplicationScoped; @@ -52,10 +53,17 @@ public class RestSemanticParser implements IRestSemanticParser { static final int MAX_CACHED_PARSERS = 100; /** - * Bounds how long an edited parser configuration keeps being served from the - * cache. Without a TTL, a config change would only take effect after a restart. + * How long a built parser is retained after it was put into the cache. + *

      + * It is deliberately not what makes an edited configuration visible: + * the cache key is the versioned resource URI ({@code parserId} + + * {@code version}) and the parser store is historized — updating a parser + * configuration writes version+1, so an edit is served under a different key + * from the moment it is saved. What the TTL bounds is retention: a parser built + * from a version that has since been superseded or deleted eventually stops + * occupying a cache slot instead of living until the process restarts. */ - private static final Duration PARSER_CACHE_TTL = Duration.ofMinutes(5); + static final Duration PARSER_CACHE_TTL = Duration.ofMinutes(5); private final IRuntime runtime; private final IResourceClientLibrary resourceClientLibrary; @@ -75,6 +83,16 @@ public class RestSemanticParser implements IRestSemanticParser { @Inject public RestSemanticParser(IRuntime runtime, IResourceClientLibrary resourceClientLibrary, @LifecycleExtensions Map> lifecycleTasks) { + this(runtime, resourceClientLibrary, lifecycleTasks, Ticker.systemTicker()); + } + + /** + * Test seam: the same construction with an injectable clock, so cache expiry + * can be driven deterministically instead of by waiting + * {@link #PARSER_CACHE_TTL}. + */ + RestSemanticParser(IRuntime runtime, IResourceClientLibrary resourceClientLibrary, + Map> lifecycleTasks, Ticker ticker) { this.runtime = runtime; this.resourceClientLibrary = resourceClientLibrary; this.parserProvider = lifecycleTasks.get("ai.labs.parser"); @@ -82,6 +100,7 @@ public RestSemanticParser(IRuntime runtime, IResourceClientLibrary resourceClien this.parserCache = Caffeine.newBuilder() .maximumSize(MAX_CACHED_PARSERS) .expireAfterWrite(PARSER_CACHE_TTL) + .ticker(ticker) .build(); } @@ -139,11 +158,15 @@ private IInputParser createParser(URI resourceUri) { /** * Drops all cached parsers so that the next request re-reads the parser - * configuration from the store. Intended as the explicit hook for parser - * configuration updates; until a store wires it up, the TTL is what bounds - * staleness. + * configuration from the store. + *

      + * Not public, and deliberately not wired into the parser store: that store is + * historized, so an update writes a new version and therefore a new cache key — + * there is nothing for an update hook to invalidate. This exists so the cache + * tests can assert that cached parsers really are rebuilt from the store, not + * as a configuration-update contract. */ - public void invalidateCache() { + void invalidateCache() { parserCache.invalidateAll(); } diff --git a/src/test/java/ai/labs/eddi/configs/agents/model/AgentConfigurationTest.java b/src/test/java/ai/labs/eddi/configs/agents/model/AgentConfigurationTest.java index 211faeffd..636627f14 100644 --- a/src/test/java/ai/labs/eddi/configs/agents/model/AgentConfigurationTest.java +++ b/src/test/java/ai/labs/eddi/configs/agents/model/AgentConfigurationTest.java @@ -4,6 +4,7 @@ */ package ai.labs.eddi.configs.agents.model; +import com.fasterxml.jackson.databind.ObjectMapper; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Nested; import org.junit.jupiter.api.Test; @@ -293,6 +294,55 @@ void setters() { assertEquals(10.0, dc.getMaxCostPerRun()); } + /** + * Ownership default: a dream cycle is configured by one agent, so it must not + * reach into other agents' memories unless the operator says so. + */ + @Test + void crossAgentMaintenanceDefaultsToFalse() { + var dc = new AgentConfiguration.DreamConfig(); + assertFalse(dc.isCrossAgentMaintenance()); + dc.setCrossAgentMaintenance(true); + assertTrue(dc.isCrossAgentMaintenance()); + } + + /** + * Config-compat: {@code maxSummarizationCalls} is deprecated but still honoured + * as a backstop for the configs that declare it. That distinction rests + * entirely on the presence marker, which must be false for a config that never + * mentions the field and true for one that does — including after a JSON round + * trip, since stored agent JSON in MongoDB is the compatibility contract. + */ + @Test + void maxSummarizationCallsMarkerTracksExplicitConfiguration() { + var untouched = new AgentConfiguration.DreamConfig(); + assertFalse(untouched.isMaxSummarizationCallsSet()); + assertEquals(10, untouched.getMaxSummarizationCalls(), "the default value itself is unchanged"); + + var configured = new AgentConfiguration.DreamConfig(); + configured.setMaxSummarizationCalls(3); + assertTrue(configured.isMaxSummarizationCallsSet()); + assertEquals(3, configured.getMaxSummarizationCalls()); + } + + @Test + void maxSummarizationCallsMarkerSurvivesJsonRoundTrip() throws Exception { + var mapper = new ObjectMapper(); + + var legacyStoredConfig = mapper.readValue("{\"enabled\":true,\"maxSummarizationCalls\":3}", + AgentConfiguration.DreamConfig.class); + assertTrue(legacyStoredConfig.isMaxSummarizationCallsSet(), "a stored config that sets the ceiling must keep it enforced"); + assertEquals(3, legacyStoredConfig.getMaxSummarizationCalls()); + + var withoutTheField = mapper.readValue("{\"enabled\":true}", AgentConfiguration.DreamConfig.class); + assertFalse(withoutTheField.isMaxSummarizationCallsSet(), "an absent field must not become a ceiling"); + + // The marker is derived, never written back into stored agent JSON + String serialized = mapper.writeValueAsString(legacyStoredConfig); + assertFalse(serialized.contains("maxSummarizationCallsSet"), "serialized: " + serialized); + assertTrue(serialized.contains("maxSummarizationCalls")); + } + @Test void parametersDefaultToEmptyAndSetterIsNullSafe() { var dc = new AgentConfiguration.DreamConfig(); diff --git a/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java index 6edb7faea..c81ff95d9 100644 --- a/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java +++ b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java @@ -25,6 +25,7 @@ import ai.labs.eddi.datastore.serialization.IJsonSerialization; import ai.labs.eddi.engine.api.IConversationService; import ai.labs.eddi.engine.api.IGroupConversationService.GroupDiscussionEventListener; +import ai.labs.eddi.engine.lifecycle.GroupConversationEventSink; import ai.labs.eddi.engine.lifecycle.model.ControlSignal; import ai.labs.eddi.engine.lifecycle.model.DiscussionControlToken; import ai.labs.eddi.engine.memory.model.ConversationOutput; @@ -78,12 +79,16 @@ * task back to IN_PROGRESS behind the reset sweep, stranding it forever. *

    • C2 — the live {@code Collections.synchronizedList} transcript was * published by reference into every member conversation's context and then - * iterated on the member's own thread.
      • + * iterated on the member's own thread. Both hand-off sites are covered: the + * parallel phase and the single-member follow-up. *
    • C6 — the {@code maxTurns} budget was enforced with a * check-then-act on an {@code AtomicInteger} shared by N parallel agent * threads.
      • *
    • C8 — the parallel-phase timeout was applied serially, so N hanging - * members cost N × timeout instead of one timeout.
      • + * members cost N × timeout instead of one timeout; the single batch deadline + * that replaced it must still cover one member's full attempt envelope, or the + * orchestrator cancels members that are inside their own budget and their + * configured retry never lands. * */ @DisplayName("GroupConversationService — concurrency regressions") @@ -246,28 +251,55 @@ void executionWave_withEightMembers_neverExceedsMaxTurns() throws Exception { } gc.setTaskList(taskList); - // Rendezvous inside the first turn of every member, so all 8 agent threads - // reach the budget check for their second task at the same moment. Once all - // 8 have arrived the latch stays open, so later turns pass straight through. - var rendezvous = new CountDownLatch(memberCount); doAnswer(inv -> { - rendezvous.countDown(); - rendezvous.await(30, TimeUnit.SECONDS); handlerOf(inv.getArgument(8)).onComplete(snapshot("contribution")); return null; }).when(conversationService).say(any(), any(), any(), any(), any(), any(), any(), anyBoolean(), any()); + // Park all 8 agent threads on a barrier at the last point the production code + // reaches BEFORE the contended budget check: the onSpeakerComplete callback of + // their first task. All 8 are released in the same instant and nothing but the + // loop back-edge stands between that release and the check for turn 9, so the + // adversarial interleaving is forced rather than hoped for. (The single-shot + // latch this test used before was reached during turn 1 and stood open by the + // time the threads got to the turn-2 check: they drifted apart across + // completeTask, and a split check-then-act budget would pass unnoticed.) + var atContendedCheck = new CyclicBarrier(memberCount); + var completions = new AtomicInteger(); + var barrierFailure = new AtomicReference(); + var listener = new GroupDiscussionEventListener() { + @Override + public void onSpeakerComplete(GroupConversationEventSink.SpeakerCompleteEvent event) { + // First turn of each member only: the increments 1..8 all happen before + // any thread is released, so a completion numbered > 8 belongs to the + // single winner of the race — which must not wait for seven threads + // that have already broken out of the loop. + if (completions.incrementAndGet() <= memberCount) { + try { + atContendedCheck.await(30, TimeUnit.SECONDS); + } catch (Throwable t) { + barrierFailure.compareAndSet(null, t); + } + } + } + }; + var turnCounter = new AtomicInteger(0); invoke(executionPhaseMethod(), gc, config(members), members, - phase(PhaseType.EXECUTE, TurnOrder.PARALLEL), protocol(30), QUESTION, 0, null, + phase(PhaseType.EXECUTE, TurnOrder.PARALLEL), protocol(30), QUESTION, 0, listener, turnCounter, maxTurns); + assertNull(barrierFailure.get(), () -> "the 8 threads never met at the contended check: " + barrierFailure.get()); long completed = gc.getTaskList().all().stream() .filter(t -> t.status() == TaskStatus.COMPLETED).count(); assertEquals((long) maxTurns, completed, "only the turns the budget allows may run"); assertEquals((long) (2 * memberCount - maxTurns), gc.getTaskList().all().stream() .filter(t -> t.status() == TaskStatus.ASSIGNED).count(), "the remaining tasks stay ASSIGNED — untouched, not half-executed"); + assertEquals(0L, gc.getTaskList().all().stream() + .filter(t -> t.status() == TaskStatus.IN_PROGRESS).count(), + "a thread that loses the budget race must not have started a task — the check and " + + "startTask are fused under the task-list monitor"); assertEquals(maxTurns, turnCounter.get(), "the turn counter must land exactly on the budget"); verify(conversationService, times(maxTurns)) .say(any(), any(), any(), any(), any(), any(), any(), anyBoolean(), any()); @@ -389,14 +421,125 @@ void parallelPhase_appliesOneDeadlineAcrossAllMembers() throws Exception { assertTrue(entered.await(30, TimeUnit.SECONDS), "all members should have been dispatched in parallel"); assertTrue(elapsedMs >= 1500, - () -> "the batch must still honour its 2s deadline, took " + elapsedMs + "ms"); + () -> "the batch must still honour its deadline (2s member budget + 1s grace), took " + elapsedMs + "ms"); assertTrue(elapsedMs < 6000, - () -> "5 hanging members must not each restart the 2s budget (10s serial), took " + elapsedMs + "ms"); + () -> "5 hanging members must not each restart the budget (10s serial), took " + elapsedMs + "ms"); assertEquals(5L, gc.getTranscript().stream() .filter(e -> e.type() == TranscriptEntryType.SKIPPED).count(), "every hanging member is recorded as SKIPPED exactly once"); } + @Test + @Timeout(120) + @DisplayName("C8: the batch deadline covers the retries onAgentFailure=RETRY promises") + void parallelPhase_batchDeadline_coversTheMemberRetryEnvelope() throws Exception { + var gc = groupConversation("gc-batch-retry"); + + // 1s per attempt, RETRY with maxRetries=2 → the member is allowed three + // attempts, so the batch budget must be 3 × 1s (+ grace), not 1 × 1s. The + // member burns two full attempt timeouts before answering on the third, i.e. + // it finishes ~1s AFTER a one-attempt batch deadline however the orchestrator + // and the member thread interleave — sized at one attempt, the orchestrator + // cancels the batch and the retried answer is replaced by a SKIPPED "unknown". + var protocol = new ProtocolConfig(1, ProtocolConfig.MemberFailurePolicy.RETRY, 2, + ProtocolConfig.MemberUnavailablePolicy.SKIP); + + var attempts = new AtomicInteger(); + doAnswer(inv -> { + // Attempts 1 and 2 never answer: the member runs into its OWN timeout and + // takes executeAgentTurn's RETRY branch. Attempt 3 answers immediately. + if (attempts.incrementAndGet() >= 3) { + handlerOf(inv.getArgument(8)).onComplete(snapshot("retried answer")); + } + return null; + }).when(conversationService).say(any(), any(), any(), any(), any(), any(), any(), anyBoolean(), any()); + + invoke(parallelPhaseMethod(), gc, config(List.of(member(0))), List.of(member(0)), + phase(PhaseType.OPINION, TurnOrder.PARALLEL), protocol, QUESTION, 0, null, + new AtomicInteger(0), 50); + + assertEquals(3, attempts.get(), "the member must get all three attempts its failure policy grants"); + assertEquals(1, gc.getTranscript().size(), "one speaker, one transcript entry"); + TranscriptEntry entry = gc.getTranscript().get(0); + assertEquals("retried answer", entry.content(), + "the retried contribution must survive — a batch deadline sized for a single attempt " + + "cancels the member before its configured retry can complete"); + assertEquals("agent-0", entry.speakerAgentId(), + "the entry belongs to the member that answered, not to the orchestrator's \"unknown\" timeout entry"); + assertEquals(TranscriptEntryType.OPINION, entry.type(), "a completed retry is a contribution, not a SKIP"); + } + + @Test + @DisplayName("C8: the batch budget is one member's attempt envelope, never the batch size") + void parallelBatchBudget_isDerivedFromOneMembersAttemptEnvelope() { + // SKIP / ABORT never retry: one attempt plus the 1s setup grace. + assertEquals(11L, GroupConversationService.parallelBatchBudgetSeconds( + new ProtocolConfig(10, ProtocolConfig.MemberFailurePolicy.SKIP, 2, + ProtocolConfig.MemberUnavailablePolicy.SKIP))); + assertEquals(11L, GroupConversationService.parallelBatchBudgetSeconds( + new ProtocolConfig(10, ProtocolConfig.MemberFailurePolicy.ABORT, 2, + ProtocolConfig.MemberUnavailablePolicy.SKIP))); + // RETRY: maxRetries + 1 attempts, so the batch cannot cut a retry short. + assertEquals(31L, GroupConversationService.parallelBatchBudgetSeconds( + new ProtocolConfig(10, ProtocolConfig.MemberFailurePolicy.RETRY, 2, + ProtocolConfig.MemberUnavailablePolicy.SKIP))); + // Unset values fall back to the same defaults executeAgentTurn applies: + // 180s per attempt, 2 retries. + assertEquals(541L, GroupConversationService.parallelBatchBudgetSeconds( + new ProtocolConfig(0, ProtocolConfig.MemberFailurePolicy.RETRY, 0, + ProtocolConfig.MemberUnavailablePolicy.SKIP))); + // An absurd config is capped instead of overflowing the deadline into the past. + assertEquals(TimeUnit.HOURS.toSeconds(24), GroupConversationService.parallelBatchBudgetSeconds( + new ProtocolConfig(Integer.MAX_VALUE, ProtocolConfig.MemberFailurePolicy.RETRY, Integer.MAX_VALUE, + ProtocolConfig.MemberUnavailablePolicy.SKIP))); + } + + // ================================================================= + // C2 — the follow-up hand-off must snapshot too + // ================================================================= + + @Test + @Timeout(60) + @DisplayName("C2: a follow-up to one member publishes a transcript snapshot, never the live list") + @SuppressWarnings("unchecked") + void followUpWithMember_publishesTranscriptSnapshot_notTheLiveList() throws Exception { + var gc = groupConversation("gc-followup"); + gc.setState(GroupConversationState.COMPLETED); + gc.setMemberConversationIds(Map.of("agent-0", "conv-followup")); + gc.addMemberDisplayName("agent-0", "Agent 0"); + for (int i = 0; i < 5; i++) { + gc.getTranscript().add(transcriptEntry("seed-" + i)); + } + + doReturn(gc).when(conversationStore).read("gc-followup"); + doReturn(true).when(conversationStore).compareAndSetState("gc-followup", + GroupConversationState.COMPLETED, GroupConversationState.IN_PROGRESS); + + var liveListPublished = new AtomicInteger(0); + var handedOver = new AtomicReference>(); + doAnswer(inv -> { + InputData inputData = inv.getArgument(6); + Object published = inputData.getContext().get("groupTranscript").getValue(); + if (published == gc.getTranscript()) { + liveListPublished.incrementAndGet(); + } + handedOver.set((List) published); + handlerOf(inv.getArgument(8)).onComplete(snapshot("follow-up answer")); + return null; + }).when(conversationService).say(any(), any(), any(), any(), any(), any(), any(), anyBoolean(), any()); + + service.followUpWithMember("gc-followup", "agent-0", "And what about X?"); + + assertEquals(0, liveListPublished.get(), + "the live synchronized transcript must never be handed to the member conversation by reference — " + + "it is serialised on the member's thread while this one keeps appending"); + assertNotNull(handedOver.get(), "the member must have received a groupTranscript"); + assertEquals(6, handedOver.get().size(), + "the member sees the transcript as of its turn: 5 seed entries + the follow-up question"); + assertEquals(7, gc.getTranscript().size(), + "the live transcript grew by the agent's answer afterwards — proof the member did not hold it"); + } + // ================================================================= // Helpers // ================================================================= diff --git a/src/test/java/ai/labs/eddi/engine/internal/RestAgentEngineTest.java b/src/test/java/ai/labs/eddi/engine/internal/RestAgentEngineTest.java index 978ca8fc3..e01341964 100644 --- a/src/test/java/ai/labs/eddi/engine/internal/RestAgentEngineTest.java +++ b/src/test/java/ai/labs/eddi/engine/internal/RestAgentEngineTest.java @@ -33,6 +33,7 @@ import java.net.URI; import java.util.List; import java.util.Map; +import java.util.concurrent.RejectedExecutionException; import static org.junit.jupiter.api.Assertions.*; import static org.mockito.Mockito.*; @@ -380,6 +381,56 @@ void resourceNotFound() throws Exception { verify(asyncResponse).resume(any(jakarta.ws.rs.NotFoundException.class)); } + @Test + @DisplayName("should resume with 503 + Retry-After, not 500, when the node is draining") + void rejectedWhileShuttingDown() throws Exception { + var asyncResponse = mock(AsyncResponse.class); + var inputData = new InputData("Hello", Map.of()); + + // ConversationService#rejectIfShuttingDown throws this synchronously on the + // request thread once GracefulShutdownService flips isShuttingDown. + doThrow(new RejectedExecutionException( + "This node is shutting down and no longer accepts new conversation turns — retry against another node")) + .when(conversationService).say(anyString(), any(), any(), any(), any(), anyBoolean(), any()); + + // Without the explicit catch this fell through to the generic handler, which + // THROWS InternalServerErrorException instead of resuming — so the assertions + // below (resume called at all, and with 503) both fail on the old code. + restAgentEngine.sayWithinContext("conv-1", false, false, + List.of(), inputData, asyncResponse); + + var captor = ArgumentCaptor.forClass(Response.class); + verify(asyncResponse).resume(captor.capture()); + Response resumed = captor.getValue(); + assertEquals(503, resumed.getStatus()); + assertEquals("5", resumed.getHeaderString("Retry-After")); + assertEquals(Map.of("error", "capacity_exceeded", + "message", "This node is shutting down and no longer accepts new conversation turns" + + " — retry against another node"), + resumed.getEntity()); + } + + @Test + @DisplayName("should resume with 503 when the coordinator rejects with a null message") + void rejectedWithoutMessage() throws Exception { + var asyncResponse = mock(AsyncResponse.class); + var inputData = new InputData("Hello", Map.of()); + + doThrow(new RejectedExecutionException()) + .when(conversationService).say(anyString(), any(), any(), any(), any(), anyBoolean(), any()); + + restAgentEngine.sayWithinContext("conv-1", false, false, + List.of(), inputData, asyncResponse); + + var captor = ArgumentCaptor.forClass(Response.class); + verify(asyncResponse).resume(captor.capture()); + Response resumed = captor.getValue(); + assertEquals(503, resumed.getStatus()); + // mirrors RejectedExecutionExceptionMapper's null-message fallback + assertEquals(Map.of("error", "capacity_exceeded", "message", "Service temporarily unavailable"), + resumed.getEntity()); + } + @Test @DisplayName("should throw ISE for generic exception") void genericException() throws Exception { @@ -392,6 +443,7 @@ void genericException() throws Exception { assertThrows(InternalServerErrorException.class, () -> restAgentEngine.sayWithinContext("conv-1", false, false, List.of(), inputData, asyncResponse)); + verify(asyncResponse, never()).resume(any(Response.class)); } } @@ -432,6 +484,24 @@ void delegatesWithRerunFlag() throws Exception { assertEquals("", capturedInput.getInput()); assertTrue(capturedInput.getContext().containsKey("lang")); } + + @Test + @DisplayName("should resume with 503 while draining (rerun shares sayInternal)") + void rejectedWhileShuttingDown() throws Exception { + var asyncResponse = mock(AsyncResponse.class); + + doThrow(new RejectedExecutionException("node draining")) + .when(conversationService).say(anyString(), any(), any(), any(), any(), anyBoolean(), any()); + + restAgentEngine.rerunLastConversationStep("conv-1", "en", false, false, + List.of(), asyncResponse); + + var captor = ArgumentCaptor.forClass(Response.class); + verify(asyncResponse).resume(captor.capture()); + Response resumed = captor.getValue(); + assertEquals(503, resumed.getStatus()); + assertEquals("5", resumed.getHeaderString("Retry-After")); + } } @Nested diff --git a/src/test/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManagerTest.java b/src/test/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManagerTest.java index c14107a7a..372868a89 100644 --- a/src/test/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManagerTest.java +++ b/src/test/java/ai/labs/eddi/engine/lifecycle/internal/LifecycleManagerTest.java @@ -492,6 +492,84 @@ void interruptedThread() { Thread.interrupted(); } } + + /** + * The in-loop interrupt check only guards the transition INTO a task, so an + * interrupt that landed while the LAST task ran was never observed: the loop + * ran out and the turn returned as if it had completed cleanly. That matters + * because the runtime watchdog abandons a timed-out turn by interrupting the + * pipeline thread and nothing else — {@code AbandonableFuture#cancel} sets its + * own {@code abandoned} flag and interrupts, it never sets the memory's cancel + * flag — and then discards the completion, so the "clean" turn went on to + * commit long-term property writes for a conversation document that was thrown + * away. + */ + @Test + @Timeout(15) + @DisplayName("an interrupt that lands WHILE the last task runs stops the turn (watchdog abandonment)") + void interruptDuringLastTaskIsObserved() throws Exception { + var task = mock(ILifecycleTask.class); + when(task.getId()).thenReturn(new TaskId("ai.labs.output")); + when(task.getType()).thenReturn("output"); + lifecycleManager.addLifecycleTask(task); + + var memory = mock(IConversationMemory.class); + var currentStep = mock(IConversationMemory.IWritableConversationStep.class); + when(memory.getCurrentStep()).thenReturn(currentStep); + when(memory.getConversationId()).thenReturn("conv1"); + when(memory.getAgentId()).thenReturn("agent1"); + when(componentCache.getComponentMap(anyString())).thenReturn(new HashMap<>()); + // The abandonment path signals ONLY by interrupt — the cancel flag stays + // false, which is exactly why re-checking isCancelled() alone is not enough. + when(memory.isCancelled()).thenReturn(false); + + var pipelineThread = Thread.currentThread(); + var taskEntered = new CountDownLatch(1); + + // The task returns only once the interrupt has actually landed — that is + // the interleaving under test and it makes the race deterministic without + // a sleep. It must NOT block interruptibly: an await() would consume the + // interrupt and clear the flag before the exit check reads it. + doAnswer(invocation -> { + taskEntered.countDown(); + long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (!Thread.currentThread().isInterrupted()) { + if (System.nanoTime() - deadlineNanos > 0) { + fail("watchdog thread never interrupted the pipeline thread"); + } + Thread.onSpinWait(); + } + return null; + }).when(task).execute(any(), any()); + + var watchdog = new Thread(() -> { + try { + assertTrue(taskEntered.await(10, TimeUnit.SECONDS), "task never started"); + pipelineThread.interrupt(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }, "watchdog-abandon"); + watchdog.start(); + + boolean interruptFlagSurvived; + try { + assertThrows(LifecycleException.LifecycleInterruptedException.class, + () -> lifecycleManager.executeLifecycle(memory, null)); + } finally { + // Read (and thereby clear) the flag BEFORE joining: join() is itself + // interruptible, so on a still-interrupted thread it would throw and + // clear the flag before it could be asserted on. + interruptFlagSurvived = Thread.interrupted(); + watchdog.join(10_000); + } + + // The task ran to completion (an interrupt is not a kill), but the turn + // must not be reported as clean — and the flag must still be set on exit + // so the runtime's own abandonment check still sees it. + verify(task).execute(any(), any()); + assertTrue(interruptFlagSurvived, "the exit check must not swallow the interrupt flag"); + } } @Nested @@ -1669,9 +1747,9 @@ void newKeyUncommitted() throws Exception { } /** - * C5 — the component cache is keyed by the task's ABSOLUTE index in the - * workflow ({@code WorkflowStoreClientLibrary} writes - * {@code createComponentKey(id, version, indexInWorkflow)}), but a selective + * C5 — the component cache is keyed by the task's ABSOLUTE position in the task + * list ({@code WorkflowStoreClientLibrary} writes + * {@code createComponentKey(id, version, indexInTaskList)}), but a selective * execution hands the loop a SUBLIST, so the loop index is sublist-relative. * Building the lookup key from that relative index resolved a key that was * never written, the task ran with {@code component == null} and no-opped — diff --git a/src/test/java/ai/labs/eddi/engine/runtime/BaseRuntimeConcurrencyTest.java b/src/test/java/ai/labs/eddi/engine/runtime/BaseRuntimeConcurrencyTest.java index 26f216f7f..026b7d0e2 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/BaseRuntimeConcurrencyTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/BaseRuntimeConcurrencyTest.java @@ -17,6 +17,7 @@ import java.util.concurrent.Callable; import java.util.concurrent.CountDownLatch; import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.Future; @@ -174,10 +175,20 @@ public void onFailure(Throwable t) { * therefore reported the turn as failed, and the coordinator's retry then * re-executed a callable that had ALREADY run (duplicate LLM calls, duplicate * tool side effects, duplicate cost). + * + *

      + * Suppressing onFailure must not turn the failure into silence: the throw is + * surfaced through the Future instead, so the submitter (which knows the + * conversation) can still log it with context and flip the state — + * {@code ConversationService.waitForExecutionFinishOrTimeout} maps the + * resulting {@link java.util.concurrent.ExecutionException} onto + * {@code logConversationError} + ERROR. Swallowing it here (log-only) would + * leave the caller unable to tell a persisted turn from a lost one. + *

      */ @Test @Timeout(30) - @DisplayName("C9: a throwing completion callback is not reported as a failure") + @DisplayName("C9: a throwing completion callback is not reported as a failure, but fails the Future") void throwingCompletionCallbackIsNotRoutedToOnFailure() throws Exception { AtomicInteger executions = new AtomicInteger(); CountDownLatch onCompleteCalled = new CountDownLatch(1); @@ -204,7 +215,11 @@ public void onFailure(Throwable t) { assertTrue(onCompleteCalled.await(10, TimeUnit.SECONDS), "onComplete should have been invoked"); // The Future settles only after the callback dispatch, so once it is done the // decision about onFailure has already been made — no polling window needed. - assertEquals("ok", future.get(10, TimeUnit.SECONDS)); + ExecutionException thrown = assertThrows(ExecutionException.class, () -> future.get(10, TimeUnit.SECONDS), + "a completion callback that blew up must reach the submitter through the Future — " + + "logging it inside BaseRuntime leaves nobody able to act on it"); + assertInstanceOf(IllegalStateException.class, thrown.getCause()); + assertEquals("completion callback blew up", thrown.getCause().getMessage()); assertEquals(1, onFailureCalled.getCount(), "onFailure must not fire after the work already executed — callers read it as 'never ran' and re-execute"); assertEquals(1, executions.get(), "the callable must be executed exactly once"); diff --git a/src/test/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibraryTest.java b/src/test/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibraryTest.java new file mode 100644 index 000000000..a88d23bb1 --- /dev/null +++ b/src/test/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibraryTest.java @@ -0,0 +1,204 @@ +/* + * Copyright EDDI contributors + * SPDX-License-Identifier: Apache-2.0 + */ +package ai.labs.eddi.engine.runtime.client.workflows; + +import ai.labs.eddi.configs.descriptors.model.DocumentDescriptor; +import ai.labs.eddi.configs.workflows.model.WorkflowConfiguration; +import ai.labs.eddi.engine.lifecycle.IComponentCache; +import ai.labs.eddi.engine.lifecycle.ILifecycleTask; +import ai.labs.eddi.engine.lifecycle.TaskId; +import ai.labs.eddi.engine.memory.IConversationMemory; +import ai.labs.eddi.engine.runtime.service.IWorkflowStoreService; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import jakarta.inject.Provider; +import java.net.URI; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.ArgumentMatchers.anyMap; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +/** + * Component-cache keying in {@link WorkflowStoreClientLibrary}. + * + *

      + * {@code LifecycleManager} resolves each task's component by the task's + * position in ITS OWN task list. {@code WorkflowStoreClientLibrary} writes + * those cache entries while walking the workflow STEP list — and a step whose + * type URI does not use the {@code eddi} scheme is skipped without producing a + * task. Keying the write off the raw step index therefore shifts every + * component after such a step one slot past the task that needs it, and those + * tasks run with {@code component == null} (i.e. silently no-op with their + * configuration missing). + */ +class WorkflowStoreClientLibraryTest { + + private static final String WORKFLOW_ID = "5a8b1c2d3e4f5a6b7c8d9e0f"; + private static final int WORKFLOW_VERSION = 1; + + private static final String PARSER_COMPONENT = "parser-config"; + private static final String OUTPUT_COMPONENT = "output-config"; + private static final String BEHAVIOR_COMPONENT = "behavior-config"; + + private IWorkflowStoreService workflowStoreService; + private RecordingComponentCache componentCache; + private Map> extensions; + + private ILifecycleTask parserTask; + private ILifecycleTask behaviorTask; + private ILifecycleTask outputTask; + + @BeforeEach + void setUp() throws Exception { + workflowStoreService = mock(IWorkflowStoreService.class); + componentCache = new RecordingComponentCache(); + extensions = new HashMap<>(); + + parserTask = task("ai.labs.parser", "expressions", PARSER_COMPONENT); + behaviorTask = task("ai.labs.behavior", "behavior_rules", BEHAVIOR_COMPONENT); + outputTask = task("ai.labs.output", "output", OUTPUT_COMPONENT); + + extensions.put("ai.labs.parser", () -> parserTask); + extensions.put("ai.labs.behavior", () -> behaviorTask); + extensions.put("ai.labs.output", () -> outputTask); + + var descriptor = new DocumentDescriptor(); + descriptor.setName("workflow"); + descriptor.setDescription("test workflow"); + descriptor.setResource(URI.create( + "eddi://ai.labs.workflow/workflowstore/workflows/" + WORKFLOW_ID + "?version=" + WORKFLOW_VERSION)); + when(workflowStoreService.getWorkflowDocumentDescriptor(WORKFLOW_ID, WORKFLOW_VERSION)).thenReturn(descriptor); + } + + private static ILifecycleTask task(String id, String type, Object component) throws Exception { + var task = mock(ILifecycleTask.class); + when(task.getId()).thenReturn(new TaskId(id)); + when(task.getType()).thenReturn(type); + when(task.configure(anyMap(), anyMap())).thenReturn(component); + return task; + } + + private static WorkflowConfiguration workflowOf(String... stepTypes) { + var configuration = new WorkflowConfiguration(); + List steps = new LinkedList<>(); + for (String stepType : stepTypes) { + var step = new WorkflowConfiguration.WorkflowStep(); + step.setType(URI.create(stepType)); + steps.add(step); + } + configuration.setWorkflowSteps(steps); + return configuration; + } + + private WorkflowStoreClientLibrary libraryFor(WorkflowConfiguration configuration) throws Exception { + when(workflowStoreService.getKnowledgeWorkflow(WORKFLOW_ID, WORKFLOW_VERSION)).thenReturn(configuration); + return new WorkflowStoreClientLibrary(workflowStoreService, componentCache, extensions); + } + + private static IConversationMemory memory() { + var memory = mock(IConversationMemory.class); + when(memory.getCurrentStep()).thenReturn(mock(IConversationMemory.IWritableConversationStep.class)); + when(memory.getConversationId()).thenReturn("conv1"); + when(memory.getAgentId()).thenReturn("agent1"); + return memory; + } + + @Test + @DisplayName("all-eddi workflow: components are cached at 0,1,2 (guards the normal path)") + void everyStepProducesATaskSoKeysAreTheStepIndices() throws Exception { + var library = libraryFor(workflowOf( + "eddi://ai.labs.parser", + "eddi://ai.labs.behavior", + "eddi://ai.labs.output")); + + library.getExecutableWorkflow(WORKFLOW_ID, WORKFLOW_VERSION); + + assertEquals(List.of(WORKFLOW_ID + ":1:0", WORKFLOW_ID + ":1:1", WORKFLOW_ID + ":1:2"), + componentCache.keysInPutOrder); + } + + @Test + @DisplayName("a non-eddi step does not shift the component keys of the steps after it") + void nonEddiStepDoesNotShiftLaterComponentKeys() throws Exception { + var library = libraryFor(workflowOf( + "eddi://ai.labs.parser", + "https://third.party/some.extension", // skipped — produces no task + "eddi://ai.labs.output")); + + library.getExecutableWorkflow(WORKFLOW_ID, WORKFLOW_VERSION); + + // The output task is task #1 in the lifecycle manager, so its component must + // be cached under index 1. Keying off the workflow step index wrote ":1:2". + assertEquals(List.of(WORKFLOW_ID + ":1:0", WORKFLOW_ID + ":1:1"), componentCache.keysInPutOrder); + } + + @Test + @DisplayName("after a non-eddi step, every remaining task still receives its component at execution time") + void tasksAfterASkippedStepStillResolveTheirComponent() throws Exception { + var library = libraryFor(workflowOf( + "eddi://ai.labs.parser", + "https://third.party/some.extension", + "eddi://ai.labs.output")); + + var workflow = library.getExecutableWorkflow(WORKFLOW_ID, WORKFLOW_VERSION); + + var memory = memory(); + workflow.getLifecycleManager().executeLifecycle(memory, null); + + verify(parserTask).execute(memory, PARSER_COMPONENT); + // Before the fix this was execute(memory, null): LifecycleManager looked the + // output task up at its task-list position (1) while the component had been + // written at the workflow-step position (2). + verify(outputTask).execute(memory, OUTPUT_COMPONENT); + } + + @Test + @DisplayName("a non-eddi step is skipped, not rejected — a stored config containing one still loads") + void nonEddiStepDoesNotFailWorkflowCreation() throws Exception { + var library = libraryFor(workflowOf( + "https://third.party/some.extension", + "eddi://ai.labs.parser")); + + var workflow = library.getExecutableWorkflow(WORKFLOW_ID, WORKFLOW_VERSION); + + assertEquals(WORKFLOW_ID, workflow.getWorkflowId()); + // The leading skip must not push the parser's component to index 1. + assertEquals(List.of(WORKFLOW_ID + ":1:0"), componentCache.keysInPutOrder); + + var memory = memory(); + workflow.getLifecycleManager().executeLifecycle(memory, null); + verify(parserTask).execute(memory, PARSER_COMPONENT); + } + + /** + * Minimal in-memory {@link IComponentCache} that also records the keys written, + * in order, so the tests can assert on the exact cache layout rather than only + * on what happens to resolve. + */ + private static final class RecordingComponentCache implements IComponentCache { + private final Map> componentsByType = new HashMap<>(); + private final List keysInPutOrder = new ArrayList<>(); + + @Override + public Map getComponentMap(String type) { + return componentsByType.computeIfAbsent(type, t -> new HashMap<>()); + } + + @Override + public void put(String type, String key, Object component) { + keysInPutOrder.add(key); + getComponentMap(type).put(key, component); + } + } +} diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/ConversationCancelPersistenceTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/ConversationCancelPersistenceTest.java index 9fc7cd0bc..044980d3f 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/ConversationCancelPersistenceTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/ConversationCancelPersistenceTest.java @@ -11,21 +11,28 @@ import ai.labs.eddi.engine.lifecycle.IConversation; import ai.labs.eddi.engine.lifecycle.ILifecycleManager; import ai.labs.eddi.engine.lifecycle.exceptions.ConversationStopException; +import ai.labs.eddi.engine.lifecycle.model.HitlDecision; import ai.labs.eddi.engine.memory.ConversationMemory; import ai.labs.eddi.engine.memory.IPropertiesHandler; +import ai.labs.eddi.engine.memory.model.ConversationState; import ai.labs.eddi.engine.runtime.IExecutableWorkflow; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Timeout; +import org.mockito.stubbing.Answer; import java.util.LinkedHashMap; import java.util.List; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyInt; import static org.mockito.Mockito.*; /** @@ -65,6 +72,16 @@ void setUp() { when(workflow.getLifecycleManager()).thenReturn(lifecycleManager); } + /** + * Registers the pipeline stub. Not a {@link java.util.function.Consumer}: the + * {@code ILifecycleManager} methods being stubbed declare checked exceptions, + * so the registration lambda has to be allowed to throw. + */ + @FunctionalInterface + private interface StubRegistration { + void register(Answer answer) throws Exception; + } + /** Mirrors {@code Agent#continueConversation}: a NEW Conversation per turn. */ private Conversation nextTurn() { return new Conversation(List.of(workflow), memory, propertiesHandler, @@ -76,16 +93,19 @@ private Conversation nextTurn() { * under a cancel: it runs, observes the flag, and aborts with * {@link ConversationStopException}. The flag itself is set by a second thread * while the pipeline is inside the workflow. + *

      + * The stub is registered by the caller because the say path and the resume path + * enter the pipeline through different {@code ILifecycleManager} methods. */ - private Thread cancelWhilePipelineRuns() throws Exception { + private Thread cancelWhilePipelineRuns(StubRegistration stubRegistration) throws Exception { var pipelineEntered = new CountDownLatch(1); var cancelApplied = new CountDownLatch(1); - doAnswer(invocation -> { + stubRegistration.register(invocation -> { pipelineEntered.countDown(); assertTrue(cancelApplied.await(10, TimeUnit.SECONDS), "canceller thread did not run"); throw new ConversationStopException(); - }).when(lifecycleManager).executeLifecycle(any(), any()); + }); var canceller = new Thread(() -> { try { @@ -100,6 +120,45 @@ private Thread cancelWhilePipelineRuns() throws Exception { return canceller; } + /** + * Models watchdog abandonment: a second thread interrupts the pipeline thread + * while the workflow is running, and the workflow then returns NORMALLY without + * observing it — the state the guard sees when the interrupt lands in the + * residual window after {@code LifecycleManager}'s own exit check. + *

      + * The stub spins on the flag instead of awaiting a latch on purpose: any + * interruptible wait would throw {@code InterruptedException} and CLEAR the + * flag, destroying the very condition under test. The spin ends exactly when + * the interrupt lands, so the interleaving is deterministic without a sleep. + */ + private Thread interruptWhilePipelineRuns(StubRegistration stubRegistration) throws Exception { + var pipelineThread = Thread.currentThread(); + var pipelineEntered = new CountDownLatch(1); + + stubRegistration.register(invocation -> { + pipelineEntered.countDown(); + long deadlineNanos = System.nanoTime() + TimeUnit.SECONDS.toNanos(10); + while (!Thread.currentThread().isInterrupted()) { + if (System.nanoTime() - deadlineNanos > 0) { + fail("watchdog thread never interrupted the pipeline thread"); + } + Thread.onSpinWait(); + } + return null; + }); + + var watchdog = new Thread(() -> { + try { + assertTrue(pipelineEntered.await(10, TimeUnit.SECONDS), "pipeline never started"); + pipelineThread.interrupt(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }, "watchdog-abandon"); + watchdog.start(); + return watchdog; + } + @Test @Timeout(30) @DisplayName("a turn cancelled mid-pipeline does not upsert the longTerm properties it set") @@ -110,7 +169,8 @@ void cancelledTurnDoesNotPersistProperties() throws Exception { memory.getConversationProperties() .put("dietary_restriction", new Property("dietary_restriction", "vegan", Scope.longTerm)); - var canceller = cancelWhilePipelineRuns(); + var canceller = cancelWhilePipelineRuns( + stub -> doAnswer(stub).when(lifecycleManager).executeLifecycle(any(), any())); try { conversation.say("I am vegan", new LinkedHashMap<>()); } finally { @@ -135,4 +195,136 @@ void stoppedButNotCancelledTurnStillPersists() throws Exception { verify(userMemoryStore, times(1)).upsert(any(UserMemoryEntry.class)); } + + // ===================================================================== + // Watchdog abandonment — the OTHER abort signal, which is an interrupt + // and NOT the cancel flag. + // ===================================================================== + + /** + * A timed-out turn is abandoned by interrupting the pipeline thread: + * {@code BaseRuntime.AbandonableFuture#cancel} sets its own {@code abandoned} + * flag and interrupts, it never sets {@code memory.setCancelled(true)}, and the + * late completion is routed to {@code onFailure} so the conversation document + * is discarded. If the interrupt lands after the pipeline's last observation + * point, the workflow returns normally and — without the interrupt half of the + * guard — the turn still upserts its changed longTerm properties into the + * user-memory store, for a turn that has no persisted conversation document. + *

      + * The stub returns normally with the flag set, which is exactly the state at + * the guard when the interrupt lands in the residual window after + * {@code LifecycleManager}'s own exit check. + */ + @Test + @Timeout(30) + @DisplayName("a turn abandoned by the watchdog (interrupt, cancel flag NOT set) does not upsert its longTerm properties") + void abandonedTurnDoesNotPersistProperties() throws Exception { + Conversation conversation = nextTurn(); + memory.getConversationProperties() + .put("dietary_restriction", new Property("dietary_restriction", "vegan", Scope.longTerm)); + + var watchdog = interruptWhilePipelineRuns( + stub -> doAnswer(stub).when(lifecycleManager).executeLifecycle(any(), any())); + + boolean interruptFlagSurvived; + try { + conversation.say("I am vegan", new LinkedHashMap<>()); + } finally { + // Read (and clear) the flag BEFORE joining: join() is interruptible, so on + // a still-interrupted thread it would throw and clear the flag first. + interruptFlagSurvived = Thread.interrupted(); + watchdog.join(10_000); + } + + assertFalse(memory.isCancelled(), "abandonment must not be confused with a cooperative cancel"); + assertTrue(interruptFlagSurvived, "the guard must only read the interrupt flag, never clear it"); + verify(userMemoryStore, never()).upsert(any(UserMemoryEntry.class)); + } + + // ===================================================================== + // resume() — the same guard, on the HITL path + // ===================================================================== + + /** Parks the memory in AWAITING_HUMAN with a RULE-pause bookmark on wf-1. */ + private void parkAwaitingHuman() { + memory.setConversationState(ConversationState.AWAITING_HUMAN); + memory.setHitlPausedWorkflowId("wf-1"); + memory.setHitlPausedAbsoluteTaskIndex(0); + } + + private static HitlDecision approved() { + var decision = new HitlDecision(); + decision.setVerdict(HitlDecision.HitlVerdict.APPROVED); + return decision; + } + + @Test + @Timeout(30) + @DisplayName("a resume cancelled mid-pipeline does not upsert the longTerm properties it set") + void cancelledResumeDoesNotPersistProperties() throws Exception { + parkAwaitingHuman(); + Conversation conversation = nextTurn(); + memory.getConversationProperties() + .put("dietary_restriction", new Property("dietary_restriction", "vegan", Scope.longTerm)); + + var canceller = cancelWhilePipelineRuns( + stub -> doAnswer(stub).when(lifecycleManager).executeLifecycleFromIndex(any(), anyInt())); + try { + conversation.resume(approved()); + } finally { + canceller.join(10_000); + } + + // The cancel wins the outcome, and nothing may reach the user-memory store. + assertEquals(ConversationState.ENDED, memory.getConversationState()); + verify(userMemoryStore, never()).upsert(any(UserMemoryEntry.class)); + } + + @Test + @Timeout(30) + @DisplayName("control: the SAME resume stop path without a cancel still persists") + void stoppedButNotCancelledResumeStillPersists() throws Exception { + parkAwaitingHuman(); + Conversation conversation = nextTurn(); + memory.getConversationProperties() + .put("dietary_restriction", new Property("dietary_restriction", "vegan", Scope.longTerm)); + + doThrow(new ConversationStopException()) + .when(lifecycleManager).executeLifecycleFromIndex(any(), anyInt()); + + conversation.resume(approved()); + + assertEquals(ConversationState.ENDED, memory.getConversationState()); + verify(userMemoryStore, times(1)).upsert(any(UserMemoryEntry.class)); + } + + @Test + @Timeout(30) + @DisplayName("a resume abandoned by the watchdog (interrupt) does not upsert its longTerm properties") + void abandonedResumeDoesNotPersistProperties() throws Exception { + parkAwaitingHuman(); + Conversation conversation = nextTurn(); + memory.getConversationProperties() + .put("dietary_restriction", new Property("dietary_restriction", "vegan", Scope.longTerm)); + + var watchdog = interruptWhilePipelineRuns( + stub -> doAnswer(stub).when(lifecycleManager).executeLifecycleFromIndex(any(), anyInt())); + + boolean interruptFlagSurvived; + try { + conversation.resume(approved()); + } finally { + // Read (and clear) the flag BEFORE joining — see the say-path twin. + interruptFlagSurvived = Thread.interrupted(); + watchdog.join(10_000); + } + + // The resume itself "succeeded" (READY, not ERROR) — the ERROR/AWAITING_HUMAN + // exclusions therefore cannot be what suppresses the write; only the + // abandonment check can. + assertEquals(ConversationState.READY, memory.getConversationState()); + assertFalse(memory.isCancelled(), "abandonment must not be confused with a cooperative cancel"); + assertTrue(interruptFlagSurvived, "the guard must only read the interrupt flag, never clear it"); + verify(userMemoryStore, never()).upsert(any(UserMemoryEntry.class)); + } } diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceExtendedTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceExtendedTest.java index bd5bd7b18..68141d028 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceExtendedTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceExtendedTest.java @@ -267,7 +267,7 @@ void costCeilingStopsProcessing() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 5000, 5000)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); } } @@ -295,7 +295,7 @@ void consolidatedCapped() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 100, 50)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); // Only 1 upsert should happen (capped to target) verify(store, times(1)).upsert(any(UserMemoryEntry.class)); @@ -324,7 +324,7 @@ void tooManyEntriesSkipped() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 100, 50)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.entriesSummarized()); verify(store, never()).upsert(any(UserMemoryEntry.class)); @@ -344,6 +344,7 @@ void selfNeverUpgradedToGlobal() throws Exception { dreamConfig.setSummarizeGroupBy("all"); dreamConfig.setPreserveAgentProvenance(false); dreamConfig.setSummarizeMinEntries(2); + dreamConfig.setCrossAgentMaintenance(true); // other agents' entries are only in scope when opted in // Two self-scoped entries per agent, from 2 different agents var entries = new ArrayList<>(List.of( @@ -366,7 +367,7 @@ void selfNeverUpgradedToGlobal() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 0, 0)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); // One self-scoped entry per contributing agent; nothing widened @@ -381,6 +382,7 @@ void groupScopedStillMerged() throws Exception { dreamConfig.setSummarizeGroupBy("all"); dreamConfig.setPreserveAgentProvenance(false); dreamConfig.setSummarizeMinEntries(2); + dreamConfig.setCrossAgentMaintenance(true); // other agents' entries are only in scope when opted in var entries = new ArrayList<>(List.of( new UserMemoryEntry("id1", "user-1", "k1", "v1", "fact", @@ -395,7 +397,7 @@ void groupScopedStillMerged() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult("[{\"key\": \"c\", \"value\": \"m\"}]", 0, 0)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); // Already shared → a single merged entry, still group-scoped @@ -414,6 +416,7 @@ class BuildGroupsTests { void groupByAll() throws Exception { dreamConfig.setSummarizeGroupBy("all"); dreamConfig.setPreserveAgentProvenance(false); + dreamConfig.setCrossAgentMaintenance(true); // the agent-2 half is only in scope when opted in var entries = makeEntries(4, "fact", "agent-1"); entries.addAll(makeEntries(2, "preference", "agent-2")); @@ -424,7 +427,7 @@ void groupByAll() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 0, 0)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); // One LLM call for the "all" group verify(summarizationService, times(1)) @@ -437,6 +440,7 @@ void groupByCategoryWithProvenance() throws Exception { dreamConfig.setSummarizeGroupBy("category"); dreamConfig.setPreserveAgentProvenance(true); dreamConfig.setSummarizeMinEntries(2); + dreamConfig.setCrossAgentMaintenance(true); // the agent-2 sub-group is only in scope when opted in var entries = new ArrayList<>(List.of( new UserMemoryEntry("id1", "user-1", "k1", "v1", "fact", @@ -458,7 +462,7 @@ void groupByCategoryWithProvenance() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 0, 0)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); // Two sub-groups: fact:agent-1 and fact:agent-2 verify(summarizationService, times(2)) @@ -486,7 +490,7 @@ void nullCategoryDefaultsFact() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(new SummarizationResult(llmResponse, 0, 0)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); verify(summarizationService, times(1)) .summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); @@ -504,7 +508,7 @@ class ProcessExceptionTests { void exceptionReturnsError() throws Exception { when(store.getAllEntries("user-1")).thenThrow(new RuntimeException("DB error")); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertFalse(result.isSuccess()); assertNotNull(result.error()); assertTrue(result.error().contains("DB error")); diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java index 6b07482e1..1705890b0 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java @@ -65,7 +65,7 @@ void process_shouldPruneStaleEntries() throws Exception { fresh)); when(store.getAllEntries("user-1")).thenReturn(entries); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(1, result.entriesPruned()); @@ -78,7 +78,7 @@ void process_shouldSkipPruningWhenDisabled() throws Exception { dreamConfig.setPruneStaleAfterDays(0); when(store.getAllEntries("user-1")).thenReturn(List.of()); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.entriesPruned()); @@ -96,8 +96,9 @@ void process_shouldDetectContradictions() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); dreamConfig.setPruneStaleAfterDays(0); + dreamConfig.setCrossAgentMaintenance(true); // the conflicting pair spans two agents - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(1, result.contradictionsFound()); @@ -109,7 +110,7 @@ void process_shouldSkipContradictionDetectionWhenDisabled() throws Exception { dreamConfig.setDetectContradictions(false); when(store.getAllEntries("user-1")).thenReturn(List.of()); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.contradictionsFound()); @@ -121,7 +122,7 @@ void process_shouldRecordMetrics() throws Exception { dreamConfig.setPruneStaleAfterDays(0); dreamConfig.setDetectContradictions(false); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertTrue(result.durationMs() >= 0); @@ -131,7 +132,7 @@ void process_shouldRecordMetrics() throws Exception { void process_shouldHandleStoreException() throws Exception { when(store.getAllEntries("user-1")).thenThrow(new ai.labs.eddi.datastore.IResourceStore.ResourceStoreException("DB down")); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertFalse(result.isSuccess()); assertNotNull(result.error()); @@ -147,7 +148,7 @@ void process_shouldLoadEntriesOnlyOnce() throws Exception { dreamConfig.setPruneStaleAfterDays(30); dreamConfig.setDetectContradictions(true); - dreamService.process("user-1", dreamConfig); + dreamService.process("user-1", "agent-1", dreamConfig); // getAllEntries should be called exactly once (shared across both operations) verify(store, times(1)).getAllEntries("user-1"); @@ -163,7 +164,7 @@ void process_shouldReloadAfterPruning() throws Exception { dreamConfig.setPruneStaleAfterDays(30); dreamConfig.setDetectContradictions(true); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(1, result.entriesPruned()); @@ -183,13 +184,17 @@ private List makeEntries(int count, String category, String age return entries; } + /** + * Note: deliberately does NOT touch {@code maxSummarizationCalls}. That legacy + * ceiling only applies to configurations that declare it, so leaving it unset + * here mirrors a stored config that never mentions the field. + */ private void enableSummarization() { dreamConfig.setPruneStaleAfterDays(0); dreamConfig.setDetectContradictions(false); dreamConfig.setSummarizeInteractions(true); dreamConfig.setSummarizeMinEntries(5); dreamConfig.setSummarizeTargetEntries(2); - dreamConfig.setMaxSummarizationCalls(10); dreamConfig.setMaxCostPerRun(0.50); } @@ -212,7 +217,7 @@ void summarize_belowThreshold_noOp() throws Exception { var entries = makeEntries(3, "fact", "agent-1"); // below threshold of 5 when(store.getAllEntries("user-1")).thenReturn(entries); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.entriesSummarized()); @@ -230,7 +235,7 @@ void summarize_aboveThreshold_consolidates() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(4, result.entriesSummarized()); // 6 originals - 2 consolidated = 4 reduced @@ -246,7 +251,7 @@ void summarize_llmReturnsEmpty_preservesEntries() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult("")); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.entriesSummarized()); @@ -262,7 +267,7 @@ void summarize_llmReturnsGarbage_preservesEntries() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult("I can't do that, sorry!")); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.entriesSummarized()); @@ -279,7 +284,7 @@ void summarize_llmReturnsMarkdownFences_parsesCorrectly() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(4, result.entriesSummarized()); @@ -299,7 +304,7 @@ void summarize_llmReturnsMoreThanOriginals_skips() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.entriesSummarized()); @@ -317,7 +322,7 @@ void summarize_insertFails_preservesEntries() throws Exception { .thenReturn(llmResult(llmResponse)); doThrow(new RuntimeException("DB write failed")).when(store).upsert(any(UserMemoryEntry.class)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.entriesSummarized()); @@ -325,15 +330,17 @@ void summarize_insertFails_preservesEntries() throws Exception { } /** - * Finding I1: the legacy {@code maxSummarizationCalls} count is no longer a - * ceiling — the dollar budget {@code maxCostPerRun} is. A config still setting - * the call count must not cap a run that is well inside its budget. + * Config-compat: {@code maxSummarizationCalls} is deprecated in favour of the + * dollar budget, but a stored agent config that sets it asked for a + * hard call ceiling. Dropping it silently turns "at most 1 consolidation call" + * into "as many calls as $1.00 buys" — at $0.00015 a call, hundreds. An + * explicitly configured count must still stop the run. */ @Test - void summarize_legacyCallCount_isNotACeiling() throws Exception { + void summarize_legacyCallCount_stillCapsWhenExplicitlyConfigured() throws Exception { enableSummarization(); - dreamConfig.setMaxSummarizationCalls(1); // legacy, ignored - dreamConfig.setMaxCostPerRun(1.00); // generous dollar budget + dreamConfig.setMaxSummarizationCalls(1); // deprecated, but explicitly configured + dreamConfig.setMaxCostPerRun(1.00); // generous dollar budget — the count is what must bite dreamConfig.setSummarizeGroupBy("category"); Instant now = Instant.now(); @@ -353,12 +360,43 @@ void summarize_legacyCallCount_isNotACeiling() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse, 10, 5)); // ~$0.00015 per call - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); - // Both category groups are consolidated despite maxSummarizationCalls=1 - verify(summarizationService, times(2)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); - assertEquals(10, result.entriesSummarized()); // (6-1) + (6-1) + // Exactly one group is consolidated — the second is stopped by the count + verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); + assertEquals(5, result.entriesSummarized()); // 6 - 1, one group only + } + + /** + * The other half of the same contract: the field's default value is + * not a ceiling. A config that never mentions {@code maxSummarizationCalls} is + * bounded by {@code maxCostPerRun} alone, so a cycle with more groups than the + * default 10 runs them all. + */ + @Test + void summarize_legacyCallCount_unsetNeverCaps() throws Exception { + enableSummarization(); // never calls setMaxSummarizationCalls + dreamConfig.setMaxCostPerRun(1.00); + dreamConfig.setSummarizeGroupBy("category"); + + Instant now = Instant.now(); + var entries = new java.util.ArrayList(); + for (int category = 0; category < 12; category++) { // 12 groups > default ceiling of 10 + for (int i = 0; i < 6; i++) { + entries.add(new UserMemoryEntry("c" + category + "-" + i, "user-1", "k" + category + "-" + i, "v", + "cat-" + category, Visibility.self, "agent-1", List.of(), "conv-1", false, 0, now, now)); + } + } + when(store.getAllEntries("user-1")).thenReturn(entries); + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenReturn(llmResult("[{\"key\": \"s1\", \"value\": \"v1\"}]", 10, 5)); + + var result = dreamService.process("user-1", "agent-1", dreamConfig); + + assertTrue(result.isSuccess()); + verify(summarizationService, times(12)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); + assertEquals(60, result.entriesSummarized()); // 12 groups x (6 - 1) } /** @@ -376,7 +414,7 @@ void summarize_passesConfiguredModelParametersToSummarizer() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult("[{\"key\": \"s\", \"value\": \"v\"}]")); - dreamService.process("user-1", dreamConfig); + dreamService.process("user-1", "agent-1", dreamConfig); verify(summarizationService).summarizeWithUsage(anyString(), anyString(), eq(dreamConfig.getLlmProvider()), eq(dreamConfig.getLlmModel()), eq(parameters)); @@ -403,7 +441,7 @@ void summarize_groupByAll_singleGroup() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); // All 6 entries in one group → 2 consolidated → 4 reduced @@ -416,6 +454,7 @@ void summarize_preserveAgentProvenance_subGroups() throws Exception { enableSummarization(); dreamConfig.setPreserveAgentProvenance(true); dreamConfig.setSummarizeMinEntries(3); + dreamConfig.setCrossAgentMaintenance(true); // sub-grouping is only observable across agents Instant now = Instant.now(); var entries = new java.util.ArrayList(); @@ -435,7 +474,7 @@ void summarize_preserveAgentProvenance_subGroups() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); // Two separate groups, each 3→1 = 2 reduced per group = 4 total @@ -454,7 +493,7 @@ void summarize_customPrompt_passedToService() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), eq(customPrompt), anyString(), anyString(), any())) .thenReturn(llmResult("[{\"key\": \"s\", \"value\": \"v\"}]")); - dreamService.process("user-1", dreamConfig); + dreamService.process("user-1", "agent-1", dreamConfig); verify(summarizationService).summarizeWithUsage(anyString(), eq(customPrompt), anyString(), anyString(), any()); } @@ -480,7 +519,7 @@ void summarize_mostRestrictiveVisibility_applied() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - dreamService.process("user-1", dreamConfig); + dreamService.process("user-1", "agent-1", dreamConfig); // Verify the upserted entry has Visibility.self (most restrictive) var captor = org.mockito.ArgumentCaptor.forClass(UserMemoryEntry.class); @@ -550,7 +589,7 @@ void prune_nullUpdatedAt_skipped() throws Exception { when(store.getAllEntries("user-1")).thenReturn(entries); dreamConfig.setDetectContradictions(false); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(1, result.entriesPruned()); // only "2" pruned, "1" skipped (null updatedAt) @@ -568,7 +607,7 @@ void prune_deleteFails_continues() throws Exception { doThrow(new RuntimeException("DB error")).when(store).deleteEntry("1"); dreamConfig.setDetectContradictions(false); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(1, result.entriesPruned()); // "1" failed, "2" succeeded @@ -586,8 +625,9 @@ void contradictions_sameKeyAndValue_noDuplicate() throws Exception { now, now)); when(store.getAllEntries("user-1")).thenReturn(entries); dreamConfig.setPruneStaleAfterDays(0); + dreamConfig.setCrossAgentMaintenance(true); // both entries must be in scope for this to mean anything - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.contradictionsFound()); // same value → not a contradiction @@ -606,7 +646,7 @@ void summarize_llmReturnsTooMany_cappedToTarget() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(6, result.entriesSummarized()); // 8 - 2 (capped) = 6 @@ -628,7 +668,7 @@ void summarize_deletePartiallyFails_logsAndContinues() throws Exception { .doThrow(new RuntimeException("DB")).doThrow(new RuntimeException("DB")).doThrow(new RuntimeException("DB")) .when(store).deleteEntry(anyString()); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); // Metric tracks actual reduction: 3 successful deletes - 1 consolidated = 2 @@ -650,7 +690,7 @@ void summarize_llmThrows_preservesEntriesAndFailsTheCycle() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenThrow(new RuntimeException("401 Unauthorized")); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertFalse(result.isSuccess()); assertNotNull(result.error()); @@ -661,8 +701,11 @@ void summarize_llmThrows_preservesEntriesAndFailsTheCycle() throws Exception { } /** - * A failing LLM is a configuration fault that would repeat for every remaining - * group — the phase aborts rather than burning the budget group by group. + * A permanent LLM failure (bad credentials, wrong endpoint, unknown + * model) would repeat for every remaining group — the phase aborts rather than + * burning the budget group by group. Transient failures are handled the + * opposite way, see + * {@link #summarize_transientLlmFailure_skipsGroupAndKeepsTheCycleSuccessful}. */ @Test void summarize_llmThrows_abortsRemainingGroups() throws Exception { @@ -681,14 +724,98 @@ void summarize_llmThrows_abortsRemainingGroups() throws Exception { } when(store.getAllEntries("user-1")).thenReturn(entries); when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) - .thenThrow(new RuntimeException("connection refused")); + .thenThrow(new RuntimeException("401 Unauthorized: invalid api key")); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertFalse(result.isSuccess()); verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); } + /** + * A transient provider failure (429/timeout/5xx) is not a configuration fault, + * and a FAILED fire consumes the schedule's dead-letter budget: three of them + * in a row disable the user's dream schedule for good. So a rate-limited group + * is skipped, the remaining groups still run, and the cycle reports success. + */ + @Test + void summarize_transientLlmFailure_skipsGroupAndKeepsTheCycleSuccessful() throws Exception { + enableSummarization(); + dreamConfig.setSummarizeGroupBy("category"); + + Instant now = Instant.now(); + var entries = new java.util.ArrayList(); + for (int i = 0; i < 6; i++) { + entries.add(new UserMemoryEntry("f-" + i, "user-1", "fk-" + i, "fv-" + i, + "fact", Visibility.self, "agent-1", List.of(), "conv-1", false, 0, now, now)); + } + for (int i = 0; i < 6; i++) { + entries.add(new UserMemoryEntry("p-" + i, "user-1", "pk-" + i, "pv-" + i, + "preference", Visibility.self, "agent-1", List.of(), "conv-1", false, 0, now, now)); + } + when(store.getAllEntries("user-1")).thenReturn(entries); + when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id"); + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenThrow(new RuntimeException("429 Too Many Requests")) + .thenReturn(llmResult("[{\"key\": \"s1\", \"value\": \"v1\"}]")); + + var result = dreamService.process("user-1", "agent-1", dreamConfig); + + assertTrue(result.isSuccess(), "a rate limit must not fail the cycle, got: " + result.error()); + assertNull(result.error()); + // The second group was still attempted and consolidated (6 originals → 1 entry) + verify(summarizationService, times(2)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); + assertEquals(5, result.entriesSummarized()); + } + + /** + * Even when every group hits the same transient failure the cycle must not be + * marked failed — otherwise a minutes-long provider outage across three cron + * fires dead-letters the schedule permanently. + */ + @Test + void summarize_transientLlmFailureOnEveryGroup_stillReportsSuccess() throws Exception { + enableSummarization(); + var entries = makeEntries(6, "fact", "agent-1"); + when(store.getAllEntries("user-1")).thenReturn(entries); + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenThrow(new RuntimeException("upstream call timed out")); + + var result = dreamService.process("user-1", "agent-1", dreamConfig); + + assertTrue(result.isSuccess(), "a timeout must not fail the cycle, got: " + result.error()); + assertNull(result.error()); + assertEquals(0, result.entriesSummarized()); + verify(store, never()).deleteEntry(anyString()); + } + + @Test + void isTransientLlmFailure_distinguishesProviderBlipsFromConfigurationFaults() { + assertTrue(DreamService.isTransientLlmFailure(new RuntimeException("429 Too Many Requests"))); + assertTrue(DreamService.isTransientLlmFailure(new RuntimeException("status 503"))); + assertTrue(DreamService.isTransientLlmFailure(new RuntimeException("Overloaded"))); + assertTrue(DreamService.isTransientLlmFailure( + new RuntimeException("llm call failed", new java.net.SocketTimeoutException("read timed out")))); + + assertFalse(DreamService.isTransientLlmFailure(new RuntimeException("401 Unauthorized"))); + assertFalse(DreamService.isTransientLlmFailure(new RuntimeException("model 'nope' does not exist"))); + assertFalse(DreamService.isTransientLlmFailure(new RuntimeException((String) null))); + // A self-referential cause chain must terminate rather than spin + assertFalse(DreamService.isTransientLlmFailure(new SelfCausedException("bad request"))); + } + + /** Exception whose cause is itself — guards the cause-chain walk. */ + private static final class SelfCausedException extends RuntimeException { + SelfCausedException(String message) { + super(message); + } + + @Override + public synchronized Throwable getCause() { + return this; + } + } + @Test void summarize_afterPruning_reloadsEntries() throws Exception { enableSummarization(); @@ -713,7 +840,7 @@ void summarize_afterPruning_reloadsEntries() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(1, result.entriesPruned()); @@ -791,7 +918,7 @@ void summarize_costCeilingReached_stopsEarly() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse, 800, 200)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); // Only 1 LLM call should have been made — cost ceiling stops second group @@ -814,6 +941,7 @@ void summarize_multiAgentSelfScope_neverWidensVisibility() throws Exception { dreamConfig.setPreserveAgentProvenance(false); dreamConfig.setSummarizeGroupBy("category"); dreamConfig.setSummarizeMinEntries(2); + dreamConfig.setCrossAgentMaintenance(true); // opt-in whole-set maintenance is what puts two agents in one group Instant now = Instant.now(); var entries = new java.util.ArrayList(); @@ -832,7 +960,7 @@ void summarize_multiAgentSelfScope_neverWidensVisibility() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - dreamService.process("user-1", dreamConfig); + dreamService.process("user-1", "agent-1", dreamConfig); // One consolidated entry per contributing agent — never a merged one var captor = org.mockito.ArgumentCaptor.forClass(UserMemoryEntry.class); @@ -855,6 +983,7 @@ void summarize_selfAndGlobalAcrossAgents_selfHalfStaysSelf() throws Exception { dreamConfig.setPreserveAgentProvenance(false); dreamConfig.setSummarizeGroupBy("all"); dreamConfig.setSummarizeMinEntries(2); + dreamConfig.setCrossAgentMaintenance(true); // opt-in whole-set maintenance is what puts two agents in one group Instant now = Instant.now(); var entries = new java.util.ArrayList(); @@ -872,7 +1001,7 @@ void summarize_selfAndGlobalAcrossAgents_selfHalfStaysSelf() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult("[{\"key\": \"s\", \"value\": \"v\"}]")); - dreamService.process("user-1", dreamConfig); + dreamService.process("user-1", "agent-1", dreamConfig); var captor = org.mockito.ArgumentCaptor.forClass(UserMemoryEntry.class); verify(store, times(2)).upsert(captor.capture()); @@ -900,7 +1029,7 @@ void summarize_preservesGroupIds() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - dreamService.process("user-1", dreamConfig); + dreamService.process("user-1", "agent-1", dreamConfig); var captor = org.mockito.ArgumentCaptor.forClass(UserMemoryEntry.class); verify(store).upsert(captor.capture()); @@ -926,7 +1055,7 @@ void summarize_nullCategory_defaultsToFact() throws Exception { when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) .thenReturn(llmResult(llmResponse)); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); verify(summarizationService, times(1)).summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any()); @@ -981,7 +1110,7 @@ void summarize_partialInsertFails_rollsBack() throws Exception { .thenReturn("inserted-1") .thenThrow(new RuntimeException("DB write failed")); - var result = dreamService.process("user-1", dreamConfig); + var result = dreamService.process("user-1", "agent-1", dreamConfig); assertTrue(result.isSuccess()); assertEquals(0, result.entriesSummarized()); @@ -1005,6 +1134,101 @@ void setSummarizeTargetEntries_rejectsNegative() { () -> config.setSummarizeTargetEntries(-1)); } + // === Agent ownership: a cycle configured by one agent must not act on another + // agent's memories === + + /** Three stale entries: one owned by agent-1, one by agent-2, one unowned. */ + private List mixedOwnershipStaleEntries() { + Instant stale = Instant.now().minus(Duration.ofDays(60)); + return List.of( + new UserMemoryEntry("own", "user-1", "k-own", "v", "fact", Visibility.self, "agent-1", List.of(), "conv-1", false, 0, stale, stale), + new UserMemoryEntry("foreign", "user-1", "k-foreign", "v", "fact", Visibility.self, "agent-2", List.of(), "conv-2", false, 0, stale, + stale), + new UserMemoryEntry("unowned", "user-1", "k-unowned", "v", "fact", Visibility.global, null, List.of(), "conv-3", false, 0, stale, + stale)); + } + + /** + * Data-loss finding: the cycle read {@code getAllEntries(userId)} — a + * userId-only, agent-unscoped query — while {@code pruneStaleAfterDays} came + * from ONE agent's dream config. Agent A's 30-day retention therefore deleted + * agent B's memories under a value B's owner never configured. Pruning must + * stay inside the firing agent's own entries, the same ownership rule + * {@code UserMemoryTool} applies before evicting. + */ + @Test + void prune_foreignAndUnownedEntries_areNeverDeleted() throws Exception { + when(store.getAllEntries("user-1")).thenReturn(mixedOwnershipStaleEntries()).thenReturn(List.of()); + dreamConfig.setDetectContradictions(false); + + var result = dreamService.process("user-1", "agent-1", dreamConfig); + + assertTrue(result.isSuccess()); + assertEquals(1, result.entriesPruned(), "only the firing agent's own entry may be pruned"); + verify(store).deleteEntry("own"); + verify(store, never()).deleteEntry("foreign"); + verify(store, never()).deleteEntry("unowned"); + } + + /** + * The escape hatch for a dedicated housekeeping agent: with + * {@code crossAgentMaintenance=true} the whole memory set is in scope again. + */ + @Test + void prune_crossAgentMaintenance_optsBackIntoTheWholeMemorySet() throws Exception { + when(store.getAllEntries("user-1")).thenReturn(mixedOwnershipStaleEntries()).thenReturn(List.of()); + dreamConfig.setDetectContradictions(false); + dreamConfig.setCrossAgentMaintenance(true); + + var result = dreamService.process("user-1", "agent-1", dreamConfig); + + assertTrue(result.isSuccess()); + assertEquals(3, result.entriesPruned()); + verify(store).deleteEntry("own"); + verify(store).deleteEntry("foreign"); + verify(store).deleteEntry("unowned"); + } + + /** + * The same boundary on the summarization side: another agent's memory text must + * not be serialized into the prompt that goes to this agent's + * configured provider/baseUrl, and its originals must not be deleted and + * replaced. + */ + @Test + void summarize_foreignAgentEntries_neverReachTheModelAndAreNotDeleted() throws Exception { + enableSummarization(); + dreamConfig.setSummarizeMinEntries(2); + + Instant now = Instant.now(); + var entries = new java.util.ArrayList(); + for (int i = 0; i < 3; i++) { + entries.add(new UserMemoryEntry("a1-" + i, "user-1", "k1-" + i, "owned-by-agent-1", + "fact", Visibility.self, "agent-1", List.of(), "conv-1", false, 0, now, now)); + } + for (int i = 0; i < 3; i++) { + entries.add(new UserMemoryEntry("a2-" + i, "user-1", "k2-" + i, "private-to-agent-2", + "fact", Visibility.self, "agent-2", List.of(), "conv-2", false, 0, now, now)); + } + when(store.getAllEntries("user-1")).thenReturn(entries); + when(store.upsert(any(UserMemoryEntry.class))).thenReturn("new-id"); + when(summarizationService.summarizeWithUsage(anyString(), anyString(), anyString(), anyString(), any())) + .thenReturn(llmResult("[{\"key\": \"s\", \"value\": \"v\"}]")); + + var result = dreamService.process("user-1", "agent-1", dreamConfig); + + assertTrue(result.isSuccess()); + var content = org.mockito.ArgumentCaptor.forClass(String.class); + verify(summarizationService, times(1)).summarizeWithUsage(content.capture(), anyString(), anyString(), anyString(), any()); + assertTrue(content.getValue().contains("owned-by-agent-1")); + assertFalse(content.getValue().contains("private-to-agent-2"), + "another agent's private memory must not be sent to this agent's model endpoint"); + for (int i = 0; i < 3; i++) { + verify(store).deleteEntry("a1-" + i); + verify(store, never()).deleteEntry("a2-" + i); + } + } + // === Finding I1: schedule wiring (processScheduledFire) === @Test @@ -1043,6 +1267,35 @@ void processScheduledFire_resolvesDreamConfigFromAgentAndRunsCycle() throws Exce verify(store).deleteEntry("1"); } + /** + * The schedule's {@code agentId} is the ownership boundary, not just the config + * source: a dream schedule for agent-1 must not prune agent-2's memories for + * the same user. + */ + @Test + void processScheduledFire_prunesOnlyTheFiringAgentsMemories() throws Exception { + var dream = new AgentConfiguration.DreamConfig(); + dream.setEnabled(true); + dream.setPruneStaleAfterDays(30); + dream.setDetectContradictions(false); + dream.setSummarizeInteractions(false); + + var memoryConfig = new AgentConfiguration.UserMemoryConfig(); + memoryConfig.setDream(dream); + var agentConfiguration = new AgentConfiguration(); + agentConfiguration.setUserMemoryConfig(memoryConfig); + when(agentStore.read("agent-1", 7)).thenReturn(agentConfiguration); + + when(store.getAllEntries("user-1")).thenReturn(mixedOwnershipStaleEntries()).thenReturn(List.of()); + + var result = dreamService.processScheduledFire("agent-1", 7, "user-1"); + + assertTrue(result.isSuccess(), "expected success, got: " + result.error()); + assertEquals(1, result.entriesPruned()); + verify(store).deleteEntry("own"); + verify(store, never()).deleteEntry("foreign"); + } + @Test void processScheduledFire_versionZero_resolvesLatestAgentVersion() throws Exception { var dream = new AgentConfiguration.DreamConfig(); diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorBranchTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorBranchTest.java index 5f49d7f0a..9a3736508 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorBranchTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorBranchTest.java @@ -126,8 +126,18 @@ void metaDataNonNull() throws Exception { @DisplayName("shutdown edge cases") class ShutdownEdgeCases { + /** + * B2: a drain that merely TIMED OUT is not an interrupt. Flagging the shutdown + * thread there aborts every remaining {@code @PreDestroy} step that performs a + * blocking/interruptible call (further drains, executor awaitTermination, the + * Mongo writes of the graceful-shutdown drain) with an immediate interrupt. + * Only the {@code InterruptedException} half of the shared catch may set the + * flag — see + * {@code NatsConversationCoordinatorExtendedTest.shutdown_interruptedException_setsInterruptFlag} + * for the positive case. + */ @Test - @DisplayName("shutdown — TimeoutException is caught and logged") + @DisplayName("shutdown — a drain TimeoutException must NOT flag the shutdown thread") void shutdownTimeoutException() throws Exception { Connection mockConn = mock(Connection.class); when(mockConn.drain(any(Duration.class))).thenThrow(new TimeoutException("drain timeout")); @@ -136,7 +146,20 @@ void shutdownTimeoutException() throws Exception { connField.setAccessible(true); connField.set(coordinator, mockConn); - assertDoesNotThrow(() -> coordinator.shutdown()); + try { + // Start from a known-clean state so the assertion below can only be + // about what shutdown() did. + Thread.interrupted(); + + assertDoesNotThrow(() -> coordinator.shutdown()); + + assertFalse(Thread.currentThread().isInterrupted(), + "a drain timeout is not an interrupt and must not flag the shutdown thread — " + + "the remaining @PreDestroy steps would abort mid-way"); + } finally { + // Never leak the flag into whatever test runs next on this thread. + Thread.interrupted(); + } } } diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorExtendedTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorExtendedTest.java index 0a738d699..10ca9818a 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorExtendedTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorExtendedTest.java @@ -503,56 +503,53 @@ void submitNext_emptyQueue_removesFromMap() { assertTrue(coordinator.getQueueDepths().isEmpty()); } - // ==================== Retry on Failure ==================== + // ==================== Failure handling (C13: no re-execution) + // ==================== @Nested - class PublishAndExecuteRetry { + class PublishAndExecuteFailure { + /** + * The callback is raised from inside the executor task, so the turn has already + * run. Re-submitting the identical callable would repeat its LLM calls, tool + * invocations and memory writes — which is why the retry loop was removed from + * BOTH coordinators. + */ @Test @SuppressWarnings("unchecked") - void onFailure_retries_whenAttemptsRemain() { + void onFailure_neverReSubmitsTheSameCallable() { Callable task = () -> null; - // 1st submission → captures the callback ArgumentCaptor> cb1 = ArgumentCaptor.forClass(IRuntime.IFinishedExecution.class); coordinator.submitInOrder("conv-retry", task); verify(runtime, times(1)).submitCallable(eq(task), cb1.capture(), isNull()); - // Trigger failure (attempt 1 of 3) → should re-submit cb1.getValue().onFailure(new RuntimeException("transient error")); - verify(runtime, times(2)).submitCallable(eq(task), any(), isNull()); + + verify(runtime, times(1)).submitCallable(eq(task), any(), isNull()); } @Test @SuppressWarnings("unchecked") - void onFailure_routesToDeadLetter_whenRetriesExhausted() throws Exception { - // maxRetries = 3 → need 3 failures to exhaust + void onFailure_routesToDeadLetter_onTheFirstFailure() throws Exception { + // maxRetries is 3 for this coordinator — and deliberately irrelevant Callable task = () -> null; ArgumentCaptor> cbCaptor = ArgumentCaptor.forClass(IRuntime.IFinishedExecution.class); coordinator.submitInOrder("conv-exhaust", task); verify(runtime, times(1)).submitCallable(eq(task), cbCaptor.capture(), isNull()); - // failure 1 → retry cbCaptor.getValue().onFailure(new RuntimeException("fail-1")); - verify(runtime, times(2)).submitCallable(eq(task), cbCaptor.capture(), isNull()); - - // failure 2 → retry - cbCaptor.getValue().onFailure(new RuntimeException("fail-2")); - verify(runtime, times(3)).submitCallable(eq(task), cbCaptor.capture(), isNull()); - // failure 3 → retries exhausted (attempt == maxRetries) → dead-letter - cbCaptor.getValue().onFailure(new RuntimeException("fail-3")); - - // Should NOT have been submitted a 4th time - verify(runtime, times(3)).submitCallable(eq(task), any(), isNull()); - - // Dead-letter published to JetStream + // Dead-letter published to JetStream on the very first failure verify(jetStream).publish( eq("eddi.deadletter.conv-exhaust"), any(byte[].class)); // totalDeadLettered incremented assertEquals(1L, coordinator.getTotalDeadLettered()); + + // ... and no second execution of a turn that already ran + verify(runtime, times(1)).submitCallable(eq(task), any(), isNull()); } } diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorIT.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorIT.java index ffcd59fb7..b83e4e7bc 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorIT.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorIT.java @@ -65,7 +65,8 @@ void setUp() throws Exception { // Create coordinator with real NATS connection (no mocks) coordinator = new NatsConversationCoordinator(runtime, null, // no metrics instance for IT - new SimpleMeterRegistry(), natsUrl, "EDDI_IT_CONVERSATIONS", "EDDI_IT_DEAD_LETTERS", 3, // maxRetries + // eddi.nats.max-retries — retained config knob, no longer re-executes turns + new SimpleMeterRegistry(), natsUrl, "EDDI_IT_CONVERSATIONS", "EDDI_IT_DEAD_LETTERS", 3, 10000); // Start coordinator (connects to NATS, creates streams) @@ -134,7 +135,7 @@ void concurrentConversations() throws Exception { } @Test - @DisplayName("should route to dead-letter after max retries exhausted") + @DisplayName("should route a failed task to dead-letter without re-executing it") void deadLetterAfterMaxRetries() throws Exception { String convId = "conv-dead-letter-test"; @@ -155,8 +156,8 @@ void deadLetterAfterMaxRetries() throws Exception { throw new RuntimeException("intentional failure for dead-letter test"); }); - // Wait for all retries to be exhausted + dead-letter published - // maxRetries=3 so we wait for attempts to complete + // The task is dead-lettered on its FIRST failure (C13: a turn that reported + // failure has already run and is never re-executed); wait for the publish. Thread.sleep(3000); // Verify message exists in dead-letter stream @@ -178,7 +179,7 @@ void deadLetterPayloadContainsConversationId() throws Exception { throw new RuntimeException("payload test failure"); }); - // Wait for retries to exhaust + // Wait for the dead-letter publish (no retries — see C13) Thread.sleep(3000); // Consume the dead-letter message diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorTest.java index 799f611d4..eea80d52a 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinatorTest.java @@ -18,6 +18,7 @@ import java.io.IOException; import java.util.concurrent.Callable; +import java.util.concurrent.RejectedExecutionException; import java.util.concurrent.atomic.AtomicInteger; import static org.junit.jupiter.api.Assertions.*; @@ -28,7 +29,8 @@ * Unit tests for {@link NatsConversationCoordinator}. * *

      - * Tests verify local ordering, retry/dead-letter logic, and metrics without + * Tests verify local ordering, failure/dead-letter handling (C13: no + * re-execution, C10: rejected submissions roll back), and metrics without * requiring a running NATS server. JetStream interactions are mocked. *

      */ @@ -168,11 +170,13 @@ void shouldProcessNextTaskAfterFailure() throws Exception { verify(runtime, times(1)).submitCallable(eq(task1), callbackCaptor.capture(), isNull()); - // Simulate task1 failure — first failure triggers retry, not next task + // C13: onFailure is raised from INSIDE the executor task, so task1 has already + // run (LLM calls, tools, memory writes, money). It must be dead-lettered once + // and the queue must move on to task2 — never re-executed. callbackCaptor.getValue().onFailure(new RuntimeException("boom")); - // task1 should be retried (attempt 1 of 3) - verify(runtime, times(2)).submitCallable(eq(task1), any(), isNull()); + verify(runtime, times(1)).submitCallable(eq(task1), any(), isNull()); + verify(runtime, times(1)).submitCallable(eq(task2), any(), isNull()); } @Test @@ -206,9 +210,15 @@ void shouldReportConnectedWhenNatsIsUp() throws Exception { // ==================== Dead-Letter Tests ==================== + /** + * C13 parity with {@link InMemoryConversationCoordinator}: a turn that reports + * failure has already executed. It is dead-lettered on the FIRST failure and + * never handed to the runtime a second time — even though + * {@code eddi.nats.max-retries} is still 3 here. + */ @Test @SuppressWarnings("unchecked") - void shouldRetryTaskBeforeDeadLettering() throws Exception { + void shouldNotReExecuteAFailedTaskAndDeadLetterImmediately() throws Exception { Callable failingTask = () -> { throw new RuntimeException("fail"); }; @@ -217,24 +227,20 @@ void shouldRetryTaskBeforeDeadLettering() throws Exception { coordinator.submitInOrder("conv-retry", failingTask); - // First execution + // First (and only) execution verify(runtime, times(1)).submitCallable(eq(failingTask), callbackCaptor.capture(), isNull()); + assertEquals(3, coordinator.getMaxRetries(), "guard: the config knob is still 3, it just must not re-execute turns"); - // Simulate failure (attempt 1) - callbackCaptor.getValue().onFailure(new RuntimeException("fail")); - verify(runtime, times(2)).submitCallable(eq(failingTask), callbackCaptor.capture(), isNull()); - - // Simulate failure (attempt 2) callbackCaptor.getValue().onFailure(new RuntimeException("fail")); - verify(runtime, times(3)).submitCallable(eq(failingTask), callbackCaptor.capture(), isNull()); - // No dead-letter yet — still have 1 more attempt - verify(jetStream, never()).publish(startsWith("eddi.deadletter."), any(byte[].class)); + verify(runtime, times(1)).submitCallable(eq(failingTask), any(), isNull()); + verify(jetStream).publish(eq("eddi.deadletter.conv-retry"), any(byte[].class)); + assertEquals(1L, coordinator.getTotalDeadLettered()); } @Test @SuppressWarnings("unchecked") - void shouldDeadLetterAfterMaxRetries() throws Exception { + void shouldDeadLetterOnFirstFailure() throws Exception { Callable failingTask = () -> { throw new RuntimeException("persistent failure"); }; @@ -242,16 +248,70 @@ void shouldDeadLetterAfterMaxRetries() throws Exception { ArgumentCaptor> callbackCaptor = ArgumentCaptor.forClass(IRuntime.IFinishedExecution.class); coordinator.submitInOrder("conv-dl", failingTask); + verify(runtime, times(1)).submitCallable(eq(failingTask), callbackCaptor.capture(), isNull()); - // Exhaust all retries (maxRetries=3) - for (int i = 0; i < coordinator.getMaxRetries(); i++) { - verify(runtime, times(i + 1)).submitCallable(eq(failingTask), callbackCaptor.capture(), isNull()); - callbackCaptor.getValue().onFailure(new RuntimeException("persistent failure")); - } + callbackCaptor.getValue().onFailure(new RuntimeException("persistent failure")); // Should publish to dead-letter subject verify(jetStream).publish(eq("eddi.deadletter.conv-dl"), any(byte[].class)); verify(deadLetterCount).increment(); + // ... and the queue must be released, not left holding the dead task + assertFalse(coordinator.getQueueDepths().containsKey("conv-dl"), + "the dead-lettered task must be dropped from the queue so the conversation stays usable"); + } + + /** + * C10 parity: when handing the task to the runtime throws, the enqueue must be + * rolled back. Otherwise every later turn for that conversation sees a + * non-empty queue, waits for a completion callback that will never come, and + * the conversation is wedged for the JVM's lifetime. + */ + @Test + void rejectedSubmissionRollsTheTaskBackOffTheQueue() throws Exception { + Callable task = () -> null; + doThrow(new RejectedExecutionException("pool is shutting down")) + .when(runtime).submitCallable(eq(task), any(), isNull()); + + assertThrows(RejectedExecutionException.class, + () -> coordinator.submitInOrder("conv-rejected", task)); + + assertFalse(coordinator.getQueueDepths().containsKey("conv-rejected"), + "a rejected submission must not leave the task queued — nothing would ever drain it"); + + // ... and the conversation must still accept the next turn. + Callable nextTask = () -> null; + coordinator.submitInOrder("conv-rejected", nextTask); + verify(runtime).submitCallable(eq(nextTask), any(), isNull()); + } + + /** + * C10 (submitNext side): the completion callback has no caller to propagate to. + * A task that cannot be scheduled must be dead-lettered and the queue must keep + * draining, otherwise the conversation wedges just the same. + */ + @Test + @SuppressWarnings("unchecked") + void unschedulableQueuedTaskIsDeadLetteredAndTheQueueKeepsDraining() throws Exception { + Callable first = () -> null; + Callable unschedulable = () -> null; + Callable third = () -> null; + + ArgumentCaptor> callbackCaptor = ArgumentCaptor.forClass(IRuntime.IFinishedExecution.class); + + coordinator.submitInOrder("conv-drain", first); + coordinator.submitInOrder("conv-drain", unschedulable); + coordinator.submitInOrder("conv-drain", third); + + verify(runtime, times(1)).submitCallable(eq(first), callbackCaptor.capture(), isNull()); + + doThrow(new RejectedExecutionException("pool is shutting down")) + .when(runtime).submitCallable(eq(unschedulable), any(), isNull()); + + // first completes → submitNext hits the un-schedulable task + callbackCaptor.getValue().onComplete(null); + + verify(jetStream).publish(eq("eddi.deadletter.conv-drain"), any(byte[].class)); + verify(runtime, times(1)).submitCallable(eq(third), any(), isNull()); } @Test diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java index 0059c67ba..fa50f8091 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/ScheduleFireExecutorTest.java @@ -18,6 +18,8 @@ import org.mockito.ArgumentCaptor; import java.time.Instant; +import java.util.ArrayList; +import java.util.List; import static org.junit.jupiter.api.Assertions.*; import static org.mockito.ArgumentMatchers.*; @@ -222,6 +224,96 @@ void fire_interruptedWhileWaiting_restoresInterruptFlagAndLogsFailed() throws Ex } } + /** + * B2, ordering half. Restoring the interrupt flag is only safe AFTER the fire + * log has been written. The synchronous MongoDB driver checks out a connection + * with {@code lockInterruptibly()} and aborts with + * {@code MongoInterruptedException} when the calling thread's flag is already + * set, so a restore placed inside the catch block makes {@code logFire} throw + * on exactly the interrupt it exists to record — the FAILED attempt then + * vanishes (the local catch swallows the store failure) and the poller can + * never see it. A plain Mockito mock is interrupt-insensitive by construction, + * so this stub reproduces that sensitivity explicitly. + */ + @Test + @Timeout(10) + void fire_interrupted_writesFireLogBeforeRestoringTheFlag() throws Exception { + var schedule = makeCronSchedule("sched-interrupt-order", "new"); + when(conversationService.startConversation(any(), any(), any(), any())) + .thenReturn(new IConversationService.ConversationResult("conv-int", null)); + doAnswer(inv -> { + Thread.currentThread().interrupt(); + return null; + }).when(conversationService).say(any(), any(), any(), anyBoolean(), anyBoolean(), any(), any(), anyBoolean(), any()); + + List persisted = interruptSensitiveFireLogStore(); + + assertFalse(Thread.currentThread().isInterrupted(), "precondition: flag starts clear"); + try { + ScheduleFireLog result = executor.fire(schedule, "instance-1", 4); + + assertEquals(1, persisted.size(), + "the FAILED attempt must reach an interrupt-sensitive store — restoring the flag before " + + "logFire() aborts the very write the interrupt path exists to perform"); + assertEquals(FireStatus.FAILED.name(), persisted.get(0).status()); + assertEquals(4, persisted.get(0).attemptNumber()); + assertEquals(FireStatus.FAILED.name(), result.status()); + // ...and the cancellation signal still reaches the caller afterwards. + assertTrue(Thread.currentThread().isInterrupted(), + "fire() must still re-assert the interrupt flag that latch.await consumed"); + } finally { + Thread.interrupted(); + } + } + + /** Same ordering guarantee on the Dream fast-path's sibling catch. */ + @Test + @Timeout(10) + void fire_dreamScheduleInterrupted_writesFireLogBeforeRestoringTheFlag() throws Exception { + var schedule = makeDreamSchedule("sched-dream-interrupt-order", "user-9"); + schedule.setAgentVersion(1); + when(dreamService.processScheduledFire(any(), any(), any())).thenAnswer(inv -> { + Thread.interrupted(); + throw new InterruptedException("consolidation interrupted"); + }); + + List persisted = interruptSensitiveFireLogStore(); + + assertFalse(Thread.currentThread().isInterrupted(), "precondition: flag starts clear"); + try { + ScheduleFireLog result = executor.fire(schedule, "instance-1", 2); + + assertEquals(1, persisted.size(), + "the Dream fast-path must log the FAILED attempt before re-asserting the interrupt flag"); + assertEquals(FireStatus.FAILED.name(), persisted.get(0).status()); + assertEquals(2, persisted.get(0).attemptNumber()); + assertEquals(FireStatus.FAILED.name(), result.status()); + assertTrue(Thread.currentThread().isInterrupted(), + "the Dream fast-path must still re-assert the interrupt flag its catch consumed"); + } finally { + Thread.interrupted(); + } + } + + /** + * Stubs {@code logFire} the way the synchronous MongoDB driver behaves: a write + * attempted while the calling thread's interrupt flag is set aborts instead of + * persisting. + * + * @return the live list of fire logs that actually made it to the store + */ + private List interruptSensitiveFireLogStore() throws Exception { + List persisted = new ArrayList<>(); + doAnswer(inv -> { + if (Thread.currentThread().isInterrupted()) { + throw new RuntimeException("MongoInterruptedException: interrupted on connection checkout"); + } + persisted.add(inv.getArgument(0)); + return null; + }).when(scheduleStore).logFire(any()); + return persisted; + } + @Test void fire_logsFireAttemptEvenOnFailure() throws Exception { var schedule = makeCronSchedule("sched-err2", "new"); diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/SchedulePollerServiceTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/SchedulePollerServiceTest.java index ea31aaeca..14996efb7 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/SchedulePollerServiceTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/SchedulePollerServiceTest.java @@ -279,6 +279,58 @@ void poll_fireFailed_deadLettersAfterMaxRetries() throws Exception { verify(scheduleStore, never()).markFailed(any(), any()); } + /** + * The residual half of the B2 interrupt work. {@code fire()} deliberately + * re-asserts the interrupt flag before returning, so the poller ran its + * bookkeeping on a still-interrupted thread — and {@code markFailed} is a Mongo + * write, which the sync driver aborts with {@code MongoInterruptedException} on + * connection checkout while that flag is set. {@code onFireFailed} swallows it, + * so {@code failCount} never incremented: the schedule stayed CLAIMED with + * {@code nextFire} in the past, was re-claimed on every lease expiry, and could + * never reach {@code maxRetries} or dead-letter. An interrupt turned a failing + * schedule into an unbounded re-fire loop. + *

      + * The store stub reproduces the driver's actual interrupt sensitivity, which a + * plain Mockito mock cannot express — and which is exactly why this went + * unnoticed: {@code verify(markFailed)} passes either way, because under the + * bug the call still HAPPENS, it just fails. So the assertion is on whether the + * write COMPLETED, not on whether it was attempted. + *

      + * The fire runs on the poller's own virtual thread, so this test cannot observe + * whether the flag survives back to the caller — that half is pinned in + * ScheduleFireExecutorTest, and this test deliberately claims no more than it + * checks. + */ + @Test + void poll_fireInterrupted_stillRecordsTheFailure() throws Exception { + var schedule = makeCronSchedule("sched-int", "0 9 * * *", "Hello"); + schedule.setFailCount(0); + when(scheduleStore.findDueSchedules(any(), any(), anyInt())).thenReturn(List.of(schedule)); + when(scheduleStore.tryClaim(any(), any(), any(), any())).thenReturn(true); + + // fire() returns FAILED and leaves the interrupt flag set, as it now does. + when(fireExecutor.fire(any(), any(), anyInt())).thenAnswer(inv -> { + Thread.currentThread().interrupt(); + return makeFireLog("sched-int", FireStatus.FAILED.name()); + }); + + // markFailed behaves like the sync Mongo driver: refuses to run interrupted. + var markFailedCompleted = new java.util.concurrent.atomic.AtomicBoolean(false); + doAnswer(inv -> { + if (Thread.currentThread().isInterrupted()) { + throw new IllegalStateException("interrupted during connection checkout"); + } + markFailedCompleted.set(true); + return null; + }).when(scheduleStore).markFailed(any(), any()); + + poller.pollDueSchedules(); + + assertTrue(markFailedCompleted.get(), + "failCount was never incremented: the schedule stays CLAIMED with nextFire in the past and re-fires forever"); + verify(scheduleStore).markFailed(eq("sched-int"), any()); + } + // --- Heartbeat scheduling --- @Test diff --git a/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java b/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java index 029c0daaa..6b0f6eec3 100644 --- a/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java +++ b/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java @@ -8,6 +8,7 @@ import ai.labs.eddi.engine.schedule.model.ScheduleConfiguration; import ai.labs.eddi.engine.schedule.model.ScheduleConfiguration.FireStatus; import ai.labs.eddi.engine.schedule.model.ScheduleConfiguration.TriggerType; +import ai.labs.eddi.engine.schedule.model.ScheduleFireLog; import ai.labs.eddi.engine.runtime.internal.ScheduleFireExecutor; import ai.labs.eddi.engine.runtime.internal.SchedulePollerService; import ai.labs.eddi.engine.security.OwnershipValidator; @@ -16,6 +17,7 @@ import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; +import java.security.Principal; import java.time.Instant; import java.util.List; import java.util.Map; @@ -258,7 +260,7 @@ void fireNow_regularSchedule_stillFires() throws Exception { var regular = makeCronSchedule("r1"); when(scheduleStore.readSchedule("r1")).thenReturn(regular); when(fireExecutor.fire(any(), any(), anyInt())) - .thenReturn(new ai.labs.eddi.engine.schedule.model.ScheduleFireLog( + .thenReturn(new ScheduleFireLog( "log-1", "r1", "fire-1", null, Instant.now(), Instant.now(), FireStatus.COMPLETED.name(), "n1", "conv-1", null, 1, 0.0)); @@ -388,6 +390,172 @@ void readAllSchedules_showsHitlForAdmin() throws Exception { assertEquals(2, result.size()); } + // --- Cross-user schedules: a schedule runs AS its userId --- + + /** + * The schedule surface is the one place a plain {@code eddi-editor} can name an + * arbitrary {@code userId}. Every fire then acts as that identity, and for a + * {@code dreamType=dream_consolidation} schedule that means + * {@code DreamService} prunes, rewrites and permanently deletes the named + * user's persistent memories — an operation the direct memory API + * ({@code IRestUserMemoryStore}, roles {@code {eddi-admin, eddi-user}}, + * {@code validateUserAccess} on every method) refuses that caller outright. + * These tests pin the guard that stops the schedule API becoming a back door + * around it. + */ + private static ScheduleConfiguration dreamSchedule(String id, String userId) { + var s = makeCronSchedule(id); + s.setName("dream-" + id); + s.setCronExpression("0 3 * * *"); + s.setUserId(userId); + s.setMetadata(Map.of("dreamType", "dream_consolidation")); + return s; + } + + /** Authenticate the mocked identity as a plain (non-admin) editor. */ + private void asEditor(String principalName) { + when(identity.hasRole("eddi-admin")).thenReturn(false); + when(identity.isAnonymous()).thenReturn(false); + when(identity.getPrincipal()).thenReturn(new TestPrincipal(principalName)); + } + + /** Authenticate the mocked identity as an admin. */ + private void asAdmin(String principalName) { + when(identity.hasRole("eddi-admin")).thenReturn(true); + when(identity.isAnonymous()).thenReturn(false); + when(identity.getPrincipal()).thenReturn(new TestPrincipal(principalName)); + } + + @Test + void createSchedule_actingAsAnotherUser_forbiddenForEditor() throws Exception { + asEditor("editor-1"); + + Response response = rest.createSchedule(dreamSchedule("d1", "victim-42")); + + assertEquals(403, response.getStatus()); + verify(scheduleStore, never()).createSchedule(any()); + } + + @Test + void createSchedule_actingAsSelf_allowedForEditor() throws Exception { + asEditor("editor-1"); + when(scheduleStore.createSchedule(any())).thenReturn("d2"); + + Response response = rest.createSchedule(dreamSchedule("d2", "editor-1")); + + assertEquals(201, response.getStatus()); + assertEquals("editor-1", ((ScheduleConfiguration) response.getEntity()).getUserId()); + verify(scheduleStore).createSchedule(any()); + } + + @Test + void createSchedule_actingAsAnotherUser_allowedForAdmin() throws Exception { + // Admins legitimately schedule work on behalf of any user. + asAdmin("root"); + when(scheduleStore.createSchedule(any())).thenReturn("d3"); + + Response response = rest.createSchedule(dreamSchedule("d3", "victim-42")); + + assertEquals(201, response.getStatus()); + assertEquals("victim-42", ((ScheduleConfiguration) response.getEntity()).getUserId()); + } + + @Test + void createSchedule_systemSchedulerPlaceholder_allowedForEditor() throws Exception { + // 'system:scheduler' is not a real principal (DreamService refuses to + // consolidate for it) and it is what applyDefaults/readSchedule hand back, so + // round-tripping it must not turn into a 403. + asEditor("editor-1"); + when(scheduleStore.createSchedule(any())).thenReturn("s1"); + + var schedule = makeCronSchedule("s1"); + schedule.setUserId("system:scheduler"); + + Response response = rest.createSchedule(schedule); + + assertEquals(201, response.getStatus()); + verify(scheduleStore).createSchedule(any()); + } + + @Test + void updateSchedule_repointingToAnotherUser_forbiddenForEditor() throws Exception { + asEditor("editor-1"); + // Stored schedule is an innocuous system schedule; the BODY re-points it at a + // victim — the conversion path the create guard would otherwise miss. + when(scheduleStore.readSchedule("r1")).thenReturn(makeCronSchedule("r1")); + + Response response = rest.updateSchedule("r1", dreamSchedule("r1", "victim-42")); + + assertEquals(403, response.getStatus()); + verify(scheduleStore, never()).updateSchedule(eq("r1"), any()); + } + + @Test + void fireNow_scheduleActingAsAnotherUser_forbiddenForEditor() throws Exception { + asEditor("editor-1"); + when(scheduleStore.readSchedule("d1")).thenReturn(dreamSchedule("d1", "victim-42")); + + Response response = rest.fireNow("d1"); + + assertEquals(403, response.getStatus()); + // The destructive dispatch must never be reached. + verify(fireExecutor, never()).fire(any(), any(), anyInt()); + } + + @Test + void fireNow_scheduleActingAsAnotherUser_allowedForAdmin() throws Exception { + asAdmin("root"); + when(scheduleStore.readSchedule("d1")).thenReturn(dreamSchedule("d1", "victim-42")); + when(fireExecutor.fire(any(), any(), anyInt())) + .thenReturn(new ScheduleFireLog( + "log-d1", "d1", "fire-1", null, Instant.now(), Instant.now(), + FireStatus.COMPLETED.name(), "n1", null, null, 1, 0.0)); + + Response response = rest.fireNow("d1"); + + assertEquals(200, response.getStatus()); + verify(fireExecutor).fire(any(), any(), anyInt()); + } + + @Test + void fireNow_scheduleActingAsSelf_allowedForEditor() throws Exception { + asEditor("editor-1"); + when(scheduleStore.readSchedule("d2")).thenReturn(dreamSchedule("d2", "editor-1")); + when(fireExecutor.fire(any(), any(), anyInt())) + .thenReturn(new ScheduleFireLog( + "log-d2", "d2", "fire-1", null, Instant.now(), Instant.now(), + FireStatus.COMPLETED.name(), "n1", null, null, 1, 0.0)); + + Response response = rest.fireNow("d2"); + + assertEquals(200, response.getStatus()); + verify(fireExecutor).fire(any(), any(), anyInt()); + } + + @Test + void fireNow_systemSchedulerSchedule_stillFiresForEditor() throws Exception { + asEditor("editor-1"); + var schedule = makeCronSchedule("r2"); + schedule.setUserId("system:scheduler"); + when(scheduleStore.readSchedule("r2")).thenReturn(schedule); + when(fireExecutor.fire(any(), any(), anyInt())) + .thenReturn(new ScheduleFireLog( + "log-r2", "r2", "fire-1", null, Instant.now(), Instant.now(), + FireStatus.COMPLETED.name(), "n1", "conv-1", null, 1, 0.0)); + + Response response = rest.fireNow("r2"); + + assertEquals(200, response.getStatus()); + verify(fireExecutor).fire(any(), any(), anyInt()); + } + + private record TestPrincipal(String name) implements Principal { + @Override + public String getName() { + return name; + } + } + private static ScheduleConfiguration makeCronSchedule(String id) { var s = new ScheduleConfiguration(); s.setId(id); diff --git a/src/test/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParserCacheTest.java b/src/test/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParserCacheTest.java index 8c340ed22..1ecb59a20 100644 --- a/src/test/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParserCacheTest.java +++ b/src/test/java/ai/labs/eddi/modules/nlp/impl/RestSemanticParserCacheTest.java @@ -29,6 +29,7 @@ import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertInstanceOf; @@ -67,6 +68,7 @@ class RestSemanticParserCacheTest { private IInputParser inputParser; private AsyncResponse asyncResponse; private RestSemanticParser parser; + private Map> lifecycleTasks; @BeforeEach void setUp() throws Exception { @@ -77,7 +79,7 @@ void setUp() throws Exception { inputParser = mock(IInputParser.class); asyncResponse = mock(AsyncResponse.class); - Map> lifecycleTasks = new HashMap<>(); + lifecycleTasks = new HashMap<>(); lifecycleTasks.put("ai.labs.parser", parserProvider); parser = new RestSemanticParser(runtime, resourceClientLibrary, lifecycleTasks); @@ -181,7 +183,7 @@ void cacheIsBoundedByMaximumSize() throws Exception { @Test @Timeout(60) - @DisplayName("invalidateCache drops cached parsers so an updated configuration is re-read") + @DisplayName("invalidateCache empties the cache so the next request rebuilds from the store") void invalidateCacheForcesReload() throws Exception { doReturn(new ParserConfiguration()).when(resourceClientLibrary).getResource(any(), eq(ParserConfiguration.class)); @@ -200,4 +202,52 @@ void invalidateCacheForcesReload() throws Exception { verify(parserProvider, times(2)).get(); verify(resourceClientLibrary, times(2)).getResource(any(), eq(ParserConfiguration.class)); } + + /** + * The TTL is the only thing that ever removes a cached parser on its own — the + * store never invalidates it. Without a fake clock this is untestable (five + * minutes of wall time), so the parser is built with an injectable + * {@code Ticker} here and the clock is advanced by hand: no sleeps, no + * flakiness. Drop {@code expireAfterWrite} from the Caffeine builder and the + * two assertions after the advance both fail — the entry is still cached and + * the store is never re-read. + */ + @Test + @Timeout(60) + @DisplayName("a cached parser survives until PARSER_CACHE_TTL and is rebuilt from the store afterwards") + void cachedParserExpiresAfterTtl() throws Exception { + doReturn(new ParserConfiguration()).when(resourceClientLibrary).getResource(any(), eq(ParserConfiguration.class)); + + AtomicLong nanos = new AtomicLong(); + RestSemanticParser expiringParser = new RestSemanticParser(runtime, resourceClientLibrary, lifecycleTasks, nanos::get); + + expiringParser.parse(CONFIG_ID, 1, "first", asyncResponse); + captureCallables(1).getFirst().call(); + assertEquals(1, expiringParser.cachedParserCount()); + verify(resourceClientLibrary, times(1)).getResource(any(), eq(ParserConfiguration.class)); + + // One nanosecond short of the TTL: still the same cached parser instance. + nanos.addAndGet(RestSemanticParser.PARSER_CACHE_TTL.toNanos() - 1); + reset(runtime); + doReturn(mock(Future.class)).when(runtime).submitCallable(any(Callable.class), any()); + expiringParser.parse(CONFIG_ID, 1, "second", mock(AsyncResponse.class)); + captureCallables(1).getFirst().call(); + + assertEquals(1, expiringParser.cachedParserCount(), "entry must not expire before the TTL elapses"); + verify(resourceClientLibrary, times(1)).getResource(any(), eq(ParserConfiguration.class)); + verify(parserProvider, times(1)).get(); + + // Past the TTL: the entry is gone and the next request rebuilds it. + nanos.addAndGet(2); + assertEquals(0, expiringParser.cachedParserCount(), "entry must be evicted once the TTL has elapsed"); + + reset(runtime); + doReturn(mock(Future.class)).when(runtime).submitCallable(any(Callable.class), any()); + expiringParser.parse(CONFIG_ID, 1, "third", mock(AsyncResponse.class)); + captureCallables(1).getFirst().call(); + + verify(resourceClientLibrary, times(2)).getResource(any(), eq(ParserConfiguration.class)); + verify(parserProvider, times(2)).get(); + assertEquals(1, expiringParser.cachedParserCount()); + } } From e1534c4ea56857f51f9a436bd50f7ac816b74d7b Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Wed, 29 Jul 2026 23:35:06 +0200 Subject: [PATCH 06/11] fix(schedule): guard the STORED owner on update, not just the request body MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit's first review of this PR (50 files, finally under its 100-file limit). Seven findings, all legitimate; three Major. The security one is a hole in the ownership fix I added earlier in this same PR. - updateSchedule checked only the request BODY's userId. A body that omits userId is exempt — it means "run as the system scheduler" — so a non-admin could PUT over a schedule STORED against victim-42 and pass the guard, retargeting its agent/cron/message or disarming the victim's dream schedule outright. fireNow already read the stored value for exactly this reason. Update needs both questions answered: may I touch THIS schedule (stored owner), and may I make it act as THAT identity (body owner). This is the same defect class as A2/A5 in wave 2a — the guard validated what the caller SENT rather than what they were reaching for. I fixed that pattern three PRs ago and reintroduced it here. Mutation-checked: without the stored guard the editor gets 200 on a victim's schedule instead of 403. A missing schedule falls through to the normal 404 rather than 403, so the guard cannot be used to probe which schedule ids exist. - A crashed Dream cycle reported itself SUCCESSFUL. DreamResult.isSuccess() is `error == null` and the catch stored e.getMessage(), which is null for plenty of real exceptions (NullPointerException among them) — so the schedule's failure bookkeeping never ran for precisely the failures nobody anticipated. A describe() helper now always yields a non-null, class-named reason. - getCurrentResourceId returns null on PostgreSQL for an agent with no deployed version, which made the operator-facing rejection read "Could not read agent 'x': null" — the one message that has to be actionable, since every rejection there explains a misconfigured Dream schedule. Handled explicitly. - The parallel-batch setup grace was a flat 1 second. Setup cost (agent lookup, conversation start, attachment grants, prior-entry verification — several store round trips) does not shrink with a short configured timeout, so 1s is a large share of a 2s budget and quite possibly less than setup takes. Now max(1s, 10% of the per-attempt budget), keeping the orchestrator's deadline behind the member's own in both directions. - routeToDeadLetter's javadoc still said "after all retries are exhausted", contradicting the no-retry code directly above it. - The user-memory.md example still set maxSummarizationCalls, so anyone copying it opted into the deprecated bound the table beside it warns against — an explicitly-set value re-arms the legacy ceiling. - The wave-3 Verification block reported 12,633 tests while the review-pass section reported 12,912. Both were real runs; they are now labelled as such rather than one silently contradicting the other. 710 tests pass across the affected classes. --- docs/changelog.md | 4 +- docs/user-memory.md | 1 - .../internal/GroupConversationService.java | 44 +++++++++++++++--- .../engine/runtime/internal/DreamService.java | 40 ++++++++++++++-- .../internal/NatsConversationCoordinator.java | 9 +++- .../schedule/rest/RestScheduleStore.java | 46 ++++++++++++++++++- ...oupConversationServiceConcurrencyTest.java | 21 +++++++-- .../schedule/rest/RestScheduleStoreTest.java | 43 +++++++++++++++++ 8 files changed, 189 insertions(+), 19 deletions(-) diff --git a/docs/changelog.md b/docs/changelog.md index 3ac937364..cf1827791 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -78,7 +78,9 @@ Per the repo owner's decision, `DreamService` is now registered with `ScheduleFi ### Verification -Clean compile passed **first attempt**, with no repairs needed despite three cross-workstream signature changes. Full suite: 12,633 tests, **0 non-environmental failures** (308 listed failures/errors all carry a loopback/selector/event-loop signature; this machine cannot bind sockets). All four mutation checks bite — C1, C5, C9 and B2 each fail a test when reverted, verified against whole test classes and with surefire reports checked to confirm the new classes actually executed rather than being silently skipped. +Clean compile passed **first attempt**, with no repairs needed despite three cross-workstream signature changes. Full suite **as of the original wave-3 work**: 12,633 tests, **0 non-environmental failures** (308 listed failures/errors all carry a loopback/selector/event-loop signature; this machine cannot bind sockets). All four mutation checks bite — C1, C5, C9 and B2 each fail a test when reverted, verified against whole test classes and with surefire reports checked to confirm the new classes actually executed rather than being silently skipped. + +> The pre-merge review pass above re-ran the suite after its fixes: **12,912 tests**, failures confined to the same 15 known network-dependent classes. Both figures are real runs at different points — the earlier one is not superseded, it just predates ~280 added tests. --- diff --git a/docs/user-memory.md b/docs/user-memory.md index 046778c8a..22cf20ebf 100644 --- a/docs/user-memory.md +++ b/docs/user-memory.md @@ -60,7 +60,6 @@ Enable advanced memory features (LLM tools, Dream consolidation, guardrails, rec "summarizeTargetEntries": 2, "summarizeGroupBy": "category", "preserveAgentProvenance": false, - "maxSummarizationCalls": 10, "maxCostPerRun": 0.50 } }, diff --git a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java index 4499b2234..b6490fbda 100644 --- a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java +++ b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java @@ -109,11 +109,25 @@ public class GroupConversationService implements IGroupConversationService { /** * Slack added on top of a member's own budget when a parallel phase arms its * batch deadline. A member reaches its {@code responseFuture.get(timeout)} only - * after agent lookup, conversation start and attachment sharing, so without a - * grace the orchestrator's deadline — armed the instant the batch is dispatched - * — expires while the member is still legitimately inside its own budget. + * after agent lookup, conversation start, attachment grants and prior-entry + * verification — several store round trips — so without a grace the + * orchestrator's deadline, armed the instant the batch is dispatched, expires + * while the member is still legitimately inside its own budget. + *

      + * The floor is absolute, but it cannot be ONLY absolute: setup cost does not + * shrink with a short configured {@code agentTimeoutSeconds}, so a flat second + * is a large fraction of a 2s budget and a rounding error against a 180s one. + * The grace is therefore {@code max(floor, timeout * fraction)} — see + * {@link #parallelBatchGraceSeconds}. */ - private static final int PARALLEL_BATCH_GRACE_SECONDS = 1; + private static final int PARALLEL_BATCH_GRACE_FLOOR_SECONDS = 1; + + /** + * Fraction of a member's per-attempt budget also allowed for setup, so the + * grace scales instead of being swamped by a large timeout or dominating a + * small one. + */ + private static final double PARALLEL_BATCH_GRACE_FRACTION = 0.1; /** * Ceiling on the derived parallel-batch budget, so an absurd @@ -1700,8 +1714,8 @@ private static boolean reserveTurn(AtomicInteger turnCounter, int maxTurns) { * It is derived from what ONE member turn may legitimately consume — its * per-attempt {@code agentTimeoutSeconds} multiplied by the number of attempts * {@code onAgentFailure} allows (only {@code RETRY} retries, and it retries at - * most {@code maxRetries} times) — plus {@link #PARALLEL_BATCH_GRACE_SECONDS}. - * The normalisation of both protocol values is deliberately identical to + * most {@code maxRetries} times) — plus {@link #parallelBatchGraceSeconds}. The + * normalisation of both protocol values is deliberately identical to * {@code executeAgentTurn}'s: if the orchestrator's deadline is shorter than * the member's own, the member's timeout handling (retry / abort / attributed * SKIP) becomes unreachable. @@ -1714,7 +1728,23 @@ static long parallelBatchBudgetSeconds(ProtocolConfig protocol) { long attempts = protocol.onAgentFailure() == ProtocolConfig.MemberFailurePolicy.RETRY ? (protocol.maxRetries() > 0 ? protocol.maxRetries() : DEFAULT_MAX_RETRIES) + 1L : 1L; - return Math.min(timeout * attempts + PARALLEL_BATCH_GRACE_SECONDS, MAX_PARALLEL_BATCH_BUDGET_SECONDS); + return Math.min(timeout * attempts + parallelBatchGraceSeconds(timeout), MAX_PARALLEL_BATCH_BUDGET_SECONDS); + } + + /** + * Setup slack for a parallel batch: the larger of an absolute floor and a + * fraction of the member's per-attempt budget. + *

      + * A purely absolute grace loses the race again whenever setup outruns it, which + * is likeliest with a SHORT configured timeout — there one second is both a big + * share of the budget and quite possibly less than the store round trips take. + * Scaling with the timeout keeps the orchestrator's deadline behind the + * member's own in both directions, which is the property that makes + * {@code executeAgentTurn}'s retry / abort / attributed-SKIP branches reachable + * at all. + */ + static long parallelBatchGraceSeconds(long perAttemptTimeoutSeconds) { + return Math.max(PARALLEL_BATCH_GRACE_FLOOR_SECONDS, (long) Math.ceil(perAttemptTimeoutSeconds * PARALLEL_BATCH_GRACE_FRACTION)); } // ================================================================= diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java index 61ba110cd..11ebae3be 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java @@ -184,11 +184,21 @@ public DreamResult processScheduledFire(String agentId, Integer agentVersion, St try { int version = agentVersion != null && agentVersion > 0 ? agentVersion - : agentStore.getCurrentResourceId(agentId).getVersion(); + : currentVersionOf(agentId); + if (version <= 0) { + // getCurrentResourceId returns null on PostgreSQL for an agent with no + // deployed version (GroupConversationService documents the same). Left + // implicit it became an NPE whose message is null, so the operator-facing + // reason read "Could not read agent 'x': null" — the one string that has + // to be actionable, since every rejection here explains a misconfigured + // Dream schedule. + return rejected(userId, start, "Agent '" + agentId + "' has no current version — deploy it before scheduling a dream cycle, " + + "or pin an explicit agentVersion on the schedule."); + } agentConfiguration = agentStore.read(agentId, version); } catch (Exception e) { LOGGER.errorf(e, "[DREAM] Could not read agent '%s' (version=%s) for a scheduled dream cycle", agentId, agentVersion); - return rejected(userId, start, "Could not read agent '" + agentId + "': " + e.getMessage()); + return rejected(userId, start, "Could not read agent '" + agentId + "': " + describe(e)); } if (agentConfiguration == null) { @@ -323,7 +333,7 @@ public DreamResult process(String userId, String agentId, AgentConfiguration.Dre cyclesFailedCounter.increment(); LOGGER.errorf(e, "[DREAM] Failed for user='%s'", userId); return new DreamResult(userId, pruned, contradictions, summarized, Duration.between(start, Instant.now()).toMillis(), estimatedCost, - e.getMessage()); + describe(e)); } } @@ -878,6 +888,30 @@ static String truncate(String text, int maxLength) { * {@code null} on success; otherwise the cause, which the schedule * dispatcher turns into a FAILED fire */ + /** + * A never-null, always-informative description of a failure. + *

      + * {@code DreamResult.isSuccess()} is {@code error == null}, and + * {@link Throwable#getMessage()} is null for plenty of real exceptions + * (NullPointerException among them) — so passing the raw message through made a + * crashed cycle report itself as a SUCCESSFUL one, and the schedule's failure + * bookkeeping never ran. + */ + private static String describe(Throwable e) { + String message = e.getMessage(); + return (message == null || message.isBlank()) ? e.getClass().getSimpleName() : e.getClass().getSimpleName() + ": " + message; + } + + /** + * Current version of an agent, or {@code -1} when it has none. Isolated so the + * null {@code getCurrentResourceId} contract is handled in one place rather + * than surfacing as an NPE at the call site. + */ + private int currentVersionOf(String agentId) throws Exception { + var currentId = agentStore.getCurrentResourceId(agentId); + return currentId == null || currentId.getVersion() == null ? -1 : currentId.getVersion(); + } + public record DreamResult(String userId, int entriesPruned, int contradictionsFound, int entriesSummarized, long durationMs, double estimatedCostUsd, String error) { diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java index 1ae817283..bea4db5e0 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/NatsConversationCoordinator.java @@ -370,8 +370,13 @@ private void recordConsumeMetrics(long startNanos) { } /** - * Route a failed message to the dead-letter stream after all retries are - * exhausted. + * Route a failed message to the dead-letter stream. + *

      + * On the FIRST failure, not after retries are exhausted: a task that reports + * failure has already run — possibly calling an LLM, executing tools and + * spending money — so re-running it would repeat those side effects. The + * retry-then-dead-letter wording this replaced described the behaviour before + * the no-retry change and contradicted the code directly above. */ private void routeToDeadLetter(String conversationId, Throwable failure) { String deadLetterSubject = DEAD_LETTER_PREFIX + sanitizeSubject(conversationId); diff --git a/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java b/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java index 4f46413b3..f658f9ed6 100644 --- a/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java +++ b/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java @@ -184,8 +184,22 @@ public Response updateSchedule(String scheduleId, ScheduleConfiguration schedule return guard; } - // Checked on the incoming BODY: this is the path that would re-point an - // otherwise harmless schedule at another user's identity. + // BOTH sides matter, and checking only one is the bug this replaces. + // + // STORED userId — "may I touch this schedule at all?". Guarding only the + // body let a non-admin PUT over a schedule owned by victim-42 as long as + // the body left userId unset (or "system:scheduler", which is exempt): + // the guard passed and the update ran, so the caller could retarget the + // agent/cron/message or effectively disarm the victim's dream schedule. + // fireNow already checks the stored value for exactly this reason. + // + // BODY userId — "may I make it act as this identity?", i.e. the re-point + // path that would aim an otherwise harmless schedule at another user. + Response storedOwnerGuard = requireOwnUserIdOfStoredSchedule(scheduleId, "update"); + if (storedOwnerGuard != null) { + return storedOwnerGuard; + } + Response ownerGuard = requireOwnUserId(schedule != null ? schedule.getUserId() : null, "update"); if (ownerGuard != null) { return ownerGuard; @@ -424,6 +438,34 @@ private Response requireOwnUserId(String userId, String operation) { .build(); } + /** + * The other half of {@link #requireOwnUserId}: may this caller touch the + * schedule that is already stored under {@code scheduleId}? + *

      + * Checking the request body alone is not enough. A body that simply omits + * {@code userId} is exempt (it means "run as the system scheduler"), so a + * body-only guard let a non-admin overwrite a schedule stored against another + * user — retargeting its agent, cron or message, or disarming it outright. + * {@code fireNow} reads the stored value for the same reason. + *

      + * A missing schedule is left to the caller's own not-found handling rather than + * being reported as forbidden, so this guard cannot be used to probe which + * schedule ids exist. + */ + private Response requireOwnUserIdOfStoredSchedule(String scheduleId, String operation) { + ScheduleConfiguration stored; + try { + stored = scheduleStore.readSchedule(scheduleId); + } catch (Exception e) { + // Not found / unreadable: let the normal path produce the 404. + return null; + } + if (stored == null) { + return null; + } + return requireOwnUserId(stored.getUserId(), operation); + } + /** * For mutating operations on a HITL timeout schedule, require the eddi-admin * role. Reads the STORED schedule so a request body cannot hide the marker. The diff --git a/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java index c81ff95d9..e7eab72be 100644 --- a/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java +++ b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceConcurrencyTest.java @@ -472,7 +472,14 @@ void parallelPhase_batchDeadline_coversTheMemberRetryEnvelope() throws Exception @Test @DisplayName("C8: the batch budget is one member's attempt envelope, never the batch size") void parallelBatchBudget_isDerivedFromOneMembersAttemptEnvelope() { - // SKIP / ABORT never retry: one attempt plus the 1s setup grace. + // The grace is max(1s, 10% of the per-attempt budget) — a flat second is a + // large share of a 2s timeout and a rounding error against a 180s one, and + // setup cost does not shrink just because the timeout is short. Whenever the + // grace is too small the orchestrator wins the race again and the member's + // own RETRY/ABORT/attributed-SKIP handling is unreachable, which is the whole + // defect this budget exists to avoid. + // + // SKIP / ABORT never retry: one attempt plus the grace (10 -> max(1, 1) = 1). assertEquals(11L, GroupConversationService.parallelBatchBudgetSeconds( new ProtocolConfig(10, ProtocolConfig.MemberFailurePolicy.SKIP, 2, ProtocolConfig.MemberUnavailablePolicy.SKIP))); @@ -483,11 +490,19 @@ void parallelBatchBudget_isDerivedFromOneMembersAttemptEnvelope() { assertEquals(31L, GroupConversationService.parallelBatchBudgetSeconds( new ProtocolConfig(10, ProtocolConfig.MemberFailurePolicy.RETRY, 2, ProtocolConfig.MemberUnavailablePolicy.SKIP))); + // A short timeout keeps the 1s FLOOR rather than 10% of 2s. + assertEquals(3L, GroupConversationService.parallelBatchBudgetSeconds( + new ProtocolConfig(2, ProtocolConfig.MemberFailurePolicy.SKIP, 0, + ProtocolConfig.MemberUnavailablePolicy.SKIP))); // Unset values fall back to the same defaults executeAgentTurn applies: - // 180s per attempt, 2 retries. - assertEquals(541L, GroupConversationService.parallelBatchBudgetSeconds( + // 180s per attempt, 2 retries -> 540 + ceil(18) = 558. + assertEquals(558L, GroupConversationService.parallelBatchBudgetSeconds( new ProtocolConfig(0, ProtocolConfig.MemberFailurePolicy.RETRY, 0, ProtocolConfig.MemberUnavailablePolicy.SKIP))); + // The grace itself: floor below 10s, proportional above it. + assertEquals(1L, GroupConversationService.parallelBatchGraceSeconds(2)); + assertEquals(1L, GroupConversationService.parallelBatchGraceSeconds(10)); + assertEquals(18L, GroupConversationService.parallelBatchGraceSeconds(180)); // An absurd config is capped instead of overflowing the deadline into the past. assertEquals(TimeUnit.HOURS.toSeconds(24), GroupConversationService.parallelBatchBudgetSeconds( new ProtocolConfig(Integer.MAX_VALUE, ProtocolConfig.MemberFailurePolicy.RETRY, Integer.MAX_VALUE, diff --git a/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java b/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java index 6b0f6eec3..b9d3d129b 100644 --- a/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java +++ b/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java @@ -490,6 +490,49 @@ void updateSchedule_repointingToAnotherUser_forbiddenForEditor() throws Exceptio verify(scheduleStore, never()).updateSchedule(eq("r1"), any()); } + /** + * The body-only guard was not enough, and this is the hole it left. A body that + * omits {@code userId} is exempt — it means "run as the system scheduler" — so + * a non-admin could PUT over a schedule STORED against a victim and pass the + * check with room to spare, retargeting its agent/cron/message or disarming it. + * Update has to answer both questions: may I touch THIS schedule (stored + * owner), and may I make it act as THAT identity (body owner)? + */ + @Test + void updateSchedule_ofAnotherUsersStoredSchedule_forbiddenForEditor() throws Exception { + asEditor("editor-1"); + // Stored schedule belongs to the victim... + when(scheduleStore.readSchedule("v1")).thenReturn(dreamSchedule("v1", "victim-42")); + + // ...and the body leaves userId unset, which the body-only guard waved through. + Response response = rest.updateSchedule("v1", makeCronSchedule("v1")); + + assertEquals(403, response.getStatus(), "a non-admin must not overwrite a schedule owned by another user"); + verify(scheduleStore, never()).updateSchedule(eq("v1"), any()); + } + + @Test + void updateSchedule_ofOwnStoredSchedule_allowedForEditor() throws Exception { + asEditor("editor-1"); + when(scheduleStore.readSchedule("m1")).thenReturn(dreamSchedule("m1", "editor-1")); + + Response response = rest.updateSchedule("m1", dreamSchedule("m1", "editor-1")); + + assertEquals(200, response.getStatus()); + verify(scheduleStore).updateSchedule(eq("m1"), any()); + } + + @Test + void updateSchedule_ofAnotherUsersStoredSchedule_allowedForAdmin() throws Exception { + asAdmin("root"); + when(scheduleStore.readSchedule("v1")).thenReturn(dreamSchedule("v1", "victim-42")); + + Response response = rest.updateSchedule("v1", dreamSchedule("v1", "victim-42")); + + assertEquals(200, response.getStatus()); + verify(scheduleStore).updateSchedule(eq("v1"), any()); + } + @Test void fireNow_scheduleActingAsAnotherUser_forbiddenForEditor() throws Exception { asEditor("editor-1"); From 0d01ef31bda6b4e2fa829eca625a9f074e855c63 Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Wed, 29 Jul 2026 23:51:38 +0200 Subject: [PATCH 07/11] fix(schedule): fail closed on an unreadable owner; reattach the DreamResult javadoc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two follow-ups from the bots' second look at the previous commit. - Copilot: requireOwnUserIdOfStoredSchedule caught Exception and returned "allow", i.e. it failed OPEN — "not found" and "could not read it" collapsed into one benign answer on a security check. Directionally right, and I checked the claim rather than taking it: on the update path it was NOT exploitable, because requireAdminForHitl reads the same schedule first and already fails closed with a 500. But a guard whose safety depends on an unrelated guard running before it is one reordering away from being a hole, so it now fails closed on its own account — mirroring that method's status and phrasing instead of inventing a second convention for the same condition. The test says exactly this rather than overclaiming: it pins that a store failure never lets the update proceed, and states in its javadoc that the 500 today comes from the HITL guard, so nobody later reads it as proof of the ownership branch. - The code-quality bot: my previous commit inserted describe() and currentVersionOf() between the DreamResult record's javadoc and the record, so that javadoc — @param estimatedCostUsd, @param error — was attached to describe(). A real slip from editing by anchor text. Helpers moved above; each doc now sits on its own declaration, and I checked the file for other adjacent javadoc blocks (none). 139 tests pass across the affected classes. --- .../engine/runtime/internal/DreamService.java | 20 +++++------ .../schedule/rest/RestScheduleStore.java | 20 +++++++++-- .../schedule/rest/RestScheduleStoreTest.java | 34 +++++++++++++++++++ 3 files changed, 62 insertions(+), 12 deletions(-) diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java index 11ebae3be..88b4ce5f7 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java @@ -878,16 +878,6 @@ static String truncate(String text, int maxLength) { return text.substring(0, maxLength - 1) + "…"; } - /** - * Result of a dream consolidation cycle. - * - * @param estimatedCostUsd - * estimated LLM spend of this cycle — reported on the schedule fire - * log so the dollar budget is observable - * @param error - * {@code null} on success; otherwise the cause, which the schedule - * dispatcher turns into a FAILED fire - */ /** * A never-null, always-informative description of a failure. *

      @@ -912,6 +902,16 @@ private int currentVersionOf(String agentId) throws Exception { return currentId == null || currentId.getVersion() == null ? -1 : currentId.getVersion(); } + /** + * Result of a dream consolidation cycle. + * + * @param estimatedCostUsd + * estimated LLM spend of this cycle — reported on the schedule fire + * log so the dollar budget is observable + * @param error + * {@code null} on success; otherwise the cause, which the schedule + * dispatcher turns into a FAILED fire + */ public record DreamResult(String userId, int entriesPruned, int contradictionsFound, int entriesSummarized, long durationMs, double estimatedCostUsd, String error) { diff --git a/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java b/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java index f658f9ed6..f61334605 100644 --- a/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java +++ b/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java @@ -451,14 +451,30 @@ private Response requireOwnUserId(String userId, String operation) { * A missing schedule is left to the caller's own not-found handling rather than * being reported as forbidden, so this guard cannot be used to probe which * schedule ids exist. + *

      + * "Not found" and "could not read it" are deliberately NOT the same outcome. An + * earlier version caught {@code Exception} and returned "allow" for both, + * collapsing two very different causes into one benign answer on a security + * check. + *

      + * On the update path that was not actually exploitable — + * {@link #requireAdminForHitl} reads the same schedule first and already fails + * closed — but a guard whose safety depends on an unrelated guard running + * before it is one reordering away from being a hole, so this one fails closed + * on its own account. It mirrors that method's status and phrasing rather than + * inventing a second convention for the same condition. */ private Response requireOwnUserIdOfStoredSchedule(String scheduleId, String operation) { ScheduleConfiguration stored; try { stored = scheduleStore.readSchedule(scheduleId); + } catch (IResourceStore.ResourceNotFoundException e) { + return null; // no schedule to protect — let the downstream op surface its 404 } catch (Exception e) { - // Not found / unreadable: let the normal path produce the 404. - return null; + LOGGER.error("Failed to verify schedule ownership for " + sanitize(scheduleId) + " (" + sanitize(operation) + ")", e); + return Response.status(Response.Status.INTERNAL_SERVER_ERROR) + .entity("Unable to verify schedule authorization; refusing to " + operation + " schedule.") + .build(); } if (stored == null) { return null; diff --git a/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java b/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java index b9d3d129b..c1b588d35 100644 --- a/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java +++ b/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java @@ -4,6 +4,7 @@ */ package ai.labs.eddi.engine.schedule.rest; +import ai.labs.eddi.datastore.IResourceStore; import ai.labs.eddi.engine.schedule.IScheduleStore; import ai.labs.eddi.engine.schedule.model.ScheduleConfiguration; import ai.labs.eddi.engine.schedule.model.ScheduleConfiguration.FireStatus; @@ -511,6 +512,39 @@ void updateSchedule_ofAnotherUsersStoredSchedule_forbiddenForEditor() throws Exc verify(scheduleStore, never()).updateSchedule(eq("v1"), any()); } + /** + * An unverifiable owner must DENY, never allow. + *

      + * Be precise about what this proves. On the update path + * {@code requireAdminForHitl} reads the same schedule first and already fails + * closed, so it is that guard producing the 500 here — the ownership guard's + * own fail-closed branch is defence in depth, not the thing standing between a + * caller and the mutation today. What the assertion genuinely pins is the + * property that matters: a store failure never results in the update + * proceeding. + */ + @Test + void updateSchedule_whenOwnershipCannotBeRead_refusesInsteadOfFailingOpen() throws Exception { + asEditor("editor-1"); + when(scheduleStore.readSchedule("x1")).thenThrow(new IResourceStore.ResourceStoreException("store down")); + + Response response = rest.updateSchedule("x1", makeCronSchedule("x1")); + + assertEquals(500, response.getStatus(), "an unverifiable owner must deny, not allow"); + verify(scheduleStore, never()).updateSchedule(eq("x1"), any()); + } + + @Test + void updateSchedule_ofMissingSchedule_stillReportsNotFound() throws Exception { + asEditor("editor-1"); + when(scheduleStore.readSchedule("gone")).thenThrow(new IResourceStore.ResourceNotFoundException("nope")); + doThrow(new IResourceStore.ResourceNotFoundException("nope")).when(scheduleStore).updateSchedule(eq("gone"), any()); + + // A missing schedule must not be reported as forbidden — otherwise the guard + // becomes an oracle for which schedule ids exist. + assertThrows(jakarta.ws.rs.NotFoundException.class, () -> rest.updateSchedule("gone", makeCronSchedule("gone"))); + } + @Test void updateSchedule_ofOwnStoredSchedule_allowedForEditor() throws Exception { asEditor("editor-1"); From 2185024464286ad67e1dd5e6c21f450cbe670694 Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Thu, 30 Jul 2026 00:24:55 +0200 Subject: [PATCH 08/11] fix(schedule): one definition of an absent time zone; apply defaults on update MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot flagged that updateSchedule never applies defaults before persisting, citing a blank timeZone throwing and an omitted userId being stored empty. Both symptoms were real. One mechanism was not, and finding that out was the useful part. - The userId half is exactly as reported. Update skipped applyDefaults while create ran it, so an absent userId persisted as NULL rather than SCHEDULER_USER_ID — and since ownership treats null as unowned, an editor updating their OWN schedule silently made it writable by every other editor. applyDefaults now runs on update too, deliberately AFTER the ownership guards so they judge what the caller sent rather than what defaulting produced. - The blank-timeZone half was NOT caused by the missing applyDefaults, and my first fix did not resolve it — the test I wrote to prove the fix kept failing, which is the only reason I looked further instead of shipping a plausible explanation. validateSchedule runs BEFORE defaulting on BOTH paths, and it was validateSchedule's own ZoneId.of that threw. The three ZoneId.of call sites disagreed about what "absent" means: one used the raw value, two null-checked but not blank — and a blank string is non-null, so it reached ZoneId.of(""). So the defect was on CREATE as well as update, in a place the report did not name. Replaced with a single zoneOf() accessor holding one definition of absent (null OR blank), used by all three sites, and covered on both paths. Mutation-checked: reverting zoneOf to a null-only check fails both tests. I corrected the code comment and the test javadoc that had recorded the wrong mechanism, rather than leaving a confident explanation that happened to be false — a misleading comment survives longer than the bug. 149 tests pass across the schedule surface. --- .../schedule/rest/RestScheduleStore.java | 41 +++++++++++-- .../schedule/rest/RestScheduleStoreTest.java | 57 +++++++++++++++++++ 2 files changed, 92 insertions(+), 6 deletions(-) diff --git a/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java b/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java index f61334605..efd8a2732 100644 --- a/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java +++ b/src/main/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStore.java @@ -207,6 +207,21 @@ public Response updateSchedule(String scheduleId, ScheduleConfiguration schedule validateSchedule(schedule); + // Same order as createSchedule, and for the same reason: PUT is a full + // replace, so a field the body omits must be defaulted rather than + // persisted empty. Update skipped this entirely, so an absent userId was + // stored as null instead of SCHEDULER_USER_ID — and because ownership + // treats null as unowned, an editor updating their OWN schedule silently + // made it writable by every other editor. + // + // (The blank-timeZone half of that report was a different bug in a + // different place: validateSchedule runs BEFORE this on both paths, and it + // was validateSchedule's own ZoneId.of that threw. Fixed in zoneOf().) + // + // Deliberately AFTER the ownership guards above, so they judge what the + // caller actually sent rather than what defaulting turned it into. + applyDefaults(schedule); + // Recompute nextFire computeInitialNextFire(schedule); @@ -539,6 +554,23 @@ private Response setEnabled(String scheduleId, boolean enabled) { } } + /** + * The schedule's time zone, or the configured default when it is absent. + *

      + * "Absent" means null OR blank. The three {@code ZoneId.of} call sites used to + * disagree about this: one passed the raw value, two null-checked but not blank + * — and a blank string is non-null, so it reached {@code ZoneId.of("")} and + * threw {@code DateTimeException}. That surfaced as a 500 on both create and + * update for a body carrying {@code "timeZone": ""}, even though + * {@link #applyDefaults} treats blank as "use the default" and + * {@link #validateSchedule} deliberately skips validating a blank value. One + * accessor, one definition of absent. + */ + private ZoneId zoneOf(ScheduleConfiguration schedule) { + String zone = schedule.getTimeZone(); + return ZoneId.of(zone == null || zone.isBlank() ? defaultTimeZone : zone); + } + private void applyDefaults(ScheduleConfiguration schedule) { // Infer trigger type from fields — must also handle the case where the // default CRON value is set but heartbeatIntervalSeconds indicates HEARTBEAT @@ -577,8 +609,7 @@ private void computeInitialNextFire(ScheduleConfiguration schedule) { // Heartbeat: first fire = now + interval schedule.setNextFire(Instant.now().plusSeconds(schedule.getHeartbeatIntervalSeconds())); } else if (schedule.getCronExpression() != null && !schedule.getCronExpression().isBlank()) { - ZoneId zoneId = ZoneId.of(schedule.getTimeZone()); - Instant nextFire = CronParser.computeNextFire(schedule.getCronExpression(), Instant.now(), zoneId); + Instant nextFire = CronParser.computeNextFire(schedule.getCronExpression(), Instant.now(), zoneOf(schedule)); schedule.setNextFire(nextFire); } else if (schedule.getOneTimeAt() != null && !schedule.getOneTimeAt().isBlank()) { schedule.setNextFire(Instant.parse(schedule.getOneTimeAt())); @@ -590,8 +621,7 @@ private Instant computeNextFireForSchedule(ScheduleConfiguration schedule) { return Instant.now().plusSeconds(schedule.getHeartbeatIntervalSeconds()); } if (schedule.getCronExpression() != null && !schedule.getCronExpression().isBlank()) { - ZoneId zoneId = ZoneId.of(schedule.getTimeZone() != null ? schedule.getTimeZone() : defaultTimeZone); - return CronParser.computeNextFire(schedule.getCronExpression(), Instant.now(), zoneId); + return CronParser.computeNextFire(schedule.getCronExpression(), Instant.now(), zoneOf(schedule)); } return null; } @@ -649,8 +679,7 @@ private void validateSchedule(ScheduleConfiguration schedule) { CronParser.validate(schedule.getCronExpression()); // Enforce minimum interval - ZoneId zoneId = ZoneId.of(schedule.getTimeZone() != null ? schedule.getTimeZone() : defaultTimeZone); - long intervalSec = CronParser.computeMinIntervalSeconds(schedule.getCronExpression(), zoneId); + long intervalSec = CronParser.computeMinIntervalSeconds(schedule.getCronExpression(), zoneOf(schedule)); if (intervalSec < minIntervalSeconds) { throw new IllegalArgumentException(String.format( "Cron interval (%ds) is below minimum allowed (%ds). " diff --git a/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java b/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java index c1b588d35..4facadb81 100644 --- a/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java +++ b/src/test/java/ai/labs/eddi/engine/schedule/rest/RestScheduleStoreTest.java @@ -545,6 +545,63 @@ void updateSchedule_ofMissingSchedule_stillReportsNotFound() throws Exception { assertThrows(jakarta.ws.rs.NotFoundException.class, () -> rest.updateSchedule("gone", makeCronSchedule("gone"))); } + /** + * A blank time zone answered 500 on BOTH create and update. The originally + * reported mechanism — update skipping {@code applyDefaults} — was not the + * cause: {@code validateSchedule} runs before defaulting on both paths, and it + * was its own unguarded {@code ZoneId.of} that threw, because the three call + * sites disagreed about whether "absent" meant null or also blank. A blank + * string is non-null. See {@code zoneOf}. + */ + @Test + void createSchedule_withBlankTimeZone_appliesTheDefaultInsteadOfThrowing() throws Exception { + when(scheduleStore.createSchedule(any())).thenReturn("new-id"); + + var body = makeCronSchedule("c1"); + body.setTimeZone(""); + + Response response = rest.createSchedule(body); + + assertEquals(201, response.getStatus(), "create carried the identical defect"); + assertEquals("UTC", ((ScheduleConfiguration) response.getEntity()).getTimeZone()); + } + + @Test + void updateSchedule_withBlankTimeZone_appliesTheDefaultInsteadOfThrowing() throws Exception { + asAdmin("root"); + when(scheduleStore.readSchedule("t1")).thenReturn(makeCronSchedule("t1")); + + var body = makeCronSchedule("t1"); + body.setTimeZone(""); + + Response response = rest.updateSchedule("t1", body); + + assertEquals(200, response.getStatus(), "a blank timeZone must default, not 500"); + assertEquals("UTC", body.getTimeZone()); + verify(scheduleStore).updateSchedule(eq("t1"), any()); + } + + /** + * The other half, and the one that touches ownership: an absent userId was + * stored as null rather than the scheduler placeholder. Ownership treats null + * as unowned, so an editor updating their OWN schedule silently made it + * writable by every other editor. + */ + @Test + void updateSchedule_withoutUserId_storesTheSchedulerPlaceholderNotNull() throws Exception { + asEditor("editor-1"); + when(scheduleStore.readSchedule("u1")).thenReturn(dreamSchedule("u1", "editor-1")); + + var body = makeCronSchedule("u1"); + body.setUserId(null); + + Response response = rest.updateSchedule("u1", body); + + assertEquals(200, response.getStatus()); + assertEquals("system:scheduler", body.getUserId(), "an omitted userId must not persist as null — null reads as unowned"); + verify(scheduleStore).updateSchedule(eq("u1"), any()); + } + @Test void updateSchedule_ofOwnStoredSchedule_allowedForEditor() throws Exception { asEditor("editor-1"); From 1bca6d31f7ca8cfa2644916614464b2cb19c4c78 Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Thu, 30 Jul 2026 00:53:15 +0200 Subject: [PATCH 09/11] fix(group): order failure writes against the abort sweep; unresolvable host is permanent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit's second review of #619. Two of its four Major findings are addressed here; two are heavy-lift behavioural changes tracked separately. - DreamService.isTransientLlmFailure classified UnknownHostException as transient, contradicting its own javadoc two lines above, which names "wrong endpoint" as the example of PERMANENT. A hostname that does not resolve is a misconfiguration: treating it as transient made every cycle skip its groups and report SUCCESS forever, so the schedule never retried, never dead-lettered, and nobody learned the endpoint was wrong. ConnectException stays transient by contrast — the host resolved and refused, which is a restarting service. - The failure paths in the task-execution wave read the cancellation token WITHOUT the task-list monitor while the success path reads it inside. My own review pass refuted this on the grounds that abortWave is cancel() THEN allOf(futures).get(...), so the sweep cannot start until every worker returned. That refutation was WRONG, and I only found out by verifying the claim I was about to write into a comment: the join is bounded by MEMBER_TURN_CANCEL_DRAIN_SECONDS and proceeds to resetStrandedInProgressTasks on TIMEOUT. The timeout is reachable — a member parked in tryResolveMemberToolPause or in a nested GROUP discuss() never observes the token. So a late failure write really can race the sweep, marking a task FAILED that the sweep just reset and appending an error entry to a finished wave. Both failure paths now take the monitor across the check AND the write. handleTaskFailure is split into recordTaskFailure (document mutation, called under the monitor) and notifyTaskFailure (SSE listener, called outside it) — holding the monitor across a client write would stall sibling workers, which is why the success path emits its event outside the lock too. Lock order stays taskList -> transcript, and recordTaskFailure never touches taskList itself, so there is no inversion. Four tests reached handleTaskFailure by reflection and so failed at RUNTIME rather than compile time; repointed at recordTaskFailure, which is the half they actually assert on. Also corrected MEMBER_TURN_CANCEL_DRAIN_SECONDS' javadoc, which claimed cancellation releases turns at their await points so the timeout is "only a safety bound". It is not: the two await points named above ignore the token, so for a turn parked there the timeout is the mechanism, not the backstop. Third docs-describe-intent finding in this stack. 578 tests pass across the group and Dream surfaces. --- .../internal/GroupConversationService.java | 98 +++++++++++++++---- .../engine/runtime/internal/DreamService.java | 15 ++- ...pConversationServiceHitlCoverage3Test.java | 18 ++-- ...GroupConversationServiceTaskForceTest.java | 10 +- .../runtime/internal/DreamServiceTest.java | 23 +++++ 5 files changed, 134 insertions(+), 30 deletions(-) diff --git a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java index b6490fbda..fa83b4e4e 100644 --- a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java +++ b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java @@ -138,9 +138,24 @@ public class GroupConversationService implements IGroupConversationService { /** * How long an aborting orchestrator waits for cooperatively cancelled member - * turns to unwind before reclaiming their tasks. Cancellation releases the - * turns at their await points immediately, so this is only a safety bound for a - * turn that is between two await points. + * turns to unwind before reclaiming their tasks. + *

      + * This is NOT merely a safety bound, which is what an earlier version of this + * comment claimed. Cancellation releases the turn promptly at + * {@code responseFuture.get(...)}, but that is not a member turn's only await + * point, and the others do not observe the token: + *

        + *
      • {@code tryResolveMemberToolPause} blocks on a {@code resumeFuture} that + * was never registered against the cancellation token;
        • + *
      • a {@code MemberType.GROUP} member is dispatched into a nested synchronous + * {@code discuss(...)} with no token at all, under its own {@code activeTokens} + * entry.
        • + *
      + * For a turn parked at either of those, this timeout is the mechanism rather + * than the backstop: the orchestrator reclaims the task once it expires while + * the child work carries on. Making those paths cancellation-aware is tracked + * separately — until then this is a bound that can genuinely be hit, not an + * unreachable guard. */ private static final int MEMBER_TURN_CANCEL_DRAIN_SECONDS = 5; @@ -2064,26 +2079,60 @@ private void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigura // document alone; the reset sweep reclaims the task. break; } catch (GroupDiscussionException e) { - if (cancellation.isCancelled()) { - break; // no writes after the orchestrator gave up on this wave - } - // Quota errors are non-retryable — abort all tasks immediately + // Quota errors are non-retryable — abort all tasks immediately. + // Checked before the cancellation guard so a quota breach is still + // reported even when the wave is already unwinding. if (e.getCause() instanceof QuotaExceededException) { errors.add(e); return; // exit the entire agent's CompletableFuture } - handleTaskFailure(gc, task, member, e.getMessage(), phaseIdx, phase, listener, errors, e); + // The check AND the write go under the monitor, exactly like the + // success path above. + // + // The tempting argument for reading the token unlocked is that + // abortWave is cancel() THEN allOf(futures).get(...), so the reset + // sweep cannot begin until every worker has returned. That argument + // does NOT hold: the join is bounded by + // MEMBER_TURN_CANCEL_DRAIN_SECONDS and proceeds to + // resetStrandedInProgressTasks on timeout — and that timeout is + // reachable, because a member parked in tryResolveMemberToolPause + // or in a nested GROUP discuss() never observes the token (see + // MEMBER_TURN_CANCEL_DRAIN_SECONDS). A late failure write can + // therefore race the sweep, marking a task FAILED that the sweep + // just reset and appending an error entry to a finished wave. + // Only the monitor orders the two. Lock order: taskList -> transcript. + boolean recorded; + synchronized (taskList) { + recorded = !cancellation.isCancelled(); + if (recorded) { + recordTaskFailure(gc, task, member, e.getMessage(), phaseIdx, phase, errors, e); + } + } + if (!recorded) { + break; // no writes after the orchestrator gave up on this wave + } + // Outside the monitor: the listener is an SSE sink, and holding + // taskList across a client write would stall sibling workers. The + // success path emits its event outside the lock for the same reason. + notifyTaskFailure(listener, member, e.getMessage(), phaseIdx, phase); if (protocol.onAgentFailure() == ProtocolConfig.MemberFailurePolicy.ABORT) { break; } } catch (IllegalStateException e) { - if (cancellation.isCancelled()) { - break; // see above - } // H5 fix: catch status transition errors (e.g., double completion) LOGGER.warnf("Task state error for '%s': %s", task.subject(), e.getMessage()); - handleTaskFailure(gc, task, member, e.getMessage(), phaseIdx, phase, listener, errors, - new GroupDiscussionException(e.getMessage(), e)); + boolean recorded; + synchronized (taskList) { // see the reasoning above + recorded = !cancellation.isCancelled(); + if (recorded) { + recordTaskFailure(gc, task, member, e.getMessage(), phaseIdx, phase, errors, + new GroupDiscussionException(e.getMessage(), e)); + } + } + if (!recorded) { + break; + } + notifyTaskFailure(listener, member, e.getMessage(), phaseIdx, phase); } } }), executorService); @@ -2545,9 +2594,17 @@ private String resolveTaskAssignment(String assignToRole, List memb * Marks the task as failed, adds an error transcript entry, emits SSE events, * and collects the error for potential ABORT propagation. */ - private void handleTaskFailure(GroupConversation gc, TaskItem task, GroupMember member, + /** + * The document-mutating half of a task failure: fail the task and append the + * error entry. Callers MUST hold the task-list monitor so this write is ordered + * against {@code abortWave}'s reset sweep — see the call sites for why the + * cancel-then-join ordering does not suffice on its own. + *

      + * Split from {@link #notifyTaskFailure} deliberately: the listener is an SSE + * sink and must not be invoked while holding the monitor. + */ + private void recordTaskFailure(GroupConversation gc, TaskItem task, GroupMember member, String errorMessage, int phaseIdx, DiscussionPhase phase, - GroupDiscussionEventListener listener, List errors, GroupDiscussionException ex) { try { gc.getTaskList().failTask(task.id(), errorMessage); @@ -2564,14 +2621,19 @@ private void handleTaskFailure(GroupConversation gc, TaskItem task, GroupMember Instant.now(), null, null)); } - // Emit error event so SSE clients see the failure + errors.add(ex); + } + + /** + * Emit the failure to SSE clients. Called OUTSIDE the task-list monitor. + */ + private void notifyTaskFailure(GroupDiscussionEventListener listener, GroupMember member, + String errorMessage, int phaseIdx, DiscussionPhase phase) { if (listener != null) { listener.onSpeakerComplete(new GroupConversationEventSink.SpeakerCompleteEvent( member.agentId(), member.displayName(), "[ERROR] " + errorMessage, phaseIdx, phase.name())); } - - errors.add(ex); } private GroupMember findMember(List members, String agentId) { diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java index 88b4ce5f7..3e883ab21 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java @@ -638,13 +638,24 @@ private SummarizationOutcome summarizeInteractions(String userId, * permanent configuration fault (bad credentials, wrong endpoint, unknown * model). Only the latter should fail the schedule fire, because a FAILED fire * consumes the schedule's dead-letter budget. + *

      + * {@link UnknownHostException} is deliberately NOT transient, though it sits + * beside the ones that are. A hostname that does not resolve is a wrong + * endpoint — the very example this javadoc gives for "permanent" — so treating + * it as transient made every cycle skip its groups and report SUCCESS forever: + * the schedule never retried, never dead-lettered, and the operator never + * learned the endpoint was misconfigured. The classifier contradicted its own + * stated contract. + *

      + * {@link ConnectException} stays transient by contrast: the host resolved and + * refused, which is what a restarting or briefly unreachable service looks + * like. */ static boolean isTransientLlmFailure(Throwable throwable) { Throwable current = throwable; // Bounded walk — a self-referential cause chain must not spin forever. for (int depth = 0; current != null && depth < 10; depth++, current = current.getCause()) { - if (current instanceof SocketTimeoutException || current instanceof TimeoutException - || current instanceof ConnectException || current instanceof UnknownHostException) { + if (current instanceof SocketTimeoutException || current instanceof TimeoutException || current instanceof ConnectException) { return true; } String message = current.getMessage(); diff --git a/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceHitlCoverage3Test.java b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceHitlCoverage3Test.java index dcb094621..cdae48d45 100644 --- a/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceHitlCoverage3Test.java +++ b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceHitlCoverage3Test.java @@ -787,12 +787,15 @@ void handleTaskFailureMarksFailed() throws Exception { var task = g.getTaskList().all().get(0); var errors = java.util.Collections.synchronizedList(new java.util.ArrayList()); - var m = method("handleTaskFailure", GroupConversation.class, TaskItem.class, GroupMember.class, - String.class, int.class, DiscussionPhase.class, GroupDiscussionEventListener.class, + // handleTaskFailure was split so the SSE listener callback is not made while + // holding the task-list monitor; recordTaskFailure is the document-mutating + // half these tests actually assert on. + var m = method("recordTaskFailure", GroupConversation.class, TaskItem.class, GroupMember.class, + String.class, int.class, DiscussionPhase.class, List.class, GroupDiscussionException.class); var ex = new GroupDiscussionException("agent broke"); - invoke(m, g, task, member(), "agent broke", 0, phase(PhaseType.EXECUTE), null, errors, ex); + invoke(m, g, task, member(), "agent broke", 0, phase(PhaseType.EXECUTE), errors, ex); assertEquals(TaskStatus.FAILED, g.getTaskList().all().get(0).status()); assertEquals(1, errors.size()); @@ -815,12 +818,15 @@ void handleTaskFailureAlreadyTerminal() throws Exception { var task = g.getTaskList().all().get(0); var errors = java.util.Collections.synchronizedList(new java.util.ArrayList()); - var m = method("handleTaskFailure", GroupConversation.class, TaskItem.class, GroupMember.class, - String.class, int.class, DiscussionPhase.class, GroupDiscussionEventListener.class, + // handleTaskFailure was split so the SSE listener callback is not made while + // holding the task-list monitor; recordTaskFailure is the document-mutating + // half these tests actually assert on. + var m = method("recordTaskFailure", GroupConversation.class, TaskItem.class, GroupMember.class, + String.class, int.class, DiscussionPhase.class, List.class, GroupDiscussionException.class); assertDoesNotThrow(() -> invoke(m, g, task, member(), "late fail", 0, phase(PhaseType.EXECUTE), - null, errors, new GroupDiscussionException("late fail"))); + errors, new GroupDiscussionException("late fail"))); assertEquals(1, errors.size(), "error is still collected even when task cannot transition"); } diff --git a/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceTaskForceTest.java b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceTaskForceTest.java index 8533f2448..c208d5594 100644 --- a/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceTaskForceTest.java +++ b/src/test/java/ai/labs/eddi/engine/internal/GroupConversationServiceTaskForceTest.java @@ -311,11 +311,13 @@ class HandleTaskFailureTests { @BeforeEach void setUp() throws Exception { + // handleTaskFailure was split so the SSE listener callback is not made + // while holding the task-list monitor; recordTaskFailure is the + // document-mutating half these tests assert on. handleMethod = GroupConversationService.class.getDeclaredMethod( - "handleTaskFailure", + "recordTaskFailure", GroupConversation.class, TaskItem.class, GroupMember.class, String.class, int.class, DiscussionPhase.class, - GroupDiscussionEventListener.class, List.class, GroupDiscussionException.class); handleMethod.setAccessible(true); } @@ -341,7 +343,7 @@ void basicFailure_marksAndRecords() throws Exception { var errors = new ArrayList(); var ex = new GroupDiscussionException("LLM timeout"); - handleMethod.invoke(service, gc, task, member, "LLM timeout", 1, phase, null, errors, ex); + handleMethod.invoke(service, gc, task, member, "LLM timeout", 1, phase, errors, ex); // Task should be FAILED assertEquals(TaskStatus.FAILED, gc.getTaskList().findById(task.id()).status()); @@ -376,7 +378,7 @@ void alreadyTerminalTask_doesNotCrash() throws Exception { var ex = new GroupDiscussionException("Second failure"); // Should NOT throw even though task is already FAILED - assertDoesNotThrow(() -> handleMethod.invoke(service, gc, failedTask, member, "Second failure", 1, phase, null, errors, ex)); + assertDoesNotThrow(() -> handleMethod.invoke(service, gc, failedTask, member, "Second failure", 1, phase, errors, ex)); // Error still collected assertEquals(1, errors.size()); diff --git a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java index 1705890b0..4192d52c5 100644 --- a/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java +++ b/src/test/java/ai/labs/eddi/engine/runtime/internal/DreamServiceTest.java @@ -804,6 +804,29 @@ void isTransientLlmFailure_distinguishesProviderBlipsFromConfigurationFaults() { assertFalse(DreamService.isTransientLlmFailure(new SelfCausedException("bad request"))); } + /** + * A hostname that does not resolve is a WRONG ENDPOINT — the very example the + * classifier's own javadoc gives for "permanent". Classifying it as transient + * made every cycle skip its groups and report SUCCESS indefinitely: the + * schedule never retried, never dead-lettered, and the operator never learned + * the endpoint was misconfigured. + *

      + * ConnectException is the contrast that makes the distinction meaningful: the + * host resolved and refused, which is what a restarting service looks like. + */ + @Test + void isTransientLlmFailure_treatsAnUnresolvableHostAsPermanent() { + assertFalse(DreamService.isTransientLlmFailure(new java.net.UnknownHostException("api.wrong-endpoint.invalid")), + "an unresolvable host is a misconfiguration; reporting success forever hides it"); + assertFalse(DreamService.isTransientLlmFailure( + new RuntimeException("llm call failed", new java.net.UnknownHostException("api.wrong-endpoint.invalid"))), + "also when wrapped — the classifier walks the cause chain"); + + // Still transient: resolved but refused, i.e. a service that may come back. + assertTrue(DreamService.isTransientLlmFailure( + new RuntimeException("llm call failed", new java.net.ConnectException("connection refused")))); + } + /** Exception whose cause is itself — guards the cause-chain walk. */ private static final class SelfCausedException extends RuntimeException { SelfCausedException(String message) { From 31120062f6ae8cb725a566bba1ce63262824697a Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Thu, 30 Jul 2026 01:09:24 +0200 Subject: [PATCH 10/11] docs(group): record why the parallel-phase transcript appends need no extra lock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit round 3, finding 1 of 2 — declined, with the reasoning recorded at the site so it stops cycling. The claim: the snapshot locks gc.getTranscript() but the success/timeout/ cancellation/error appends do not, so a delayed worker could snapshot mid-write and hit ConcurrentModificationException. Verified instead of complied with: GroupConversation guarantees the transcript is always a Collections.synchronizedList — the field initializer wraps it and setTranscript re-wraps, with no path assigning a bare list. That wrapper's mutex IS the wrapper object, which is precisely what the snapshot block locks. So add() and the snapshot already exclude one another; the explicit monitor is needed only because List.copyOf ITERATES, which the wrapper cannot make atomic. Wrapping every append would add lock scope in a hot path without removing a race. Worth being explicit that this is the OPPOSITE conclusion from the taskList guard fixed in the previous commit, where a near-identical-looking asymmetry was a real bug. The difference is what the two sides are: there, a cancellation read and a document write ordered only by the monitor; here, two operations on a single synchronized collection. Recorded in the comment so the next reviewer gets the distinction rather than the pattern match. The second finding (a late conversationService.say callback mutating gc after the abort sweep) is real and heavy-lift; tracked separately rather than redesigned as round-7 feedback on an approved PR. --- .../engine/internal/GroupConversationService.java | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java index fa83b4e4e..943bf8d8a 100644 --- a/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java +++ b/src/main/java/ai/labs/eddi/engine/internal/GroupConversationService.java @@ -2703,6 +2703,21 @@ private void executeParallelPhase(GroupConversation gc, AgentGroupConfiguration // SAFETY: Snapshot the transcript so parallel tasks each see a consistent view. // Iterating a Collections.synchronizedList requires holding its monitor. + // + // The bare gc.getTranscript().add(...) calls further down this method are NOT + // an oversight, and reviewers have asked about the asymmetry: GroupConversation + // guarantees the transcript is always a Collections.synchronizedList (both the + // field initializer and setTranscript wrap it — no path assigns a bare list), + // and that wrapper's mutex IS the wrapper object, i.e. exactly what this block + // locks. So add() and this snapshot already exclude one another; the explicit + // monitor is required only because List.copyOf ITERATES, which the wrapper + // cannot make atomic on its own. Wrapping every append would add lock scope + // without removing a race. + // + // This is deliberately the opposite conclusion from the taskList guard in the + // task-execution wave, where the asymmetry WAS a real bug: there the two sides + // were a cancellation read and a document write ordered only by the monitor, + // not two operations on one synchronized collection. List snapshotTranscript; synchronized (gc.getTranscript()) { snapshotTranscript = List.copyOf(gc.getTranscript()); From 8c533232e5f610586da5b027e914c28eb10655b0 Mon Sep 17 00:00:00 2001 From: Gregor Jarisch Date: Thu, 30 Jul 2026 01:27:13 +0200 Subject: [PATCH 11/11] fix(security): sanitize the log sites this PR introduced (CWE-117) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Copilot flagged the new skipped-workflow-step WARN, which logged documentDescriptor.getResource() and extensionType raw. Both come from stored workflow configuration, so this is the same log-forging class fixed eight times earlier in this stack — this time in code added by the review-fix pass itself. Rather than wait for a bot to find the rest one at a time, I swept the PR's own diff with a statement-aware scan (the first grep over-reported, because multi-line LOGGER calls put their arguments on continuation lines, and because counts/costs/enum names are not forging vectors). The scan separated inherited from introduced, which is what decided the scope: 79 of the flagged statements exist verbatim on main already, and 10 were introduced or reworded by this PR — all of them in DreamService. Only the latter belong in this PR. DreamService's LOGGER statements now route userId, agentId, group.getKey() and the rejection reason through LogSanitizer (19 statements in that file, being all of them rather than only the 10 newly added — the file is already substantially rewritten here, and leaving half its logs unsanitized would just invite the same finding next round). Counts, costs and durations are untouched; only arguments were rewritten, never the format strings. The 79 pre-existing sites in other files are NOT touched here. They are a real codebase-wide gap, but sanitizing them would balloon an already-approved 52-file PR at round seven of review, and they are tracked separately. 122 tests pass across the Dream and schedule surfaces. --- .../workflows/WorkflowStoreClientLibrary.java | 4 +- .../engine/runtime/internal/DreamService.java | 44 +++++++++++-------- 2 files changed, 28 insertions(+), 20 deletions(-) diff --git a/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java b/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java index 3d4e6c6ce..117314acd 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/client/workflows/WorkflowStoreClientLibrary.java @@ -17,6 +17,7 @@ import ai.labs.eddi.engine.runtime.service.IWorkflowStoreService; import ai.labs.eddi.engine.runtime.service.ServiceException; import ai.labs.eddi.configs.descriptors.model.DocumentDescriptor; +import ai.labs.eddi.utils.LogSanitizer; import ai.labs.eddi.utils.RestUtilities; import org.jboss.logging.Logger; @@ -92,7 +93,8 @@ private IExecutableWorkflow createExecutableWorkflow(final DocumentDescriptor do // disables a pipeline step, so say so instead of no-opping quietly. LOGGER.warnf("Workflow '%s' declares step '%s', which does not use the '%s' URI scheme — " + "the step is skipped and will not run.", - documentDescriptor.getResource(), extensionType, URI_SCHEME_ID); + LogSanitizer.sanitize(String.valueOf(documentDescriptor.getResource())), + LogSanitizer.sanitize(String.valueOf(extensionType)), URI_SCHEME_ID); continue; } diff --git a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java index 3e883ab21..3693aa5e8 100644 --- a/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java +++ b/src/main/java/ai/labs/eddi/engine/runtime/internal/DreamService.java @@ -10,6 +10,7 @@ import ai.labs.eddi.configs.properties.model.Property.Visibility; import ai.labs.eddi.configs.properties.model.UserMemoryEntry; import ai.labs.eddi.modules.llm.impl.SummarizationService; +import ai.labs.eddi.utils.LogSanitizer; import com.fasterxml.jackson.core.io.JsonStringEncoder; import com.fasterxml.jackson.core.type.TypeReference; import com.fasterxml.jackson.databind.ObjectMapper; @@ -197,7 +198,8 @@ public DreamResult processScheduledFire(String agentId, Integer agentVersion, St } agentConfiguration = agentStore.read(agentId, version); } catch (Exception e) { - LOGGER.errorf(e, "[DREAM] Could not read agent '%s' (version=%s) for a scheduled dream cycle", agentId, agentVersion); + LOGGER.errorf(e, "[DREAM] Could not read agent '%s' (version=%s) for a scheduled dream cycle", LogSanitizer.sanitize(agentId), + agentVersion); return rejected(userId, start, "Could not read agent '" + agentId + "': " + describe(e)); } @@ -216,7 +218,7 @@ public DreamResult processScheduledFire(String agentId, Integer agentVersion, St } private DreamResult rejected(String userId, Instant start, String reason) { - LOGGER.errorf("[DREAM] %s", reason); + LOGGER.errorf("[DREAM] %s", LogSanitizer.sanitize(reason)); cyclesFailedCounter.increment(); return new DreamResult(userId, 0, 0, 0, Duration.between(start, Instant.now()).toMillis(), 0.0, reason); } @@ -255,7 +257,7 @@ private static List scopeToOwningAgent(List en if (foreign > 0) { LOGGER.infof("[DREAM] Skipping %d of %d memory entries not owned by agent '%s' — set " + "userMemoryConfig.dream.crossAgentMaintenance=true if this agent is meant to maintain " - + "the user's memories across agents.", foreign, entries.size(), agentId); + + "the user's memories across agents.", foreign, entries.size(), LogSanitizer.sanitize(agentId)); } return owned; } @@ -284,7 +286,7 @@ public DreamResult process(String userId, String agentId, AgentConfiguration.Dre String summarizationError = null; try { - LOGGER.infof("[DREAM] Starting dream cycle for user='%s', agent='%s'", userId, agentId); + LOGGER.infof("[DREAM] Starting dream cycle for user='%s', agent='%s'", LogSanitizer.sanitize(userId), LogSanitizer.sanitize(agentId)); // Load entries once — shared across pruning and contradiction detection List allEntries = scopeToOwningAgent(userMemoryStore.getAllEntries(userId), agentId, dreamConfig); @@ -320,18 +322,20 @@ public DreamResult process(String userId, String agentId, AgentConfiguration.Dre if (summarizationError != null) { cyclesFailedCounter.increment(); LOGGER.errorf("[DREAM] Completed WITH ERRORS for user='%s': pruned=%d, contradictions=%d, summarized=%d, " - + "estimatedCost=$%.4f, duration=%dms, error=%s", userId, pruned, contradictions, summarized, estimatedCost, + + "estimatedCost=$%.4f, duration=%dms, error=%s", LogSanitizer.sanitize(userId), pruned, contradictions, summarized, + estimatedCost, duration.toMillis(), summarizationError); } else { LOGGER.infof("[DREAM] Completed for user='%s': pruned=%d, contradictions=%d, summarized=%d, " - + "estimatedCost=$%.4f, duration=%dms", userId, pruned, contradictions, summarized, estimatedCost, duration.toMillis()); + + "estimatedCost=$%.4f, duration=%dms", LogSanitizer.sanitize(userId), pruned, contradictions, summarized, estimatedCost, + duration.toMillis()); } return new DreamResult(userId, pruned, contradictions, summarized, duration.toMillis(), estimatedCost, summarizationError); } catch (Exception e) { cyclesFailedCounter.increment(); - LOGGER.errorf(e, "[DREAM] Failed for user='%s'", userId); + LOGGER.errorf(e, "[DREAM] Failed for user='%s'", LogSanitizer.sanitize(userId)); return new DreamResult(userId, pruned, contradictions, summarized, Duration.between(start, Instant.now()).toMillis(), estimatedCost, describe(e)); } @@ -352,13 +356,13 @@ private int pruneStaleEntries(String userId, List allEntries, i pruned++; entriesPrunedCounter.increment(); } catch (Exception e) { - LOGGER.warnf("[DREAM] Failed to prune entry '%s' for user '%s': %s", entry.key(), userId, e.getMessage()); + LOGGER.warnf("[DREAM] Failed to prune entry '%s' for user '%s': %s", entry.key(), LogSanitizer.sanitize(userId), e.getMessage()); } } } if (pruned > 0) { - LOGGER.infof("[DREAM] Pruned %d stale entries (>%d days) for user='%s'", pruned, staleAfterDays, userId); + LOGGER.infof("[DREAM] Pruned %d stale entries (>%d days) for user='%s'", pruned, staleAfterDays, LogSanitizer.sanitize(userId)); } return pruned; @@ -379,7 +383,8 @@ private int detectContradictions(String userId, List allEntries if (!Objects.equals(existing.value(), entry.value())) { contradictions++; contradictionsFoundCounter.increment(); - LOGGER.infof("[DREAM] Contradiction found for user='%s', key='%s': '%s' vs '%s'", userId, entry.key(), existing.value(), + LOGGER.infof("[DREAM] Contradiction found for user='%s', key='%s': '%s' vs '%s'", LogSanitizer.sanitize(userId), entry.key(), + existing.value(), entry.value()); } } @@ -450,7 +455,7 @@ private SummarizationOutcome summarizeInteractions(String userId, // the primary ceiling, because a call count says nothing about spend. if (estimatedCostAccumulated >= config.getMaxCostPerRun()) { LOGGER.infof("[DREAM] Cost ceiling ($%.4f >= $%.2f) reached for user='%s' " + - "after %d calls", estimatedCostAccumulated, config.getMaxCostPerRun(), userId, llmCallsMade); + "after %d calls", estimatedCostAccumulated, config.getMaxCostPerRun(), LogSanitizer.sanitize(userId), llmCallsMade); break; } @@ -463,7 +468,7 @@ private SummarizationOutcome summarizeInteractions(String userId, LOGGER.warnf("[DREAM] Legacy call ceiling maxSummarizationCalls=%d reached for user='%s' " + "after $%.4f of an allowed $%.2f. This field is deprecated — configure maxCostPerRun " + "instead, which bounds actual spend.", - config.getMaxSummarizationCalls(), userId, estimatedCostAccumulated, config.getMaxCostPerRun()); + config.getMaxSummarizationCalls(), LogSanitizer.sanitize(userId), estimatedCostAccumulated, config.getMaxCostPerRun()); break; } @@ -502,7 +507,8 @@ private SummarizationOutcome summarizeInteractions(String userId, + "consolidation is ABORTED for this cycle. If this is an authentication or endpoint error, set the " + "credentials on the agent under userMemoryConfig.dream.parameters (e.g. \"apiKey\": \"${vault:my-key}\") " + "— a background dream cycle has no parent LLM task to inherit them from.", - userId, group.getKey(), config.getLlmProvider(), config.getLlmModel(), parameterKeys(config)); + LogSanitizer.sanitize(userId), LogSanitizer.sanitize(group.getKey()), config.getLlmProvider(), config.getLlmModel(), + parameterKeys(config)); return new SummarizationOutcome(totalConsolidated, estimatedCostAccumulated, "Memory consolidation LLM call failed (" + config.getLlmProvider() + "/" + config.getLlmModel() + "): " + e.getMessage()); @@ -515,14 +521,14 @@ private SummarizationOutcome summarizeInteractions(String userId, if (consolidated.isEmpty()) { LOGGER.warnf("[DREAM] Summarization returned empty/invalid result for " + - "user='%s', group='%s'. Preserving original entries.", userId, group.getKey()); + "user='%s', group='%s'. Preserving original entries.", LogSanitizer.sanitize(userId), LogSanitizer.sanitize(group.getKey())); continue; } // 5. Validate: consolidated must be fewer than originals if (consolidated.size() >= groupEntries.size()) { LOGGER.warnf("[DREAM] LLM returned %d entries (>= %d originals). " + - "Skipping group '%s'.", consolidated.size(), groupEntries.size(), group.getKey()); + "Skipping group '%s'.", consolidated.size(), groupEntries.size(), LogSanitizer.sanitize(group.getKey())); continue; } @@ -554,7 +560,7 @@ private SummarizationOutcome summarizeInteractions(String userId, if (distinctAgents.size() > 1 && mergedVisibility == Visibility.self) { LOGGER.errorf("[DREAM] Refusing to merge self-scoped entries from %d agents for user='%s', " + "group='%s' — that would expose one agent's private memories to the others.", - distinctAgents.size(), userId, group.getKey()); + distinctAgents.size(), LogSanitizer.sanitize(userId), LogSanitizer.sanitize(group.getKey())); continue; } Instant earliestCreated = groupEntries.stream() @@ -581,7 +587,7 @@ private SummarizationOutcome summarizeInteractions(String userId, } catch (Exception e) { LOGGER.warnf("[DREAM] Failed to insert consolidated entries for " + "user='%s', group='%s': %s. Originals preserved, rolling back %d inserts.", - userId, group.getKey(), e.getMessage(), insertedIds.size()); + LogSanitizer.sanitize(userId), LogSanitizer.sanitize(group.getKey()), e.getMessage(), insertedIds.size()); // Rollback: delete any partially-inserted consolidated entries for (String insertedId : insertedIds) { try { @@ -617,7 +623,7 @@ private SummarizationOutcome summarizeInteractions(String userId, if (transientFailures > 0) { LOGGER.warnf("[DREAM] %d of %d groups were skipped for user='%s' after transient LLM failures — " - + "they are retried on the next dream cycle.", transientFailures, groups.size(), userId); + + "they are retried on the next dream cycle.", transientFailures, groups.size(), LogSanitizer.sanitize(userId)); } return new SummarizationOutcome(totalConsolidated, estimatedCostAccumulated, null); @@ -774,7 +780,7 @@ private static Map> splitSelfScopedGroupsByAgent( } LOGGER.infof("[DREAM] Group '%s' holds self-scoped memories from %d agents — consolidating each agent " - + "separately so no private memory is widened.", group.getKey(), byAgent.size()); + + "separately so no private memory is widened.", LogSanitizer.sanitize(group.getKey()), byAgent.size()); byAgent.forEach((agentId, agentEntries) -> result.put(group.getKey() + ":" + agentId, agentEntries)); } return result;