Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,56 @@
> **Purpose:** Living document tracking all changes, decisions, and reasoning during implementation. Updated as work progresses for easy reference and review.


---

## 🧹 Refactor: Remove duplicate `RetryConfiguration` shim in LlmConfiguration (2026-07-16)

**Repo:** EDDI (`feat/error-handling-recovery`)

### Summary

Follow-up to the error-handling PR (#593): resolved a code-quality finding (nested class with the same simple name as its superclass). The `LlmConfiguration.RetryConfiguration` nested class was an empty subclass of the extracted `ai.labs.eddi.configs.shared.RetryConfiguration`, kept as a backward-compat shim. It overrode nothing and shadowed the imported shared type within `LlmConfiguration`'s body.

### Key Changes

- **`LlmConfiguration.java`**: Deleted the empty nested `RetryConfiguration` subclass. The `retry` field, getter, and setter now bind directly to the imported shared `RetryConfiguration` (import already present).
- **9 test files**: Replaced `new LlmConfiguration.RetryConfiguration()` with the shared `RetryConfiguration` (added the `configs.shared` import). Includes `LlmConfigurationTest`, which had pulled the nested type in via a `LlmConfiguration.*` wildcard import.

### Design Decisions

- **Deleted the shim rather than renaming it** (a reviewer suggested `LegacyRetryConfiguration`). The subclass added zero fields/overrides, so a rename would keep a misleadingly-named dead class for the same test churn. Per project philosophy, internal-API removal is safe — the only backward-compat concern is stored JSON, and because the removed subclass added no fields, the `retry` JSON structure is byte-for-byte identical (existing MongoDB/ZIP configs deserialize unchanged).

### Verification

- `mvnw clean test-compile` clean; 123 affected unit tests pass (0 failures).
- Repo-wide grep confirms zero remaining references to the nested type (dotted, JVM binary-name, reflection strings, or wildcard imports).

---

## ⚡ Feat: Holistic Error Handling and Recovery Infrastructure (2026-07-07)

**Repo:** EDDI (`feat/error-handling-recovery`)

### Summary

Complete overhaul of error handling across LLM, HTTP, and MCP call subsystems plus cross-cutting infrastructure for admin visibility, recovery, and monitoring. 19 source files changed, 7 test files added (83+ new tests). Code-reviewed and all review findings addressed before commit.

### Key Changes

- **Shared RetryConfiguration**: Extracted reusable retry logic with exponential backoff, retryable error classification, configurable per subsystem.
- **LifecycleManager**: Error classification, failure audit entries, SSE `task_failed` events, Micrometer counters tagged by `error.type`.
- **Admin state reset**: `PATCH /{conversationId}/state` endpoint to recover stuck conversations.
- **HTTP error body storage**: 4xx/5xx response bodies stored in memory; JSON parse softened.
- **MCP continueOnError + retry + circuit breaker**: Config-driven error resilience per MCP call.
- **LLM ResponseValidation**: Config-driven policies for empty/truncated/filtered responses.
- **Streaming retry**: Zero-token failures retried; partial responses returned with metadata.

### Design Decisions

- Retry at call site (not pipeline level) per user directive.
- Strict-write default kept as `false` (opt-in) to avoid breaking existing agents.
- No `"retry"` validation action — retry handled by RetryConfiguration at call level.

---

## 🧹 Multimodal Attachments Completion — Remove dead config knob `reattachTurns` (2026-07-13)
Expand Down
33 changes: 33 additions & 0 deletions src/main/java/ai/labs/eddi/configs/mcpcalls/model/McpCall.java
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

import ai.labs.eddi.configs.apicalls.model.PostResponse;
import ai.labs.eddi.configs.apicalls.model.PreRequest;
import ai.labs.eddi.configs.shared.RetryConfiguration;

import java.util.List;
import java.util.Map;
Expand Down Expand Up @@ -68,6 +69,22 @@ public class McpCall {
*/
private PostResponse postResponse;

/**
* When {@code true}, a failed MCP call stores the error in memory but does not
* abort the pipeline. Downstream tasks and post-response processing can inspect
* the error via the {@code <responseObjectName>Error} memory key. Defaults to
* {@code false} (fail-fast).
*/
private Boolean continueOnError = false;

/**
* Optional retry configuration for this MCP call. When set, transient failures
* (timeouts, rate limits, connection errors) are retried with exponential
* backoff before the call is considered failed. Uses the shared
* {@link RetryConfiguration} model.
*/
private RetryConfiguration retry;

// --- Getters and Setters ---

public String getName() {
Expand Down Expand Up @@ -141,4 +158,20 @@ public PostResponse getPostResponse() {
public void setPostResponse(PostResponse postResponse) {
this.postResponse = postResponse;
}

public Boolean getContinueOnError() {
return continueOnError;
}

public void setContinueOnError(Boolean continueOnError) {
this.continueOnError = continueOnError;
}

public RetryConfiguration getRetry() {
return retry;
}

public void setRetry(RetryConfiguration retry) {
this.retry = retry;
}
}
215 changes: 215 additions & 0 deletions src/main/java/ai/labs/eddi/configs/shared/RetryConfiguration.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
/*
* Copyright EDDI contributors
* SPDX-License-Identifier: Apache-2.0
*/
package ai.labs.eddi.configs.shared;

import ai.labs.eddi.engine.lifecycle.exceptions.LifecycleException;
import org.jboss.logging.Logger;

import java.util.concurrent.Callable;

/**
* Shared retry configuration and execution utility.
* <p>
* Used by all subsystems (LLM, MCP, etc.) to define per-call retry policies and
* execute actions with exponential backoff.
*
* @since 6.0.0
*/
public class RetryConfiguration {
private static final Logger LOGGER = Logger.getLogger(RetryConfiguration.class);

private Integer maxAttempts = 3;
private Long backoffDelayMs = 1000L;
private Double backoffMultiplier = 2.0;
private Long maxBackoffDelayMs = 10000L;

public Integer getMaxAttempts() {
return maxAttempts;
}

public void setMaxAttempts(Integer maxAttempts) {
this.maxAttempts = maxAttempts;
}

public Long getBackoffDelayMs() {
return backoffDelayMs;
}

public void setBackoffDelayMs(Long backoffDelayMs) {
this.backoffDelayMs = backoffDelayMs;
}

public Double getBackoffMultiplier() {
return backoffMultiplier;
}

public void setBackoffMultiplier(Double backoffMultiplier) {
this.backoffMultiplier = backoffMultiplier;
}

public Long getMaxBackoffDelayMs() {
return maxBackoffDelayMs;
}

public void setMaxBackoffDelayMs(Long maxBackoffDelayMs) {
this.maxBackoffDelayMs = maxBackoffDelayMs;
}

// ========================== Static Retry Utility ==========================

/**
* Executes a generic action with retry logic based on configuration.
* <p>
* Uses exponential backoff: {@code delay * multiplier^(attempt-1)}, capped at
* {@code maxBackoffDelayMs}. Retryable errors are identified by walking the
* exception cause chain for known transient error types (timeout, connection,
* rate limit, HTTP 429/502/503/504).
*
* @param action
* the action to execute
* @param retryConfig
* retry settings (null = defaults: 3 attempts, 1s backoff, 2.0x
* multiplier, 10s cap)
* @param actionDescription
* human-readable description for logging
* @param <T>
* return type
* @return the action's result on success
* @throws LifecycleException
* if all attempts fail or a non-retryable error occurs
*/
public static <T> T executeWithRetry(Callable<T> action, RetryConfiguration retryConfig,
String actionDescription)
throws LifecycleException {

if (retryConfig == null) {
retryConfig = new RetryConfiguration();
}

int maxAttempts = retryConfig.getMaxAttempts() != null ? retryConfig.getMaxAttempts() : 3;
long backoffDelay = retryConfig.getBackoffDelayMs() != null ? retryConfig.getBackoffDelayMs() : 1000L;
double backoffMultiplier = retryConfig.getBackoffMultiplier() != null ? retryConfig.getBackoffMultiplier() : 2.0;
long maxBackoffDelay = retryConfig.getMaxBackoffDelayMs() != null ? retryConfig.getMaxBackoffDelayMs() : 10000L;

int attempt = 0;
long currentBackoff = backoffDelay;
Exception lastException = null;

while (attempt < maxAttempts) {
attempt++;

try {
LOGGER.debug(actionDescription + " attempt " + attempt + "/" + maxAttempts);

T result = action.call();

LOGGER.info(actionDescription + " succeeded on attempt " + attempt);
return result;

} catch (Exception e) {
// HITL tool pause: a gated tool call must abort the retry loop and
// travel up UNCHANGED — it is not a retryable failure. Rethrow before
// any retry/backoff or LifecycleException wrapping so the pause signal
// reaches LifecycleManager intact. Applies to LLM and MCP callers.
if (e instanceof ai.labs.eddi.engine.hitl.tools.ToolApprovalRequiredException tare) {
throw tare;
}
lastException = e;

if (attempt < maxAttempts) {
if (isRetryableError(e)) {
LOGGER.warn(actionDescription + " failed (attempt " + attempt + "/" + maxAttempts
+ "), retrying after " + currentBackoff + "ms: " + e.getMessage());

try {
Thread.sleep(currentBackoff);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new LifecycleException("Retry interrupted", ie);
}

currentBackoff = Math.min((long) (currentBackoff * backoffMultiplier), maxBackoffDelay);
} else {
LOGGER.error(actionDescription + " failed with non-retryable error: " + e.getMessage());
throw new LifecycleException(actionDescription + " failed: " + e.getMessage(), e);
}
} else {
LOGGER.error(actionDescription + " failed after " + maxAttempts + " attempts");
}
}
}

throw new LifecycleException(actionDescription + " failed after " + maxAttempts + " attempts", lastException);
}

/**
* Sleeps for the appropriate backoff duration for the given attempt. Useful for
* callers that manage their own retry loop (e.g., streaming).
*/
public static void backoff(int attempt, RetryConfiguration retryConfig) {
if (retryConfig == null) {
retryConfig = new RetryConfiguration();
}
long baseDelay = retryConfig.getBackoffDelayMs() != null ? retryConfig.getBackoffDelayMs() : 1000L;
double multiplier = retryConfig.getBackoffMultiplier() != null ? retryConfig.getBackoffMultiplier() : 2.0;
long maxDelay = retryConfig.getMaxBackoffDelayMs() != null ? retryConfig.getMaxBackoffDelayMs() : 10000L;

int exponent = Math.max(0, attempt - 1);
long delay = Math.min((long) (baseDelay * Math.pow(multiplier, exponent)), maxDelay);
try {
Thread.sleep(delay);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}

// ========================== Retryable Error Detection
// ==========================

/**
* Determines if an error is retryable by checking:
* <ol>
* <li>Known exception types (timeout, connection)</li>
* <li>HTTP status codes from HttpException or WebApplicationException (429,
* 502, 503, 504)</li>
* <li>Message string patterns (fallback for wrapped/untyped exceptions)</li>
* </ol>
*/
public static boolean isRetryableError(Exception e) {
Throwable current = e;

while (current != null) {
// 1. Typed exception matching
if (current instanceof java.net.SocketTimeoutException
|| current instanceof java.util.concurrent.TimeoutException
|| current instanceof java.net.ConnectException
|| current instanceof java.net.UnknownHostException) {
return true;
}

// 2. HTTP status code matching from typed exceptions
if (current instanceof jakarta.ws.rs.WebApplicationException wae) {
int status = wae.getResponse().getStatus();
if (status == 429 || status == 502 || status == 503 || status == 504) {
return true;
}
}

// 3. String-based fallback for wrapped exceptions (conservative matching)
String message = current.getMessage() != null ? current.getMessage().toLowerCase() : "";

if (message.contains("timeout") || message.contains("rate limit") || message.contains("too many requests")
|| message.contains("connection refused") || message.contains("connection reset")
|| message.contains("service unavailable") || message.contains("bad gateway")
|| message.contains("gateway timeout")) {
return true;
}

current = current.getCause();
}

return false;
}
}
11 changes: 11 additions & 0 deletions src/main/java/ai/labs/eddi/engine/api/IConversationService.java
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,13 @@ ConversationResult startConversation(Environment environment, String agentId, St
*/
void endConversation(String conversationId);

/**
* Reset the state of a stuck conversation (ERROR or EXECUTION_INTERRUPTED) to
* READY. Updates both persistent storage and the in-memory cache atomically.
* Admin-only operation.
*/
void resetConversationState(String conversationId, ConversationState targetState);

/**
* End a conversation with actor attribution. When the conversation was
* {@code AWAITING_HUMAN}, ending it terminally resolves the pending approval:
Expand Down Expand Up @@ -153,6 +160,10 @@ interface StreamingResponseHandler {

void onError(Throwable error);

default void onTaskFailed(TaskId taskId, String taskType, long durationMs,
String errorType, String errorSummary) {
}

/**
* The turn was dropped without consuming the input (see
* {@link ConversationResponseHandler#onSkipped}). Defaults to
Expand Down
17 changes: 16 additions & 1 deletion src/main/java/ai/labs/eddi/engine/api/IRestAgentEngine.java
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,20 @@ void sayWithinContext(@PathParam("conversationId") String conversationId,
@APIResponse(responseCode = "200", description = "Redo successful.")
Response redo(@PathParam("conversationId") String conversationId);

// --- Admin: state reset ---

@PATCH
@Path("/{conversationId}/state")
@Consumes(MediaType.APPLICATION_JSON)
@RolesAllowed("eddi-admin")
@Operation(summary = "Reset conversation state",
description = "Resets a stuck conversation (ERROR or EXECUTION_INTERRUPTED) to READY. Admin-only.")
@APIResponse(responseCode = "200", description = "State reset successful.")
@APIResponse(responseCode = "409", description = "Cannot reset from current state.")
Response resetState(@PathParam("conversationId") String conversationId,
@QueryParam("state")
@DefaultValue("READY") String targetState);

// --- Cancel ---

@POST
Expand Down Expand Up @@ -229,5 +243,6 @@ Response getApprovalStatus(@PathParam("conversationId") String conversationId,
@Operation(summary = "List pending approvals",
description = "Lists conversations currently awaiting human approval (bounded by limit, max 1000).")
@APIResponse(responseCode = "200", description = "List of pending approvals.")
List<PendingApprovalSummary> listPendingApprovals(@QueryParam("limit") @DefaultValue("200") Integer limit);
List<PendingApprovalSummary> listPendingApprovals(@QueryParam("limit")
@DefaultValue("200") Integer limit);
}
Loading
Loading