Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,36 @@
> **Purpose:** Living document tracking all changes, decisions, and reasoning during implementation. Updated as work progresses for easy reference and review.


---

## 🔧 fix(agents): null-version undeploy no-op, deploy under a CHM bin lock, EXECUTE wave deadline, protocol defaults (2026-08-09)

**Repo:** EDDI (`fix/agent-lifecycle-and-group-deadlines`)

Wave A of a deep review of the Agent / Group Agent surface. The review's finding was that the *group feature layer* has been reviewed exhaustively while the *agent lifecycle layer it stands on* has not — four of the five fixes here come from `AgentFactory`, two of them from a single six-line method.

1. **`undeployAgent(env, agentId, null)` was a silent no-op — ephemeral agents leaked.** `AgentFactory.AgentId` keys on `(id, version)`, so `new AgentId(id, null)` equals no key the environment map ever holds. Both dynamic-agent teardown paths pass null (`GroupLifecycleOps#cleanupEphemeralAgents` after *every* group discussion, and `TeardownAgentTool`), so `agentEnvironment.remove(...)` and `deployedAgents.remove(...)` both did nothing while the caller logged `"undeployed agent '%s'"` at INFO. `agentStore.deleteAllPermanently(agentId)` then ran anyway: the constructed agent stayed resolvable through `getLatestReadyAgent` **after its configuration had been deleted from the store**, and `eddi_agents_deployed` grew monotonically for the lifetime of the process. A null version now means *every* deployed version of that agent — the only reading that matches what the teardown callers mean. The REST/admin path passes a real version and stays exact. Pinned by `AgentFactoryUndeployVersionTest` (7 tests; 4 fail against the old behaviour — verified by mutation).
2. **`deployAgent` ran store I/O and full workflow construction inside `ConcurrentHashMap.compute`.** `compute` holds the bin lock for the key while the mapping function runs, and `ConcurrentHashMap` documents that the function must be short and must not touch other mappings; this held a bin lock across multi-second I/O, and any re-entrant agent resolution during construction would have deadlocked. The claim is now a `putIfAbsent` of an IN_PROGRESS placeholder (atomic, no long hold) with the load outside the map. Side benefit: the placeholder is now actually *published*, which `compute` never did — the dummy was only ever returned on the failure path — so a concurrent `getAgent()` can observe "deployment in progress" instead of a bare null. An ERROR entry is re-claimed with `replace(key, expected, placeholder)` so two racing redeploys cannot both proceed.
3. **`deployedAgents` was mutated unsynchronized.** `deployAgent` appended under `synchronized (deployedAgents)` while `undeployAgent` removed with no lock at all, and the Micrometer gauge reads `size()` from the metrics thread — three unordered accesses to a `LinkedList`. Now a `CopyOnWriteArrayList`, with `addIfAbsent` replacing a non-atomic `contains()`-then-`add()`.
4. **The TASK_FORCE EXECUTE wave gave up before the turns it was waiting for could have.** The wave waited `agentTimeoutSeconds × maxTasksPerAgent`, which ignores retries — under `onAgentFailure=RETRY` a member legitimately gets `timeout × (maxRetries + 1)` — and carries no setup grace, so even a one-task no-retry wave could expire while the member was still inside its own budget (a member turn reaches its response wait only after agent lookup, conversation start and attachment grants). That is exactly what `PARALLEL_BATCH_GRACE_FLOOR_SECONDS` exists to prevent, and both the parallel debate batch and the bid round *in the same file* already sized themselves through `parallelBatchBudgetSeconds`. The wave now does too, via a new extracted `TaskForceEngine#waveBudgetSeconds` so the derivation is assertable without timing a real wave.
5. **The documented 180s `agentTimeoutSeconds` default was unreachable on the common path.** `resolveProtocol`'s fallback handed out a literal `60`, and `McpGroupTools.create_group` hard-coded `60` — while the constant's own Javadoc, the four shipped templates and the published table in `docs/group-conversations.md` all said 180 (the value introduced *because* 60 timed out thinking models during synthesis). Since nothing backfills a `protocol` block at save time, a group saved without one — the common shape — ran at 60. The defaults now live on `ProtocolConfig` (`DEFAULT_AGENT_TIMEOUT_SECONDS`, `DEFAULT_MAX_RETRIES`) as the single source of truth, referenced by the engine, the MCP tool and the follow-up path (`resolveAgentTimeoutSeconds`, which had its own stray `return 60`).

Also: removed an unreachable version comparison in `getAllLatestAgents` (it compared the result of `getLatestAgent`, which already returns the highest version, against itself), and downgraded `waitForDeploymentCompletion`'s "still IN_PROGRESS" ERROR to DEBUG when no deployment future was registered — with the placeholder now published that state is reachable and ordinary, not a failure. Wiring the registration properly is Wave F.

**Final review round (model-independent second pass, PR #648).** Two further items:
- **`waitForDeploymentCompletion` now waits with a timed `get()` and no longer mutates the shared registration future.** It previously armed `orTimeout(60s)` on that future, and `orTimeout` mutates the future it is called on — while `DeploymentListener` hands the same instance to every waiter and to whoever registered the deployment. Pre-#651 only the ZIP importer ever held one, so the mutation was near-unreachable; with #651 registering on every REST deploy, one impatient `getAgent` caller would, at its own 60s deadline, complete the shared future exceptionally for every other consumer and evict the registration before the real deployment event arrived. The timed `get()` waits without writing. The newly-reachable `InterruptedException` restores the interrupt flag, and a timeout logs at WARN as this caller's outcome rather than ERROR as the deployment's.
- **The `eddi_agents_deployed` gauge counts READY deployments per environment, and its comment now says so.** The comment previously claimed the derived gauge "matched the old semantics", but one delta is real and deliberate: the old list keyed on (id, version) with no environment, so an agent deployed to both `production` and `test` counted once, whereas counting map entries counts it once per environment — that is, once per actual deployment. The comment and this entry now state that change directly.

**Review round 1 (PR #648).** Four findings, all fixed:
- **CodeRabbit (Major), and a genuine regression this PR introduced:** moving the store load out of `ConcurrentHashMap.compute` opened a window in which an `undeployAgent` for the same id could land mid-load, after which the unconditional `put` of the finished agent resurrected it — a deployment silently winning a race it started before the undeploy was even requested, and leaving the metric describing a registry that no longer held it. Publication is now `replace(key, OUR placeholder, agent)`, so an interleaved undeploy or competing redeploy keeps its outcome. The reviewer's further point — "coordinate the map and metric tracking together, do not only swap put for replace" — is addressed by **deleting the parallel structure entirely**: `deployedAgents` is gone and `eddi_agents_deployed` is now a `Gauge` derived from the environment maps (READY entries only, matching the old semantics). Two structures holding one fact could never be linearized against each other; one structure needs no linearization. Pinned by `undeployDuringLoadIsNotOverwritten`, which blocks the store lookup, undeploys, then releases — mutation-verified against a `put`.
- **CodeQL (4× log injection):** the three claim-path `log.debug(String.format(...))` calls and the deploy-failure `log.error` interpolated an unsanitized `agentId`. All now go through `LogSanitizer.sanitize` (and the debug calls use `debugf` rather than pre-formatting).
- **CodeRabbit (Minor):** `ProtocolConfig`'s `@param agentTimeoutSeconds` Javadoc still said "default: 60".
- **Code-quality bot:** a redundant assertion in `TaskForceEngineWaveBudgetTest` (`budget > 30` is implied by `budget >= 90`) replaced with `assertNotEquals(30L, budget)`, which keeps the "not the old formula's output" intent without the always-true comparison.

**Behaviour change, deliberate:** groups with no `protocol` block go from a 60s to a 180s per-turn timeout, and MCP-created groups likewise. This makes the code agree with the documentation rather than the reverse.

Suites: `AgentFactory*`, `AgentDeploymentManagement*`, `TaskForceEngine*`, `GroupConversationService*`, `McpGroupTools*`, `AgentGroupConfiguration*`, `RestAgentAdministration*` — 820 tests green, plus 19 new.

---

## 🔎 fix(groups): pre-merge deep review — facilitator HITL bypass, CALL_VOTE guards, metric cardinality, template honesty (2026-08-08)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -913,7 +913,9 @@ public enum ContextScope {
* failure policies, and safety caps.
*
* @param agentTimeoutSeconds
* per-agent timeout in seconds (default: 60)
* per-agent timeout in seconds. Non-positive, or no {@code protocol}
* block at all, falls back to {@link #DEFAULT_AGENT_TIMEOUT_SECONDS}
* (180)
* @param onAgentFailure
* policy when an agent fails (SKIP, RETRY, ABORT)
* @param maxRetries
Expand All @@ -940,6 +942,23 @@ public enum ContextScope {
public record ProtocolConfig(int agentTimeoutSeconds, MemberFailurePolicy onAgentFailure, int maxRetries,
MemberUnavailablePolicy onMemberUnavailable, int maxTurns, Double maxCostPerDiscussion, CostPolicy onCostExceeded) {

/**
* Per-agent-turn timeout used when a group configures no {@code protocol}
* block, and when one supplies a non-positive {@code agentTimeoutSeconds}. 180s
* covers thinking models (e.g. claude-sonnet-5) and synthesis phases
* comfortably.
* <p>
* Lives here rather than only on the engine so every writer of a default
* protocol agrees: the MCP {@code create_group} tool used to hard-code 60 while
* the engine documented 180 and the templates shipped 180, which meant the
* published default was true of exactly one of the three ways a group gets
* created.
*/
public static final int DEFAULT_AGENT_TIMEOUT_SECONDS = 180;
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

/** Retry attempts per member turn when {@code onAgentFailure=RETRY}. */
public static final int DEFAULT_MAX_RETRIES = 2;

/**
* Canonical constructor — normalizes a {@code null} {@link #onCostExceeded} to
* {@link CostPolicy#SYNTHESIZE_NOW} so every reader can treat the field as
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,16 +103,21 @@ public class GroupConversationService implements IGroupConversationService {
* {@code protocol.agentTimeoutSeconds}. 180s covers thinking models (e.g.
* claude-sonnet-5) and synthesis phases comfortably. Was 60s, which caused
* timeouts on synthesis with extended thinking.
* <p>
* Aliases {@link ProtocolConfig#DEFAULT_AGENT_TIMEOUT_SECONDS} rather than
* restating it: the value is also needed by writers of a default protocol
* outside this engine (the MCP {@code create_group} tool), and two independent
* literals is how they drifted apart in the first place.
*/
private static final int DEFAULT_AGENT_TIMEOUT_SECONDS = 180;
private static final int DEFAULT_AGENT_TIMEOUT_SECONDS = ProtocolConfig.DEFAULT_AGENT_TIMEOUT_SECONDS;

/**
* Default number of retries per member turn when not configured via
* {@code protocol.maxRetries}. Shared by the retry loop in
* {@code executeAgentTurn} and the batch budget a parallel phase derives from
* it, so the two cannot drift apart.
*/
private static final int DEFAULT_MAX_RETRIES = 2;
private static final int DEFAULT_MAX_RETRIES = ProtocolConfig.DEFAULT_MAX_RETRIES;

/**
* Slack added on top of a member's own budget when a parallel phase arms its
Expand Down Expand Up @@ -1711,16 +1716,32 @@ private void persistRuntimePhases(GroupConversation gc, List<DiscussionPhase> ph
conversationStore.update(gc);
}

/**
* The protocol a discussion runs under, or the engine defaults when the group
* configured none.
* <p>
* The fallback timeout is {@link #DEFAULT_AGENT_TIMEOUT_SECONDS}, not the
* literal 60 it used to be. A group saved without a {@code protocol} block is
* the COMMON shape — nothing backfills one at save time — so that literal made
* 180 reachable only for the odd config that supplies a protocol with a
* non-positive timeout, i.e. essentially never. Both the constant's own
* rationale (thinking models time out at 60s during synthesis) and the
* published default in {@code docs/group-conversations.md} said 180; only this
* line disagreed.
*/
private ProtocolConfig resolveProtocol(AgentGroupConfiguration config) {
return config.getProtocol() != null
? config.getProtocol()
: new ProtocolConfig(60, ProtocolConfig.MemberFailurePolicy.SKIP, 2, ProtocolConfig.MemberUnavailablePolicy.SKIP);
: new ProtocolConfig(DEFAULT_AGENT_TIMEOUT_SECONDS, ProtocolConfig.MemberFailurePolicy.SKIP, DEFAULT_MAX_RETRIES,
ProtocolConfig.MemberUnavailablePolicy.SKIP);
}

/**
* Resolve the per-agent timeout (seconds) for a follow-up turn from the group's
* protocol config, so follow-ups honor the same configurable limit as
* discussion turns. Defaults to 60 if the config cannot be loaded.
* discussion turns. Falls back to {@link #DEFAULT_AGENT_TIMEOUT_SECONDS} if the
* config cannot be loaded — a follow-up is an ordinary member turn and has no
* reason to get a tighter budget than one inside the discussion.
* <p>
* Public: called back from GroupLifecycleOps.followUpWithMember (Wave R, R1
* step 8).
Expand All @@ -1741,7 +1762,7 @@ public int resolveAgentTimeoutSeconds(GroupConversation gc) {
LOGGER.debugf("Could not resolve agent timeout for group %s, using default: %s",
LogSanitizer.sanitize(gc.getGroupId()), e.getMessage());
}
return 60;
return DEFAULT_AGENT_TIMEOUT_SECONDS;
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -327,7 +327,6 @@ public void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigurat
// not transcript context.

List<GroupDiscussionException> errors = Collections.synchronizedList(new ArrayList<>());
int timeout = protocol.agentTimeoutSeconds() > 0 ? protocol.agentTimeoutSeconds() : defaultAgentTimeoutSeconds;
int maxWaves = 100; // safety cap to prevent infinite loops
final SharedTaskList taskList = gc.getTaskList();

Expand Down Expand Up @@ -508,8 +507,23 @@ public void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigurat
futures.add(future);
}

// Wait for this wave — timeout based on max tasks per agent (H2 fix)
// Wait for this wave. The budget is per-member-turn × the longest task
// chain one agent holds, and the per-turn figure comes from
// parallelBatchBudgetSeconds — NOT from a bare agentTimeoutSeconds.
//
// A bare timeout was wrong in both directions. It ignored retries, so
// under onAgentFailure=RETRY a member legitimately consuming
// timeout × (maxRetries + 1) blew a deadline sized for one attempt and
// the wave aborted mid-flight; and it carried no setup grace, so even
// with one task and no retries the orchestrator's clock (armed at
// dispatch) could expire while the member was still inside its own
// budget, because a member turn reaches responseFuture.get() only after
// agent lookup, conversation start and attachment grants. That is the
// exact defect PARALLEL_BATCH_GRACE_FLOOR_SECONDS exists to prevent, and
// both the debate batch (PhaseExecutionEngine) and the bid round below
// already size themselves this way.
int maxTasksPerAgent = tasksByAgent.values().stream().mapToInt(List::size).max().orElse(1);
long waveBudgetSeconds = waveBudgetSeconds(protocol, maxTasksPerAgent);
try {
var allOf = CompletableFuture.allOf(futures.toArray(CompletableFuture[]::new));
// NEW-3: Register the blocking future so IMMEDIATE cancel can interrupt.
Expand All @@ -523,7 +537,7 @@ public void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigurat
allOf.cancel(true);
}
}
allOf.get(timeout * (long) maxTasksPerAgent, TimeUnit.SECONDS);
allOf.get(waveBudgetSeconds, TimeUnit.SECONDS);
} catch (TimeoutException e) {
LOGGER.warnf("Task execution timed out for group %s (wave %d)",
LogSanitizer.sanitize(gc.getGroupId()), wave + 1);
Expand Down Expand Up @@ -570,6 +584,29 @@ public void executeTaskExecutionPhase(GroupConversation gc, AgentGroupConfigurat
}
}

/**
* Wall-clock budget one EXECUTE wave gets before the orchestrator gives up on
* the agents still running.
* <p>
* Derived from {@link GroupConversationService#parallelBatchBudgetSeconds} —
* the per-member-turn figure that already accounts for retries and setup grace
* — multiplied by the longest task chain any single agent holds, since one
* agent runs its own tasks sequentially.
* <p>
* Extracted so the derivation is assertable without timing a real wave. It was
* previously an inline {@code agentTimeoutSeconds × maxTasksPerAgent}, which
* under-budgeted a RETRY wave by a factor of {@code maxRetries + 1} and carried
* no setup grace at all.
*
* @param maxTasksPerAgent
* the largest number of tasks assigned to one agent in this wave;
* values below 1 are treated as 1
*/
static long waveBudgetSeconds(ProtocolConfig protocol, int maxTasksPerAgent) {
long perTurn = GroupConversationService.parallelBatchBudgetSeconds(protocol);
return perTurn * Math.max(1, maxTasksPerAgent);
}

/**
* Atomically reserve one turn from the shared budget.
* <p>
Expand Down
9 changes: 6 additions & 3 deletions src/main/java/ai/labs/eddi/engine/mcp/McpGroupTools.java
Original file line number Diff line number Diff line change
Expand Up @@ -289,10 +289,13 @@ public String create_group(@ToolArg(description = "Group name") String name,
}
}

// Protocol with maxTurns safety cap
// Protocol with maxTurns safety cap. The timeout/retry figures come from
// ProtocolConfig's own defaults — hard-coding 60 here meant every
// MCP-created group silently ran at the value the 180s default was
// introduced to replace.
int mt = parseIntOrDefault(maxTurns, 0);
config.setProtocol(new ProtocolConfig(60, ProtocolConfig.MemberFailurePolicy.SKIP, 2,
ProtocolConfig.MemberUnavailablePolicy.SKIP, mt));
config.setProtocol(new ProtocolConfig(ProtocolConfig.DEFAULT_AGENT_TIMEOUT_SECONDS, ProtocolConfig.MemberFailurePolicy.SKIP,
ProtocolConfig.DEFAULT_MAX_RETRIES, ProtocolConfig.MemberUnavailablePolicy.SKIP, mt));

Response response = groupStore.createGroup(config);
String location = response.getLocation() != null ? response.getLocation().toString() : "";
Expand Down
Loading
Loading