Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions DESIGN_SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -915,7 +915,7 @@ hybrid:

`AgentEngine` (in `engine/agent_engine.py`) is the top-level entry point for running an agent on a task. It composes the execution loop with prompt construction, context management, tool invocation, and cost tracking into a single `run()` call.

**`async run(identity, task, completion_config?, max_turns?, memory_messages?) -> AgentRunResult`**
**`async run(identity, task, completion_config?, max_turns?, memory_messages?, timeout_seconds?) -> AgentRunResult`**

Pipeline steps:

Expand All @@ -925,23 +925,24 @@ Pipeline steps:
4. **Seed conversation** — injects system prompt, optional memory messages, and formatted task instruction as initial messages.
5. **Transition task** — `ASSIGNED` → `IN_PROGRESS` (pass-through if already `IN_PROGRESS`).
6. **Prepare tools and budget** — creates `ToolInvoker` from registry and `BudgetChecker` from task budget limit.
7. **Delegate to loop** — calls `ExecutionLoop.execute()` with context, provider, tool invoker, budget checker, and completion config.
7. **Delegate to loop** — calls `ExecutionLoop.execute()` with context, provider, tool invoker, budget checker, and completion config. If `timeout_seconds` is set, wraps the call in `asyncio.wait_for`; on expiry the run returns with `TerminationReason.ERROR` but cost recording and post-execution processing still occur.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DESIGN_SPEC references asyncio.wait_for but implementation uses asyncio.wait

The spec description (line 928) states:

If timeout_seconds is set, wraps the call in asyncio.wait_for

However, the actual implementation in _run_loop_with_timeout (lines 278–300) deliberately uses asyncio.wait instead, with clear rationale: this prevents conflating an internal TimeoutError from the loop with the engine's wall-clock deadline (see the method's docstring at lines 280–282).

The implementation choice is correct and well-reasoned; the spec just needs to match:

Suggested change
7. **Delegate to loop** — calls `ExecutionLoop.execute()` with context, provider, tool invoker, budget checker, and completion config. If `timeout_seconds` is set, wraps the call in `asyncio.wait_for`; on expiry the run returns with `TerminationReason.ERROR` but cost recording and post-execution processing still occur.
7. **Delegate to loop** — calls `ExecutionLoop.execute()` with context, provider, tool invoker, budget checker, and completion config. If `timeout_seconds` is set, wraps the call in `asyncio.wait` (not `asyncio.wait_for`, to avoid conflating internal `TimeoutError` with the engine's wall-clock deadline); on expiry the run returns with `TerminationReason.ERROR` but cost recording and post-execution processing still occur.
Prompt To Fix With AI
This is a comment left during a code review.
Path: DESIGN_SPEC.md
Line: 928

Comment:
**DESIGN_SPEC references `asyncio.wait_for` but implementation uses `asyncio.wait`**

The spec description (line 928) states:

> If `timeout_seconds` is set, wraps the call in `asyncio.wait_for`

However, the actual implementation in `_run_loop_with_timeout` (lines 278–300) deliberately uses `asyncio.wait` instead, with clear rationale: this prevents conflating an internal `TimeoutError` from the loop with the engine's wall-clock deadline (see the method's docstring at lines 280–282).

The implementation choice is correct and well-reasoned; the spec just needs to match:

```suggestion
7. **Delegate to loop** — calls `ExecutionLoop.execute()` with context, provider, tool invoker, budget checker, and completion config. If `timeout_seconds` is set, wraps the call in `asyncio.wait` (not `asyncio.wait_for`, to avoid conflating internal `TimeoutError` with the engine's wall-clock deadline); on expiry the run returns with `TerminationReason.ERROR` but cost recording and post-execution processing still occur.
```

How can I resolve this? If you propose a fix, please make it concise.

8. **Record costs** — records accumulated `TokenUsage` to `CostTracker` (if available). Cost recording failures are logged but do not affect the result.
9. **Return result** — wraps `ExecutionResult` in `AgentRunResult` with engine-level metadata.
9. **Apply post-execution transitions** — on `COMPLETED` termination: IN_PROGRESS → IN_REVIEW → COMPLETED (two-hop auto-complete in M3; reviewers deferred to M4+). All other termination reasons leave the task in its current state. Transition failures are logged but do not discard the successful execution result.
10. **Return result** — wraps `ExecutionResult` in `AgentRunResult` with engine-level metadata.

Error handling: `MemoryError` and `RecursionError` propagate unconditionally. All other exceptions are caught and wrapped in an `AgentRunResult` with `TerminationReason.ERROR`.

Constructor accepts: `provider` (required), `execution_loop` (defaults to `ReactLoop`), `tool_registry`, `cost_tracker`. The `run()` method also accepts `memory_messages` — optional working memory to inject between the system prompt and task instruction (memory retrieval is M5; the engine provides the injection hook).

Logs structured events under the `execution.engine.*` namespace (10 constants in `events/execution.py`): creation, start, prompt built, completion, errors, invalid input, task transitions, and cost recording outcomes.
Logs structured events under the `execution.engine.*` namespace (12 constants in `events/execution.py`): creation, start, prompt built, completion, errors, invalid input, task transitions, cost recording outcomes, task metrics, and timeout.

**`AgentRunResult`** — frozen Pydantic model wrapping `ExecutionResult` with engine metadata:

- `execution_result` — outcome from the execution loop
- `system_prompt` — the `SystemPrompt` used for this run
- `duration_seconds` — wall-clock run time
- `agent_id`, `task_id` — identifiers
- Computed fields: `termination_reason`, `total_turns`, `total_cost_usd`, `is_success`
- Computed fields: `termination_reason`, `total_turns`, `total_cost_usd`, `is_success`, `completion_summary`

### 6.6 Agent Crash Recovery

Expand Down Expand Up @@ -1463,12 +1464,15 @@ Every LLM provider call is tracked with comprehensive metadata for financial rep

Every completion call produces a `CompletionResponse` with `TokenUsage` (token counts and cost). The engine layer creates a `CostRecord` (with agent/task context) and records it into `CostTracker` — the provider itself does not have agent/task context. In M3, the engine additionally logs **proxy overhead metrics** at task completion:

- `turns_per_task` — number of LLM turns to complete the task (from `AgentContext.turn_count`)
- `turns_per_task` — number of LLM turns to complete the task (from `AgentRunResult.total_turns`)
- `tokens_per_task` — total tokens consumed (from `AgentContext.accumulated_cost.total_tokens`)
- `cost_per_task` — total USD cost (from `TaskExecution.accumulated_cost.cost_usd`)
- `cost_per_task` — total USD cost (from `AgentContext.accumulated_cost.cost_usd` via `AgentRunResult.total_cost_usd`)
- `duration_seconds` — wall-clock execution time in seconds (from `AgentRunResult.duration_seconds`)

These are natural overhead indicators — a task consuming 15 turns and 50k tokens for a one-line fix signals a problem.

These metrics are captured in `TaskCompletionMetrics` (in `engine/metrics.py`), a frozen Pydantic model with a `from_run_result()` factory method. The engine logs these metrics at task completion via the `EXECUTION_ENGINE_TASK_METRICS` event.

#### M4: Call Categorization + Orchestration Ratio

When multi-agent coordination exists, each `CostRecord` is tagged with a **call category**:
Expand Down Expand Up @@ -2153,6 +2157,7 @@ ai-company/
│ │ ├── task_execution.py # TaskExecution + StatusTransition
│ │ ├── context.py # AgentContext + AgentContextSnapshot
│ │ ├── loop_protocol.py # ExecutionLoop protocol + result models
│ │ ├── metrics.py # TaskCompletionMetrics proxy overhead model
│ │ ├── react_loop.py # ReAct loop implementation
│ │ ├── run_result.py # AgentRunResult outcome model
│ │ ├── agent_engine.py # Agent execution engine
Expand Down
2 changes: 2 additions & 0 deletions src/ai_company/engine/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@
TerminationReason,
TurnRecord,
)
from ai_company.engine.metrics import TaskCompletionMetrics
from ai_company.engine.prompt import (
DefaultTokenEstimator,
PromptTokenEstimator,
Expand Down Expand Up @@ -58,6 +59,7 @@
"ReactLoop",
"StatusTransition",
"SystemPrompt",
"TaskCompletionMetrics",
"TaskExecution",
"TerminationReason",
"TurnRecord",
Expand Down
168 changes: 153 additions & 15 deletions src/ai_company/engine/agent_engine.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
tool invocation, and budget tracking into a single ``run()`` entry point.
"""

import asyncio
import time
from datetime import UTC, datetime
from typing import TYPE_CHECKING
Expand All @@ -17,6 +18,7 @@
TerminationReason,
TurnRecord,
)
from ai_company.engine.metrics import TaskCompletionMetrics
from ai_company.engine.prompt import SystemPrompt, build_system_prompt
from ai_company.engine.react_loop import ReactLoop
from ai_company.engine.run_result import AgentRunResult
Expand All @@ -31,7 +33,9 @@
EXECUTION_ENGINE_INVALID_INPUT,
EXECUTION_ENGINE_PROMPT_BUILT,
EXECUTION_ENGINE_START,
EXECUTION_ENGINE_TASK_METRICS,
EXECUTION_ENGINE_TASK_TRANSITION,
EXECUTION_ENGINE_TIMEOUT,
)
from ai_company.providers.enums import MessageRole
from ai_company.providers.models import ChatMessage
Expand Down Expand Up @@ -90,14 +94,15 @@ def __init__(
has_cost_tracker=self._cost_tracker is not None,
)

async def run(
async def run( # noqa: PLR0913
self,
*,
identity: AgentIdentity,
task: Task,
completion_config: CompletionConfig | None = None,
max_turns: int = DEFAULT_MAX_TURNS,
memory_messages: tuple[ChatMessage, ...] = (),
timeout_seconds: float | None = None,
) -> AgentRunResult:
Comment on lines +98 to 107

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-high high

The run method orchestrates the agent execution lifecycle but fails to verify that the provided agent is actually assigned to the task. While it validates the agent's status and the task's status, it does not check the assigned_to field of the task against the agent's ID. This allows any active agent to execute any task that is in an ASSIGNED or IN_PROGRESS state, potentially leading to unauthorized access to task details and unauthorized state transitions. Although this check might be performed at a higher level, as the 'Top-level orchestrator', the AgentEngine should enforce this authorization boundary defensively.

"""Execute an agent on a task.

Expand All @@ -108,6 +113,10 @@ async def run(
max_turns: Maximum LLM turns allowed (must be >= 1).
memory_messages: Optional working memory messages to inject
between the system prompt and task instruction.
timeout_seconds: Optional wall-clock timeout in seconds.
When exceeded, the execution loop is cancelled and the
run returns with ``TerminationReason.ERROR``. Cost
recording and post-execution processing still occur.

Returns:
``AgentRunResult`` with execution outcome and metadata.
Expand All @@ -118,7 +127,8 @@ async def run(
Raises:
ExecutionStateError: If pre-flight validation fails (agent
not ACTIVE or task not ASSIGNED/IN_PROGRESS).
ValueError: If ``max_turns`` is less than 1.
ValueError: If ``max_turns`` is less than 1, or if
``timeout_seconds`` is not positive.
MemoryError: Re-raised unconditionally (non-recoverable).
RecursionError: Re-raised unconditionally (non-recoverable).
"""
Expand All @@ -135,6 +145,16 @@ async def run(
)
raise ValueError(msg)

if timeout_seconds is not None and timeout_seconds <= 0:
msg = f"timeout_seconds must be > 0, got {timeout_seconds}"
logger.warning(
EXECUTION_ENGINE_INVALID_INPUT,
agent_id=agent_id,
task_id=task_id,
reason=msg,
)
raise ValueError(msg)

self._validate_agent(identity, agent_id)
self._validate_task(task, agent_id, task_id)

Expand Down Expand Up @@ -167,6 +187,7 @@ async def run(
ctx=ctx,
system_prompt=system_prompt,
start=start,
timeout_seconds=timeout_seconds,
)
except MemoryError, RecursionError:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python 2 except comma syntax — not valid Python 3

except MemoryError, RecursionError: is Python 2 syntax and is a SyntaxError in Python 3. Python 2 parsed this as "catch MemoryError, bind it to the name RecursionError" — it does NOT catch both exception types. The Python 3 form to catch multiple exceptions requires parentheses. This same pattern appears on lines 584 and 669 as well.

Suggested change
except MemoryError, RecursionError:
except (MemoryError, RecursionError):
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/ai_company/engine/agent_engine.py
Line: 192

Comment:
**Python 2 `except` comma syntax — not valid Python 3**

`except MemoryError, RecursionError:` is Python 2 syntax and is a `SyntaxError` in Python 3. Python 2 parsed this as "catch `MemoryError`, bind it to the name `RecursionError`" — it does NOT catch both exception types. The Python 3 form to catch multiple exceptions requires parentheses. This same pattern appears on lines **584** and **669** as well.

```suggestion
        except (MemoryError, RecursionError):
```

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The except A, B: syntax is from Python 2 and will cause a SyntaxError in Python 3. The correct syntax is to use a tuple: except (MemoryError, RecursionError):.

        except (MemoryError, RecursionError):

logger.error(
Expand Down Expand Up @@ -200,6 +221,7 @@ async def _execute( # noqa: PLR0913
ctx: AgentContext,
system_prompt: SystemPrompt,
start: float,
timeout_seconds: float | None = None,
) -> AgentRunResult:
"""Run the execution loop, record costs, and build result."""
budget_checker = _make_budget_checker(task)
Expand All @@ -212,16 +234,49 @@ async def _execute( # noqa: PLR0913
estimated_tokens=system_prompt.estimated_tokens,
)

execution_result = await self._loop.execute(
context=ctx,
provider=self._provider,
tool_invoker=tool_invoker,
budget_checker=budget_checker,
completion_config=completion_config,
)
duration = time.monotonic() - start
try:
coro = self._loop.execute(
context=ctx,
provider=self._provider,
tool_invoker=tool_invoker,
budget_checker=budget_checker,
completion_config=completion_config,
)
if timeout_seconds is not None:
execution_result = await asyncio.wait_for(
coro,
timeout=timeout_seconds,
)
else:
execution_result = await coro
except TimeoutError:

Copilot AI Mar 6, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

asyncio.wait_for() raises asyncio.TimeoutError, but this handler catches the broad built-in TimeoutError. If the underlying loop/provider/tooling raises its own TimeoutError while timeout_seconds is set, it will be misclassified as a wall-clock timeout and the original error context will be lost. Prefer catching asyncio.TimeoutError (or catching around only the wait_for call) so non-wall-clock timeouts propagate into the normal fatal-error path/logging.

Suggested change
except TimeoutError:
except asyncio.TimeoutError:

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

asyncio.wait_for raises asyncio.TimeoutError. While this is an alias for the built-in TimeoutError in Python 3.11+, using except TimeoutError: will not catch the exception on older Python 3 versions. For better portability, it's recommended to explicitly catch asyncio.TimeoutError.

        except asyncio.TimeoutError:

if timeout_seconds is None:
raise
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Outdated
duration = time.monotonic() - start
error_msg = (
f"Wall-clock timeout after {duration:.1f}s (limit: {timeout_seconds}s)"
)
logger.warning(
EXECUTION_ENGINE_TIMEOUT,
agent_id=agent_id,
task_id=task_id,
duration_seconds=duration,
timeout_seconds=timeout_seconds,
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
execution_result = ExecutionResult(
context=ctx,
termination_reason=TerminationReason.ERROR,
error_message=error_msg,
)

await self._record_costs(execution_result, identity, agent_id, task_id)
execution_result = self._apply_post_execution_transitions(
execution_result,
agent_id,
task_id,
)

duration = time.monotonic() - start

result = AgentRunResult(
execution_result=execution_result,
Expand All @@ -230,7 +285,7 @@ async def _execute( # noqa: PLR0913
agent_id=agent_id,
task_id=task_id,
)
self._log_completion(result, execution_result, agent_id, task_id, duration)
self._log_completion(result, agent_id, task_id, duration)
return result

# ── Setup ────────────────────────────────────────────────────
Expand Down Expand Up @@ -341,6 +396,78 @@ def _transition_task_if_needed(
)
return ctx

def _apply_post_execution_transitions(
self,
execution_result: ExecutionResult,
agent_id: str,
task_id: str,
) -> ExecutionResult:
"""Apply post-execution task transitions based on termination reason.

Only ``TerminationReason.COMPLETED`` triggers transitions:
IN_PROGRESS → IN_REVIEW → COMPLETED (two-hop auto-complete).
All other reasons leave the task in its current state.

Note:
The IN_REVIEW → COMPLETED auto-complete is M3 scaffolding
(no reviewers yet). Later milestones will gate COMPLETED
on reviewer approval.

Transition failures are logged but do not discard the
successful execution result — a bookkeeping error must never
destroy the agent's work.

Args:
execution_result: Result from the execution loop.
agent_id: Agent identifier for logging.
task_id: Task identifier for logging.

Returns:
New ``ExecutionResult`` with updated context if transitions
were applied, or the original result unchanged.
"""
ctx = execution_result.context
if ctx.task_execution is None:
return execution_result

if execution_result.termination_reason != TerminationReason.COMPLETED:
return execution_result

try:
ctx = ctx.with_task_transition(
TaskStatus.IN_REVIEW,
reason="Agent completed execution",
)
logger.info(
EXECUTION_ENGINE_TASK_TRANSITION,
agent_id=agent_id,
task_id=task_id,
from_status=TaskStatus.IN_PROGRESS.value,
to_status=TaskStatus.IN_REVIEW.value,
)
# TODO(M4): Replace auto-complete with review gate
ctx = ctx.with_task_transition(
TaskStatus.COMPLETED,
reason="Auto-completed (no reviewers in M3)",
)
logger.info(
EXECUTION_ENGINE_TASK_TRANSITION,
agent_id=agent_id,
task_id=task_id,
from_status=TaskStatus.IN_REVIEW.value,
to_status=TaskStatus.COMPLETED.value,
)
except (ValueError, ExecutionStateError) as exc:
logger.exception(
EXECUTION_ENGINE_ERROR,
agent_id=agent_id,
task_id=task_id,
error=f"Post-execution transition failed: {exc}",
)
return execution_result

return execution_result.model_copy(update={"context": ctx})

def _make_tool_invoker(self) -> ToolInvoker | None:
"""Create a ToolInvoker from the registry, or None."""
if self._tool_registry is None:
Expand All @@ -350,23 +477,34 @@ def _make_tool_invoker(self) -> ToolInvoker | None:
def _log_completion(
self,
result: AgentRunResult,
execution_result: ExecutionResult,
agent_id: str,
task_id: str,
duration: float,
) -> None:
"""Log structured completion event for the finished run."""
"""Log structured completion event and proxy overhead metrics."""
accumulated = result.execution_result.context.accumulated_cost
logger.info(
EXECUTION_ENGINE_COMPLETE,
agent_id=agent_id,
task_id=task_id,
termination_reason=result.termination_reason.value,
total_turns=result.total_turns,
total_tokens=execution_result.context.accumulated_cost.total_tokens,
total_tokens=accumulated.total_tokens,
duration_seconds=duration,
cost_usd=result.total_cost_usd,
)

metrics = TaskCompletionMetrics.from_run_result(result)
logger.info(
EXECUTION_ENGINE_TASK_METRICS,
agent_id=agent_id,
task_id=task_id,
Comment on lines +570 to +574

Copilot AI Mar 6, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EXECUTION_ENGINE_TASK_METRICS is logged unconditionally, even when the run ends with TerminationReason.ERROR / MAX_TURNS / BUDGET_EXHAUSTED. Given the naming (TaskCompletionMetrics) and spec wording (“logged at task completion”), this will emit misleading “completion” metrics for incomplete tasks. Consider either (a) only logging this event when result.is_success/termination_reason == COMPLETED, or (b) include termination_reason in the metrics event (and update naming/docs) to make it clear these are per-run metrics.

Copilot uses AI. Check for mistakes.
turns_per_task=metrics.turns_per_task,
tokens_per_task=metrics.tokens_per_task,
cost_per_task=metrics.cost_per_task,
duration_seconds=metrics.duration_seconds,
)

async def _record_costs(
self,
result: ExecutionResult,
Expand Down Expand Up @@ -545,7 +683,7 @@ def _handle_fatal_error( # noqa: PLR0913
error=f"Failed to build error result: {build_exc}",
original_error=error_msg,
)
raise exc from None
raise exc from build_exc
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Outdated


def _format_task_instruction(task: Task) -> str:
Expand Down
Loading