Skip to content
Merged
54 changes: 54 additions & 0 deletions ai_docs/failure_matrix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Provider Failure Matrix

Populated failure matrix mapping operations, providers, error types, and recovery behavior.

## Error Categories

| Category | Description | Recovery Strategy |
|-----------------|--------------------------------------------|--------------------------------------------|
| `transient` | Temporary failure, may succeed on retry | Retry with exponential backoff (max 3) |
| `permanent` | Unrecoverable error | Fail immediately, surface to user |
| `configuration` | Misconfigured provider/environment | Re-resolve config, prompt user to fix |
| `unavailable` | Provider is down or unreachable | Degrade gracefully, suggest fallback |

## Failure Matrix

| Operation | Provider | Error Class | Category | Current Behavior | Desired Behavior |
|--------------------|----------------|--------------------------------------|-----------------|---------------------------------|-------------------------------------------|
| `startSession` | codex (harness)| `ProviderAdapterProcessError` | `configuration` | Error propagated to transport | Check binary exists before spawn |
| `startSession` | codex (harness)| `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Retry spawn with backoff |
| `startSession` | claudeAgent | `ProviderAdapterProcessError` | `configuration` | Error propagated to transport | Validate API key before spawn |
| `startSession` | claudeAgent | `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Retry with backoff |
| `startSession` | cursor (harness)| `ProviderAdapterProcessError` | `configuration` | Error propagated to transport | Check cursor binary path |
| `startSession` | opencode (harness)| `ProviderAdapterProcessError` | `configuration` | Error propagated to transport | Check opencode binary path |
| `startSession` | any | `ProviderValidationError` | `permanent` | Error propagated to transport | Correct (no change needed) |
| `startSession` | any | `ProviderUnsupportedError` | `configuration` | Error propagated to transport | Suggest enabling provider in settings |
| `sendTurn` | codex (harness)| `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Auto-retry once, then surface |
| `sendTurn` | codex (harness)| `ProviderAdapterSessionNotFoundError`| `permanent` | Recovery via `recoverSessionForThread` | Correct (recovery already implemented) |
| `sendTurn` | claudeAgent | `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Auto-retry once, then surface |
| `sendTurn` | claudeAgent | `ProviderAdapterSessionClosedError` | `permanent` | Error propagated to transport | Prompt user to start new session |
| `sendTurn` | any | `ProviderValidationError` | `permanent` | Error propagated to transport | Correct (validation is permanent) |
| `interruptTurn` | codex (harness)| `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Best-effort interrupt, log failure |
| `interruptTurn` | claudeAgent | `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Best-effort interrupt, log failure |
| `respondToRequest` | codex (harness)| `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Retry once for approval responses |
| `respondToRequest` | claudeAgent | `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Retry once for approval responses |
| `stopSession` | codex (harness)| `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Force-stop on timeout |
| `stopSession` | claudeAgent | `ProviderAdapterProcessError` | `unavailable` | Error propagated to transport | Kill process, mark session closed |
| `rollbackConversation` | codex (harness)| `ProviderAdapterRequestError` | `transient` | Recovery via resume + rollback | Correct (already recovers) |
| `rollbackConversation` | claudeAgent | `ProviderAdapterRequestError` | `transient` | Error propagated to transport | Resume session first, then rollback |
| `listSessions` | any | (no errors expected) | - | Returns empty on failure | Correct |
| `getCapabilities` | any | `ProviderUnsupportedError` | `configuration` | Error propagated to transport | Return empty capabilities for unknown |

## Error Classification by `_tag`

| Error `_tag` | Default Category | Notes |
|----------------------------------------------|------------------|--------------------------------------------|
| `ProviderAdapterRequestError` | `transient` | Promoted to `permanent` if detail contains "not found", "unauthorized", "forbidden" |
| `ProviderAdapterProcessError` | `transient` | Promoted to `configuration` if ENOENT/permission; `unavailable` if crashed/signal |
| `ProviderAdapterValidationError` | `permanent` | Invalid input to adapter API |
| `ProviderAdapterSessionNotFoundError` | `permanent` | Session ID does not exist |
| `ProviderAdapterSessionClosedError` | `permanent` | Session exists but is closed |
| `ProviderValidationError` | `permanent` | Invalid input to ProviderService API |
| `ProviderUnsupportedError` | `configuration` | Provider not registered |
| `ProviderSessionNotFoundError` | `permanent` | Thread not found in session directory |
| `ProviderSessionDirectoryPersistenceError` | `transient` | SQLite/persistence layer failure |
158 changes: 158 additions & 0 deletions ai_docs/provider_onboarding.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
# Provider Onboarding Playbook

Guide for adding a new coding agent provider to T3 Code.

## Architecture Overview

T3 Code uses a layered adapter architecture:

```
Transport (WebSocket/RPC)
|
ProviderService (cross-provider facade)
|
ProviderAdapterRegistry (adapter lookup)
|
ProviderAdapter (provider-specific runtime)
|
Provider CLI/SDK (codex, claude, cursor, opencode, ...)
```
Comment on lines +9 to +19

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add language specifier to fenced code block.

The ASCII diagram code block is missing a language specifier, triggering markdownlint MD040. Use text or plaintext for compatibility.

📝 Proposed fix
-```
+```text
 Transport (WebSocket/RPC)
     |
 ProviderService (cross-provider facade)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```
Transport (WebSocket/RPC)
|
ProviderService (cross-provider facade)
|
ProviderAdapterRegistry (adapter lookup)
|
ProviderAdapter (provider-specific runtime)
|
Provider CLI/SDK (codex, claude, cursor, opencode, ...)
```
🧰 Tools
🪛 markdownlint-cli2 (0.22.0)

[warning] 9-9: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@ai_docs/provider_onboarding.md` around lines 9 - 19, The fenced ASCII diagram
block (the code fence containing the Transport/ProviderService/ProviderAdapter
diagram) lacks a language specifier; update that code fence to use a neutral
specifier such as text or plaintext (e.g., change ``` to ```text) so
markdownlint MD040 is satisfied, leaving the diagram contents unchanged; locate
the ASCII diagram block in the Provider onboarding documentation and add the
specifier to the opening fence.


**Two runtime paths**:

- **Node SDK adapters** (ClaudeAdapter, CodexAdapter) -- run in-process via JS/TS SDKs
- **Harness adapters** (HarnessClientAdapter) -- bridge to Elixir GenServer sessions via WebSocket

Most new providers will use the **harness path** (Elixir GenServer) unless they have a first-class Node.js SDK.

## Step 1: Declare Provider Kind

Add the new provider to the `ProviderKind` schema in `packages/contracts/src/orchestration.ts`:

```typescript
export const ProviderKind = Schema.Literal(
"codex",
"claudeAgent",
"cursor",
"opencode",
"YOUR_PROVIDER",
);
```

## Step 2: Choose Runtime Path

### Path A: Elixir Harness (recommended for CLI-based providers)

1. **Create session module**: `apps/harness/lib/harness/providers/your_session.ex`
- Implement `@behaviour Harness.Providers.ProviderBehaviour`
- All 8 callbacks: `start_link/1`, `send_turn/2`, `interrupt_turn/3`, `respond_to_approval/3`, `respond_to_user_input/3`, `read_thread/2`, `rollback_thread/3`, `stop/1`
- Use `GenServer, restart: :temporary`
- Register via `{:via, Registry, {Harness.SessionRegistry, thread_id, "your_provider"}}`

2. **Register in SessionManager**: `apps/harness/lib/harness/session_manager.ex`
- Add clause to `provider_module/1`: `defp provider_module("your_provider"), do: {:ok, YourSession}`

3. **Declare capabilities**: Add entry in `HARNESS_PROVIDER_CAPABILITIES` in `apps/server/src/provider/Layers/HarnessClientAdapter.ts`

4. **Register in serverLayers.ts**: Add the provider to the `HARNESS_PROVIDERS` array in `makeServerProviderLayer()`

### Path B: Node SDK Adapter (for providers with JS/TS SDKs)

1. **Create service tag**: `apps/server/src/provider/Services/YourAdapter.ts`
- Follow `ClaudeAdapter.ts` pattern
- Extend `ProviderAdapterShape<ProviderAdapterError>`

2. **Create layer**: `apps/server/src/provider/Layers/YourAdapter.ts`
- Implement all methods of `ProviderAdapterShape`
- Include `translateMcpConfig` (return null if provider manages its own MCP)

3. **Register in adapter registry**: Update `ProviderAdapterRegistryLive` or `makeServerProviderLayer()`

## Step 3: Implement ProviderAdapterShape Methods

Every adapter must implement these methods (see `apps/server/src/provider/Services/ProviderAdapter.ts`):

| Method | Description |
| -------------------- | ---------------------------------------------- |
| `startSession` | Start a provider-backed session |
| `sendTurn` | Send a conversational turn |
| `interruptTurn` | Interrupt an active turn |
| `respondToRequest` | Respond to approval requests |
| `respondToUserInput` | Respond to user input requests |
| `stopSession` | Stop one session |
| `listSessions` | List active sessions |
| `hasSession` | Check session ownership |
| `readThread` | Read thread snapshot |
| `rollbackThread` | Roll back N turns |
| `stopAll` | Stop all sessions |
| `translateMcpConfig` | Translate MCP config to provider-native format |

## Step 4: Declare Capabilities

Set `ProviderAdapterCapabilities` for your provider:

```typescript
{
sessionModelSwitch: "in-session" | "restart-session" | "unsupported",
supportsUserInput: boolean,
supportsRollback: boolean,
supportsFileChangeApproval: boolean,
}
```

The contract test suite (`apps/server/integration/contract.integration.test.ts`) uses these to auto-skip tests for unsupported capabilities.

## Step 5: Runtime Event Mapping

Your adapter must emit `ProviderRuntimeEvent` objects with these core event types:

- `turn.started` / `turn.completed` -- turn lifecycle
- `content.delta` -- streaming text output
- `item.started` / `item.completed` -- tool/file-change items
- `request.opened` / `request.resolved` -- approval flow
- `session.status` -- session state changes

See `packages/contracts/src/providerRuntime.ts` for the full event schema.

## Step 6: MCP Configuration

If your provider supports external MCP servers:

- Implement `translateMcpConfig()` to convert `ResolvedMcpConfig` to provider-native format
- The harness path stores `mcp_config` in the session state (see `codex_session.ex`)

If your provider manages its own MCP (like Claude):

- Return `null` from `translateMcpConfig()`

## Step 7: Settings Integration

Add provider entry in `packages/contracts/src/settings.ts` server settings schema so users can enable/disable and configure the provider.

## Step 8: Testing

1. **Unit tests**: Add adapter-level tests in `apps/server/src/provider/Layers/YourAdapter.test.ts`
2. **Contract tests**: The contract suite auto-includes all registered providers
3. **Integration tests**: Add provider-specific integration scenarios if needed

## Step 9: Model Discovery (Optional)

If your provider supports model listing:

- Elixir: implement model discovery in the session module or a separate GenServer
- Node: expose via the adapter shape

Register in `ProviderRegistry` (`apps/server/src/provider/Services/ProviderRegistry.ts`) for the UI model picker.

## Checklist

- [ ] Provider kind added to `ProviderKind` schema
- [ ] Session module created (Elixir) or adapter layer created (Node)
- [ ] `@behaviour ProviderBehaviour` implemented (Elixir path)
- [ ] Capabilities declared
- [ ] Runtime event mapping implemented
- [ ] MCP config translator implemented
- [ ] Registered in SessionManager (Elixir) or adapter registry (Node)
- [ ] Settings entry added
- [ ] Unit tests passing
- [ ] Contract tests passing (or properly skipping unsupported capabilities)
1 change: 1 addition & 0 deletions apps/harness/lib/harness/application.ex
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ defmodule Harness.Application do
{Registry, keys: :unique, name: Harness.SessionRegistry},
{DynamicSupervisor, name: Harness.SessionSupervisor, strategy: :one_for_one},
Harness.Storage,
Harness.Metrics,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Harness.Metrics placed before critical services under rest_for_one supervisor causes cascading restarts

In the supervision tree, Harness.Metrics is inserted at position 5 (before Harness.SnapshotServer at 6 and HarnessWeb.Endpoint at 7) under a :rest_for_one strategy. If Harness.Metrics crashes, the supervisor terminates and restarts all children started after it — including the SnapshotServer and the WebSocket Endpoint — causing a service interruption for all connected clients. The Metrics module is a non-critical counter server designed to be crash-tolerant (its safe_cast/2 already handles the case where the GenServer is down). Placing it before critical services contradicts the root AGENTS.md priority: "Reliability first" and "Keep behavior predictable under load and during failures." The fix is to either move Harness.Metrics to the end of the children list, or supervise it under a separate supervisor so its failure doesn't cascade.

Suggested change
Harness.Metrics,
Harness.SnapshotServer,
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Harness.SnapshotServer,
HarnessWeb.Endpoint
]
Expand Down
109 changes: 106 additions & 3 deletions apps/harness/lib/harness/metrics.ex
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,125 @@ defmodule Harness.Metrics do
Collects BEAM runtime metrics for stress testing.

Exposes per-process memory, GC stats, scheduler utilization,
and SnapshotServer health — all the numbers needed to compare
OTP session isolation vs shared-heap runtimes.
SnapshotServer health, and session lifecycle counters — all
the numbers needed to compare OTP session isolation vs
shared-heap runtimes.
"""

use GenServer

# ---------------------------------------------------------------------------
# Public API
# ---------------------------------------------------------------------------

@doc """
Start the Metrics counter server (linked to the calling process).
"""
def start_link(_opts \\ []) do
GenServer.start_link(__MODULE__, %{}, name: __MODULE__)
end

@doc """
Collect a full metrics snapshot.
Collect a full metrics snapshot, including lifecycle counters.
"""
def collect do
counters = get_counters()

%{
beam: beam_metrics(),
sessions: session_metrics(),
snapshot_server: snapshot_server_metrics(),
lifecycle: counters,
timestamp: System.system_time(:millisecond)
}
end

@doc """
Record a session start event for the given provider.
"""
def record_session_start(provider) when is_binary(provider) do
safe_cast(:record, {:start, provider})
end

@doc """
Record a session end event for the given provider.
"""
def record_session_end(provider) when is_binary(provider) do
safe_cast(:record, {:end, provider})
end

@doc """
Record a session resume event for the given provider.
"""
def record_session_resume(provider) when is_binary(provider) do
safe_cast(:record, {:resume, provider})
end

@doc """
Record a turn duration sample (milliseconds) for the given provider.
"""
def record_turn_duration(provider, duration_ms)
when is_binary(provider) and is_number(duration_ms) do
safe_cast(:record_duration, {provider, duration_ms})
end

@doc """
Return the current lifecycle counter map.
"""
def get_counters do
if Process.whereis(__MODULE__) do
GenServer.call(__MODULE__, :get_counters)
else
default_counters()
end
end

# ---------------------------------------------------------------------------
# GenServer callbacks
# ---------------------------------------------------------------------------

@impl true
def init(_opts) do
{:ok, default_counters()}
end

@impl true
def handle_cast({:record, {event, provider}}, state) do
key = counter_key(event, provider)
{:noreply, Map.update(state, key, 1, &(&1 + 1))}
end

def handle_cast({:record_duration, {provider, duration_ms}}, state) do
durations_key = "turn_durations.#{provider}"
existing = Map.get(state, durations_key, [])
# Keep last 100 samples to bound memory.
samples = Enum.take([duration_ms | existing], 100)
{:noreply, Map.put(state, durations_key, samples)}
end

@impl true
def handle_call(:get_counters, _from, state) do
{:reply, state, state}
end

# ---------------------------------------------------------------------------
# Private helpers
# ---------------------------------------------------------------------------

defp safe_cast(msg, payload) do
if Process.whereis(__MODULE__) do
GenServer.cast(__MODULE__, {msg, payload})
end

:ok
end

defp counter_key(:start, provider), do: "start_count.#{provider}"
defp counter_key(:end, provider), do: "end_count.#{provider}"
defp counter_key(:resume, provider), do: "resume_count.#{provider}"

defp default_counters, do: %{}

# --- BEAM-level metrics ---

defp beam_metrics do
Expand Down
Loading