Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions packages/agent-core-v2/docs/en/llm.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ llm is a standalone LLM request library inside the human layer (`src/human/llm/`
## Design Principles

1. **Minimal boundary: llm = "a single request"**. llm only handles request encoding/decoding and event emission. auth, usage accounting, HistoryMessage/meta, compaction, switch, the media file system, and Tool Message assembly are all out of scope — they either move up to the turn/agent layer or plug in as contribution points.
2. **Streaming-native; events are the contract**. The only outward surface is a single, purely serializable event stream (requester level: `llm.sent / streaming.headers / streaming.part / streaming.usage / streaming.finish / streaming.message_id / failed.syntax / failed.remote / done`; the machine level adds `llm.retrying / llm.recovering`, and `llm.sent` carries the most recent recovery record). Streaming and non-streaming are isomorphic (non-streaming also accumulates over the stream, just without deltas). Events are emitted as they arrive — no caching, no fallback.
3. **format masks inter-protocol differences; traits express provider customizations**. format lives at the protocol layer and handles encoding/decoding of requests, responses, errors, usage, and finish. Each protocol owns a typed trait interface (`OpenAITrait` / `OpenAIResponsesTrait` / `AnthropicTrait` / `GoogleGenAITrait`) exposing only the customization points that protocol actually consumes — a hook a protocol ignores is unrepresentable, never silently dead. format and trait never import each other: both speak only the neutral wire/chunk types in the protocol's `contract.ts`. The requester is the composition root — `generate` runs a fixed per-protocol pipeline (`planOpenAIRequest` and friends) that alternates pure format stages (lower → assemble → encode → stream parser) with trait hooks (cacheKey/thinking → convertMessage → mergeHistory → convertTool → buildParams → extractUsage), so customization is explicit data flow instead of a closure captured inside format. Endpoint/env resolution and default headers form the provider `connection`, error classification is a requester option, and model capability is a provider-variant field — none of them are format business. Each base's public seam is contract + trait + requester; format, lower, and patterns are internal to the requester pipeline — only bases code and tests may import them (lint-enforced). Protocol differences must not leak into the machine or into requester decorators.
2. **Streaming-native; events are the contract**. The only outward surface is a single, purely serializable event stream (requester level: `llm.sent / streaming.headers / streaming.part / streaming.usage / streaming.finish / streaming.message_id / failed.syntax / failed.remote / done`; the turn level adds `llm.retrying / llm.recovering`, and `llm.sent` carries the most recent recovery record). Streaming and non-streaming are isomorphic (non-streaming also accumulates over the stream, just without deltas). Events are emitted as they arrive — no caching, no fallback.
3. **format masks inter-protocol differences; traits express provider customizations**. format lives at the protocol layer and handles encoding/decoding of requests, responses, errors, usage, and finish. Each protocol owns a typed trait interface (`OpenAITrait` / `OpenAIResponsesTrait` / `AnthropicTrait` / `GoogleGenAITrait`) exposing only the customization points that protocol actually consumes — a hook a protocol ignores is unrepresentable, never silently dead. format and trait never import each other: both speak only the neutral wire/chunk types in the protocol's `contract.ts`. The requester is the composition root — `generate` runs a fixed per-protocol pipeline (`planOpenAIRequest` and friends) that alternates pure format stages (lower → assemble → encode → stream parser) with trait hooks (cacheKey/thinking → convertMessage → mergeHistory → convertTool → buildParams → extractUsage), so customization is explicit data flow instead of a closure captured inside format. Endpoint/env resolution and default headers form the provider `connection`, error classification is a requester option, and model capability is a provider-variant field — none of them are format business. Each base's public seam is contract + trait + requester; format, lower, and patterns are internal to the requester pipeline — only bases code and tests may import them (lint-enforced). Protocol differences must not leak into the turn or into requester decorators.
4. **Two-layer error model**. Internally, code throws the SDK's native errors; local request validation throws the shared `SyntaxRequestFormatError` (`llm/syntax-errors.ts`), which the requester converts uniformly via `toLlmSyntaxErrorMessage`, with no intermediate layer. Externally there are only `llm.failed.syntax` (local message syntax errors, never retried) and `llm.failed.remote` (remote streaming errors, subdivided into connection / timeout / rate_limit / quota_exhausted / context_overflow / request_structure, etc.), converted by format at the boundary.
5. **Stateless core + state machine shell**. `generate(config, content, control)` is a stateless function; errors are delivered via onEvent, never thrown. The llm machine wraps a single request (messageResolvers, abort scope, event forwarding) and drives retry and recovery through the pure policy functions in retry.ts / recovery.ts: recovery re-sends with replacement messages produced by the pure `propose` function (attempt resets to 1), retry backs off in the `retrying` state (honoring Retry-After), and the machine emits `llm.recovering / llm.retrying` for each. Empty response is judged by `withEmptyResponseGuard` at the requester boundary and raised as `llm.failed.remote`, entering the same retry path. Abort is carried by an AbortController owned by the turn: the controller is passed into the machine and the request actor via `LlmInput.signal`, and the turn aborts it directly on `turn.abort`, with the request ending as `llm.failed.remote`; the request actor neither creates its own controller nor touches any signal on teardown, so a finished request can never abort a shared signal. The accumulator is held by the turn and fed by the event stream; on `llm.retrying / llm.recovering` the turn rolls it back and recreates it, so every attempt accumulates from zero while as much interrupted state as possible is preserved (the turn finishes the complete message out of the accumulator at `llm.done`).
5. **Stateless core + turn-driven orchestration**. `generate(config, content, control)` is a stateless function; errors are delivered via onEvent, never thrown. The turn machine invokes the request actor (`createRequestActor`) directly: the actor wraps a single request (messageResolvers, abort scope, event sendBack), and the turn drives retry and recovery through the pure policy functions in retry.ts / recovery.ts: recovery re-sends with replacement messages produced by the pure `propose` function (attempt resets to 1), retry backs off in the `retrying` state (honoring Retry-After), and the turn emits `llm.recovering / llm.retrying` for each. Empty response is judged by `withEmptyResponseGuard` at the requester boundary and raised as `llm.failed.remote`, entering the same retry path. Abort is carried by an AbortController owned by the turn: the controller is passed into the request actor via `LlmInput.signal`, and the turn aborts it directly on `turn.abort`, with the request ending as `llm.failed.remote`; the request actor neither creates its own controller nor touches any signal on teardown, so a finished request can never abort a shared signal. The accumulator is held by the turn and fed by the event stream; on `llm.retrying / llm.recovering` the turn rolls it back and recreates it, so every attempt accumulates from zero while as much interrupted state as possible is preserved (the turn finishes the complete message out of the accumulator at `llm.done`).
6. **No silent fallback**. Configuration is taken exactly as given. For beta features, thinking, empty response, and similar scenarios, define explicit error conditions first, fail at request time, and guide the user to fix the configuration — never fall back silently.
7. **Every variable capability is a contribution point**. Providers, media upload/degradation, usage, traceId, and error recovery (compaction / media degradation) all plug in through extension points; the llm core contains none of these concepts.
8. **Data is data**. A model is pure, function-free data (endpoint url + model uniquely identifies a model), serializable and directly usable as generate input. The catalog is a derived `provider -> models` cache; the dependency direction only goes from models-dev into llm internals, never the reverse.
Expand All @@ -34,9 +34,9 @@ llm/
├── requester/
│ ├── requester.ts LlmRequester.generate(config, content, control);
│ │ ExtraParams typed per protocol {openai?, responses?, anthropic?, googleGenai?}
│ ├── machine.ts llm state machine (single request + retry/recovery + empty response
│ │ judgment; emits llm.retrying / llm.recovering)
│ ├── retry.ts / recovery.ts pure retry/recovery policy functions (driven by the llm machine; propose is pure)
│ ├── actor.ts request actor: a fromCallback wrapping a single request
│ │ (messageResolvers, abort scope, event sendBack); invoked by the turn
│ ├── retry.ts / recovery.ts pure retry/recovery policy functions (driven by the turn machine; propose is pure)
│ ├── empty-response.ts withEmptyResponseGuard: judges empty responses at finish and raises llm.failed.remote
│ └── bases/ four protocol bases: openai / openai-responses / anthropic / google-genai
│ each with contract / format / lower / patterns / capability / extra-params / trait / requester
Expand All @@ -53,11 +53,12 @@ llm/
└── media/ media contribution points: cache / degrade / ref / resolver / store / upload
```

Request lifecycle: `generate` receives (config, content, control) → the requester's `plan*` function composes pure format stages with trait hooks into protocol requestParams (format lowers the generic Message[] through the Pattern Rewriter; trait adjusts kwargs, converted messages, history, tools, and final params in between) → internalGenerate calls the official SDK → streaming chunks are converted by the stateless parser callbacks into `llm.streaming.part / streaming.usage / streaming.finish / streaming.message_id` events → errors are converted by format into `llm.failed.*`; on success the requester emits `llm.done`, on failure it ends with `llm.failed.syntax / llm.failed.remote` and never emits `llm.done`. `withEmptyResponseGuard` judges empty responses at finish and raises `llm.failed.remote`; the llm machine first tries recovery on `llm.failed.remote` (replacement messages from the pure `propose` function, emitting `llm.recovering`), then retries with backoff (honoring Retry-After, emitting `llm.retrying`), and only lands in the failed final state once attempts are exhausted. The upper-layer turn holds the HistoryAccumulator, fed by the event stream, rolls it back and recreates it on `llm.retrying / llm.recovering`, and finishes the complete message at `llm.done`; usage accounting, tracing, compaction, and media degradation all attach to the event stream as plugins/contribution points.
Request lifecycle: `generate` receives (config, content, control) → the requester's `plan*` function composes pure format stages with trait hooks into protocol requestParams (format lowers the generic Message[] through the Pattern Rewriter; trait adjusts kwargs, converted messages, history, tools, and final params in between) → internalGenerate calls the official SDK → streaming chunks are converted by the stateless parser callbacks into `llm.streaming.part / streaming.usage / streaming.finish / streaming.message_id` events → errors are converted by format into `llm.failed.*`; on success the requester emits `llm.done`, on failure it ends with `llm.failed.syntax / llm.failed.remote` and never emits `llm.done`. At `llm.done` the turn judges empty responses via `emptyResponseError` and re-raises them as `llm.failed.remote`; the turn machine first tries recovery on `llm.failed.remote` (replacement messages from the pure `propose` function, emitting `llm.recovering`), then retries with backoff (honoring Retry-After, emitting `llm.retrying`), and only fails the turn once attempts are exhausted. The turn holds the HistoryAccumulator, fed by the event stream, rolls it back and recreates it on `llm.retrying / llm.recovering`, and finishes the complete message at `llm.done`; usage accounting, tracing, compaction, and media degradation all attach to the event stream as plugins/contribution points.

## Rejected Schemes (do not reintroduce)

- Splitting llmActor / llmStreamActor into two actors — a single machine; non-streaming also accumulates over the stream.
- Splitting the request actor into llmActor / llmStreamActor — one actor per request; non-streaming also accumulates over the stream.
- A dedicated llm state machine wrapping the request actor — the turn machine invokes the actor directly and owns retry/recovery; the extra machine layer carried no state anyone consumed.
- DDD domain-method wrapping (Generation Domain, etc.) — use the format/trait/provider layering instead.
- A single cross-protocol trait bag holding every vendor hook (the old ProtocolTrait) — per-protocol typed traits, composed by the requester's request pipeline.
- Binding the trait into the format (a `createOpenAIFormat(trait)` closure, or trait hooks passed as formatRequest options) — the requester pipeline alternates format stages and trait hooks explicitly; the two sides only share the neutral `contract.ts` types.
Expand Down
Loading
Loading