Skip to content

refactor(engine): give the workflow runtime's state named owners (W=1 state split) - #2044

Merged
justinchuby merged 1 commit into
mainfrom
justinchuby/session-runtime-state-split
Aug 24, 2026
Merged

justinchuby merged 1 commit into
mainfrom
justinchuby/session-runtime-state-split

Conversation

@justinchuby

Copy link
Copy Markdown
Owner

What this is

The first bounded step of docs/architecture/SESSION_CONCURRENCY.md §13
Phase 1's state split: WorkflowRuntime's flat field list becomes the owners
§3 already names. W = 1 structural prerequisite only — this PR makes
nothing concurrent and claims nothing about concurrency.

Before, the runtime held the compiled plan, the ORT and native backend
handles, the caches derived from them, every counter and the interpreter's
session cells in one list, so which §3 category a field belonged to was a
fact about the comment above it. A future worker could be handed state it
must not share without anything failing to compile.

The split

Owner Category Holds
WorkflowPlan behind Arc §3.1 immutable workflow, compiled graph, derived output sets, memory plan, backend, adapter service, preprocessing, speculative contract
WorkerBackend §3.2 backend handles component sessions, execution islands, native component set
WorkerRuntimeState §3.2 storage, §3.3 access bindings, allocators, device values, embedding tables, session cells, session leases, iteration variants, adapter cache, counters
WorkflowPass §3.3 per-execution the pass id and its telemetry, created when a pass opens and dropped when it ends

WorkflowPlan carries a static assertion that it is Send + Sync — a
mutable cell added to it fails the build rather than quietly making the plan
unshareable. Both worker types are structurally !Send/!Sync through a
PhantomData marker: no new unsafe impl, no blanket lock, and static
assertions keep it that way. Engine's existing unsafe impl Send is
untouched.

Id namespaces

Made explicit, because a future worker minting from the wrong one hands two
conversations the same id.

  • Worker-local: PassId (was the bare workflow_execution_generation
    counter) and GraphCaptureId. Both are only ever compared against state
    the same worker owns. Component capture ids came from bindings.len(),
    which is a counter only for as long as nothing is removed from that map;
    islands seeded a hand-rolled island_id * 1000 stride, kept because it
    makes a captured graph's id say which island captured it. Exhaustion is now
    a typed error rather than a wrap into ORT's -1 uncaptured sentinel.
  • Shared: session ids are the only ones that leave the engine, so they
    mint from engine::ids::SharedSessionIds, an atomic allocator. Ids still
    start at 1 and still increment by 1 — nothing externally observed
    changes.

#2019's contracts are preserved

WorkflowRuntime::drop still clears island bindings, then values, then
allocators: a Value is a bare OrtValue handle with no back-reference to
its allocator, so nothing in the type system orders that release. The clears
moved into WorkerRuntimeState::release_ort_state, called from the same
Drop. worker is now declared before backend, and _ort_environment
stays last, so the field order agrees with the explicit teardown instead of
contradicting it.

Explicitly not in this PR

No N > 1. No pre-enqueue lease. No server routing. EngineOwner's
unsafe impl Send is not removed. Batching semantics unchanged. No public
API change — the split is private to the crate and every accessor keeps its
signature.

Tests

New, in pipeline/runtime_state.rs, pipeline/mod.rs and engine/ids.rs:

  • static: the plan is Send + Sync behind Arc; WorkflowRuntime is
    neither Send nor Sync; each worker type and id allocator likewise.
  • the plan really crosses a thread boundary behind an Arc and reads the
    same on the other side.
  • pass ids start at "none", are fresh per pass, and an iteration variant
    mints in its own namespace without advancing its parent's.
  • capture ids are unique and non-negative for component bindings, disjoint
    across islands, and exhaustion is refused rather than wrapped.
  • session ids start at 1, never repeat, and are collision-free across eight
    concurrent minters.
  • a conversation still continues from — and is still released through — the
    worker's own cells, and the inner session lease still refuses a second
    holder by name.
  • the drop path still releases cached values, before the allocators that own
    their memory.

Verification

cargo fmt --all -- --check                                              # clean
cargo clippy -p onnx-genai-engine --all-targets -- -D warnings          # clean
cargo clippy -p onnx-genai-engine --all-targets --features native-backend -- -D warnings  # clean
cargo clippy -p onnx-genai-server --all-targets -- -D warnings          # clean
cargo test -p onnx-genai-engine --lib                                   # 494 passed, 1 ignored
cargo test -p onnx-genai-engine --lib --features native-backend         # 637 passed, 1 ignored
cargo test -p onnx-genai-server --lib                                   # 273 passed, 2 ignored
cargo test -p onnx-genai-engine --test workflow_session_continuation    # 16 passed
cargo test -p onnx-genai-engine --test multi_session                    # 2 passed
cargo test -p onnx-genai-engine --test static_cache_workflow            # 7 passed
cargo test -p onnx-genai-engine --test workflow_performance_conformance # 1 passed, 2 ignored
cargo test -p onnx-genai-engine --test authored_body_selects_executor \
  --test authored_iteration_executors --test model_package \
  --test batching_capability                                            # all passed
python3 scripts/check_feature_gate_coverage.py                          # pass
python3 .github/scripts/verify_cuda_test_honesty.py --source-scan       # pass

cargo clippy --workspace cannot run in this worktree: git worktree add
does not populate the crates/onnx-runtime-cpuinfo/vendor/cpuinfo
submodule, which is an environment fault rather than a result. The crates
this change touches are covered above.

Docs

SESSION_CONCURRENCY.md §13 Phase 1 gains a note that the state split
proper has landed for the interpreter, and that the phase is not done:
the Box<Session> holders and the thread_affinity extension to the native
handles remain. Citations into pipeline/mod.rs that the move invalidated
are repointed at pipeline/runtime_state.rs.

`WorkflowRuntime` held the compiled plan, the ORT and native backend
handles, the caches derived from them, every counter and the interpreter's
session cells as one flat field list. Which of
`docs/architecture/SESSION_CONCURRENCY.md` §3's three categories a field
belonged to was therefore a fact about the comment above it, and a second
worker could be given state it must not share without anything failing to
compile.

This splits the list into the owners §3 already names, and nothing else:

- `WorkflowPlan` (§3.1) holds the frozen package data — workflow, compiled
  graph, derived output sets, memory plan, backend, adapter service,
  preprocessing, speculative contract — behind an `Arc`, with a static
  assertion that it is `Send + Sync`. A mutable cell added to it fails the
  build rather than quietly making the plan unshareable.
- `WorkerBackend` (§3.2) holds the component sessions, execution islands and
  native component set.
- `WorkerRuntimeState` (§3.2 storage under §3.3 access) holds the bindings,
  allocators, device values, embedding tables, session cells, session
  leases, iteration variants, adapter cache and counters.
- `WorkflowPass` (§3.3) is created when a pass opens and dropped when it
  ends, carrying the pass id and the telemetry that used to be two unrelated
  locals.

Both worker types are structurally `!Send`/`!Sync` through a `PhantomData`
marker — not a new `unsafe impl`, not a lock — and static assertions hold
them there. `Engine`'s existing `unsafe impl Send` is untouched.

The id namespaces are made explicit at the same time, because a future
worker that mints from the wrong one hands two conversations the same id:

- `PassId` (was the bare `workflow_execution_generation` counter) and
  `GraphCaptureId` mint from worker-local allocators, since both are only
  ever compared against state the same worker owns. Component capture ids
  came from `bindings.len()`, which is a counter only while nothing is ever
  removed from that map; islands seeded a hand-rolled `island_id * 1000`
  stride, which is kept for log readability. Exhaustion is now a typed error
  instead of a wrap into ORT's `-1` uncaptured sentinel.
- Session ids are the only ones that leave the engine, so they mint from
  `engine::ids::SharedSessionIds`, an atomic allocator. Ids still start at 1
  and still increment by 1, so nothing externally observed changes.

#2019's lifetime contracts are preserved deliberately. `WorkflowRuntime`'s
`Drop` still clears island bindings and then values before allocators — a
`Value` is a bare `OrtValue` handle with no back-reference to its allocator,
so nothing in the type system orders that release — and the clears now live
in `WorkerRuntimeState::release_ort_state`, called from the same `Drop`.
`worker` is declared before `backend` and `_ort_environment` stays last, so
the field order agrees with the explicit teardown instead of contradicting
it.

This is a `W = 1` structural change only. There is still one worker and one
thread, no pre-enqueue lease, no server routing, and batching semantics are
untouched.

Tests: static assertions for the plan being shareable and the runtime being
neither `Send` nor `Sync`; the plan really crossing a thread boundary behind
an `Arc`; pass ids being fresh, worker-local and starting at "none"; capture
ids being unique, non-negative and disjoint per island; session ids being
collision-free across eight threads; a conversation still continuing and
still being released through the worker's own cells; the inner session lease
still refusing a second holder by name; and the drop path still releasing
cached values before the allocators that own their memory.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Aug 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.49%. Comparing base (398cff8) to head (a2bebe6).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2044      +/-   ##
==========================================
+ Coverage   80.25%   80.49%   +0.23%     
==========================================
  Files         409      425      +16     
  Lines      189791   209135   +19344     
  Branches   189791   209135   +19344     
==========================================
+ Hits       152319   168334   +16015     
- Misses      32048    35141    +3093     
- Partials     5424     5660     +236     
Flag Coverage Δ
cli-ort-linux 72.51% <ø> (?)
cli-ort-windows 72.10% <ø> (?)
mlas 85.10% <ø> (?)
offline 80.62% <ø> (+0.37%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 67 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@justinchuby
justinchuby merged commit 2d380b7 into main Aug 24, 2026
17 checks passed
@justinchuby
justinchuby deleted the justinchuby/session-runtime-state-split branch August 24, 2026 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants