Skip to content

refactor(engine): share ORT session ownership - #2122

Merged
justinchuby merged 1 commit into
mainfrom
justinchuby/share-ort-session-resources
Aug 25, 2026
Merged

justinchuby merged 1 commit into
mainfrom
justinchuby/share-ort-session-resources

Conversation

@justinchuby

Copy link
Copy Markdown
Owner

Summary

  • convert the remaining primary, draft, MTP, and EAGLE-3 model session resources to structurally shared Arc<Session> ownership
  • add a named borrowed/shared ORT session owner for persistent decode runners, eliminating the synthetic 'static session borrow
  • keep every IoBinding, allocator, device Value, KV/cache, and graph-capture state runner/worker-local
  • preserve Session::supports_concurrent_run() as a capability signal and add stable-address, lifecycle, and W=1 continuation parity coverage
  • update SESSION_CONCURRENCY.md only for this completed prerequisite

Scope

This PR is a shared immutable ORT ownership prerequisite only. It does not enable N>1 workers, concurrent session runs, new server worker configuration, or concurrent use of providers/configurations that decline supports_concurrent_run().

Validation

  • cargo test -p onnx-genai-engine --lib
  • cargo test -p onnx-genai-engine --lib --features native-backend
  • cargo test -p onnx-genai-engine --test multi_session --test canonical_execution_parity --test chained_proposer_real
  • cargo test -p onnx-genai-ort
  • cargo test -p onnx-genai-server --lib
  • cargo check -p onnx-genai-server --all-targets
  • cargo clippy -p onnx-genai-ort -p onnx-genai-engine -p onnx-genai-server --all-targets -- -D warnings
  • cargo clippy -p onnx-genai-engine -p onnx-genai-server --all-targets --features native-backend -- -D warnings
  • cargo fmt --all -- --check

Convert the remaining primary, draft, and EAGLE-3 session holders to Arc and give persistent decode runners a structural shared-session owner. Remove the synthetic static session borrow while keeping bindings, allocators, values, KV, and graph-capture state worker-local.

Preserve the concurrent-run capability signal and W=1 behavior, add lifecycle/parity coverage, and update the session concurrency phase status without enabling multi-worker execution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Copilot <223556219+Copilot@users.noreply.github.com>
@justinchuby
justinchuby merged commit 37e99e5 into main Aug 25, 2026
16 checks passed
@justinchuby
justinchuby deleted the justinchuby/share-ort-session-resources branch August 25, 2026 15:35
@codecov

codecov Bot commented Aug 25, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.70%. Comparing base (74e6cb6) to head (af7b1ab).
⚠️ Report is 21 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2122      +/-   ##
==========================================
+ Coverage   80.29%   80.70%   +0.40%     
==========================================
  Files         426      429       +3     
  Lines      205244   215150    +9906     
  Branches   205244   215150    +9906     
==========================================
+ Hits       164801   173636    +8835     
- Misses      34792    35742     +950     
- Partials     5651     5772     +121     
Flag Coverage Δ
cli-ort-linux 72.51% <ø> (?)
cli-ort-windows 72.10% <ø> (+0.09%) ⬆️
mlas 85.90% <ø> (?)
offline 80.82% <ø> (+0.31%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 64 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants