Repository navigation
EP plugin export: run our Rust execution providers inside upstream ONNX Runtime - #762
Merged
Merged
Conversation
… session Baseline commit of the outbound ORT plugin-EP export workstream recovered from an interrupted session. Contains the shared adapter crate skeleton, the CPU plugin shim, inventory/ABI docs, and decision inbox drops. Known-broken at this commit (fixed in follow-ups): - crates/onnx-runtime-ep-plugin does not compile (OrtEp field mismatch) - 4 CRITICAL/HIGH FFI security findings outstanding - compute path returns NOT_IMPLEMENTED Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… compute path Makes the outbound ORT plugin-EP export adapter compile, be panic-safe, and actually execute kernels instead of returning ORT_NOT_IMPLEMENTED. FFI hardening (Nabil): - Fix OrtEp initializer: ValidateCompiledModelCompatibilityInfo belongs to OrtEpFactory, not OrtEp (which has GetCompiledModelCompatibilityInfo). - catch_unwind on all extern "C" callbacks (no UB across the C ABI). - HOST_ORT_API as AtomicPtr with Release/Acquire (was `static mut`). - Null-check incoming ORT pointers; return ORT_INVALID_ARGUMENT. - Remove unsound `unsafe impl Send/Sync` on OutboundGraphReader. - check() now reports the real ORT error string via GetErrorMessage. Compute path (Deckard): - Real CreateState/Compute/ReleaseState lifecycle. - OrtKernelContext <-> TensorView bridge with dtype/shape marshaling. - Output shapes via ElementwiseBroadcast / SameAsInput inference; fails closed with an actionable message when a shape cannot be resolved. Conformance harness (Pris): - L1 exported-symbol assertions (nm -D, readelf --dyn-syms). - L2 dlopen/dlsym factory lifecycle + fail-closed bad-API-version. - L3 real ORT 1.27.0 vtable drive (3 tests ignored pending device fix). - New unsupported-op fixture (nonzero_1x4). Docs (Roy): corrected stale "v1 implemented" claims, pinned the exact ORT compatibility boundary, resolved the onnxruntime-mlx scope question. Validation: cargo build/clippy clean; 23 adapter unit tests pass; 8 cpu-plugin conformance tests pass, 3 ignored on a known blocker (GetSupportedDevices returns 0 devices -> ORT segfaults at registration). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Upstream ONNX Runtime 1.27.0 now genuinely loads, registers, and executes our Rust CPU execution provider as a plugin EP. Device enumeration (Nabil): - GetSupportedDevices filters ORT's discovered hardware for CPU devices, creates OrtEpDevice via OrtEpApi::CreateEpDevice, and registers CPU memory info via EpDevice_AddAllocatorInfo. Bounded by max_ep_devices. - Previously returned 0 devices, which made ORT segfault inside RegisterExecutionProviderLibrary. Verified end-to-end against real ORT 1.27.0: CreateEnv -> RegisterExecutionProviderLibrary -> GetEpDevices (finds 'cpu_ep') -> SessionOptionsAppendExecutionProvider_V2 -> CreateSession -> Run -> outputs [6.0, 8.0, 10.0, 12.0] (correct). Shape inference (Deckard): - 22 inference rules: MatMul (1-D/2-D/batched-ND broadcast), Gemm, Concat, Transpose, Gather/GatherND/GatherBlockQuantized, Reshape, Slice, Squeeze/ Unsqueeze, Shape, reductions, Conv, LayerNorm family, MHA/GQA/RoPE. - Data-dependent shapes (Reshape, Slice) read their control tensors at Compute time, honoring -1 inference and allowzero. - Unmodeled ops return Declined -> hard error, never a guessed shape. - Multi-node fused subgraphs thread intermediates in topological order. Adapter unit tests: 66 passed (up from 23). Known follow-ups (tracked, not shipped): ep.rs still calls for_op rather than for_node, so attribute-aware rules are not yet wired into the live path; Holden's re-audit returned RED on three panic-safety//dim-validation findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gs, broaden conformance Closes the integration gap that left the richer shape-inference rules dead, resolves Holden's ship-blocking audit findings, and turns a single smoke test into a real upstream-ORT conformance suite. Capability + shape-inference integration (Nabil): - ep.rs now calls ShapeInference::for_node (was for_op), so the 22 attribute-aware rules are live instead of always returning Declined. - GetCapability filters out nodes whose shape cannot be inferred, so we stop over-claiming ops we cannot execute (NonZero now falls back to ORT). - SubgraphRouting wired for multi-node fused subgraphs. - graph_reader reads node attributes and int64 initializers, copying them into owned Rust data during Compile (no OrtGraph/OrtNode pointer caching, per ORT's lifetime contract). Opset-13 Unsqueeze/Squeeze axes-as-input is resolved from initializers. Security remediation of Holden's RED verdict: - N2 (HIGH, Leon): dims from ORT are validated - negative dims rejected by index and value instead of wrapping -1i64 to usize::MAX; element-count and byte-length use checked_mul; zero dims accepted as legal. - N3 (MEDIUM, Isidore): macro-generated CreateEpFactories/ReleaseEpFactory are panic-guarded. ReleaseEpFactory's return type corrected to void per the ORT ABI. Exported symbols unchanged. - N1 (CRITICAL): compute_execute panic guard verified present and covered by a regression test. Reviewer Rejection Protocol was enforced: Holden nominated the original authors as fixers, so the coordinator reassigned N1/N2 to Leon and N3 to Isidore under strict lockout. Conformance suite (Pris) - all against real ORT 1.27.0: - broadcast, dynamic/symbolic dim, int32, multi-node chain (A+B)*C+D, MatMul 2-D, and a mixed graph that forces ORT to partition between our EP and its own CPU EP. Each asserts independently computed values. Validation: 82 adapter unit tests pass; cpu-plugin suite 16 passed, 0 failed, 3 ignored on a known device-descriptor lifetime bug. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixes a use-after-free that corrupted our advertised EP device and left ORT unable to find an allocator at Run time. BUG 1 and BUG 2 shared one root cause. 1. Use-after-free: OrtMemoryInfo was released immediately after EpDevice_AddAllocatorInfo, but ORT stores the raw pointer rather than copying it. Across repeated register/Run/unregister cycles the freed memory was reused, so the device descriptor decayed into garbage (DeviceType:-112 MemoryType:-85). ORT now owns the memory info via the EpDevice; we only release it on the failure path. 2. Wrong API: CreateCpuMemoryInfo is the pre-1.22 entry point and does not populate the OrtMemoryInfoDeviceType/OrtDeviceMemoryType fields that the EP device system reads, yielding DeviceType:64 MemoryType:28 and an allocator lookup miss at Run. Replaced with CreateMemoryInfo_V2 using explicit OrtMemoryInfoDeviceType_CPU / OrtDeviceMemoryType_DEFAULT. Reviewer Rejection Protocol enforced: Pris rejected the device-enumeration work, so the fix was reassigned from Nabil (original author, locked out) to Deckard. Validation against real ORT 1.27.0: - ort_loads_our_ep_and_runs_model: all stages pass (register -> GetEpDevices -> CreateSession -> Run -> correct outputs). - conformance_multiple_run_calls: passes; repeated cycles no longer corrupt the descriptor. - No regressions in the previously passing conformance tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rage
Leon found that validate_dims() was never actually called: read_inputs still
cast ORT dims with `d as usize`, so every ORT-supplied dimension bypassed
validation and a -1 dynamic sentinel would still have wrapped to usize::MAX.
The helper is now wired into read_inputs, closing the path for real.
Pris (conformance):
- conformance_two_sessions was a test bug, not an implementation bug:
EpDevice_EpName returns OrtEpFactory::GetName ("cpu_ep"), not the key
passed to RegisterExecutionProviderLibrary. Assertion corrected.
- stress_register_run_unregister_cycles: 25 full register/Run/unregister
cycles, 4x the former corruption threshold, locking in the OrtMemoryInfo
use-after-free fix so it cannot silently regress.
- conformance_matmul_batched_nd: [2,3,4]x[2,4,2] -> [2,3,2] through real ORT
dispatch, with hand-computed expected values.
- All three previously ignored tests now run.
Holden's final re-audit: YELLOW (may ship). N1/N2/N3 all RESOLVED with
evidence; the factory.rs ownership fix confirmed correct (ORT owns the
memory info on success, we release only on the failure path, so the paths
are mutually exclusive and double-free is impossible). Two LOW advisories
recorded as post-merge follow-ups.
Validation: 82 adapter unit tests pass; cpu-plugin suite 21 passed,
0 failed, 0 ignored, stable across repeated runs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Isidore (lib.rs): drop redundant mut on out_num; replace a type-ascribed panic! with a bare panic! statement so it is no longer a diverging sub-expression. Panic guards, null checks, and the fail-closed ORT API version check are untouched. Deckard (ep.rs): replace the 1usize as *mut OrtEp test sentinel with std::ptr::dangling_mut, the idiomatic form for an intentional non-null, never-dereferenced placeholder. Cosmetic lint, confined to test code; the pointer never crossed the ABI. cargo clippy --all-targets -D warnings is now clean and rustfmt is clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Roy: rewrote docs/EP_PLUGIN_EXPORT_PR.md against verified reality. The earlier draft was written mid-flight and understated the result - it still described conformance_two_sessions as ignored, the OrtEpDevice corruption as open, a mutex-poison cascade that no longer occurs, and Holden's verdict as pending. All corrected; every number re-observed before being quoted. Also documented honestly: end-to-end f16/bf16 remains unproven because the EP implements no GetKernelRegistry, so type-constraint metadata is never registered with ORT (our CPU kernels do implement f16/bf16; no fake test was written). CUDA EP is blocked on both halves - no toolkit/GPU on this host, and genuine adapter design work remains for allocators, streams, and device pointers crossing the ABI. Scribe: merged 34 decision drops into .squad/decisions.md, wrote 16 orchestration-log entries and the session log, and compacted Deckard's history. Two ABI contracts recorded prominently, since both caused real bugs here: - OrtMemoryInfo passed to EpDevice_AddAllocatorInfo must outlive the EpDevice; ORT stores the pointer and does not copy it. - OrtGraph*/OrtNode* must not be cached beyond Compile; copy what you need into owned Rust data. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ctive Standing directive from @justinchuby: push progress to origin frequently, open draft PR(s) early rather than waiting for completion, and convert to ready for review once implementation and validation are complete. Captured as a decision drop because this session demonstrated the failure mode it prevents: a lost coordinator session left the entire EP plugin export workstream uncommitted, and the recovered branch then accumulated local-only commits that were invisible from GitHub because no push credentials exist on this host. Also adds the corollary that a blocked push must be reported immediately and loudly rather than deferred to a final summary, since it silently breaks this contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…, f16/bf16 marshaling Milestone 2, stacked on squad/ep-plugin-export. Extends the outbound plugin adapter beyond the host-only CPU path. Device surfaces (Nabil) - crates/onnx-runtime-ep-plugin/src/device.rs: - device_type_to_ort_hardware maps our DeviceType to OrtHardwareDeviceType (Cuda/Rocm -> GPU, Qnn/CoreML/MLX -> NPU, Cpu -> CPU). - DeviceSupport declares which hardware types an EP serves, plus allocator name, vendor id, and stream-awareness. - DeviceAllocator projects ExecutionProvider allocate/deallocate through the OrtAllocator vtable; DeviceSyncStream projects sync() through OrtSyncStreamImpl. Both panic-guarded and null-checked. - Fail-closed validators for device mismatch, allocator type mismatch, and stream creation on a non-stream-aware EP. - ~26 unit tests driven by MockGpuEp/MockCpuEp doubles, so the device paths are exercised on a machine with no GPU. New crate crates/onnx-runtime-ep-cuda-plugin: feature-gated behind `cuda`, workspace member but not a default member, so `cargo check --workspace` still succeeds without a CUDA toolkit. Kernel registry (Deckard): GetKernelRegistry callback, OrtKernelRegistryHolder, build_ort_kernel_registry, and create_ep_factories_with_registry. Confirmed from the ORT header that GetKernelRegistry coexists with Compile rather than replacing it, so the working compile-based path is preserved. NEW-2 resolved (Deckard): ORT's ownership contract for OrtNodeComputeInfo on a mid-loop Compile failure is unspecified in the header, so ep_compile_inner now frees and nulls already-written entries. That is safe under both readings - if ORT calls ReleaseNodeComputeInfos it sees nulls, and if it does not, we have already freed. NEW-1 resolved (Leon): compute_release_state is now panic-guarded; it was the last unguarded extern "C" callback. f16/bf16 dtype mapping verified against the generated bindings (FLOAT16=10, BFLOAT16=16, 2 bytes each) and covered by the existing checked_mul overflow guards. Validation: 120 adapter unit tests pass (was 82); cpu-plugin conformance still 21 passed, 0 failed, 0 ignored. Known-broken at this commit, fixed in follow-ups: the new trait/C-ABI parity test does not compile (DataType::Float should be Float32; KernelMatch lacks Debug), and device.rs trips clippy's forget_non_drop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ts, prove trait parity Trait <-> C-ABI parity (Pris): 9 integration tests in crates/onnx-runtime-ep-plugin/tests/trait_cabi_parity.rs now compile and pass, giving §524's Rust-trait half real evidence rather than assertion. The committed parity rule holds, but the Declined set is smaller than assumed: ShapeInference::for_node can infer Squeeze (empty axes), ReduceMean, and Conv from input shapes, so those are NOT declined. The confirmed real declined case is opset>=13 Unsqueeze with axes as a data-dependent input. Tests were corrected to use that real example rather than the assumed one. Also fixed a latent test-harness bug: the graph builder never set opset_imports, so effective_opset() returned 0 and supports_op declined everything - which would have made the parity assertions vacuous. Device generalization (Nabil): factory.rs now filters via DeviceSupport::serves(hw_type) instead of a hard-coded CPU constant. IsStreamAware reads support.stream_aware; CreateAllocator builds a DeviceAllocator for device EPs and defers to ORT's default for CPU; CreateSyncStream builds a DeviceSyncStream for stream-aware EPs and fails closed otherwise. The OrtMemoryInfo ownership fix is preserved and still guarded by the 25-cycle stress test. The mem::forget flagged by clippy in device.rs was dead code, not an ownership bug: DeviceBuffer has no Drop, and the allocation lives in the EP's allocator until device_free calls ep.deallocate(). Type constraints (Deckard): build_cpu_registry_with_descriptors() in crates/onnx-runtime-ep-cpu/src/kernels/mod.rs derives supported dtypes by recording every register() call through the same path that builds the live OpRegistry, then classifying by the kernel's actual dispatch macro. This projects the Rust registry rather than hand-maintaining a list that could drift. Ops whose dtype support cannot be determined advertise f32 only - fail closed, so we never claim a dtype we cannot honor. Validation: 127 adapter unit tests, 9 parity tests, 21 cpu-plugin conformance tests. f16/bf16 end-to-end remains honestly ignored pending verification that ORT consults the kernel registry for compile-based EPs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… end-to-end f16 and bf16 models now execute through our EP inside upstream ONNX Runtime, verified with exact bit-pattern assertions: conformance_add_float16 -> [0x4000, 0x4400, 0x4600, 0x4800] conformance_add_bfloat16 -> [0x4000, 0x4080, 0x40C0, 0x4100] Both were previously ignored. This is our EP claiming and executing the nodes, not ORT falling back - the distinction was checked explicitly. Dtype-aware claiming (Deckard): ep_get_capability_inner now runs node_passes_dtype_filter() on every candidate node, rejecting any node whose input/output element types we do not support, whose dtype is Undefined, or whose op has no registry entry. Previously we claimed nodes without checking element type at all - the same over-claiming class already fixed for NonZero, where claiming denies ORT the chance to fall back. The claim predicate and the advertised type constraints consume the same Vec<KernelRegistryEntry> from one call to build_kernel_registry_entries(), so they cannot drift apart. Drift here would mean advertising a dtype we cannot honor, i.e. silently wrong results inside ORT. Test hardening (Pris): the unused `ep` binding flagged by clippy was hiding a weak assertion - error_parity_declined_shape_inference_is_cabi_only checked shape inference twice and never verified the trait/C-ABI divergence it documented. Rewritten to prove all three steps: the trait claims Unsqueeze, for_node returns Declined, and the C-ABI filter yields empty. Ownership cleanup (Nabil): the two remaining mem::forget sites were in #[cfg(test)] mock EPs and were no-ops - DeviceBuffer has no Drop, so the manual dealloc was already correct. Replaced with a discard binding. Holden independently confirmed this assessment. Security (Holden, milestone-2 audit): YELLOW - may ship. NEW-1 and NEW-2 both RESOLVED. One MEDIUM found: factory_create_sync_stream leaks the EP because stream_release drops only the DeviceSyncStream and never reclaims the Box::into_raw'd ep, unlike factory_release_allocator which does. Tracked as M2-1 and reassigned away from the author under strict lockout. Validation: 132 adapter unit tests, 9 trait/C-ABI parity tests, 23 cpu-plugin conformance tests (6 + 17), 0 ignored. cargo check --workspace clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…s and docs M2-1 (MEDIUM, Holden): factory_create_sync_stream handed out an EP via Box::into_raw, but stream_release only dropped the DeviceSyncStream and never reclaimed it, leaking a full ExecutionProvider per sync-stream creation inside a long-lived host process. stream_release now reclaims the EP with Box::from_raw behind a null guard. Double-free was ruled out from the header rather than assumed: onnxruntime_ep_c_api.h:207-216 specifies ORT calls OrtSyncStreamImpl::Release exactly once per stream, and no other path reclaims that EP (the allocator path owns a separate instance). New regression test stream_release_reclaims_owned_ep_no_leak uses an EP with a Drop counter and asserts the drop actually runs, so the leak cannot silently return. M2-2 (LOW): the DeviceAllocator::memory_info comment claimed "Owned; freed on drop" when the pointer is borrowed from ORT and must not be freed. Corrected - an inaccurate ownership comment is a real hazard here, since a lifetime misjudgement in exactly this area caused milestone 1's use-after-free. Reviewer Rejection Protocol enforced again: Holden found both in Nabil's file and explicitly recommended reassignment, so Leon owned the fix under strict lockout. Also: removed two needless borrows in ep.rs test helpers (Deckard, who confirmed the surrounding dtype-filter assertions are genuinely meaningful rather than vacuous), and updated the docs for both milestones. Final validation on this branch: - cargo clippy -p onnx-runtime-ep-plugin --all-targets -D warnings: clean - cargo test -p onnx-runtime-ep-plugin: 133 lib + 9 parity, 0 failed - cargo test -p onnx-runtime-ep-cpu-plugin: 23 (6 + 17), 0 failed, 0 ignored - cargo check --workspace: clean - rustfmt: clean Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…0ded Third and final accuracy pass. The previous draft still listed the clippy regression, M2-1 (EP leak in stream_release) and M2-2 (misleading ownership comment) as open pre-merge blockers; all three were resolved in 3ab0ded. Corrections: removed the pre-merge blocker paragraph, moved M2-1/M2-2 into resolved findings with Leon as fixer and the regression test as evidence, refreshed counts (133 lib + 9 parity = 142; 23 cpu-plugin, 0 ignored), and marked the engineer status table complete. Holden's milestone-2 verdict is retained as YELLOW with an explicit note that his findings are resolved but a re-verification pass was not separately run - deliberately not upgraded to GREEN on his behalf. Kept honest: the native nxrt dynamic ABI half of the extension contract is still unimplemented, and CUDA remains blocked on both missing hardware and real device-pointer/data-transfer work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Owner
Author
PR structure note (supersedes the two-stacked-PR recommendation in the description)Per @justinchuby's direction, this is a single draft PR tracking the whole EP-compatibility milestone, rather than the two stacked PRs recommended in the body above. Subsequent milestone commits will be pushed to this same draft. It will stay in draft until the broader EP compatibility milestone is complete and validated. The base branch Current state at
|
Push blocker cleared - gh is authenticated, so both squad/ep-plugin-export (M1, 9 commits) and squad/ep-plugin-parity-cuda (M2, 5 commits) are now on origin and no longer local-only. Delivered as a single draft PR (#762, base main) tracking the whole EP compatibility milestone, per @justinchuby's direction. This supersedes the two-stacked-PR recommendation in docs/EP_PLUGIN_EXPORT_PR.md; a PR comment records the deviation so reviewers are not misled by the description. Stays in draft until the broader milestone is complete: the native nxrt dynamic ABI is unimplemented, and CUDA is blocked on both hardware and real device-pointer/data-transfer work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
🔴 Benchmark Regression DetectedComparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).
Visual flags: Host infoWhat this cannot catch
|
…ring Advances the EP-compatibility milestone on three fronts. The nxrt half is committed as WIP because it has a real integration gap, documented below. Native nxrt ABI (Nabil) - crates/onnx-runtime-ep-nxrt-abi: - Exports NxrtNegotiate + NxrtCreateEpFactories with #[repr(C)] vtables projecting ExecutionProvider. - Version negotiation: major mismatch is a hard reject; minor is min(plugin, host) with plugin-newer-than-host rejected; every struct carries struct_size for forward compatibility; unknown capability bits are rejected via NXRT_CAP_KNOWN_MASK (fail closed). - Ownership made explicit rather than implied: host owns factory/EP/kernel/ allocator and must call vtable.release; borrowed pointers are valid only within the callback frame. This is the contract whose absence caused the OrtMemoryInfo use-after-free on the ORT side. - Panic containment on every extern "C" entry. 19 unit tests. nxrt host loader (Isidore) - crates/onnx-runtime-ep-nxrt-host: - dlopen + eager symbol resolution, version check, adapts a loaded plugin back to dyn ExecutionProvider. Library lifetime enforced structurally via Arc<Library> held by every derived object, so an EP cannot outlive its library. Negative paths return actionable errors. Device transfer (Leon) - crates/onnx-runtime-ep-plugin/src/transfer.rs: - Projects ExecutionProvider copy/copy_async/sync through ORT's OrtDataTransferImpl. Copy-direction matrix implemented with honest CanCopy: host<->device and same-device supported, cross-device and host->host declined rather than accepted-then-failed. - Device pointers are tagged non-host-accessible so the adapter never dereferences them on the host. CUDA wiring (Deckard) - the cuda-plugin now constructs a real CudaExecutionProvider behind the `cuda` feature and advertises GPU device support, instead of panicking. `cargo check -p onnx-runtime-ep-cuda-plugin --features cuda` passes here because cudarc uses dynamic loading. prefetch_lazy_weight is deliberately left as Ok(false): CudaWeightResidency has no try_without_eviction API and the standing directive forbids a prefetch that may evict. KNOWN INTEGRATION GAP (fixed in follow-up, do not read this as working): - onnx-runtime-ep-nxrt-host does NOT depend on onnx-runtime-ep-nxrt-abi. It carries a private duplicate abi_contract.rs expecting nxrt_abi_version / nxrt_create_ep, while the ABI crate exports NxrtNegotiate / NxrtCreateEpFactories. The two halves are incompatible. - crates/onnx-runtime-ep-nxrt-testplugin declares its own [workspace], so cargo check --workspace never builds it. Consequently the whole tree compiles only because the incompatible pieces are never linked together. §524's native ABI is NOT yet proven end to end. CUDA remains UNVALIDATED: no GPU or driver on this host. Compiling is not working, and nothing here should be read as a working CUDA EP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… native ABI Makes §524's native nxrt dynamic ABI real end to end. The previous commit shipped two halves that each compiled but were never linked; this connects them and proves the round trip. The gap (found by Pris and Roy, confirmed by inspection): - onnx-runtime-ep-nxrt-host had NO dependency on onnx-runtime-ep-nxrt-abi. It carried a private duplicate contract in abi_contract.rs expecting nxrt_abi_version / nxrt_create_ep, while the shipped ABI exports NxrtNegotiate / NxrtCreateEpFactories. - onnx-runtime-ep-nxrt-testplugin declared its own [workspace], so cargo check --workspace never built it. The tree therefore compiled green while nothing was connected - the "instruments that lie" failure mode this team has recorded. Resolution: Nabil's ABI is the single source of truth (it has the versioned struct_size forward-compat, major/minor negotiation, and NXRT_CAP_KNOWN_MASK fail-closed capability handling that §524 requires). - Isidore deleted abi_contract.rs; the host now depends on the ABI crate and implements the real negotiate -> create_factories -> create_ep -> release sequence, copying borrowed name pointers into owned Strings rather than retaining them. - Nabil shipped export_nxrt_ep_factories! plus override macros (NxrtNegotiateOverride / NxrtCreateFactoriesOverride) so deliberately broken plugins can be built for negative testing without hand-rolling a duplicate ABI - closing the hole that invited the duplicate in the first place. - Pris rebuilt the fixture on the shipped ABI and made it a real workspace member. Round trip now proven against the shipped ABI: negotiate -> create factories -> create EP -> query -> release, with Drop counters returning to zero. Negative paths all fail closed: incompatible major, plugin minor newer than host, unknown capability bits, missing symbol, missing library, factory error, zero devices, and panic containment. Two test-integrity fixes along the way: - Fixture resolution followed the cdylib into the shared workspace target/, honoring CARGO_TARGET_DIR and profile, with auto-build and a loud failure rather than a silent skip. - An ENV_MUTEX now serializes the tests that set NXRT_TEST_PANIC / NXRT_TEST_FACTORY_ERROR; without it a parallel run could contaminate the lifecycle test (Roy hit exactly this). Duplicate round-trip suites were collapsed into one authoritative suite so they cannot drift apart. CUDA remains UNVALIDATED. scripts/cuda_conformance_runner.sh correctly exits 2 (UNVALIDATED) on this GPU-less host rather than reporting success. Validation: clippy --all-targets -D warnings clean across all four EP crates; rustfmt clean; cargo check --workspace clean; nxrt-abi 30, nxrt-host 4+10, ep-plugin 154+9, cpu-plugin 6+17 - all passing, 0 failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…h PR doc CI (Rust quality, job 93646716235) failed with: NXRT_ABI: documented in EP_PLUGIN_EXPORT_PR.md but no crate reads it. NXRT_ABI is not an environment variable. It was matched out of the documentation filename docs/NXRT_ABI.md, which EP_PLUGIN_EXPORT_PR.md links to, because ENV_PATTERN could not distinguish a doc filename from a variable: \b(?:ONNX_GENAI|NXRT)_[A-Z0-9_]+\b This is a false-positive class rather than a one-off: any doc named NXRT_*.md or ONNX_GENAI_*.md trips the gate as soon as another doc links to it. Fix (Gaff): add a negative lookahead so a match immediately followed by a file extension is not treated as an env var. A real env var is never written with a .md suffix, so the gate keeps its full power. The module docstring records the exclusion and the reasoning so it is not "fixed" back later. Deliberately NOT done: adding NXRT_ABI to KNOWN_UNIMPLEMENTED. That list means "a real env var we knowingly do not read"; NXRT_ABI is not an env var, so listing it would be a false statement inside the honesty gate itself. Also not done: deleting the docs/NXRT_ABI.md links to dodge the checker, which would leave the landmine armed for the next doc. Gate re-proved rather than assumed: injecting a fake NXRT_FAKE_KNOB into a scratch doc still fails the check (exit 1); scratch file removed. Docs (Roy): swept EP_PLUGIN_EXPORT_PR.md and siblings for stale claims. Removed the push-blocked language (no GH_TOKEN / no SSH key / branches local-only / "a user must open the PR") - PR #762 has been open on origin since credentials landed. Removed "1 test failing", "9/10 passing", and "Pris's fixture isolation fix not yet landed"; the nxrt round-trip suite is 10/10 and stable. Updated stale SHAs 99560c8 and 3ab0ded to 087d348. CUDA status unchanged and still honest: UNVALIDATED, nothing has run on a GPU, and there is no self-hosted GPU workflow in this repo, so CI cannot validate it. scripts/cuda_conformance_runner.sh exits 2 (UNVALIDATED) here. Validation: - python3 .github/scripts/verify_documented_env_vars.py -> exit 0, "63 documented, 9 known-unimplemented, all accounted for" - python3 .github/scripts/verify_cuda_test_honesty.py -> exit 0, 385 tests/51 targets, 381 fail-loud, 0 passed on this no-CUDA host - cargo fmt --all -- --check clean; cargo metadata --locked resolves - clippy --all-targets -D warnings clean across the four EP crates - 230 tests passing, 0 failed Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Second failure in PR #762's Rust quality job. The env-var honesty gate now passes; this one was the "Run clippy on all offline crates" step. crates/onnx-runtime-ep-cpu/src/kernels/mod.rs:2293 used `delta >= 0 && delta < 50`, which trips clippy::manual_range_contains. It appears only as a warning in a plain local run, but the CI job sets RUSTFLAGS: "-D warnings" at job level, so it is a hard error there. Replaced with `(0..50).contains(&delta)`. Checked rather than assumed that the assertion is sound: `delta` is a static count difference between two in-memory registry maps built from compiled code - no wall clock, no I/O, no concurrency - so the bound cannot flake on a loaded runner. Worth confirming, since a timing-based assertion hidden behind a lint would have been a worse problem than the lint. Validation now runs the CI lane exactly rather than only the touched crates: - RUSTFLAGS="-D warnings" cargo clippy --locked --all-targets over all 31 offline-lane crates -> clean - same flags over the four EP crates -> clean - verify_documented_env_vars.py -> 63 documented, 9 known-unimplemented - verify_cuda_test_honesty.py -> 381 fail-loud, 0 passed on this no-CUDA host - cargo fmt --all -- --check -> clean CUDA remains UNVALIDATED; no GPU runner exists in this repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rtifact Fixes the five CI lanes (Fast Linux, Rust Windows ARM64, Rust coverage on Linux/Windows/macOS) failing at "Test Linux offline crates": Could not find libonnx_runtime_ep_cpu_plugin.so Searched: [".../target/debug/...", ".../target/release/..."] test result: FAILED. 2 passed; 4 failed The cpu-plugin tests hardcoded target/debug and target/release. They passed locally only because an earlier manual `cargo build` had left the .so in target/debug. Moving that file aside reproduced the CI failure exactly, so the suite was never reproducible from a clean tree - our headline "23 conformance tests passing" was an artifact-dependent false pass. That is the same instruments-that-lie class as the duplicated nxrt ABI, hiding inside the number we were reporting. Fix: extract crates/onnx-runtime-ep-cpu-plugin/tests/cdylib_resolve.rs as a shared helper used by both plugin_export_abi.rs and plugin_ort_e2e.rs, rather than duplicating the lookup a second time - duplicated resolution logic is what let these two paths drift apart to begin with. Resolution order: NXRT_CPU_PLUGIN_PATH override, then CARGO_TARGET_DIR with the active profile, then the workspace target dir, then auto-build via `cargo build -p onnx-runtime-ep-cpu-plugin`, panicking loudly if that build fails. It never silently skips. Also fixes a latent cause of the Windows and macOS failures specifically: the library name was hardcoded to .so, so those lanes could not have found the artifact under any search path. The helper now selects .so / .dylib / .dll via cfg!(target_os). Audited every other test that loads a built artifact by path: the nxrt round-trip suite already had this pattern, trait_cabi_parity loads none, and onnx-genai-bench's profile_native has an env override with a graceful skip. No other test can pass on a stale artifact. Verified from genuinely clean state (artifact deleted first, not a warm tree): 6 + 17 = 23 passing, with the cdylib rebuilt by the test run itself. Also clean: clippy -D warnings on the four EP crates, rustfmt, the env-var honesty gate, and workspace_test_packages verify. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… Windows
Fixes the Rust (Windows ARM64) lane failing at "Test cross-platform offline
crates" with 12 instances of:
error[E0308]: mismatched types
--> crates\onnx-runtime-ep-cpu-plugin\tests\plugin_ort_e2e.rs:227:13
| ep_path_c.as_ptr(),
| ^^^^^^^^^^^^^^^^^^ expected `*const u16`, found `*const i8`
ONNX Runtime's path-taking APIs use ORTCHAR_T, which is wchar_t (UTF-16,
*const u16) on Windows and char (*const c_char) on Unix. The tests built
paths with CString, which is Unix-only, so the ORT end-to-end suite has never
compiled on Windows - it only ever ran on Linux. Same shape as the stale
artifact bug: a suite quoted as green that was not green everywhere it
claimed to run.
Fix: crates/onnx-runtime-ep-cpu-plugin/tests/ort_path.rs provides OrtPathBuf,
which encodes to a NUL-terminated Vec<u16> via OsStrExt::encode_wide under
cfg(windows) and to a CString otherwise, exposing as_ptr() at the type the
bindings expect. Applied at all 12 path call sites - six
RegisterExecutionProviderLibrary and six CreateSession.
OrtPathBuf owns its buffer and every use is bound to a named local that
outlives the FFI call, so there is no as_ptr()-on-a-temporary dangling
pointer. That is the same ownership trap behind the OrtMemoryInfo
use-after-free earlier in this workstream.
Registration names are deliberately left as CString: those parameters are
char* on every platform, unlike the ORTCHAR_T paths.
Audited the rest: no other test passes a path into an ORT C API, and both the
nxrt round-trip and plugin_export_abi suites go through libloading, which
takes an OsStr and is already portable.
Windows compilation is UNVERIFIED locally - this host is Linux, and
`cargo check --target x86_64-pc-windows-msvc` fails in bindgen for lack of
Windows SDK headers. The Windows branch is type-correct by construction
against the generated bindings; CI must confirm it.
Linux validation, from a clean artifact state: 6 + 17 passing, clippy
-D warnings clean, rustfmt clean, and no regression in ep-plugin (154 + 9),
nxrt-abi (30), or nxrt-host (4 + 10).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fixes the last two CI failures in the cpu-plugin suite:
Missing model fixture: .../tests/fixtures/add_1x4/model.onnx
called `Result::unwrap()` on an `Err` value: PoisonError { .. }
.gitignore line 3 is `*.onnx`, and git check-ignore confirmed it matched
every fixture. `git ls-files` on tests/fixtures returned only
generate_fixtures.py: all 11 models existed on the developer's disk and in no
commit, so the ORT end-to-end suite was never runnable from a fresh clone.
This is the third instance this session of one failure class - stale build
artifact, Windows-only compile error, now untracked fixtures - each of which
was green locally and impossible in CI. The rule worth keeping: a test asset
that is not tracked by git does not exist in CI.
Fix follows the convention already in .gitignore, which carries explicit `!`
negations for the 15 .onnx fixtures the repo already tracks. Added one
negation per fixture (11 files, 44K total) so the broad `*.onnx` ignore keeps
protecting against committing real model weights.
The PoisonError was a cascade, not a second bug: the first test panicked
while holding the shared ORT lock, so unrelated tests then failed with a
misleading error and pointed at the wrong test. A lock_ort_ep() helper now
recovers via PoisonError::into_inner() and reports the real cause, so one
failure can no longer masquerade as several.
generate_fixtures.py was missing generators for 4 of the 11 committed models;
added them so the script is once again the complete source of truth and the
fixtures can be regenerated byte-identically.
Audited every *.onnx/*.bin/*.pb/*.npz under crates/*/tests against
git ls-files: no other untracked test asset exists.
Validation from a clean artifact state: cpu-plugin 6 + 17, ep-plugin 154 + 9,
nxrt-abi 30, nxrt-host 4 + 10 - all passing. clippy -D warnings clean,
rustfmt clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…bol checks portable
Fixes the three remaining non-Linux lanes (Rust Windows ARM64, Rust coverage
Windows, Rust coverage macOS arm64). All Linux lanes are already green.
BUG 1 - undefined behaviour that passed on Linux by luck.
plugin_export_abi.rs declared:
type ReleaseEpFactory = unsafe extern "C" fn(*mut OrtEpFactory) -> *mut OrtStatus;
but the real export returns nothing (onnx-runtime-ep-plugin/src/lib.rs:163),
because ReleaseEpFactory is void in the ORT ABI. Calling a void function
through a pointer typed as returning *mut OrtStatus reads whatever is in the
return register. On x86-64 Linux that garbage was null so the assertion
passed; on arm64 it is non-null, producing:
ReleaseEpFactory returned non-null status (error)
So the test was asserting on uninitialised memory and had never actually
verified anything. Corrected the type to the void signature and removed the
meaningless status assertion.
Audited every other extern "C" fn-pointer type in the test suites -
CreateEpFactories, CreateFn, GetApiBaseFn - all match their real exports. No
further mismatches.
BUG 2 - the L1 symbol checks used ELF-only tooling. `readelf` does not exist
on macOS or Windows, and `nm -D` is ELF-specific ("File format has no dynamic
symbol table" on Mach-O). Replaced with:
- l1_required_symbols_resolve: resolves both exports through libloading
(dlsym/LoadLibrary) on every platform, which is a stronger and more
portable assertion than parsing nm output.
- l1_no_symbol_leakage: keeps the "nothing else is exported" property via
`nm --dynamic` on Linux, skipping with an explicit message elsewhere rather
than silently passing or failing for a missing tool.
That is four instances this session of one failure class - stale artifact,
Windows-only compile error, untracked fixtures, and now a signature mismatch
- each green on Linux and impossible or wrong elsewhere.
Verified on Linux from a clean artifact state: 6 + 17 passing, ep-plugin
154 + 9, nxrt-abi 30, nxrt-host 4 + 10, clippy -D warnings clean, rustfmt
clean. macOS and Windows cannot be exercised on this host; CI must confirm.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The four tests added for build_cpu_registry_with_descriptors each constructed the full 166-op registry, and RecordingOpRegistry builds it a second time to record descriptors, so the suite performed roughly eight full constructions. They now share a OnceLock-backed fixture. Every assertion is preserved verbatim. Context: Rust (Windows ARM64) is the last red lane, failing spmd_adaptive_calibrated_decode_is_bit_identical_to_flat in matmul_nbits.rs - a file this branch does not touch. Its child subprocess printed its test header and then nothing at all: no assertion, no stderr, which is the signature of a killed or timed-out process rather than a failed check. The hypothesis was that our extra registry-building tests raised memory/CPU pressure enough to starve that child on the most constrained runner. Measured rather than assumed, and the numbers refute it: wall clock 32.20s -> 32.25s, peak RSS 254 MB -> 253 MB. So the consolidation is not the fix; the Windows ARM64 failure looks like a pre-existing flake on a constrained runner. Recorded honestly rather than claimed as resolved. The change is kept because building the registry once instead of eight times is correct regardless. Deliberately not done: touching matmul_nbits.rs or ignoring the SPMD test. Editing an unrelated, untouched test to turn CI green is the exact dishonesty this workstream has spent the session correcting. Recommend to its owner that the SPMD parity test gain a timeout and diagnostics - an empty-output failure gives no actionable information. Validation on Linux: ep-cpu green, clippy -D warnings clean, rustfmt clean, cpu-plugin 6 + 17, ep-plugin 154 + 9, nxrt-abi 30, nxrt-host 4 + 10. Windows ARM64 cannot be reproduced on this host; CI must confirm. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
9 tasks
Owner
Author
GPU validation is tracked in #768 — this PR stays draftConfirmed external hardware blocker: this repo has zero self-hosted runners ( #768 carries the exact prerequisites, the Status of this PREverything except GPU validation is green — all CI lanes pass, including Rust quality, Miri, Windows ARM64, macOS, and Linux coverage.
Marking ready-for-review is gated on #768 returning exit 0 with evidence. |
Owner
Author
CI baseline comparisonThis PR is being marked ready with red checks. That is deliberate, and here is the evidence that none of the red is caused by this branch. Every failing job shares one root causeAll five failing jobs — It fails identically on
|
| check | result |
|---|---|
EP crates (ep-plugin, ep-cpu-plugin, ep-cuda-plugin, nxrt-abi, nxrt-host) |
283 passed, 0 failed |
cargo clippy --all-targets -- -D warnings on all five EP crates |
clean |
cargo fmt --check |
clean |
cargo miri test -p onnx-runtime-ep-plugin |
173 lib tests pass |
The remaining workspace clippy errors are in onnx-runtime-cuda-memory, which I confirmed also fails on origin/main and which CI's clippy allowlist does not lint.
Reproducing the Rust quality job locally
Every step in that job passes here: check_publish_order, workspace_test_packages verify, check_platform_naming, check_dispatch_reachability, check_dispatch_manifest, check_feature_gate_coverage, verify_documented_env_vars, check_cross_compile.sh, cargo fmt --all --check.
One note on measurement method
cargo test --workspace fail-fasts and silently truncates its totals — it once reported "1555 passed" where the true figure was 4598. Every count above uses --no-fail-fast. Similarly, test binaries were rebuilt before counting; a stale binary reported 42/1 where a fresh build gave 41/2 on a sibling PR.
If a maintainer would rather this wait until main is green, say so and I will return it to draft.
…on/session logs - Merged 6 inbox drops (rachael, coco, isidore, gaff, freysa, iran) into decisions.md with 5 durable lessons from the PR #762 ready-for-review wave - Deleted processed inbox files; README.md preserved - Wrote orchestration logs for Rachael, Coco, Isidore, Gaff, Freysa (2026-08-12T08-00-00Z) - Wrote session log 2026-08-12T08-00-00Z-762-ready.md - Appended #762-ready entries to all five agent history files - Compacted Rachael (11→3 dated entries), Coco (9→2), Isidore (10→2) into history-archive.md - decisions.md: 24043→26588 bytes (well under 51200 limit, no archive needed) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Merged decisions/inbox/batty-32003-ptx.md into decisions.md - Appended PR #32003 narrative-fix + PTX lessons to decisions.md - Wrote orchestration-log/2026-08-12T08-30-00Z-batty.md - Wrote log/2026-08-12T08-30-00Z-32003-ready.md - Updated agents/batty/history.md with PR #32003 completion PR #32003 marked ready for review. 12/12 PTX pairs byte-identical. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion - decisions.md: merged 3 inbox drops (resch-32001-validation.md, holden-32001-focused.md, zhora-32001-dedupe.md); added PR #32001 wave entry with 5 durable lessons on copy-drift, dead code, parse-time validation, error messages, and body-vs-code agreement; 29 216 → 32 531 bytes - orchestration-log: 3 new entries (resch, holden, zhora) at 2026-08-12T09-45-00Z - log: 2026-08-12T09-45-00Z-32001-ready.md - history.md: appended #32001 update to resch, holden, zhora - history compaction: resch (12 → 4 live entries) and zhora (10 → 4 live entries) older entries moved to respective history-archive.md - inbox: deleted 3 processed drops; README preserved Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…NaN/QEMU evidence - decisions.md: merged 2 inbox drops (iran-31993-nan-runtime.md, sapper-31974-ab.md); added 5 durable lessons on QEMU evidence, NaN payload portability, compiles-vs-runs, undisclosed type changes, and verifying 'no-behaviour-change' claims - orchestration-log: iran and sapper entries for final-review-wave - log: session log 2026-08-12T10-15-00Z-final-review-wave.md - iran/history.md: compacted (8 dated entries → archive); 2026-07-27 through 2026-08-11 moved to history-archive-2026-08-12T10-15-00Z.md; live file now holds role summary + 2026-08-12 entries (5,415 bytes) - sapper/history.md: appended PrePack A/B + PR body correction entry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…arks, x86 wording Merged 3 inbox drops (batty-31973-n1.md, challenger-31973-delta.md, deckard-31973-wording.md) into decisions.md. Appended history updates for Batty, Challenger, and Deckard. Decisions recorded: - Gate arch-specific test assertions with a predicate mirroring the production #if; never restate thresholds as literals - Name and scope-bound benchmark baselines (Welford fp32 mirrors layer_norm_impl.cc; per-row in-process microbenchmark) - Readability edits can narrow accuracy (AMD64/IX86 → x86-64 → x86) - This host (AVX2/FMA) can measure x86 performance honestly Board: #762 ready · #31985 merged · #31973 ready · #31974 ready · #32001 ready · #32003 ready · #31993 draft · #31988 draft Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ories - decisions.md: appended #31973 evidence-accuracy wave (Mariette B1/B2, Gaff review, Coordinator correction); 5 durable lessons; also merged prior-wave inbox drops (coco-32001-lint, pris-31993-ctest, rachael-32001-crosstarget, holden-final-two) - agents/mariette/history.md: appended 2026-08-12 evidence-accuracy entry - agents/gaff/history.md: appended 2026-08-12 evidence-accuracy review entry - Pre-existing staged changes from prior wave also included (coco, holden, pris, rachael history updates) - decisions.md: 37050 → 40970 bytes (under 51200 gate; no archive) - No history compaction: both files < 8 dated entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…action - Merged 2 inbox drops (leon-31974-prepack, coco-31974-regression) into decisions.md - Added #31974 regression wave section with 4 durable lessons - Wrote orchestration logs for Leon and Coco (2026-08-12T14-00-00Z) - Wrote session log: 2026-08-12T14-00-00Z-31974-regression.md - Appended PR #31974 regression entry to leon/history.md and coco/history.md - Compacted leon/history.md (9 dated entries → 2 current + archive pointer) - Archived 7 older Leon entries to leon/history-archive.md - decisions.md: 40,970 → 43,278 bytes (under 51,200 gate; no archiving needed) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…m fixes Syncs with main (11 commits) so CI's automatic PR-merge preview builds against current upstream, and pulls in the test_support cfg-gating bug (#807) that this branch needs to fix forward (see follow-up commit). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cfg(test) does not propagate to integration test crates — they link the library as a regular dependency. The cfg gate `any(test, feature = "gpu-tests")` therefore never activates for tests/*.rs binaries unless gpu-tests is explicitly passed on the CLI. This broke CI's honesty-checker base (cuda-only) build for 7 of PR #762's CI jobs (Rust coverage x3, Rust ARM64, CUDA compile Linux, Rust quality, Fast Linux), all sharing the identical compile error at vmm_kv_layout_residency_gpu.rs:38. Fix: narrow the lib.rs gate to `feature = "gpu-tests"` and conditionally compile the import + body in the one integration test that uses TestStream. The test binary remains in inventory in both configurations (416 tests / 66 targets), and without gpu-tests the test is #[ignore]d with an empty body, matching this crate's existing cfg_attr(not(feature = "gpu-tests")) skip convention. The remaining 2 of the original 9 failed jobs (CLI ORT Linux/Windows) fail on an unrelated, pre-existing flaky assertion in registry::tests::failed_runtime_shrink_preserves_policy_and_ledger_limit (documented in PR #758 as pre-existing multi-threaded VRAM-ledger flakiness) and are out of scope for this fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Per task instructions this branch must not mutate .squad/ state files. The decision-record file added in afbd2c1 belonged to a sibling effort on squad/fix-cuda-memory-test-support-gate, not this PR; drop it here so this branch only carries the product-code CI fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
enabled auto-merge (squash)
August 12, 2026 15:07
This was referenced Aug 12, 2026
Merged
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
Adds a PyPI publish pipeline for the two ORT plugin-EP cdylibs introduced in #762. ## Packages - **`nxrt-ep-cpu`** (`import nxrt_ep_cpu`) — bundles `libonnx_runtime_ep_cpu_plugin.{so,dll,dylib}`. - **`nxrt-ep-cuda`** (`import nxrt_ep_cuda`, experimental) — bundles the CUDA plugin cdylib built with `--features cuda`, **CUDA 13** runtime deps (`nvidia-*-cu13>=13`). Hardware validation tracked in #768. Both expose `get_library_path() -> str` and `register(session_options=None)` for ORT's `register_execution_provider_library`. The cdylibs are pure C-ABI (`CreateEpFactories`/`ReleaseEpFactory`) and **do not link libonnxruntime** — the ORT API is passed through the plugin-EP ABI at load time, so the wheels carry no ORT dependency. ## Build setuptools + plain `cargo` (not maturin — the cdylib is a C-ABI lib, not a PyO3 extension). `build_py` runs `cargo build --release -p <crate>`, vendors the `.so`, and `bdist_wheel` tags it `py3-none-<platform>` (auditwheel/delvewheel repair to manylinux/win). Validated locally: CPU wheel builds, installs, `get_library_path()` returns a real file, and `register()` accepts it against onnxruntime 1.28. ## Workflow — `.github/workflows/publish-ep-plugins.yml` - **Trusted Publishing (OIDC)**, environment `pypi`, no API tokens. - `workflow_dispatch` inputs `publish_cpu` / `publish_cuda` / `testpypi`; also `push` tag `nxrt-ep-v*`. - **`cpu` and `cuda` jobs are fully independent** (no cross-`needs:`). PyPI blocks two pending trusted publishers from one repo at once, so `nxrt-ep-cpu` publishes first and `nxrt-ep-cuda` later — the CUDA job never touches PyPI unless `publish_cuda=true`. - `publish-cpu needs:[cpu]` / `publish-cuda needs:[cuda]`: a failed build blocks the corresponding publish. No publish happens on merge — releases are triggered manually via `workflow_dispatch`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
Fast-follow to #762. A test-review of the EP-plugin suite found **no product correctness bugs** but 3 places where a test claimed more than it proved. All fixed; every affected test runs and passes locally (ORT available — nothing skipped). ## 1. Tautological CUDA diagnostic test → asserts real output `cuda_fail_closed.rs::cuda_plugin_diagnostic_message_is_actionable` previously asserted against string literals defined inside the test body (always passed). Now the plugin exposes `fail_closed_diagnostic()` returning its *actual* fail-closed message (shared `NO_CUDA_FEATURE_DIAGNOSTIC` const, also used by `CreateEpFactories`), and the test asserts the real string contains the actionable substrings. ## 2. Six non-reproducible fixtures → generators restored `generate_fixtures.py` only reproduced 22 of 28 fixtures; the 6 missing were the BL1/BL2/BL3 regression models (opaque blobs). Added generators for all 6 and wired them into `__main__`. **All 28 fixtures now regenerate byte-for-byte identical** to the committed models (`git diff` empty). ## 3. Value-blind "no_overflow" tests → real oracles `layer_norm_{f16,bf16}_absent_output_no_overflow` asserted only non-null. They now decode `Y` and compare against a hand-computed LayerNorm oracle (tol 0.02 f16 / 0.05 bf16), matching the SkipLayerNorm f16/bf16 siblings. ## Validation `cargo test -p onnx-runtime-ep-cpu-plugin` (40 passed / 0 failed / 0 skipped) · `cargo test -p onnx-runtime-ep-cuda-plugin` (12 passed) · fixtures regenerate identical · `cargo fmt` + `clippy --tests` clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Aug 12, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…detail (#825) Justin asked to update `docs/PROGRESS.md` with the latest progress and drop stale info. It had grown to **1166 lines** of day-by-day changelog spanning 2026-07-15 → 2026-07-27, with the entire recent arc missing. ## Changes (1166 → 303 lines) - **Added recent milestones (2026-07-28 → 2026-08-12), newest-first:** - EP plugins run inside ORT via the plugin-EP C ABI (#762) and ship on PyPI — **`nxrt-ep-cpu` + `nxrt-ep-cuda` 0.1.0.dev5 both LIVE** (#819/#824/#820). - Absent-optional-output memory-safety wave (#762 review; Miri-clean). - VMM / KV-layout / offload / batching residency wave — managed no-spill VMM default, layout-governed KV floor (768× reduction), offload+capture coexist, #736 over-reservation audit (#755–#814). - Issue triage + autonomous fixes (#784/#785/#786/#788/#789); CPU-EP TopK correctness+perf (#774/#775). - Qwen3.5/3.6-27B hybrid Gated-DeltaNet native CUDA (#779); GLM-4-9B + DeepSeek-V2-Lite native + ORT 1.28 3-config fairness (#766/#770/#771). - Fused QMoE decode kernel + QMoE arc concluded; CUDA-graph capture trilogy, version-selectable CUDA (#708/#728/#757/#760), megakernel study (#769). - **Compressed stale detail:** the ~40 dated July day-by-day sections + verbose diffusion/LoRA/ORT2 narrative collapsed into a ~35-line "Foundational milestones" spine; superseded perf-wave numbers dropped; §1–§40 table condensed to a compact grid. - **Header refreshed:** last-updated → 2026-08-12; Published line now lists both EP plugins on PyPI + Miri CI. - **Preserved & updated** the DeepSeek/GLM known-gaps checklists and added new open items. Docs-only; full narrative remains in `.squad/decisions-archive/` and `docs/benchmarks/`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
## What Documentation-only sync of five memory/KV design docs to the merged, executable-test-backed facts about VMM commit geometry, KV layout, weight offload, CUDA-graph capture, and plugin export. No code or tests changed. `MEMORY_ARCHITECTURE.md` is treated as the single source for KV geometry and the audit table; the other four link into it rather than restating (per #791). ## Most misleading claim found `MEMORY_ARCHITECTURE.md` — the document explicitly designated the single source of truth — still asserted the engine *"explicitly invalidates the graph on every KV bucket growth for both layouts"* and that *"the engine does not yet use that fact to retain the captured graph,"* with a table showing seq-major at `captures=4, invalidations=4`. That was falsified by **#811**: conditional invalidation has landed, seq-major fixed-stride growth now **keeps** the captured graph (growth-attributable invalidations **3 → 0**, `captures=1/invalidations=1`, matching the no-growth control), verified by `mobius_seqmajor_growth_parity_native_cuda.rs` including a negative oracle. Because this doc is the single source others link to, the stale claim had the widest blast radius. ## Changes - **MEMORY_ARCHITECTURE.md** — correct the #811 conditional-invalidation prose + table (verified against `cuda.rs` `binding_growth_signature`/`record_growth_decision` and the parity test); add the ORT-backend KV-routing + #762 plugin-export note (verified `allocator.rs:232` `for_session_device` selects by device, not implementation); thread #822 (granule queried), #812 (engine-side consumer gate), and a #800/#802 record correcting this doc's earlier fp16 sizing of Phase-2a scores (verified `attention.rs:368` uses `dtype.element_size()`). - **CUDA_GRAPH_CAPTURE.md** — replace the mutual-exclusion result with #796 (stable VA removes it), #798 (managed no-spill default), #811 (conditional invalidation), and the #804/#807 `captures=0` process-artifact correction. - **WEIGHT_OFFLOAD.md** — correct the "mutually exclusive today" claim (#796) and the north-star status for the managed no-spill default (#798). - **KV_INSERTION_DESIGN.md** (most stale, 2026-07-15) — add a synchronization banner linking to `MEMORY_ARCHITECTURE.md`; note the layout line post-dates the memo, prefix sharing is a landed primitive (#777), and block-table paged attention remains a proposal (verified: no `block_table` in any CUDA attention kernel). - **DESIGN.md** — forward-pointer in the already-superseded §39.5 noting layout (seq-major/token-major, #787/#792) is now the primary residency lever. ## Method Every retained performance claim cites the PR that measured it; measured fact / design intent / not-yet-built are distinguished; superseded designs are labelled, not deleted. The "still open" item (engine commits flat buckets, not the dense live-prefix ranges `kv_commit.rs` computes, so the `layers×2` floor is not yet realised through the engine) is described as in-flight, not delivered. ## Note on #762 The task framing had #762 (EP plugin export) as open; `gh pr view 762` shows it **merged** today. It is described accurately per that ground truth: the plugin-EP mechanism landed for the CPU EP (validated end-to-end), while the CUDA plugin shim compiles but is unvalidated on hardware — so plugin-EP-hosted VMM KV remains in-progress design intent, not a shipped capability. ## Validation Documentation only — `git diff --stat` touches exactly the five `.md` files; working tree otherwise clean. **Not merged; no issue closed.** Refs #736, #762, #776, #777, #783, #787, #791, #792, #794, #796, #797, #798, #801, #802, #804, #807, #811, #812, #822 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…-up to #762's CI fix) (#818) ## Context PR #762 merged into `main` at `e9c0ab6a4` and already carries the fix for the CI-wide compile break introduced by #807: `crates/onnx-runtime-cuda-memory/src/lib.rs` gates `pub mod test_support;` on `#[cfg(feature = "gpu-tests")]` only (previously `#[cfg(any(test, feature = "gpu-tests"))]`), and the one integration test that uses `TestStream` (`tests/vmm_kv_layout_residency_gpu.rs`) cfg-gates its import and body accordingly. **Root cause recap:** `cfg(test)` is only set when a crate compiles as its own unit-test binary. Integration tests under `tests/*.rs` link the crate as a regular (non-`--test`) dependency, so `cfg(test)` never applies to them. The old gate `any(test, feature = "gpu-tests")` therefore never actually exposed `test_support` to `tests/vmm_kv_layout_residency_gpu.rs` unless `gpu-tests` was passed explicitly — which is exactly what broke CI's honesty-checker base (cuda-only, no `gpu-tests`) build for 7 of 9 failed jobs on #762's own CI run. ## What this PR does Since the functional fix already reached `main` via #762, **this PR is a minimal, standalone doc correction**: `test_support.rs`'s module doc still said the module was compiled "behind the `gpu-tests` feature (and the crate's own `cfg(test)`)" — stale wording that contradicts the actual (correct) gate in `lib.rs` and could mislead a future reader into re-introducing the same bug by assuming `cfg(test)` alone helps here. Updated the doc to explicitly state the gate is `gpu-tests`-only and explain why `cfg(test)` doesn't apply to integration tests. **No functional/behavioral change.** No `.squad/` state included. ## Delta vs. Opus-approved fix (afbd2c1 on #762) `git diff afbd2c1 origin/main -- crates/onnx-runtime-cuda-memory/src/lib.rs crates/onnx-runtime-cuda-memory/tests/vmm_kv_layout_residency_gpu.rs` is **empty** — main already has byte-identical content for both files via the #762 merge. The only delta introduced by this branch beyond what Opus already reviewed is the `test_support.rs` doc-comment correction (7 insertions / 3 deletions), which was never part of the reviewed diff. Flagging this so a tiny follow-up look is optional but available before merge. ## Validation (local, CPU-only host, matching CI's commands) - `cargo test --locked -p onnx-runtime-cuda-memory --features cuda --tests --no-run` — compiles clean (base/no-gpu-tests config) - `cargo test --locked -p onnx-runtime-cuda-memory --features cuda,gpu-tests --tests --no-run` — compiles clean (gpu-tests config) - `python .github/scripts/verify_cuda_test_honesty.py` — passes: 416 tests/66 targets identical in both configs (416 ignored without gpu-tests; 412 fail-loud/4 ignored/0 passed with gpu-tests on this no-CUDA host) - `cargo clippy --locked -p onnx-runtime-cuda-memory --features cuda --all-targets -- -D warnings` — clean - `cargo clippy --locked -p onnx-runtime-cuda-memory --features cuda,gpu-tests --all-targets -- -D warnings` — clean - `cargo check --locked -p onnx-runtime-ep-cuda -p onnx-runtime-python --features onnx-runtime-python/cuda` — clean - `cargo test --locked -p onnx-runtime-ep-cuda --features cuda --test capture_sync_contract` — 1 passed - `cargo clippy -p onnx-runtime-ep-cuda --features cuda -- -D warnings` — clean - `cargo fmt --all -- --check` — clean ## Notes - #762 (branch `squad/ep-plugin-parity-cuda`) temporarily carried this same cfg-gating fix directly on its own branch (to unblock its own CI) and has now merged into `main`, so `main`'s CI should already be green going forward. This standalone branch exists per explicit request so the repo-wide CI unblock has its own paper trail, separate from #762's larger diff. - Kept scope minimal: no CI workflow changes, no other files touched. - Leaving as **Draft** until CI is confirmed green on this PR's own run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 12, 2026
…823) ## Summary Cherry-pick of commit `dd6ab807f` which missed the #762 merge window by 30 minutes and is stranded on the already-merged branch. **This fixes CURRENTLY-RED lanes on main:** - Rust Windows ARM64 - Rust coverage Windows x86_64 - Rust coverage macOS arm64 The root cause: 7 hardcoded `libonnxruntime.so` joins in `plugin_ort_e2e.rs` that fail on Windows (`onnxruntime.dll`) and macOS (`libonnxruntime.dylib`). **Fix:** Route all ORT library path construction through `ort_discovery::ort_lib_name()` which returns the correct platform-specific filename. Also switches `diag_ort_ep_api_nullcheck` to use the `skip_if_missing!` macro. **Verification (Linux x86_64):** - `cargo test -p onnx-runtime-ep-cpu-plugin --no-fail-fast`: 24 passed, 0 failed, 0 ignored - `cargo clippy -p onnx-runtime-ep-cpu-plugin --all-targets -- -D warnings`: clean - No hardcoded lib names outside `ort_discovery.rs` **Windows/macOS correctness is BY CONSTRUCTION** (via `ort_lib_name()` returning the right filename per `cfg!(target_os)`) and must be confirmed by CI — we have no such hardware in this environment. Original commit: `dd6ab807f` on branch `squad/ep-plugin-parity-cuda` (PR #762, merged 2026-08-12T15:10:54Z; commit dated 30 min after merge). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 13, 2026
Merge 3 inbox drops into one "Decode latency-floor: node-collapse arc" section in decisions.md. Batch-1 native CUDA int4 decode is confirmed at its launch-amortized latency floor from three independent directions: megakernel NO-GO (#898), glue-collapse +0.9% ceiling (#899/#900), skip-RMSNorm fold -1.5% regression (#903). - #899 (Sebastian): glue node-collapse GO under graph replay (74-75%). - #900 (Batty): bf16 SiLU/SwiGLU-mul glue collapse SHIP (+0.9%, byte-exact, SiLU-fusion bf16 portability fix under Rule 11). - #903 (Sebastian): bf16 skip-RMSNorm KERNEL byte-exact SHIP; standalone FOLD -1.5% NO-SHIP (opt-in behind ONNX_GENAI_CUDA_ENABLE_SKIP_RMSNORM_FUSION, default OFF). Size gate: archived 2026-08-12 VMM/offload + #762 durable-lessons narratives to decisions-archive/2026-08.md to keep decisions.md under 50 KB (49,833 B). Appended cross-agent history for sebastian + batty; delete 3 processed drops. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EP Plugin Export — Rust Execution Providers as ORT Plugin EPs
What This PR Does
Exports our Rust execution providers through the ORT plugin-EP C ABI so upstream ONNX Runtime can load them via
RegisterExecutionProviderLibrary. The full ORT lifecycle —RegisterExecutionProviderLibrary→GetEpDevices→SessionOptionsAppendExecutionProvider_V2→CreateSession→Run— succeeds with numerically correct outputs for the CPU EP.New crates
onnx-runtime-ep-pluginExecutionProviderthrough ORT's C ABIonnx-runtime-ep-cpu-pluginCpuExecutionProvideronnx-runtime-ep-cuda-pluginCudaExecutionProvider(compiles; unvalidated on hardware)onnx-runtime-ep-nxrt-abi#[repr(C)]types, vtable definitions,export_nxrt_ep_factories!macroonnx-runtime-ep-nxrt-hostdlopen→ negotiate → factories)onnx-runtime-ep-nxrt-testpluginReview History — Six Rounds, Three Late-Breaking Blockers
This PR went through six independent adversarial review rounds. The first two established the basic adapter and conformance suite. Rounds 3–6 found and fixed three blocking defects that would have caused silent wrong results or undetectable fallback:
Blocker 1 — Optional slot fidelity (found round 3, fixed
6ce94f033–fbd565160)Omitted optional outputs were compacted with
filter_map, so canonicalSkipLayerNormalization(output,"","",sum)silently wrote mean into sum. Absent optional inputs aliased to input 0. Both now preserve positional slots; absence is explicit (NodeInputSource::Absent,TensorView::absent()).Blocker 2 — LayerNorm axis resolution (found round 3, fixed
168e40c3e)The raw axis was pre-resolved against a rank truncated by
filter_map(as_static)which drops symbolic dims —[B,S,H]became[H]. The raw axis is now preserved and resolved at runtime against actual input rank.Blocker 3 — Absent-output representation (found round 5, fixed
38625fb38)An earlier fix used a name-based sentinel (
__absent_output_*) — forgeable from model content. Replaced with an out-of-bandHashSet<ValueId>;ValueIdis an arena index the reader assigns, so model content cannot forge membership.Additional fixes since round 2
Vec<Option<usize>>, so length always equals rank;build_convfails closed when a required dim is unknown.unwrap_or(DataType::Float32)fallbacks deleted — short dtype vectors now fail closed.94bbbe545):NxrtStatuscode carried asu32with a checked conversion (unknown enum discriminant from a foreign.sowas UB);struct_sizehonoured before vtable access; CUDA diagnostic's status now survives to ORT.Test Story — The Big Change
Previously a passing test could not distinguish "our EP ran it" from "ORT's built-in CPU EP ran it" — and in one case a shipped fix turned out to be dead code for exactly that reason.
How assignment is now proven
session.disable_cpu_ep_fallback=1— forbids ORT's silent fallback to its own CPU EP. If our EP cannot handle an op, the session fails instead of silently succeeding through a different code path.Session_GetEpGraphAssignmentInfo(ORT ≥ 1.24) — directly queries which EP was assigned which ops. 14 tests now assert specific ops are owned bycpu_ep:Verified falsifiable: asserting an unclaimed op yields
Expected op 'Relu' assigned to cpu_ep, but assignment was: [("cpu_ep", "Add")].Verified Test Numbers
EP crates: 269 passed, 0 failed across
ep-plugin,cpu-plugin,cuda-plugin,nxrt-abi,nxrt-host.All five EP crates are clippy-clean under
-D warningsandcargo fmt --checkclean.Workspace (
cargo test --workspace --no-fail-fast): 4580 passed / 20 failed / 436 ignored. The 20 failures are pre-existing on base675b697bc— acrossonnx-genai-server,onnx-runtime-session,onnx-genai-engine(base: 910 passed / 21 failed; head: 911 / 20). No EP crate is among the failures. Note: plaincargo test --workspacefail-fasts and silently truncates totals.~30 dead-code clippy warnings in
crates/onnx-genai-benchare pre-existing at base and untouched by this PR.What Is NOT Proven — CUDA Hardware
The CUDA EP compiles and type-checks; it is unvalidated on hardware. There is no GPU and no CUDA toolkit on the CI host, and no self-hosted GPU runner exists.
onnx-runtime-ep-cuda-pluginfails closed by design — returns 0 factories when no GPU is available, with an actionableOrtStatus.Architecture
ORT Plugin Adapter (
onnx-runtime-ep-plugin)export_ep_factories!macro generatesCreateEpFactories/ReleaseEpFactorywith fullcatch_unwindpanic containmentAtomicPtrfor ORT API storage (nostatic mut)Declinedfor unresolvable shapesCreateState/Compute/ReleaseStatewith topological kernel executionDeviceAllocator,DeviceSyncStream,DeviceDataTransferFull— all projecting through ORT's vtable ABICUDA Ownership Model (B1 fix)
Components hold
Arc<Mutex<Box<dyn ExecutionProvider + Send>>>clones viaEpRef::Shared. Each use locks briefly for the EP method call duration. No raw pointers fromMutexGuards.Native nxrt ABI (
onnx-runtime-ep-nxrt-abi)NxrtNegotiate/NxrtCreateEpFactories— major/minor version negotiationstruct_sizeforward-compatibility on all vtablesNxrtStatus(inline[u8; 256]buffer — no cross-module free)export_nxrt_ep_factories!macro with panic containmentSecurity
Holden's audit verdict: 🟡 YELLOW — May ship. All CRITICAL/HIGH findings resolved. Two LOW advisories recorded for post-merge. See
docs/EP_PLUGIN_EXPORT_SECURITY_AUDIT.md.Files Changed
55 commits; primary changes in
crates/onnx-runtime-ep-plugin/,crates/onnx-runtime-ep-cpu-plugin/,crates/onnx-runtime-ep-cuda-plugin/,crates/onnx-runtime-ep-nxrt-abi/,crates/onnx-runtime-ep-nxrt-host/,crates/onnx-runtime-ep-nxrt-testplugin/, anddocs/.