test(inference): require Rust-owned observed information

[seonghobae]

Production numerical-ownership gap

Advances issue #625 from protected main 7b57f77747b9a9bd911660c76b1be0ca0dc2d145. Public observed_information() still constructs the dense finite-difference Hessian, off-diagonal stencil and final symmetrization directly in Python, and public second_order_test() still symmetrizes/eigendecomposes the information matrix in NumPy, even though covariance inversion and SE diagonal reduction are already Rust-owned on protected main.

Exact fail-first RED evidence

Exact source head: d80c26ed715c942a30530b79580bf94a56a84818, directly based on protected main 7b57f77747b9a9bd911660c76b1be0ca0dc2d145.

CI run 31492628277 checked out synthetic integration d934f37b67e8130b3ee75771d72a120cdc3fc1e5 (d80c26e... into 7b57f77...). Both CPython lanes built the Rust/PyO3 extension and verified Rust as the resolved production backend before executing the full suite. Python 3.12 completed 2 failed, 3282 passed, 2 skipped; the only failures are the two intentional ownership tests in tests/test_observed_information_rust_ownership.py:

• test_public_observed_information_delegates_matrix_to_rust: compiled-core sentinel call count is 0, proving public Hessian construction remains Python-owned;
• test_public_second_order_diagnostic_delegates_to_rust: compiled-core sentinel call count is 0, proving public positive-definiteness/eigenvalue diagnostics remain Python-owned.

The pre-existing _log_sigmoid RuntimeWarning is tracked by #583/#564 and is not owned by this slice. Rust workspace/PyO3, package/wheel/reinstall/release acceptance, enterprise sales-readiness smoke, GPU-no-skip and fuzz jobs are GREEN. Exact-head Security Scan 31492628299 and SAST Semgrep 31492628444 are GREEN. This is valid runtime RED after normal package/core setup, not setup/import/fixture failure.

Minimum GREEN

1. Move the current dense central finite-difference observed-information construction into mlsirm-core, including diagonal/off-diagonal stencil evaluation and final symmetry semantics that affect the returned matrix.
2. Move public positive-definiteness/eigenvalue/second-order diagnostic arithmetic into mlsirm-core; Python must not retain an independent eigvalsh production path for the governed diagnostic.
3. Reuse existing Rust objective/gradient/model primitives where the estimand matches; do not create a second formula implementation merely to cross the boundary.
4. Expose the thinnest PyO3 entrypoints. Python owns bounded input/config validation, immutable marshalling and result transport only; missing/incompatible core capability fails closed rather than silently retaining Python Hessian/eigenvalue arithmetic.
5. Preserve the current penalized negative-log-likelihood estimand, CPU-f64 default, step contract, parameter packing/order, symmetric-result contract and tolerance/error semantics unless a separately reviewed scientific correction establishes a defect.
6. Replace the nominal 5,000-parameter dense ceiling with evidence-backed resource limits before production acceptance: dense Hessian memory and finite-difference objective-call count are O(p²). Benchmark representative dimensions and fail closed before impractical allocation/evaluation work.
7. Add direct Rust↔public parity/property tests over ordinary, symmetric, singular/near-singular, indefinite, extreme-scale and malformed cases, caller immutability and stable non-reflective errors. For uncertainty claims add simulation/coverage or SE-calibration evidence appropriate to the estimator/design; algebraic parity or correlation alone is insufficient.
8. Keep CPU f64 as reference. Benchmark before GPU; any GPU path requires measured material benefit and parity.
9. Add complete rustdoc/docstrings, APA 7 equation/source traceability, authoritative changelog material only after GREEN, exact changed-production statement/branch coverage, and one unchanged final head with all applicable Python 3.12/3.14, Rust/PyO3, package/reinstall, GPU-existing, fuzz, Security Scan, SAST and current-head review gates.
10. Canonical Architecture/TRD/ADR/UML/ERD/traceability remains owned solely by docs: establish canonical product and architecture baseline #604/docs: close canonical documentation fitness gaps without parallel authority #621 after implementation stabilizes; do not create a parallel documentation authority.

The existing Oakes information implementation remains a distinct Rust-owned MMLE path and is not changed by this first JMLE/general observed-information ownership slice.

Keep Draft through RED→GREEN. Advances #625.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4abea10d-8699-4ec9-ae70-47a4449234d3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

@opencode-agent address

Exact bounded GREEN implementation handoff for Draft #734 / issue #625. Final pre-write guard is mandatory: refetch source head d80c26ed715c942a30530b79580bf94a56a84818, live protected main 7b57f77747b9a9bd911660c76b1be0ca0dc2d145, comments/reviews/threads and exact inference.py / Rust core / PyO3 target blobs; abort without writing if either ref moved or another writer targets this branch/paths. Preserve the two-commit fail-first history; no force push, destructive rebase, temporary writer workflow or competing documentation branch.

Exact RED is proven by CI 31492628277, synthetic integration d934f37b67e8130b3ee75771d72a120cdc3fc1e5. Both CPython lanes built the Rust/PyO3 extension and proved Rust-primary backend resolution. Python 3.12 completed 2 failed, 3282 passed, 2 skipped; the ONLY failures are the two ownership tests, both because the proposed compiled-core entrypoint was called zero times. Rust workspace/PyO3, package/wheel/reinstall/release acceptance, enterprise sales readiness, explicit GPU-no-skip and fuzz are GREEN; Security Scan 31492628299 and SAST 31492628444 are GREEN. The marginal _log_sigmoid warning is #583/#564 and must not be changed here.

Implement the minimum root-cause GREEN only:

1. Move dense central finite-difference observed-information/Hessian construction into mlsirm-core, including diagonal/off-diagonal stencil evaluation and final symmetry semantics that determine the matrix. Preserve current packed-parameter ordering, current penalized negative-log-likelihood estimand and step behavior unless a separate test-first scientific defect proves otherwise.
2. Move public second-order positive-definiteness/eigenvalue arithmetic into Rust as well. Python must not retain an ordinary production np.linalg.eigvalsh/symmetrization owner for this governed result.
3. Expose thin PyO3 entrypoints matching the ownership tests (_core.observed_information, _core.second_order_test or an equivalently reviewed public-core registration adjusted test-first). Python is limited to bounded validation, immutable marshalling/config resolution and typed result transport. Missing/incompatible core capability fails closed; do not retain the Python numerical path as production fallback.
4. Reuse existing Rust objective/model/packing helpers where mathematically identical; do not create a parallel formula implementation merely to cross the language boundary. Keep Oakes MMLE information separate.
5. Replace/reassess the nominal 5,000-parameter dense allowance from measured resource evidence. Dense memory and finite-difference evaluations are O(p²); benchmark representative p and define an evidence-backed fail-closed budget before impractical allocation/evaluation. CPU f64 is reference; do not add GPU without measured material benefit and parity.
6. Add Rust unit/property tests and direct public↔Rust parity for ordinary, symmetric, singular/near-singular, indefinite, extreme-scale and malformed matrices, caller immutability and stable non-reflective errors. Add appropriate simulated uncertainty/SE coverage calibration evidence where public inference interpretation depends on it; algebraic parity/correlation alone is not validity.
7. Add complete rustdoc/docstrings and primary-source/equation APA 7 doctoring for the finite-difference information/second-order inference contract. Add authoritative changelog fragment only after GREEN, then render/check managed CHANGELOG.
8. Run focused ownership/parity/resource/recovery tests, exact changed-production 100% statement/branch coverage and public docs, then full Python 3.12/3.14, Rust/PyO3, package/reinstall, existing GPU, fuzz, Security Scan, SAST and current-head review gates on one unchanged final head. Keep Draft while any gate/finding remains.
9. Cross-cutting Architecture/TRD/ADR/UML/ERD/traceability belongs only to canonical docs: establish canonical product and architecture baseline #604/docs: close canonical documentation fitness gaps without parallel authority #621 after this implementation stabilizes. Do not edit that authority from this branch.

Stop writing immediately if head/base/lease state moves.

[seonghobae]

@opencode-agent address

The prior GREEN handoff 5253417035 is stale because protected main advanced from 7b57f77747b9a9bd911660c76b1be0ca0dc2d145 to 914127ba227d3e02d0564aeeb4f27d76137610f9. That handoff required abort on base movement; do not write from predecessor-base assumptions or transfer its synthetic-merge/check evidence.

Fresh bounded lease for Draft #734 / issue #625: source head remains d80c26ed715c942a30530b79580bf94a56a84818; live main is 914127ba227d3e02d0564aeeb4f27d76137610f9. Before any write refetch both refs, comments/reviews/threads and exact inference.py / Rust / PyO3 blobs, and abort if anything moved or another writer owns the paths.

First reconcile current protected main non-destructively while preserving the intentional two-test ownership RED and all accepted behavior through #733. Recreate installed-package RED on the reconciled exact head; setup/import/fixture/changelog failure is invalid. If observed_information() still constructs the finite-difference Hessian in Python and second_order_test() still performs NumPy eig/PD diagnostics, proceed with the existing bounded minimum GREEN: move result-affecting Hessian stencil/symmetry and second-order eigen/PD arithmetic into mlsirm-core, expose thin PyO3 transport, leave Python validation/marshalling/reporting only, fail closed without compatible Rust capability, preserve estimand/step/packing/tolerance semantics, and replace nominal dense limits with measured O(p²) resource bounds. Add direct parity plus appropriate uncertainty calibration/recovery, doctoring/changelog after GREEN, and one unchanged final head with full Python 3.12/3.14, Rust/PyO3, package/reinstall, existing GPU/fuzz, Security, SAST, coverage/docstrings and review gates.

Keep Draft. Do not widen into estimator redesign, providers, dependencies/workflows/version/release, hosted state or canonical #604 docs.

[seonghobae]

Superseded by clean current-main Draft #757.

Fresh lineage proof before closure:

• protected main: 8a56d7b1b024e18a5a96a94db4ede1bf6648ead1;
• test(inference): require Rust-owned observed information #734 exact head d80c26ed715c942a30530b79580bf94a56a84818 is diverged ahead 2 / behind 30 from current main;
• test(inference): require Rust-owned observed information #734's only unique delta is tests/test_observed_information_rust_ownership.py;
• its prior OpenCode handoff was explicitly bound to old main 7b57f77747b9a9bd911660c76b1be0ca0dc2d145 and therefore became stale when main moved;
• clean replacement test(inference): refresh Rust observed-information ownership RED on current main #757 exact head 9938913de69cefb4c40ca9af3569d6f55be98ed7 is a direct one-commit descendant of current protected main;
• predecessor and replacement test files have identical blob SHA edd2e9be1cce3fcb5b23e098c234f7e0d1640619.

No #734 checks/reviews/approval/base-sensitive evidence transfers. #757 is now the sole current-lineage RED/implementation authority for the observed-information and second-order numerical-ownership slice. Closing #734 drops no unique value.