chore: improve embedded native-runtime compatibility guidance - #1043
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughEmbedded serving now checks for a compatible cached native runtime before startup, loads it when available, and reports version, ABI, and cache requirements when missing. The SDK exposes initialization and compatibility helpers, while README guidance and tests document and validate the installation sequence. ChangesEmbedded native runtime readiness
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant SDKConsumer
participant start_embedded_node
participant prepare_embedded_native_runtime
participant RuntimeCache
SDKConsumer->>SDKConsumer: Check and install required runtime
SDKConsumer->>start_embedded_node: Start embedded serving
start_embedded_node->>prepare_embedded_native_runtime: Validate embedded runtime readiness
prepare_embedded_native_runtime->>RuntimeCache: Load compatible cached runtime
RuntimeCache-->>prepare_embedded_native_runtime: Runtime loaded or unavailable
prepare_embedded_native_runtime-->>start_embedded_node: Continue startup or return error
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review. |
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/mesh-llm-sdk/README.md`:
- Around line 96-101: Update the runtime selection example around
matching_runtime to resolve compatibility using the same host-profile and
RuntimeSelection checks as startup, rather than only
native_runtime_versions_match_current_sdk. Prefer install_native_runtime for
explicit compatible resolution when appropriate, ensuring unavailable
platform/backend libraries trigger installation before proceeding to start.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 238b2c9a-e587-4f85-b6ac-d48d920d21b2
📒 Files selected for processing (9)
crates/mesh-llm-embedded-runtime/src/lib.rscrates/mesh-llm-host-runtime/src/sdk.rscrates/mesh-llm-host-runtime/src/sdk/embedded_startup.rscrates/mesh-llm-host-runtime/src/sdk/native_runtime.rscrates/mesh-llm-host-runtime/src/system/native_runtime.rscrates/mesh-llm-runtime-install/src/lib.rscrates/mesh-llm-sdk/README.mdcrates/mesh-llm-sdk/src/lib.rscrates/mesh-llm-sdk/tests/native_runtime_guidance.rs
ndizazzo
changed the title
86cf9c1 to
acbf2ab
Compare
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/mesh-llm-sdk/README.md (1)
79-82: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDeclare the dependencies used by this example.
The runnable example uses
tokio::main,anyhow::Result, andanyhow::ensure!, but theCargo.toml dependency block only declaresmesh-llm-sdk. Add compatibletokioandanyhow` dependencies so the documented setup compiles as written.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/mesh-llm-sdk/README.md` around lines 79 - 82, Add compatible tokio and anyhow entries to the README example’s [dependencies] block alongside mesh-llm-sdk, enabling tokio::main, anyhow::Result, and anyhow::ensure! used by the runnable example. Preserve the existing mesh-llm-sdk dependency and serving feature.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@crates/mesh-llm-sdk/README.md`:
- Around line 79-82: Add compatible tokio and anyhow entries to the README
example’s [dependencies] block alongside mesh-llm-sdk, enabling tokio::main,
anyhow::Result, and anyhow::ensure! used by the runnable example. Preserve the
existing mesh-llm-sdk dependency and serving feature.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 463f81bd-f865-4382-a872-56aa12463f71
📒 Files selected for processing (9)
crates/mesh-llm-embedded-runtime/src/lib.rscrates/mesh-llm-host-runtime/src/sdk.rscrates/mesh-llm-host-runtime/src/sdk/embedded_startup.rscrates/mesh-llm-host-runtime/src/sdk/native_runtime.rscrates/mesh-llm-host-runtime/src/system/native_runtime.rscrates/mesh-llm-runtime-install/src/lib.rscrates/mesh-llm-sdk/README.mdcrates/mesh-llm-sdk/src/lib.rscrates/mesh-llm-sdk/tests/native_runtime_guidance.rs
🚧 Files skipped from review as they are similar to previous changes (8)
- crates/mesh-llm-sdk/src/lib.rs
- crates/mesh-llm-embedded-runtime/src/lib.rs
- crates/mesh-llm-host-runtime/src/sdk/native_runtime.rs
- crates/mesh-llm-runtime-install/src/lib.rs
- crates/mesh-llm-host-runtime/src/sdk/embedded_startup.rs
- crates/mesh-llm-sdk/tests/native_runtime_guidance.rs
- crates/mesh-llm-host-runtime/src/system/native_runtime.rs
- crates/mesh-llm-host-runtime/src/sdk.rs
i386
left a comment
i386
left a comment
There was a problem hiding this comment.
The new native-runtime example is not self-contained as written: its dependency block only adds mesh-llm-sdk, while the code uses #[tokio::main], anyhow::Result, and anyhow::ensure!. Copying it into a fresh crate therefore fails to resolve tokio and anyhow. Please include the required tokio (runtime + macros) and anyhow dependencies, ideally with a compile test for the full example.
acbf2ab to
c1f4644
Compare
* origin/main: Fix Metal small-batch matmul parity for GLM verification (#1078) Handle K-only transposed KV page import and export (#1084) Refresh llama.cpp upstream patch queue (#1085) chore: improve embedded native-runtime compatibility guidance (#1043) fix(console-ui): chat transcript snapping during live status updates (#1083) ci: bump Linux CUDA slim container to gha-convention base runner image fix: record activation cache prefix identities (#1041) fix: read-only model download caches (#1042) ci: disable sccache for Windows ROCm native runtime build (#1087) ci: fix v0.74 release GPU builds (sccache disk-only + force_hosted_runners) (#1086) Make release sccache failures non-fatal (#1079) Keep client-only nodes out of model election (#1074) # Conflicts: # crates/mesh-llm-host-runtime/src/runtime/auto_join.rs # crates/mesh-llm-host-runtime/src/runtime/tests/auto_join.rs # third_party/llama.cpp/patches/0004-Add-lanes-external-media-and-chat-grammar-support.patch
2 participants
Closes #1016
What changed
Why
Embedded SDK users could previously reach a low-level FFI failure when the process had no compatible native runtime loaded. The shipped binary performs native-runtime loading during startup, but embedded serving did not make that prerequisite clear or fail early with corrective guidance.
Validation
Summary by CodeRabbit