Repository navigation
refactor(llm): modularize dynamo-llm into leaf, transport, protocol, model, and worker crates - #101
Closed
ishandhanani wants to merge 48 commits into
Closed
ishandhanani wants to merge 48 commits into
ishandhanani wants to merge 48 commits into
Conversation
`lib/llm/src/kv.rs` and `lib/llm/src/kv/` (5.5k lines) have had no `mod kv;` declaration in `lib.rs`, so none of it has compiled; the last touch was the 2026 copyright sweep. Remove it and drop its CODEOWNERS area entries (CODEOWNERS regenerated). Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
`dynamo_llm::perf` (stream recorder + logprob sensitivity analysis, 2.2k lines) had a single consumer: `tests/logprob_analysis_integration.rs`. Move it to `lib/llm/tests/common/perf/` and include it from that test with `#[path]`, so the production crate stops shipping test tooling. Its unit tests still run as part of the integration test binary. CODEOWNERS regenerated for the removed `lib/llm/src/perf/` entry. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… UI optional Four dependency groups were unconditional in `dynamo-llm` although each is used from a single site: - `dialoguer` (interactive text input) -> feature `text-repl`. The `Input::Text`/`Input::Stdin` arms bail without it, and the engine preparation used only by that input moves from `input/common.rs` into `input/text.rs`. - `aws-config`, `aws-credential-types`, `aws-types`, `object_store` (S3 LoRA source) -> feature `lora-s3`. The S3 source and its tests move to `lora/source/s3.rs` behind one `cfg`. The existing `request-trace-s3` feature now owns its `object_store` dependency instead of relying on it being always present. - `modelexpress-client` (ModelExpress server download) -> feature `modelexpress`. `hub::from_hf` falls back to the direct download path, which uses `modelexpress-common` and stays unconditional. - `utoipa-swagger-ui` -> feature `swagger-ui`. `/openapi.json` is always served; `/docs` only with the feature. All four are in `default`, so default builds are unchanged. Consumers that build `dynamo-llm` without default features drop the dependencies; the vLLM sidecar, which constructs `S3LoRASource`, opts into `lora-s3`, and the non-Linux Python bindings keep their previous surface. Validation: `cargo check -p dynamo-llm --all-targets` with and without default features, `cargo check` of the sidecars, backend-common, sidecar testkit and ext-proc, `cargo clippy -p dynamo-llm --all-targets -- -D warnings`, `cargo metadata --locked` on all four lockfiles. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
`AdmissionRequest for PreprocessedRequest` and `AdmissionResponse for LLMEngineOutput` lived in the wire-type modules and reached back into `admission_gate`. Both read only public fields, so they move to `admission_gate.rs`, where the traits are defined. The wire modules no longer depend on the worker admission gate. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
`RequestTracker::observe_first_token_gauges` and `observe_finish_gauges` wrote the `WORKER_LAST_*` Prometheus gauges, which made the request timing module depend on `http::service::metrics`. They become free functions `observe_worker_first_token_gauges` and `observe_worker_finish_gauges` next to the gauges they write, using only the tracker's public getters; their four tests move with them. The timing module keeps no HTTP or metrics dependency. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
`SamplingOptionsProvider`, `StopConditionsProvider`, and `OutputOptionsProvider` are frontend adapter traits implemented by blanket impls over the OpenAI request traits in `protocols::openai`. A blanket impl requires the trait to be local to this crate, so they cannot travel with the wire types they produce. Move them to `protocols::providers`; `protocols::common` re-exports them so no caller changes. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…e config `validate_model_runtime_config` compared `tool_call_parser` against the parsers linked into this build (`dynamo_parsers` registry plus the v2 unified families), tying the serde-level config type to the frontend's parser crates. The structural validator keeps everything else; `ModelRuntimeConfig::validate_tool_call_parser` takes the supported names from the caller, and `dynamo_llm::local_model::validate_runtime_config` performs both checks with `supported_tool_call_parsers()`. The in-crate registration path and the Python bindings use the combined check, so behavior is unchanged; the parser test moves to `local_model`. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…as plain config `ModelDeploymentCard::media_decoder` and `media_fetcher` held the frontend's runtime types: `MediaDecoder` (whose `ImageDecoder` carries the process-local libjpeg backend choice, `#[serde(skip)]`) and `MediaFetcher` (which owns the HTTP policy methods). A wire type should not carry decoder state, so the card now holds `MediaDecoderConfig`, `ImageDecoderConfig`, `VideoDecoderConfig`, and `MediaFetcherConfig` from `model_card::media`, together with the (moved) `ImageDecoderLimits` and `VideoDecoderLimits`. The runtime types convert `From` them; the preprocessor converts when it builds its `MediaLoader`. Wire shape is unchanged: field names, defaults, `deny_unknown_fields`, and `skip_serializing_if` match the old types. One deliberate change: `video` is now always part of the card's shape instead of appearing only in `media-ffmpeg` builds; a frontend without that feature warns and leaves videos to the worker, which is what the silently-ignored unknown field amounted to before. `backend-common` and the Python bindings construct the config types; `MediaFetcher::from_env` and `Default` delegate to the config. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…card `TensorModelConfig`, `TensorMetadata`, `DataType`, `ParameterValue`, and `Parameters` are carried on `ModelDeploymentCard` but were defined in `protocols::tensor` next to the tensor request and response types. Move them to `model_card::tensor`; `protocols::tensor` re-exports them so no caller changes. `DataType::size` spells out the two 16-bit float sizes so the module does not depend on `half`. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
`model_card.rs` mixed the `ModelDeploymentCard` wire type (the struct, its enums, `mdcsum`, JSON helpers, setters) with everything that produces one from a checkout: disk discovery, Hugging Face and ModelExpress fetches, metadata staging, `config.json` parsing (`HFConfig`, `ModelInfo`), and the shared tokenizer cache. The loader half (3.2k lines) moves to the child module `model_card::loader`; the wire half stays in `model_card.rs` (0.8k lines). This is a move within the crate: the loader keeps its inherent `impl ModelDeploymentCard` methods and its access to the card's private fields, so no caller changes. `model_card` re-exports `ModelInfo` and `is_weight_file` at their old paths. The one test that re-runs itself in a child process now derives its test name from `module_path!()` instead of a hardcoded path. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
`MultimodalData::Decoded` carried `preprocessor::media::RdmaMediaDataDescriptor`, which embedded `dynamo_memory::nixl::NixlDescriptor` (a `nixl-sys` type) and an `Arc<NixlRegistered<SystemStorage>>` that keeps the registered memory alive. That tied the request wire type to NIXL and CUDA-linked crates. The descriptor and its metadata types (`MediaTensorInfo`, `DataType`, `DecodedMediaMetadata`, `ImageMetadata`, `VideoMetadata`) now live in `protocols::common::media` with a `NixlMemoryDescriptor`/`NixlMemoryType` mirror of the NIXL types and an opaque `MediaStorageHandle` for the keep-alive. `preprocessor::media` converts at the boundary and re-exports the types at their old paths. The `Video` metadata variant is always present on the wire; the decoder feature only decides whether a frontend can produce it. Byte parity: field names, order, `flatten`, `skip`, and the mirrored `MemType` variant names are unchanged; a test serializes both forms for every memory type and asserts identical JSON. Validated with the default features and with `mm-routing`. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
The loader half of `model_card` implemented inherent methods on `ModelDeploymentCard`, `ModelInfoType`, `GenerationConfig`, `PromptFormatterArtifact`, and `TokenizerKind`. Inherent impls must live in the type's own crate, so they would pin these wire types to `dynamo-llm`. They become extension traits in `model_card::loader`: - `ModelDeploymentCardLoader`: `load_from_disk`, `download_config`, `move_to_url`, `local_dir`, `tokenizer`, `embedding_tokenizer_with_options` - `ModelInfoTypeLoader`, `GenerationConfigLoader`, `PromptFormatterArtifactLoader`, `TokenizerKindLoader`: `from_disk` and friends Private helpers become free functions. The only private state the loader touched, the memoized tokenizer id bound, is now reached through `tokenizer_id_bound` / `init_tokenizer_id_bound` on the card. Call sites keep their syntax and gain a trait import; `model_card` re-exports the traits. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
`ModelDeploymentCard::router_config` is part of the card's wire shape and checksum, but `RouterConfig` was defined in `entrypoint` and its `LoadThresholdConfig` in `discovery::worker_monitor`, so the card's wire type reached into the frontend's startup and worker-monitoring code. Both move to `model_card::router_config`; `entrypoint` and `discovery` re-export them at the old paths. Pure data move; `RouterMode` still comes from `dynamo_runtime::pipeline` for now. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Five tests under the wire-type modules exercised frontend or worker code instead: two in `protocols::common::extensions` and one in `metadata_upload` validate OpenAI request types through `ValidateRequest`, and one each in `llm_backend` and the wire `preprocessor` exercise the worker admission gate. They move to `protocols::openai::validate` and `admission_gate`, where those traits and types live. The wire modules' remaining tests reference only their own module and the runtime vocabulary. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Move the dependency-light vocabulary shared by every Dynamo crate out of
dynamo-runtime into a new leaf crate at lib/runtime-protocols:
- src/error.rs -> error taxonomy (DynamoError, ErrorClass, ...)
- src/protocols.rs, protocols/ -> EndpointId, Annotated, MaybeError
- src/engine.rs -> AsyncEngine traits and type erasure
- src/config/environment_names.rs -> canonical environment variable names
The files move verbatim (git mv). dynamo-runtime re-exports each module at
its historical path (dynamo_runtime::{engine, error, protocols} and
dynamo_runtime::config::environment_names), so no consumer changes.
dynamo_runtime::pipeline keeps re-exporting crate::engine unchanged.
The new crate depends only on serde, serde_json, anyhow, async-trait and
futures, with tokio as a dev-dependency for the engine.rs tests. The four
runnable doc-tests now import from dynamo_runtime_protocols, and the
engine.rs intra-doc links into crate::pipeline become code spans since
that module lives in dynamo-runtime. Two test-module warnings that
dynamo-runtime's crate-level allow(dead_code, unused_imports) had masked
are fixed rather than re-allowed: an unused TryFrom import and the
unreferenced compile-time trait-bound assertion helpers in error.rs.
Also: register the crate in the workspace members and dependencies,
regenerate all four lockfiles, move the CODEOWNERS entry from
lib/runtime/src/protocols/ to lib/runtime-protocols/ (areas.yaml +
regenerated CODEOWNERS), and update the two docs links that pointed at
the old environment_names.rs path.
Validation:
- cargo fmt --all -- --check
- cargo check -p dynamo-runtime-protocols -p dynamo-runtime -p dynamo-kv-router -p dynamo-llm --all-targets
- cargo test -p dynamo-runtime-protocols
- cargo clippy -p dynamo-runtime-protocols -p dynamo-runtime --all-targets -- -D warnings
- cargo metadata --locked --offline --format-version 1 --manifest-path <m> for Cargo.toml, lib/bindings/python/Cargo.toml, lib/bindings/kvbm/Cargo.toml, lib/runtime/examples/Cargo.toml
- python .github/codeowners/build_codeowners.py --areas .github/codeowners/areas.yaml --repo . --strict
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Move the Prometheus registry hierarchy out of dynamo-runtime into a new
leaf crate `dynamo-metrics` at lib/metrics so that crates which only need
metric constructors and metric name constants no longer have to depend on
the full runtime.
What moved (verbatim, same item order):
- lib/runtime/src/metrics.rs core -> lib/metrics/src/lib.rs:
`PrometheusMetric` and its impls, `create_metric`, `MetricOptions`,
`create_metric_with_options`, `Metrics<H>`, `MetricsHierarchy`, the
callback type aliases, `CollectedFamilies`, `MetricsRegistry`,
`FamilyMerger`, plus the three test modules that only need the
registry (`test_metricsregistry_units`,
`test_create_metric_with_options`, `test_metric_families_combined`).
- lib/runtime/src/metrics/prometheus_names.rs and prom_typed.rs ->
lib/metrics/src/ (git mv).
- The `test_helpers` module -> lib/metrics/src/test_helpers.rs, now
`pub` behind `#[cfg(any(test, feature = "testing"))]`.
Dead imports the runtime crate-level `#![allow(unused_imports)]` was
hiding are dropped (`ComponentBuilder`, the `crate::pipeline` block,
`Annotated`, `stream`/`StreamExt`, `DistributedRuntimeProvider`,
`parking_lot::Mutex`, `once_cell::sync::Lazy`, `regex::Regex`,
`std::any::Any`, `work_handler`, `sanitize_prometheus_name`, and the
no-op `use anyhow;`). `crate::metrics::prom_typed::BuiltFamily` paths
become `crate::prom_typed::BuiltFamily`.
Shim: lib/runtime/src/metrics.rs keeps the eight runtime-specific
submodules (frontend_perf, otlp_export, quic_response, request_plane,
tokio_perf, transport_metrics, work_handler_perf, work_handler_pool)
and re-exports every moved public item plus the `prom_typed` and
`prometheus_names` modules, so all existing `dynamo_runtime::metrics::*`
paths (including `dynamo_runtime::MetricsRegistry`) keep resolving.
Impls of `MetricsHierarchy` for DistributedRuntime/Namespace/Component/
Endpoint stay in runtime (local type, foreign trait).
Test split and `testing` feature: the two test modules that build a
DistributedRuntime (`test_metricsregistry_prefixes`,
`test_metricsregistry_prometheus_fmt_outputs`, both gated on the
`integration` feature) stay in the runtime shim unchanged apart from one
`use std::sync::Arc;` that the parent module used to provide. They reach
`test_helpers` through a `#[cfg(test)] use dynamo_metrics::test_helpers;`
in the shim. dynamo-metrics gains `testing = ["dep:serde"]`, which also
enables the serde `Deserialize` derives on the `prom_typed` fixture
types that runtime's otlp_export tests deserialize; previously those were
`cfg(test)`-gated, which does not propagate across crates. dynamo-runtime
lists `dynamo-metrics = { workspace = true, features = ["testing"] }` in
dev-dependencies. The `clamp_u64_to_i64` doctest now imports from
`dynamo_metrics`, and the `parse_prometheus_metric` doc example gets a
hidden `use` so it compiles as a doctest.
Codegen: gen_python_prometheus_names.rs, its template, and README now
point at lib/metrics/src/prometheus_names.rs; the generated
lib/bindings/python/src/dynamo/prometheus_names.py is regenerated (header
comment only). components/src/dynamo/common/utils/prometheus.py comments
cite the new paths.
Dependencies: dynamo-metrics depends on anyhow, once_cell, prometheus,
regex, tracing, and optionally serde (dev-dependency as well). Root
Cargo.toml adds the workspace member and `dynamo-metrics` workspace
dependency; dynamo-runtime keeps once_cell/regex because other modules
still use them. CODEOWNERS: lib/metrics/ is claimed by the observability
area in areas.yaml and CODEOWNERS is regenerated. All four lockfiles are
regenerated.
Validation:
- cargo fmt --all -- --check
- cargo check -p dynamo-metrics -p dynamo-runtime -p dynamo-kv-router -p dynamo-llm -p dynamo-backend-common --all-targets
- cargo test -p dynamo-metrics --all-features (18 unit + 2 doc tests) and cargo test -p dynamo-metrics
- cargo test -p dynamo-runtime --lib metrics
- NATS_SERVER=nats://127.0.0.1:14222 cargo test -p dynamo-runtime --lib --features integration metrics (against an ephemeral nats-server)
- cargo test -p dynamo-codegen
- cargo clippy -p dynamo-metrics -p dynamo-runtime --all-targets -- -D warnings
- cargo metadata --locked --offline --format-version 1 --manifest-path <m> for Cargo.toml, lib/bindings/python/Cargo.toml, lib/bindings/kvbm/Cargo.toml, lib/runtime/examples/Cargo.toml
- .github/codeowners/build_codeowners.py --strict and emit_codeowners.py
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…cols Both ride on the model deployment card: `Slug` is the card's identity and `RouterMode` sits inside its per-worker-set `RouterConfig`. A wire crate that carries the card must name them without depending on the full runtime, so they join the leaf vocabulary crate. `slug.rs` moves verbatim (it needs `blake3` for its suffix hash). `RouterMode` and its pure helpers (`telemetry_label`, `is_kv_routing`, `is_direct_routing`, `requires_occupancy`) move to `dynamo_runtime_protocols::routing`; the mapping to the runtime's private `RoutePolicy` stays in `push_router` as a free function. `dynamo-runtime` re-exports both at their old paths (`dynamo_runtime::slug`, `dynamo_runtime::pipeline::RouterMode`), so no consumer changes. All four lockfiles regenerated. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
- Drop `impl storage::kv::Versioned for ModelDeploymentCard`. Nothing publishes a card through `KeyValueStoreManager::publish`, the only consumer of that trait, and the impl was a no-op (`revision() == 0`). - Delete `common::versioned::Versioned`, a trait with no implementors or users. - Header handling in `protocols::common::extensions`, `protocols::agents`, and `metadata_upload` used `axum::http::*`; it now uses the `http` crate directly (the same types, which axum re-exports), so these modules no longer name the HTTP server framework. Lockfiles regenerated for the new direct `http` dependency. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Move the HTTP transport out of `dynamo-llm` into `lib/http-server` (`dynamo-http-server`), a crate that knows nothing about routes, protocols, discovery, or the distributed runtime: - `server`: the Dynamo-owned accept loop (TLS termination, per-connection shutdown signals, grace-period bounded drain), moved verbatim from `http::service::server`. - `write_timeout`: connection write-stall and transfer deadlines, moved verbatim. - `lifecycle`: `ServiceStage`, `ServiceObserver`, `InflightPermit`, the `x-request-id` echo middleware, and `track_inflight_response`, which keeps a permit alive for the whole response body. The frontend's admission middleware keeps its protocol-specific rejection and calls the helper for the body wrapping. - `listener`: bind-address parsing, backlog, write-idle timeout, and the graceful-shutdown budget, all read from the existing `DYN_HTTP_*` variables via `dynamo-runtime-protocols`. TLS material loading stays with the composition root: `HttpService` builds the `rustls` server config through `dynamo_runtime::tls_utils` and hands the acceptor to `serve`. `service_v2` re-exports the lifecycle types at their old paths. Tests that drove the transport through the batch proxy move to `http::service::transport_tests` in `dynamo-llm`; the pure deadline and TLS-handshake tests move with the crate. Dependencies: axum, hyper-util, tokio, tokio-rustls, tokio-util, tower, futures, parking_lot, tracing, anyhow, dynamo-runtime-protocols. `dynamo-llm` drops its direct `hyper-util` dependency. Lockfiles and CODEOWNERS regenerated. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Two generic utilities lived in `dynamo-llm` but are needed by code that is leaving it: the KV block manager uses the preconfigured `tmq` PUB/SUB/PULL sockets, and the KVBM Python bindings use the rotating JSONL `Recorder`. Both move verbatim into `dynamo-runtime` (`transports::zmq::sockets` and `utils::recorder`); `dynamo-llm` re-exports them at `utils::zmq` and `recorder`, so no caller changes. The two test-only socket helpers are exposed behind the runtime's `testing` feature, which `dynamo-llm`'s tests already enable. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
The selection, indexer, slot-tracker, and shared-cache services each hand-rolled `TcpListener::bind` + `axum::serve(..).with_graceful_shutdown`. They now go through one `services::common::server::serve_until`, built on `dynamo-http-server`: a listener with the configured backlog, TCP_NODELAY on accepted connections, write-idle deadlines, and a shutdown that stops accepting and then gives open connections the `DYN_HTTP_*` grace period instead of waiting on them without bound. The dependency is optional and enabled by the `standalone-*` features; `dynamo-kv-router` still does not depend on `dynamo-runtime`. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…erver `ConnectionStatus`, `ConnectionHandle`, and the monitor task that kills the engine context when a client vanishes are transport mechanics, not protocol. They move to `dynamo_http_server::disconnect`. The monitor's only tie to the frontend was two Prometheus increments on an unexpected close; it now takes an optional `UnexpectedCloseHook` callback instead of the frontend's `Metrics`. `dynamo-llm` keeps `create_connection_monitor` with its old signature as a thin wrapper that supplies the metrics hook, so none of its twelve call sites change, and re-exports the handle types at their old paths. The SSE stream monitors, which apply protocol error policy, stay in `dynamo-llm`. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
The EPP's `/metrics` listener was a bare `axum::serve`. It now binds through the shared listener helpers (configured backlog) and serves on `dynamo-http-server`, gaining TCP_NODELAY and write-idle deadlines with no change to the metrics themselves. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…gration tests `CheckedFile::from_disk` was unit-tested from `src/common/checked_file.rs` against `tests/data/sample-models/TinyLlama_v1.1/config.json` via `CARGO_MANIFEST_DIR`. The wire types in `common` are about to move into a separate crate that does not own that fixture, so the test now lives next to the other TinyLlama-backed checks in `tests/model_card.rs`. No behavior change; the remaining `checked_file` unit tests are fixture-free. Validation: - cargo test -p dynamo-llm --test model_card (13 passed) Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Move the frontend-to-worker wire contract out of dynamo-llm into a new leaf
crate, lib/llm-protocol (package dynamo-llm-protocol). dynamo-llm re-exports
every moved module at its historical path, so consumers (sidecars, mockers,
backend-common, ext-proc, the Python bindings) compile unchanged.
What moved (git mv, same relative paths so internal crate:: paths resolve):
- protocols/common.rs and protocols/common/{extensions, llm_backend, media,
metadata_upload (+tests), metrics, postprocessor, preprocessor, timing}.rs
- protocols/agents.rs, protocols/response_merge.rs
- model_card.rs (wire half) and model_card/{media, tensor, router_config}.rs
- local_model/runtime_config.rs
- model_type.rs, endpoint_type.rs, worker_type.rs
- common.rs and common/{checked_file, dtype}.rs
Stays in dynamo-llm: protocols/common/input_trigger.rs, protocols/providers.rs,
model_card/loader.rs, and all frontend/backend code.
Mirror layout in the new crate: lib.rs declares common, endpoint_type,
local_model, model_card, model_type, protocols, worker_type; protocols.rs
declares agents, common, response_merge and defines TokenIdType;
local_model.rs declares runtime_config.
Shims in dynamo-llm that keep the old paths:
- src/protocols/common.rs: `pub use dynamo_llm_protocol::protocols::common::*`
plus `pub mod input_trigger` and the *OptionsProvider re-exports.
- src/model_card.rs: `pub use dynamo_llm_protocol::model_card::*` plus
`pub mod loader` and the loader re-exports; loader.rs now imports its
std/anyhow/serde/tokenizers/Slug/CheckedFile/TokenizerBackend/TokenIdType
dependencies explicitly instead of through `use super::*`, builds the card
via `ModelDeploymentCard::with_name_only` instead of a struct literal over
private fields, and uses the `init_tokenizer_id_bound` / `slug()` accessors.
- src/protocols.rs: `pub use dynamo_llm_protocol::protocols::{TokenIdType,
agents, response_merge}`.
- src/local_model.rs: `pub use dynamo_llm_protocol::local_model::runtime_config`.
- src/lib.rs: `pub use dynamo_llm_protocol::{common, endpoint_type,
model_type, worker_type}`.
Non-mechanical edits:
- dynamo_runtime paths in moved files now point at dynamo-runtime-protocols
(error, protocols, environment_names, slug::Slug, routing::RouterMode) and
dynamo-truthy (is_truthy, env_is_truthy, parse_bool, parse_bool_opt).
- router_config.rs imports SessionAffinityMode from
dynamo_kv_router::services::selection::affinity; that module is only
reachable behind the kv-router `standalone-selection` feature, which is the
single kv-router feature the new crate enables (`runtime-protocols` is not).
- model_card.rs gets a private verbatim copy of dynamo_tokenizers::log_json_err
so the wire crate does not depend on the tokenizer stack; the unused
`tokenizers::Tokenizer as HfTokenizer` import moved to the loader.
- Orphan-rule fixes for impls dynamo-llm had on the moved types:
`Display`/`FromStr` for tensor::DataType move into the wire crate;
`TryFrom<StreamingCompletionResponse> for dynamo_protocols::types::Choice`
moves from protocols/openai/completions.rs into protocols/common.rs;
`DataType::to_kserve` becomes the `TensorDataTypeExt` extension trait in
grpc/service/tensor.rs; the NIXL `From` conversions in
preprocessor/media/rdma.rs become `nixl_mem_type_to_wire`,
`nixl_mem_type_from_wire`, `nixl_descriptor_to_wire`,
`nixl_descriptor_from_wire` (re-exported from preprocessor::media); the
cfg(mm-routing + media-ffmpeg) inherent video helpers on
RdmaMediaDataDescriptor become the `RdmaMediaDataDescriptorExt` trait.
- Visibility widened from pub(crate) to pub for items dynamo-llm uses across
the crate boundary: protocols::common::invalid_argument_error;
runtime_config::{default_exclude_tools_when_tool_choice_none,
ModelRuntimeConfig::{normalize_structural_tag_compatibility,
runtime_flag_enabled, supports_runtime_capability}};
ModelDeploymentCard::{for_mdc_wire, set_source_path};
DecoderCheckpoint::{new, into_parts}; prompt_logprobs_from_engine_data;
SessionAffinityId::new; extensions::{merge_response_nvext,
validate_completion_token_ids_single_choice,
validate_metadata_upload_single_choice,
validate_metadata_upload_stop_strings}; preprocessor::{MigrationState and
its methods, owned_abort_error, PreprocessedRequest::{migration_state,
staged_kv_cleanup, jail_seed, expanded_prompt_token_count}};
RequestTracker::record_finish_if_missing.
- Doc comments that named `crate::protocols::tensor`,
`crate::preprocessor::media`, and `dynamo_llm::model_type` now use plain
text or the new crate path.
Dependencies: the new crate depends on dynamo-kv-router (standalone-selection),
dynamo-parsers, dynamo-protocols (protocol-schema), dynamo-renderer,
dynamo-runtime-protocols, dynamo-truthy, and the third-party crates the moved
code already used (anyhow, base64, bitflags, blake3, chrono, derive_builder,
either, http, image, parking_lot, serde, serde_json, sha2, strum, tokio,
tracing, url, utoipa, uuid, validator); dev-deps proptest, rmp-serde,
serial_test, temp-env. dynamo-llm gains dynamo-llm-protocol and drops strum,
which only the moved files used (cargo machete). The workspace registers the
crate in members and workspace.dependencies; all four lockfiles regenerated.
CODEOWNERS: lib/llm-protocol/ joins the frontend area, the agents.rs override
follows the file, and the now-empty lib/llm/src/common/ glob is removed.
lib/llm-protocol/AGENTS.md carries the N-2 wire compatibility contract and
lib/llm/AGENTS.md points at it.
Validation:
- cargo fmt --all -- --check
- cargo check -p dynamo-llm-protocol -p dynamo-llm -p dynamo-backend-common
-p dynamo-vllm-sidecar -p dynamo-sglang-sidecar -p dynamo-trtllm-sidecar
-p dynamo-sidecar-testkit -p dynamo-ext-proc -p dynamo-trtllm-mocker
-p dynamo-vllm-mocker -p dynamo-sglang-mocker --all-targets
- cargo check -p dynamo-llm --no-default-features --all-targets
- cargo test -p dynamo-llm-protocol (190 unit tests + 1 doctest)
- cargo test -p dynamo-llm --lib (3359 passed, 15 ignored)
- cargo test -p dynamo-llm --test model_card --test preprocessor
--test backend --test postprocessor_parsing_stream --test admission_dp_rank
--test admission_rank_refusal --test admission_worker_ownership
--test openai_completions --test http-service
- cargo test -p dynamo-backend-common --lib (161 passed)
- cargo clippy -p dynamo-llm-protocol -p dynamo-llm -p dynamo-backend-common
-p dynamo-ext-proc --all-targets -- -D warnings
- cd lib/bindings/python && cargo check
- cargo metadata --locked --offline for Cargo.toml, lib/bindings/python,
lib/bindings/kvbm, lib/runtime/examples
- cargo check --workspace --all-targets
- python .github/codeowners/build_codeowners.py --strict
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…d protocols module `SessionAffinityMode` is carried on the model deployment card's `RouterConfig`, so the wire crate needs it, but it was defined inside `services::selection::affinity`, which is only compiled with the `standalone-selection` feature. That forced `dynamo-llm-protocol` to enable the standalone selection service (axum, reqwest, zmq, ...) just to name a two-variant enum. Move the enum and its `FromStr` to `dynamo_kv_router::protocols`; the affinity module re-exports it, and the wire crate now depends on `dynamo-kv-router` with default features. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…mo-kvbm `lib/llm/src/block_manager` (40k lines) had no consumer inside `dynamo-llm`; its only user is the KVBM Python extension. It dragged `nixl-sys`, `cudarc`, `nix`, and `aligned-vec` into every default build of the frontend through the `block-manager` default feature. It is now its own crate, `lib/kvbm` (`dynamo-kvbm`): - `block_manager.rs` and `block_manager/` move to `lib/kvbm/src` (`crate::block_manager::` paths become `crate::`), together with the KVBM-specific `tokens` module (its `TokenBlock` carries the external hashes and is not the router's `dynamo-tokens` type), the CUDA kernels, the `vectorized_copy.fatbin` embedding build step, the transfer benchmark, and `bench_local_transfer_v2`. `block_manager.md` becomes the crate README. - The KVBM `/metrics` endpoint no longer borrows the frontend's metrics router; it encodes its own registry with a small axum handler. - `DType` comes from `dynamo-llm-protocol` (it is also on the OpenAI pooling wire), the tmq socket helpers from `dynamo_runtime::transports::zmq::sockets`. - Features move with the code: `nccl`, `testing-cuda`, `testing-nixl`, `testing-full`, `testing-etcd`, `bench`. `dynamo-llm` keeps `block-manager` as an empty feature so existing invocations still parse, keeps `testing-*` for its own RDMA media tests, drops the `block-manager-bench` feature and the now-unused `bincode`, `bitflags`, `derive-getters`, and `offset-allocator` dependencies (`cargo machete`), and its build script only builds protos. - `lib/bindings/kvbm` depends on `dynamo-kvbm` and `dynamo-runtime` and no longer on `dynamo-llm` at all. CODEOWNERS globs retargeted to `lib/kvbm/`; lockfiles regenerated. Validation: `cargo check --workspace --all-targets`; `cargo test -p dynamo-kvbm --lib` (221 passed); `cargo clippy -p dynamo-kvbm -p dynamo-llm -p dynamo-llm-protocol -p dynamo-ext-proc --all-targets -- -D warnings`; `cargo check` of the `lib/bindings/kvbm` and `lib/bindings/python` workspaces; `cargo machete lib/kvbm lib/llm` (only the pre-existing `bs62` remains); `cargo metadata --locked` on all four lockfiles. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…y map Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Update comments and docs that still cited source files by their pre-extraction paths: - lib/runtime/src/metrics/prometheus_names.rs -> lib/metrics/src/prometheus_names.rs (five docs pages and the metrics developer guide link) - lib/llm/src/protocols/common/preprocessor.rs -> lib/llm-protocol/src/protocols/common/preprocessor.rs (sglang and squeeze_evolve Python comments, router AGENTS.md table) - lib/llm/src/model_card.rs -> lib/llm/src/model_card/loader.rs in the operator EPP comments, since the HOME-based model card cache root is resolved by the loader, which stayed in dynamo-llm while the card types moved to dynamo-llm-protocol Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
`v2::memory` defines `pub type Result<T> = std::result::Result<T, StorageError>`, so `filter_map(Result::ok)` in the UCX-unavailable diagnostic of the `testing-nixl` test module resolved to the alias and failed to type-check against nixl-sys's `Result<&str, NixlError>` items. Name the std type explicitly. This was already broken before the block manager moved out of dynamo-llm; it surfaced once `cargo test -p dynamo-kvbm --all-features` was run against the new crate. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Move the self-healing listener out of the runtime's system status server into dynamo-http-server, and serve the status server through that crate's accept loop, so the runtime no longer carries its own copy of HTTP server mechanics. dynamo-http-server: - `server::Accept` is the accept-loop's source of connections. `serve` takes `impl Accept` instead of a `TcpListener`; `TcpListener` implements it with the previous behavior (retry transient accept errors after 50 ms, set TCP_NODELAY). The nodelay step is shared through `set_nodelay`. - `listener::RebindingTcpListener` is the runtime's `RebindingTcpListener` (previously an `axum::serve::Listener` impl) as an `Accept` impl, with the same error classification: connection errors and resource exhaustion keep the socket, anything else drops it and rebinds the same concrete address after the back-off. Rebinds now go through `bind_listener`, so they get SO_REUSEADDR and the configured accept backlog like every other listener in this crate. Log messages lose their "System status" prefix. - `is_connection_error` / `is_resource_exhaustion_error` and the six listener tests move along (the shutdown test now drives `serve` instead of `axum::serve`; the EMFILE child test derives its own test path from `module_path!()`). Adds `libc` and the `socket2` dev-dependency. dynamo-runtime: - `spawn_system_status_server` builds the same router and rebinding listener and hands them to `dynamo_http_server::server::serve`. The status server therefore now honors the write-idle timeout (`DYN_HTTP_WRITE_IDLE_TIMEOUT_SECS`) and bounds shutdown by the graceful shutdown budget (`DYN_HTTP_GRACEFUL_SHUTDOWN_TIMEOUT_SECS`, default 5 s) instead of waiting for open connections indefinitely. - Removes the listener, its error classifiers, and their tests from system_status_server.rs; adds the dynamo-http-server dependency. `libc` and `socket2` stay (used elsewhere; cargo machete is clean). Dependency direction: dynamo-runtime -> dynamo-http-server -> dynamo-runtime-protocols. dynamo-http-server still does not depend on the runtime. All four lockfiles regenerated. Validation: - cargo fmt --all -- --check - cargo check -p dynamo-http-server -p dynamo-runtime --all-targets - cargo clippy -p dynamo-http-server -p dynamo-runtime --all-targets -- -D warnings - cargo test -p dynamo-http-server (12 unit tests, incl. the 6 moved listener tests) - cargo test -p dynamo-runtime --lib system_status_server (11 passed) - cargo machete lib/runtime lib/http-server - cargo metadata --locked for the four lockfiles Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
The media descriptor refactor replaced `source_storage: Option<Arc<NixlRegistered<SystemStorage>>>` with the type-erased `MediaStorageHandle`, but the two `testing-nixl`-gated loader tests still used the concrete type: one called `is_registered()` on the handle, the other compared handles with `Arc::ptr_eq`. Downcast to the registered system storage in the first, and give `MediaStorageHandle` a `ptr_eq` so the cache test can keep asserting that a cache hit returns the same registration. Validation (FFmpeg 9 and the NIXL wheel runtime on PATH): - cargo clippy -p dynamo-llm --features media-ffmpeg,mm-routing,testing-nixl --all-targets -- -D warnings - cargo test -p dynamo-llm --features media-ffmpeg,mm-routing,testing-nixl --lib -- preprocessor::media (94 passed; the one failure, test_unsupported_codec_error_is_actionable, is in untouched decoder code and depends on the host FFmpeg build) Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Move `lib/llm/src/kv_dc_relay.rs` and `lib/llm/src/kv_dc_relay/` (actor,
discovery, host, identity, load, membership watch, namespace sources, pool
registry, publication, resolution, topology, the gRPC WAN adapter with
`relay.proto`, the in-tree `docs/` and protocol README, about 23k lines) to
the new workspace crate `dynamo-kv-dc-relay` at `lib/kv-dc-relay`. The
relay-only test fixture `lib/llm/tests/fixtures/relay-sources-canonical.json`
moves to `lib/kv-dc-relay/tests/fixtures/` with it. Inside the moved code,
`crate::kv_dc_relay::*` collapses to `crate::*`, `pub(in crate::kv_dc_relay)`
becomes `pub(crate)`, and the llm items it consumed (`discovery`,
`kv_router::indexer`, `kv_router::metrics_subscriber`, `local_model`,
`model_card`, `model_type`, `worker_type`) are addressed as `dynamo_llm::*`.
Dependency direction: the relay depends on `dynamo-llm`, like the sidecars.
`dynamo-llm` does not depend on the relay and keeps no re-export shim, since
either would be a cycle; `pub mod kv_dc_relay` is removed from
`lib/llm/src/lib.rs`. The crate carries its own `ckf-diagnostics` feature
that forwards to `dynamo-llm/ckf-diagnostics`.
build.rs split: the second `tonic_build::configure()` block in
`lib/llm/build.rs` (relay.proto, `relay_descriptor.bin` file descriptor set,
and its `rerun-if-changed`) moves to `lib/kv-dc-relay/build.rs` with paths
rebased to `src/wan/grpc/protocol/`. `tonic::include_proto!` and the
`include_bytes!(OUT_DIR/relay_descriptor.bin)` reflection descriptor keep
working unchanged. The kserve block stays in llm.
Dependencies: `tonic-reflection`, `tonic-health`, and `futures-util` had no
remaining user in `dynamo-llm` and move to the relay crate; `tonic`, `prost`,
and `tonic-build` stay in llm for `lib/llm/src/grpc` and are also declared
here. `cargo machete` reports only the pre-existing unused `bs62` in llm and
nothing unused in the relay crate.
Items widened from `pub(crate)` to `pub` in `dynamo-llm` for the relay:
- `discovery::readiness` (module), `discovery::KvSourceMembershipCoordinator`
(struct, `start`, `subscribe`),
`discovery::KvSourceMembershipView::matches_binding_inputs`
- `kv_router::metrics_subscriber` (module), `KvMetricsSubscriber` (struct,
`for_endpoint_id`, `next`)
- `kv_router::indexer::{DEFAULT_RECOVERY_ATTEMPT_TIMEOUT, RecoveryResetReason,
RecoveryTarget, RecoverySupervisor (client, shutdown),
TargetFaultDisposition, start_target_subscriber}` and, under
`ckf-diagnostics`, `WorkerQueryHealthSnapshot` (struct and fields) plus
`WorkerQueryClient::health_snapshot`
- `WorkerQueryClient` (struct, `sync_membership`, `handle_live_batch`,
`handle_target_fault`, `reject_source`); its private default type parameter
`T = IndexerRecoveryTarget` is dropped so the public struct does not expose a
crate-private type (no call site relied on the default)
- `WorkerQueryTransport` (trait)
- New `dynamo-llm` feature `testing = []`, mirroring `dynamo-kv-router` and
`dynamo-runtime`: the `#[cfg(test)]` re-exports of `WorkerQueryClient`,
`WorkerQueryTransport`, and `WorkerQueryClient::new_target_for_test` become
`#[cfg(any(test, feature = "testing"))]` so the relay's unit tests can reach
them; the relay enables it only in `[dev-dependencies]`.
Python bindings (`lib/bindings/python`): add the `dynamo-kv-dc-relay` path
dependency, forward `ckf-diagnostics` to it, and point
`rust/llm/kv_dc_relay.rs` at `dynamo_kv_dc_relay::*` (dropping the now unused
`use super::*`). The bindings' own `llm::kv_dc_relay` PyO3 module is unchanged.
Docs and ownership: `components/src/dynamo/kv_dc_relay/README.md` and the three
Fern pages now link to `lib/kv-dc-relay/src/`; the protocol README names
`dynamo_kv_dc_relay::wan::grpc::protocol`. `.github/codeowners/areas.yaml`
replaces the three old global-router paths with `lib/kv-dc-relay/` and
`CODEOWNERS` is regenerated. Root `Cargo.toml` adds the workspace member and
`dynamo-kv-dc-relay` workspace dependency; `AGENTS.md` lists the crate in the
`lib/` row. `Cargo.lock` and `lib/bindings/python/Cargo.lock` are regenerated;
the kvbm bindings and runtime examples lockfiles are unchanged.
Validation:
- cargo fmt --all -- --check
- cargo check --workspace --all-targets
- cargo test -p dynamo-kv-dc-relay (222 passed)
- cargo test -p dynamo-llm --lib (2901 passed, 12 ignored)
- cargo clippy -p dynamo-kv-dc-relay -p dynamo-llm --all-targets -- -D warnings
- PYO3_PYTHON=.venv/bin/python cargo check --manifest-path lib/bindings/python/Cargo.toml
- cargo machete lib/llm lib/kv-dc-relay
- cargo metadata --locked --offline --format-version 1 --manifest-path <m> for
Cargo.toml, lib/bindings/python/Cargo.toml, lib/bindings/kvbm/Cargo.toml,
lib/runtime/examples/Cargo.toml
- python .github/codeowners/build_codeowners.py --areas .github/codeowners/areas.yaml --repo . --strict
- python .github/codeowners/emit_codeowners.py --areas .github/codeowners/areas.yaml --out CODEOWNERS
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Move `generate_log_buckets` and `round_to_sig_figs`, with their seven unit
tests, from the frontend's HTTP metrics module into
`dynamo_metrics::buckets`, so crates below dynamo-llm can build histogram
families with the same bucket shape. dynamo-llm re-exports both at their
old path (`http::service::metrics::{generate_log_buckets, round_to_sig_figs}`)
and gains a direct `dynamo-metrics` dependency instead of reaching them
through the runtime shim. The env-driven `parse_bucket_config` stays in
dynamo-llm with the frontend knobs it reads.
Validation:
- cargo test -p dynamo-metrics buckets (7 passed)
- cargo check -p dynamo-llm --all-targets
- cargo clippy -p dynamo-metrics -p dynamo-llm --all-targets -- -D warnings
- cargo test -p dynamo-llm --lib http::service::metrics (57 passed)
- cargo metadata --locked for the four lockfiles
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…server Split the frontend's `Metrics` into its protocol-agnostic request accounting and its LLM families. The seven request-level collectors (requests started and completed, inflight and active gauges, queued requests, client disconnects, request duration) now live in `dynamo_http_server::metrics::RequestMetrics`; `dynamo_llm`'s `Metrics` holds one and delegates to it, keeping its public API, the `InflightGuard`/`HttpQueueGuard` RAII types, and the endpoint, request-type, status, and error-type enums, which are passed down as label strings. `RequestMetrics::new(prefix, request_duration_buckets)` builds the same metric names (the `frontend_service` constants from dynamo-metrics under the caller's prefix), the same label sets, the same help strings, and registers the same seven collectors, so the `/metrics` output is unchanged; the http_metrics integration tests pass as they are. Collector accessors replace the direct field reads the unit tests relied on. dynamo-http-server gains `dynamo-metrics` and `prometheus` dependencies and stays independent of the runtime. Validation: - cargo check -p dynamo-http-server -p dynamo-llm --all-targets - cargo clippy -p dynamo-http-server -p dynamo-llm --all-targets -- -D warnings - cargo test -p dynamo-http-server (13 passed) - cargo test -p dynamo-llm --lib -- http::service::metrics http::service::disconnect (78 passed) - cargo test -p dynamo-llm --test http_metrics --test http-service (33 + 9 passed) - cargo metadata --locked for the four lockfiles Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…-worker
Move the worker-side runtime glue out of dynamo-llm into the new workspace
crate dynamo-llm-worker at lib/llm-worker. The crate depends on
dynamo-runtime, dynamo-kv-router (metrics, runtime-protocols,
standalone-selection), and dynamo-llm-protocol only; it never depends on
dynamo-llm, dynamo-http-server, or dynamo-backend-common. dynamo-llm depends
on it and re-exports every moved item at its historical path, so the Python
and C bindings, the mocker servers, the sidecars, the testkit, and ext-proc
compile unchanged.
What moved (git mv, then path rewrites; verbatim otherwise):
- src/first_token.rs -> lib/llm-worker/src/first_token.rs
- src/admission_gate.rs and src/admission_gate/metrics.rs ->
lib/llm-worker/src/admission_gate{.rs,/metrics.rs}
- src/kv_router/sequence.rs and sequence/direct_zmq.rs ->
lib/llm-worker/src/sequence{.rs,/direct_zmq.rs}
- src/kv_router/publisher/ (all modules, tests.rs, README.md, AGENTS.md,
CLAUDE.md) -> lib/llm-worker/src/publisher/
- src/discovery/kv_source_membership.rs ->
lib/llm-worker/src/kv_source_membership.rs
- src/direct_zmq_sub_pool.rs -> lib/llm-worker/src/direct_zmq_sub_pool.rs
- src/direct_zmq_fan_in.rs -> lib/llm-worker/src/direct_zmq_fan_in.rs
(sequence/direct_zmq.rs needs it; it only uses dynamo-runtime and the
sub pool)
- src/kv_router/indexer/recovery/worker_query_endpoint.rs ->
lib/llm-worker/src/worker_query_endpoint.rs (the publisher's state agent
starts the worker-local LocalKvIndexer query endpoint; the module only
uses dynamo-kv-router and dynamo-runtime)
- the wire contracts of src/discovery/kv_state_agent.rs (topic constants,
attachment_record_id, KvStateIngressProtocol, the host/intent/source/
attachment advertisements, KvStateHostControlRequest, KvStateHostStatus,
KvStateUnknownReason, KvStateProjectionResolution,
aggregate_residency_projection, resolve_kv_state_projection, and their
tests) -> lib/llm-worker/src/kv_state.rs. The publisher uses all of them,
not only the enum. KvStateProjectionController stays in llm because it
holds the router Indexer.
- KvPublisherMetrics, its OnceLock, and kv_publisher_metrics() from
src/kv_router/metrics.rs -> lib/llm-worker/src/publisher/metrics.rs;
ActiveSequenceZmqIngressMetrics and its OnceLock ->
lib/llm-worker/src/sequence/metrics.rs (only sequence/direct_zmq.rs uses
it). Nothing left in llm uses either, so llm does not re-export them.
- KV_METRICS_SUBJECT, MULTIMODAL_EMBEDDING_CACHE_SUBJECT,
ACTIVE_SEQUENCES_SUBJECT, WORKER_KV_INDEXER_BUFFER_SIZE ->
lib/llm-worker/src/lib.rs as pub const.
Path rewrites inside the moved code: crate::kv_router::sequence -> crate::
sequence; crate::worker_type, crate::model_card, crate::protocols,
crate::local_model::runtime_config -> dynamo_llm_protocol::*;
crate::utils::zmq -> dynamo_runtime::transports::zmq::sockets (llm's
utils::zmq was already a re-export of that module);
crate::discovery::kv_state_agent -> crate::kv_state;
crate::discovery::KvEventSource -> crate::kv_source_membership::
KvEventSource; crate::kv_router::metrics::kv_publisher_metrics ->
crate::publisher::metrics::kv_publisher_metrics; crate::kv_router::
indexer::{start_worker_kv_query_endpoint*, KvStateAgentStatusCell} ->
crate::worker_query_endpoint; the four subjects -> crate::*.
KV_EVENT_SUBJECT already came from dynamo_kv_router::protocols.
Shims in dynamo-llm:
- src/lib.rs: `pub use dynamo_llm_worker::{admission_gate, first_token}`,
`pub(crate) use dynamo_llm_worker::{direct_zmq_fan_in,
direct_zmq_sub_pool}` (both were private modules; direct_zmq_fan_in.rs,
kv_router/metrics_subscriber.rs, session_affinity/replica_sync.rs, and
the indexer recovery modules keep using them).
- src/kv_router.rs: `pub use dynamo_llm_worker::{publisher, sequence}` and
the four constants; the now-unused `protocols::KV_EVENT_SUBJECT` import
is dropped.
- src/discovery.rs: `pub use dynamo_llm_worker::kv_source_membership` with
the existing name re-exports unchanged.
- src/discovery/kv_state_agent.rs: `pub use dynamo_llm_worker::kv_state::*`
plus KvStateProjectionController.
- src/kv_router/indexer/{mod.rs,recovery/mod.rs}: drop the
worker_query_endpoint module and its pub(crate) re-exports; the publisher
was their only consumer.
- src/kv_router/metrics.rs: module docs point at the worker crate.
Visibility widened from pub(crate) to pub in the new crate because llm
reaches them across the crate boundary:
- direct_zmq_sub_pool: ENDPOINTS_PER_SUB_ENV, DEFAULT_ENDPOINTS_PER_SUB,
KV_ZMQ_RCVHWM, endpoints_per_sub_from_env, DirectZmqSubItem,
RawZmqFrames, DirectZmqSubPool (new, new_raw, with_shared_endpoints,
expect_publisher, forget_publisher, connect, register_raw, shutdown,
group_count), DirectZmqSubRegistration and its fields,
DirectZmqSubConnection (close, group_id, disconnected)
- direct_zmq_fan_in: ContinuityMode, FanInEvent, FanInObservation and its
fields, start_direct_zmq_fan_in, start_direct_zmq_fan_in_for_endpoint_id,
start_direct_zmq_fan_in_async
- sequence: ActiveSequenceEventPublisher::channel,
RuntimeSequenceSubscriber::for_endpoint, host_replica_channels,
ReplicaIngress and ReplicaIngress::start
- first_token: FirstTokenSource::from_publisher, now gated
`#[cfg(any(test, feature = "testing"))]` instead of `#[cfg(test)]`
- worker_query_endpoint: KvStateAgentStatusCell (new, load, store,
store_outbound_cursor), start_worker_kv_query_endpoint,
start_worker_kv_query_endpoint_with_status
- publisher::metrics: KvPublisherMetrics (struct and build),
kv_publisher_metrics
Tests: 242 moved unit tests run in dynamo-llm-worker. Two stay in llm:
- kv_publisher_registration_tests in src/kv_router/metrics.rs keeps its
FakeHierarchy fixture, which the RouterRequestMetrics tests in the same
file share; it imports KvPublisherMetrics from
dynamo_llm_worker::publisher::metrics (build is pub for this).
- entrypoint/input/endpoint.rs::legacy_token_engine_notifies_once_after_
empty_output uses FirstTokenSource::from_publisher through the new
`testing` feature, which llm enables only in [dev-dependencies].
Features on dynamo-llm-worker: `integration` (forwards to
dynamo-runtime/integration for the publisher's NATS tests, as llm did) and
`testing` (test-only hooks, mirroring kv-router, runtime, and llm).
Dependencies: anyhow, arc-swap, async-trait, blake3, bytes, futures,
parking_lot, prometheus, rand, rmp-serde, serde, serde_json, thiserror,
tmq, tokio, tokio-util, tracing, uuid; dev: dynamo-protocols (admission
gate tests), dynamo-runtime/testing, serial_test, temp-env, tempfile,
tokio/test-util, tracing-subscriber. dynamo-llm gains dynamo-llm-worker.
cargo machete reports nothing unused in the new crate and only the
pre-existing bs62 in llm.
Housekeeping: workspace members and workspace.dependencies register the
crate; Cargo.lock and lib/bindings/python/Cargo.lock regenerated (the kvbm
and runtime/examples lockfiles do not include dynamo-llm and are
unchanged); CODEOWNERS: lib/llm-worker/ replaces lib/llm/src/kv_router/
publisher/ in the router area, so the whole crate is router-owned (the
other moved files were under the runtime catch-all); the root AGENTS.md
repository map lists llm-worker; the two component comments that named
lib/llm/src/kv_router/publisher.rs and lib/llm/src/admission_gate.rs now
name the new paths.
Validation:
- cargo fmt --all -- --check
- cargo check -p dynamo-llm-worker --all-targets
- cargo check -p dynamo-llm --all-targets
- cargo check --workspace --all-targets
- cargo test -p dynamo-llm-worker (242 passed)
- cargo test -p dynamo-llm --lib (2652 passed, 12 ignored)
- cargo clippy -p dynamo-llm-worker -p dynamo-llm -p dynamo-backend-common
-p dynamo-sglang-sidecar --all-targets -- -D warnings
- PYO3_PYTHON=.venv/bin/python cargo check --manifest-path
lib/bindings/python/Cargo.toml
- cargo tree -p dynamo-llm-worker -e normal -i dynamo-llm (no match)
- cargo metadata --offline --locked for Cargo.toml, lib/bindings/python,
lib/bindings/kvbm, lib/runtime/examples
- python .github/codeowners/build_codeowners.py --strict (100% owned)
- cargo machete lib/llm lib/llm-worker lib/backend-common lib/sidecar/sglang
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… own crates
Point dynamo-backend-common and the SGLang sidecar at dynamo-llm-protocol,
dynamo-llm-worker, and dynamo-kv-router for everything those crates now own,
so the only dynamo-llm imports left are the two items that still live in
dynamo-llm.
Import rewrites in lib/backend-common/src:
- dynamo_llm::protocols::common::* -> dynamo_llm_protocol::protocols::
common::* (adapter.rs, engine.rs, testing.rs, validate.rs)
- dynamo_llm::local_model::runtime_config::* -> dynamo_llm_protocol::
local_model::runtime_config::* (args.rs, topology.rs, worker.rs)
- dynamo_llm::{model_type, worker_type} and dynamo_llm::model_card::media
-> dynamo_llm_protocol::{model_type, worker_type, model_card::media}
(lib.rs, worker.rs)
- dynamo_llm::kv_router::protocols::KvTransferEnforcement ->
dynamo_kv_router::protocols::KvTransferEnforcement (topology.rs)
- dynamo_llm::{first_token, admission_gate} and dynamo_llm::kv_router::
publisher -> dynamo_llm_worker::{first_token, admission_gate, publisher}
(adapter.rs, engine.rs, publisher.rs, snapshot_publisher.rs, worker.rs)
Import rewrites in lib/sidecar/sglang/src:
- dynamo_llm::kv_router::publisher -> dynamo_llm_worker::publisher
(headless.rs)
- dynamo_llm::discovery::KvEventSource -> dynamo_llm_worker::
kv_source_membership::KvEventSource (headless.rs test)
- dynamo_llm::local_model::runtime_config -> dynamo_llm_protocol::
local_model::runtime_config (engine.rs)
- the native_http.rs doc comment names dynamo_llm_protocol::protocols::
response_merge::PREFILL_OUTPUT_ANNOTATION
Dependencies: backend-common gains dynamo-kv-router, dynamo-llm-protocol,
and dynamo-llm-worker; the SGLang sidecar gains dynamo-llm-protocol and
dynamo-llm-worker. All `{ workspace = true }`. cargo machete reports nothing
unused in either crate. lib/backend-common/examples/mocker only uses
dynamo-backend-common and dynamo-mocker and is unchanged.
dynamo-llm is NOT dropped from either crate:
- lib/backend-common/src/worker.rs uses dynamo_llm::local_model::
{LocalModel, LocalModelBuilder, update_model_taints} to build and
register the worker's model card. LocalModel is frontend-coupled (it
pulls ModelDeploymentCardLoader, entrypoint::RouterConfig,
frontend_config, the tool-parser registry, reasoning_field, and
request_template), so it cannot move to the worker crate without
restructuring dynamo-llm. The Cargo.toml comment records this as the
sole remaining reason for the dependency.
- lib/sidecar/sglang/src/headless.rs uses dynamo_llm::discovery::
{RuntimeConfigWatch, runtime_config_watch} (discovery/runtime_configs.rs
stays in dynamo-llm), and the sidecar reaches dynamo-llm transitively
through dynamo-backend-common and dynamo-sidecar-common regardless.
- lib/sidecar/vllm keeps dynamo-llm for LocalModel, the LoRA helpers, and
the multimodal routing preprocessor; untouched.
Validation:
- cargo fmt --all -- --check
- cargo check -p dynamo-backend-common -p dynamo-sglang-sidecar
--all-targets
- cargo check -p dynamo-backend-common --features testing --all-targets
- cargo check --workspace --all-targets
- cargo test -p dynamo-backend-common -p dynamo-sglang-sidecar
(161 + 122 + 3 + 2 passed, 5 ignored)
- cargo clippy -p dynamo-backend-common -p dynamo-sglang-sidecar
--all-targets -- -D warnings
- PYO3_PYTHON=.venv/bin/python cargo check --manifest-path
lib/bindings/python/Cargo.toml
- cargo tree -p dynamo-backend-common -e normal -i dynamo-llm:
dynamo-llm <- dynamo-backend-common (direct, for LocalModel only)
- cargo tree -p dynamo-sglang-sidecar -e normal -i dynamo-llm:
dynamo-llm <- dynamo-backend-common <- dynamo-sglang-sidecar,
dynamo-llm <- dynamo-sidecar-common <- dynamo-sglang-sidecar,
dynamo-llm <- dynamo-sglang-sidecar (direct, for runtime_config_watch)
- cargo tree -p dynamo-llm-worker -e normal -i dynamo-llm (no match)
- cargo metadata --offline --locked for Cargo.toml, lib/bindings/python,
lib/bindings/kvbm, lib/runtime/examples
- cargo machete lib/backend-common lib/sidecar/sglang
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…ynamo-http-server Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…cal-model
Move the local model loading stack out of dynamo-llm into the new workspace
crate dynamo-local-model at lib/local-model, the per-worker runtime-config
watch into dynamo-llm-worker, and the tool-call parser vocabulary and the
LoRA registration runtime key into dynamo-llm-protocol. dynamo-local-model
depends on dynamo-runtime, dynamo-llm-protocol, dynamo-tokenizers, and
dynamo-parsers (plus hf-hub, modelexpress-*, and third-party crates) only; it
never depends on dynamo-llm, dynamo-llm-worker, dynamo-http-server, or
dynamo-backend-common. dynamo-llm depends on it and re-exports every moved
item at its historical path, so the Python bindings, the mocker servers, the
testkit, the sidecars, and ext-proc compile unchanged.
What moved (git mv, then path rewrites; verbatim otherwise):
- src/local_model.rs -> lib/local-model/src/lib.rs (LocalModel,
LocalModelBuilder, DEFAULT_HTTP_PORT, ENV_SELF_HOST_METADATA,
derive_lora_suffix, register_model_card, update_model_taints,
supported_tool_call_parsers, validate_runtime_config, and the
runtime_config re-export); src/local_model/{AGENTS.md,CLAUDE.md} ->
lib/local-model/src/
- src/model_card/loader.rs -> lib/local-model/src/model_card/loader.rs,
under a new lib/local-model/src/model_card.rs that mirrors llm's: the
protocol card types glob plus the loader module and its five extension
traits (ModelDeploymentCardLoader, ModelInfoTypeLoader,
GenerationConfigLoader, PromptFormatterArtifactLoader, TokenizerKindLoader)
and ModelInfo
- src/hub.rs and src/hub/huggingface.rs -> lib/local-model/src/hub{.rs,/}
- src/frontend_config.rs, src/reasoning_field.rs, src/request_template.rs
-> lib/local-model/src/
- src/discovery/runtime_configs.rs -> lib/llm-worker/src/runtime_configs.rs
(RuntimeConfigWatch, runtime_config_watch; it only needs dynamo-runtime
discovery, dynamo-kv-router WorkerId, and the protocol card types)
- the unified-parser vocabulary of src/protocols/openai/chat_completions/
{unified_parser.rs,tool_parser_v2.rs} -> lib/llm-protocol/src/
local_model/tool_call_parsers.rs: V2_FAMILIES, the eight *_UNIFIED_FAMILY
constants, DYNAMO_FAMILY_ALIASES, FAMILY_NAMES, and unified_family_names().
unified_family_names() was `&unified_parser::FAMILY_NAMES`, which is
REGISTERED_UNIFIED_FAMILIES plus the Dynamo aliases, so the whole list
moves together; dynamo-llm-protocol gains the dynamo-parsers-v2 dependency
(crates.io, already a dependency of dynamo-llm). canonical_family() and
QWEN3_TOOL_CALL_PARSER stay in unified_parser.rs.
- LORA_REQUIRES_REGISTRATION from src/lora.rs -> dynamo_llm_protocol::
local_model::runtime_config (it is a runtime-config key).
Path rewrites inside the moved code: crate::entrypoint::RouterConfig ->
dynamo_llm_protocol::model_card::router_config::RouterConfig;
crate::common::checked_file, crate::model_type, crate::worker_type,
crate::protocols::TokenIdType -> dynamo_llm_protocol::*;
crate::tokenizers::*, crate::file_json_field, crate::log_json_err ->
dynamo_tokenizers::*; crate::local_model::runtime_config ->
crate::runtime_config; crate::local_model::LocalModelBuilder ->
crate::LocalModelBuilder; the tool_parser_v2 vocabulary imports ->
dynamo_llm_protocol::local_model::tool_call_parsers; in runtime_configs.rs
crate::{local_model::runtime_config, model_card} -> dynamo_llm_protocol::*.
The moved tests read their fixtures from ../llm/tests/data/sample-models
(the data stays with llm's own tests, which share it).
Shims in dynamo-llm:
- src/lib.rs: `pub use dynamo_local_model::{frontend_config, hub,
reasoning_field, request_template}`.
- src/local_model.rs: explicit `pub use dynamo_local_model::{...}` of every
item listed above; runtime_config keeps resolving to the protocol crate.
- src/model_card.rs: `pub use dynamo_local_model::model_card::loader` and the
same `pub use loader::{...}` trait list as before; the pub(crate)
is_weight_file re-export is dropped (both users moved).
- src/discovery.rs: `pub(crate) use dynamo_llm_worker::runtime_configs` with
the existing `pub use runtime_configs::{RuntimeConfigWatch,
runtime_config_watch}` unchanged.
- src/lora.rs: `pub use dynamo_llm_protocol::local_model::runtime_config::
LORA_REQUIRES_REGISTRATION`.
- unified_parser.rs: `pub(crate) use` of the six family constants and
DYNAMO_FAMILY_ALIASES it still routes on, plus a `#[cfg(test)] pub(crate)
use` of DEEPSEEK_V4_UNIFIED_FAMILY, FAMILY_NAMES, GEMMA4_UNIFIED_FAMILY,
and KIMI_K2_UNIFIED_FAMILY, which only tests in llm reach through
`unified_parser::`. tool_parser_v2.rs: `pub use ...::unified_family_names`
(the Python bindings import it from there).
Visibility widened because the new crate boundary separates the users:
- hub::huggingface: HfRepoSpec, HfRepoSpec::from_uri, huggingface_cache,
cached_hf_snapshot, finalize_hf_snapshot, download_hf_snapshot
(pub(crate) -> pub; llm's lora/source.rs uses all of them)
- tool_call_parsers: V2_FAMILIES (`#[cfg(test)] pub(crate)` -> pub, so the
moved registration test can reach it), the *_UNIFIED_FAMILY constants
(pub(crate) -> pub), DYNAMO_FAMILY_ALIASES and FAMILY_NAMES (private /
pub(crate) -> pub)
self_host_base_url and is_weight_file stay pub(crate) inside the new crate.
Features: dynamo-local-model has `modelexpress = ["dep:modelexpress-client"]`
(hub.rs has the only cfg sites); dynamo-llm's `modelexpress` now forwards to
it, and llm's default features are unchanged. No moved file uses lora-s3 or
any other llm feature (only cfg(target_os) and cfg(unix)).
Tests: 91 moved unit tests run in dynamo-local-model and 4 in
dynamo-llm-worker (runtime_configs); dynamo-llm --lib drops from 2652 to
2557 by exactly those 95. The model_card integration test stays in llm and
exercises the loader through the shim.
Dependencies: dynamo-local-model uses anyhow, futures, hf-hub (tokio,
rustls-tls, ureq), libc, modelexpress-client (optional), modelexpress-common,
reqwest, serde, serde_json, tempfile, tokio, tokio-util, tracing, url, uuid,
json-five, tokenizers, and sysinfo on Linux; dev: dynamo-kv-router, mockito,
serial_test, temp-env. dynamo-llm gains dynamo-local-model and drops
modelexpress-client and modelexpress-common (normal and dev), json-five, and
the Linux sysinfo dependency, all of which only the moved files used.
dynamo-llm-protocol gains dynamo-parsers-v2; dynamo-llm-worker gains the
tokio-stream dev-dependency for the runtime_configs tests. cargo machete
reports nothing unused in the new crate, llm-worker, or llm-protocol, and
only the pre-existing bs62 in llm.
Housekeeping: workspace members and workspace.dependencies register the
crate; Cargo.lock, lib/bindings/python/Cargo.lock, and
lib/bindings/kvbm/Cargo.lock regenerated (kvbm depends on llm-protocol and
so now locks dynamo-parsers-v2; lib/runtime/examples/Cargo.lock is
unchanged); CODEOWNERS: lib/local-model/ replaces lib/llm/src/local_model/
in the runtime area (lib/llm/src/model_card.rs stays listed for the shim);
the root AGENTS.md repository map lists local-model and the runtime-config
watch under llm-worker; the two operator comments naming
lib/llm/src/model_card/loader.rs and the token_budget.py comment naming the
runtime_config path now name lib/local-model/src/model_card/loader.rs and
lib/llm-protocol/src/local_model/runtime_config.rs.
Validation:
- cargo fmt --all -- --check
- cargo check -p dynamo-local-model --all-targets
- cargo check -p dynamo-llm-worker -p dynamo-llm-protocol --all-targets
- cargo check -p dynamo-llm --all-targets
- cargo check --workspace --all-targets
- cargo test -p dynamo-local-model -p dynamo-llm-worker -p dynamo-llm-protocol
(91 + 246 + 190 passed)
- cargo test -p dynamo-llm --lib (2557 passed, 12 ignored)
- cargo test -p dynamo-llm --test model_card (13 passed)
- cargo clippy -p dynamo-local-model -p dynamo-llm-worker -p dynamo-llm
-p dynamo-backend-common -p dynamo-sglang-sidecar -p dynamo-vllm-sidecar
--all-targets -- -D warnings
- PYO3_PYTHON=.venv/bin/python cargo check --manifest-path
lib/bindings/python/Cargo.toml, with and without --features mm-routing
- cargo tree -p dynamo-local-model -e normal -i dynamo-llm (no match);
same for dynamo-llm-worker and dynamo-llm-protocol
- cargo metadata --offline --locked for Cargo.toml, lib/bindings/python,
lib/bindings/kvbm, lib/runtime/examples
- python .github/codeowners/build_codeowners.py --strict (100% owned);
emit_codeowners.py leaves CONTRIBUTORS.md unchanged
- cargo machete lib/llm lib/local-model lib/llm-worker lib/llm-protocol
lib/backend-common lib/sidecar/sglang lib/sidecar/vllm
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Point dynamo-backend-common, the SGLang sidecar, and the vLLM sidecar at
dynamo-local-model, dynamo-llm-worker, and dynamo-llm-protocol for the items
those crates now own. dynamo-backend-common and the SGLang sidecar no longer
depend on dynamo-llm at all; the vLLM sidecar keeps it for the LoRA
downloader, which still lives in dynamo-llm.
Import rewrites:
- lib/backend-common/src/worker.rs: dynamo_llm::local_model::{LocalModel,
LocalModelBuilder, update_model_taints} -> dynamo_local_model::{...}
- lib/sidecar/sglang/src/headless.rs: dynamo_llm::discovery::
{RuntimeConfigWatch, runtime_config_watch} -> dynamo_llm_worker::
runtime_configs::{...}
- lib/sidecar/vllm/src: dynamo_llm::local_model::derive_lora_suffix and
LocalModel -> dynamo_local_model (lora.rs, engine.rs);
dynamo_llm::lora::LORA_REQUIRES_REGISTRATION and dynamo_llm::local_model::
runtime_config::* -> dynamo_llm_protocol::local_model::runtime_config::*
(model.rs, engine.rs, tests.rs); dynamo_llm::model_card::{LoraInfo,
ModelDeploymentCard} and dynamo_llm::worker_type::WorkerType ->
dynamo_llm_protocol::{model_card, worker_type} (lora.rs, tests.rs);
dynamo_llm::protocols::common::{preprocessed_mm_identifier,
preprocessed_mm_routing_hash} -> dynamo_llm_protocol::protocols::common
(convert.rs, test_fixtures.rs)
Dependencies:
- dynamo-backend-common: drops dynamo-llm and its "only for local_model"
comment, gains dynamo-local-model. lib/backend-common/examples/mocker and
the `testing` feature use nothing from dynamo-llm, so no dev-dependency
on it is needed.
- dynamo-sglang-sidecar: drops dynamo-llm and its "only for
runtime_config_watch" comment; dynamo-sidecar-common only depends on
dynamo-backend-common and dynamo-runtime, so nothing reaches dynamo-llm
transitively any more.
- dynamo-vllm-sidecar: gains dynamo-llm-protocol and dynamo-local-model;
dynamo-llm stays a required dependency with `features = ["lora-s3"]`.
The brief expected the only remaining use to be preprocessor::mm_routing::
image::resolve_exact_routing_image_token_id behind `mm-routing`, but
lora.rs and engine.rs also use dynamo_llm::lora::{HuggingFaceLoRASource,
LoRACache, LoRADownloader, LoRASource, LocalLoRASource, S3LoRASource,
lora_serving_enabled} unconditionally, and lib/llm/src/lora/
{cache,downloader,source}.rs stay in dynamo-llm. Making the dependency
optional would therefore require moving the LoRA downloader out of
dynamo-llm, which is out of scope here, so the `mm-routing` feature and
`default = ["mm-routing"]` are unchanged and the Cargo.toml comment names
the two remaining reasons. The lora-s3 opt-in is for the sidecar's own
S3LoRASource, not for feature unification: on the Python bindings,
`cargo tree -e features -i dynamo-llm` shows lora-s3 enabled by
dynamo-llm's `default` feature (the Linux dependency uses defaults) and by
dynamo-vllm-sidecar, before and after this change, and the non-Linux
dependency lists lora-s3 explicitly, so nothing drops out either way.
cargo machete reports nothing unused in the three crates.
The lockfiles for the root workspace and lib/bindings/python are regenerated
(dynamo-llm leaves the dependency lists of backend-common and the SGLang
sidecar); lib/bindings/kvbm and lib/runtime/examples are unchanged.
Validation:
- cargo fmt --all -- --check
- cargo check -p dynamo-backend-common -p dynamo-sglang-sidecar
-p dynamo-vllm-sidecar --all-targets
- cargo check -p dynamo-backend-common --features testing --all-targets
- cargo check -p dynamo-vllm-sidecar --no-default-features --all-targets
- cargo check --workspace --all-targets
- cargo test -p dynamo-backend-common -p dynamo-sglang-sidecar
-p dynamo-vllm-sidecar (161 + 122 + 3 + 2 + 132 + 3 passed, 5 ignored)
- cargo clippy -p dynamo-local-model -p dynamo-llm-worker -p dynamo-llm
-p dynamo-backend-common -p dynamo-sglang-sidecar -p dynamo-vllm-sidecar
--all-targets -- -D warnings
- PYO3_PYTHON=.venv/bin/python cargo check --manifest-path
lib/bindings/python/Cargo.toml, with and without --features mm-routing
- cargo tree -p dynamo-backend-common -e normal -i dynamo-llm (no match)
- cargo tree -p dynamo-sglang-sidecar -e normal -i dynamo-llm (no match)
- cargo tree -p dynamo-local-model -e normal -i dynamo-llm (no match)
- cargo tree -p dynamo-llm-worker -e normal -i dynamo-llm (no match)
- cargo tree -p dynamo-vllm-sidecar --no-default-features -e normal
-i dynamo-llm: dynamo-llm <- dynamo-vllm-sidecar (direct, for the LoRA
downloader; see above)
- cargo metadata --offline --locked for Cargo.toml, lib/bindings/python,
lib/bindings/kvbm, lib/runtime/examples
- cargo machete lib/backend-common lib/sidecar/sglang lib/sidecar/vllm
lib/local-model lib/llm (only the pre-existing bs62 in llm)
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…odel
Move the LoRA adapter sources (local filesystem, Hugging Face, and S3
behind `lora-s3`), the downloader, the cache, and the `DYN_LORA_ENABLED`
switch from `dynamo_llm::lora` into `dynamo_local_model::lora`. These
modules fetch model artifacts and have no frontend dependency: their only
in-crate edge was the Hugging Face hub helper, which already lives in
dynamo-local-model. LoRA routing, allocation, filtering, and the
controller stay in dynamo-llm, which re-exports every moved item at its
old path (`dynamo_llm::lora::{LoRACache, LoRADownloader, LoRASource,
HuggingFaceLoRASource, LocalLoRASource, S3LoRASource, lora_serving_enabled,
LORA_REQUIRES_REGISTRATION}`), so llm's entrypoints and model manager are
unchanged.
The `lora-s3` feature and its optional `object_store` and `aws-*`
dependencies move to dynamo-local-model; llm's `lora-s3` (still in its
default set) forwards to it, and llm keeps `object_store` only for
`request-trace-s3`. dynamo-local-model gains `async-trait` and
`parking_lot`. Lockfiles regenerated.
Validation:
- cargo check -p dynamo-local-model -p dynamo-llm --all-targets, and
-p dynamo-local-model --features lora-s3 --all-targets
- cargo clippy -p dynamo-local-model -p dynamo-llm --all-targets --features dynamo-local-model/lora-s3 -- -D warnings
- cargo test -p dynamo-local-model --features lora-s3 (115 passed)
- cargo test -p dynamo-llm --lib lora (124 passed)
- cargo machete lib/llm lib/local-model (only the pre-existing bs62)
- cargo tree -p dynamo-local-model -e normal -i dynamo-llm: no match
- cargo metadata --locked for the four lockfiles
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Take the LoRA downloader, cache, sources, and `lora_serving_enabled` from dynamo-local-model, where they now live, and enable that crate's `lora-s3` feature for `S3LoRASource` instead of enabling it on dynamo-llm. The one remaining dynamo-llm use is `preprocessor::mm_routing::image:: resolve_exact_routing_image_token_id` inside the `mm-routing`-gated image token resolver, so `dynamo-llm` becomes an optional dependency that the `mm-routing` feature turns on. The default feature set still includes `mm-routing`, so the standalone sidecar binary is unchanged; the Python bindings, which build the sidecar without default features, no longer pull the frontend crate through it. Validation: - cargo check -p dynamo-vllm-sidecar --all-targets, and --no-default-features --all-targets - cargo clippy -p dynamo-vllm-sidecar --all-targets -- -D warnings, and --no-default-features - cargo test -p dynamo-vllm-sidecar (132 + 3 passed) - cargo tree -p dynamo-vllm-sidecar --no-default-features -e normal -i dynamo-llm: no match - cargo machete lib/sidecar/vllm - cargo metadata --locked for the four lockfiles Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…protocol Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…cal-model Move `preprocessor::mm_routing` (per-model image and video placeholder layouts, the image token counter backed by `llm-multimodal`, and the Qwen3 and Nemotron video routing specs) from dynamo-llm into `dynamo_local_model::mm_routing`, behind the crate's new `mm-routing` feature. The module resolves routing tokens from a model directory and a tokenizer, which is model knowledge, not request processing: its only in-crate edges were the protocol token type, the tokenizer crate, and `pub(in crate::preprocessor)` visibility qualifiers, now plain `pub`. dynamo-llm re-exports it at `preprocessor::mm_routing` and its `mm-routing` feature forwards to the new crate (keeping the direct `llm-multimodal`/`llm-tokenizer` dependencies the preprocessor itself uses), so the frontend and the Python bindings are unchanged. One unit test, `image_count_failure_discards_partial_routing_and_metrics`, drives `OpenAIPreprocessor` and therefore stays in dynamo-llm as `preprocessor/mm_routing_tests.rs`; the failing image counter it needs is exposed as `mm_routing::image::testing::failing_counter` behind a new `testing` feature, which also gates `VideoRoutingProcessor::test_stub` (previously `cfg(test)`) so dynamo-llm's own tests can reach it. The frontend-only `cfg_attr(not(feature = "media-ffmpeg"), allow(dead_code))` is dropped: every facade item is public in the new crate. Lockfiles regenerated; the repository map describes the crate's new contents. Validation: - cargo check -p dynamo-local-model --features mm-routing,testing --all-targets - cargo check -p dynamo-llm --all-targets, and --features mm-routing --all-targets - cargo clippy -p dynamo-local-model --features mm-routing,testing,lora-s3 --all-targets -- -D warnings - cargo clippy -p dynamo-llm --features mm-routing --all-targets -- -D warnings - cargo test -p dynamo-local-model --features mm-routing,testing,lora-s3 (162 passed) - cargo test -p dynamo-llm --features mm-routing --lib -- preprocessor (328 passed) - PYO3_PYTHON=... cargo check --manifest-path lib/bindings/python/Cargo.toml --features mm-routing - cargo tree -p dynamo-local-model -e normal -i dynamo-llm: no match - cargo machete lib/llm lib/local-model (only the pre-existing bs62) - cargo metadata --locked for the four lockfiles Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
The exact multimodal KV routing image token now comes from `dynamo_local_model::mm_routing`, so the sidecar's `mm-routing` feature forwards to that crate instead of pulling the frontend crate. With the LoRA download stack and the model loader already in dynamo-local-model, nothing in the sidecar references dynamo-llm any more, in any feature configuration; the dependency is removed. The standalone binary keeps `mm-routing` on by default and the Python bindings keep forwarding their own `mm-routing` opt-in. Validation: - cargo check -p dynamo-vllm-sidecar --all-targets, and --no-default-features --all-targets - cargo clippy -p dynamo-vllm-sidecar --all-targets -- -D warnings, and --no-default-features - cargo test -p dynamo-vllm-sidecar (132 + 3 passed) - cargo tree -p dynamo-vllm-sidecar -e normal -i dynamo-llm (default features): no match - cargo machete lib/sidecar/vllm - cargo metadata --locked for the four lockfiles Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
… bench `cargo clippy -p dynamo-kvbm --all-features --all-targets -- -D warnings` flagged four pre-existing lints in code that CI's default-feature clippy never compiles: a `vec!` iterated by value in the local transfer bench, two `testing-cuda`/`testing-nixl` test helpers that advanced a byte by hand as a loop counter (now zipped with `start_value..`), and a four-element tuple return type in the mixed-layout pool fixture (now `MixedLayoutPools`). Behavior is unchanged. Validation: - cargo clippy -p dynamo-kvbm --all-features --all-targets -- -D warnings - cargo test -p dynamo-kvbm (221 passed) and cargo test -p dynamo-kvbm --all-features --no-run Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
had a problem deploying
to
external_collaborator
October 23, 2026 06:55 — with 
GitHub Actions
Failure
GitHub Actions
Failure
github-actions
Bot
added
refactor
documentation
…ith the zmq transport
Move the two pure transport helpers out of dynamo-llm-worker into the
runtime's zmq transport, next to the socket helpers they build on:
- lib/llm-worker/src/direct_zmq_sub_pool.rs ->
lib/runtime/src/transports/zmq/direct_sub_pool.rs
- lib/llm-worker/src/direct_zmq_fan_in.rs ->
lib/runtime/src/transports/zmq/direct_fan_in.rs
Both are declared as `pub mod` in lib/runtime/src/transports/zmq.rs. They
only ever used the runtime's own transports::event_plane, transports::zmq::
sockets, component, discovery, protocols, and traits modules plus anyhow,
futures, parking_lot, tokio, and tokio-util, so the move is a path rewrite:
`dynamo_runtime::` -> `crate::` and `crate::direct_zmq_sub_pool` ->
`super::direct_sub_pool`. The 25 unit tests travel with them and run as
`transports::zmq::{direct_fan_in,direct_sub_pool}::tests`.
No visibility changes: every item the publisher, sequence, and router code
reaches was already `pub` since the llm-worker extraction; the remaining
`#[cfg(test)]` hooks stay crate-private test helpers.
Consumers:
- dynamo-llm-worker: sequence/direct_zmq.rs, publisher/state_agent.rs, and
publisher/state_agent_host.rs import from
dynamo_runtime::transports::zmq::{direct_fan_in,direct_sub_pool}; lib.rs
re-exports those modules as direct_zmq_fan_in / direct_zmq_sub_pool so the
crate's historical paths still resolve. tmq becomes unused in llm-worker
(only the sub-pool test used it) and is dropped.
- dynamo-llm: the `pub(crate) use` aliases in src/lib.rs point at the
runtime modules directly; kv_router/metrics_subscriber.rs, the indexer
recovery modules, and session_affinity/replica_sync.rs are untouched.
Dependencies: dynamo-runtime gains serial_test = "3" in [dev-dependencies]
for the three serialized fan-in lifecycle tests; no normal dependencies
change, so dynamo-runtime still has no path to dynamo-kv-router,
dynamo-llm-protocol, or dynamo-backend-common. Cargo.lock and
lib/bindings/python/Cargo.lock regenerated (the kvbm and runtime/examples
lockfiles are unchanged).
Validation:
- cargo fmt --all -- --check
- cargo check -p dynamo-runtime -p dynamo-llm-worker --all-targets
- cargo check -p dynamo-llm --all-targets
- cargo check --workspace --all-targets
- cargo clippy --workspace --all-targets -- -D warnings
- cargo test -p dynamo-runtime --lib -- transports::zmq (25 passed)
- cargo test -p dynamo-runtime --lib (862 passed, 3 ignored)
- cargo test -p dynamo-llm-worker (225 passed)
- cargo test -p dynamo-llm --lib (2533 passed, 12 ignored)
- cargo machete lib/llm-worker lib/runtime lib/llm (only the pre-existing
bs62 in llm)
- cargo tree -p dynamo-backend-common -e normal -i dynamo-llm (no match)
- cargo tree -p dynamo-runtime -e normal -i dynamo-kv-router (no match)
- cargo tree -p dynamo-vllm-sidecar -e normal -i dynamo-llm (no match)
- PYO3_PYTHON=.venv/bin/python cargo check --manifest-path
lib/bindings/python/Cargo.toml
- cargo metadata --offline --locked for Cargo.toml, lib/bindings/python,
lib/bindings/kvbm, lib/runtime/examples
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…er library
Dissolve the dynamo-llm-worker crate into dynamo-backend-common, the worker
library the sidecars already build on, and delete lib/llm-worker. dynamo-llm
now depends on dynamo-backend-common for its in-process worker paths and
re-exports every item at its historical path, so no dynamo_llm:: import
changes anywhere.
What moved (git mv; verbatim apart from the path rewrites below):
- src/publisher/ (all modules, tests.rs, metrics.rs, README.md, AGENTS.md,
CLAUDE.md) -> lib/backend-common/src/kv_publisher/
- src/worker_query_endpoint.rs, src/admission_gate.rs + admission_gate/,
src/first_token.rs, src/sequence.rs + sequence/, src/runtime_configs.rs,
src/kv_state.rs, src/kv_source_membership.rs ->
lib/backend-common/src/<same name>
- KV_METRICS_SUBJECT, MULTIMODAL_EMBEDDING_CACHE_SUBJECT,
ACTIVE_SEQUENCES_SUBJECT, WORKER_KV_INDEXER_BUFFER_SIZE -> `pub const`
in lib/backend-common/src/lib.rs
- the direct_zmq_* re-export that the previous commit left in llm-worker
goes away; the modules live in dynamo_runtime::transports::zmq.
Module-name collision: backend-common already had a private `mod publisher`
(src/publisher.rs, the setup_publishers wiring owned by Worker) and
src/snapshot_publisher.rs. The existing module keeps its name and stays
private; the moved publisher directory is mounted as `pub mod kv_publisher`.
Inside the moved code the only references to the old name were
`crate::publisher::metrics::kv_publisher_metrics` in event_processor.rs,
zmq_listener.rs, and state_agent.rs, now `crate::kv_publisher::metrics::...`.
Every other `crate::` path (sequence, kv_state, kv_source_membership,
worker_query_endpoint, the four constants) resolves unchanged because the
siblings moved together. backend-common's own imports change from
`dynamo_llm_worker::{first_token, admission_gate, publisher}` to
`crate::{first_token, admission_gate, kv_publisher}` in adapter.rs,
engine.rs, publisher.rs, snapshot_publisher.rs, and worker.rs.
Visibility: no changes. Everything dynamo-llm and the sidecars reach was
already `pub` since the llm-worker extraction, and backend-common's own
publisher.rs / snapshot_publisher.rs / worker.rs keep their pub(crate)
items.
Features on dynamo-backend-common:
- `testing` (already `testing = []`, gating the conformance kit) now also
gates `FirstTokenSource::from_publisher`, which keeps its
`#[cfg(any(test, feature = "testing"))]`. dynamo-llm enables it in
[dev-dependencies] only, for
entrypoint/input/endpoint.rs::legacy_token_engine_notifies_once_after_
empty_output.
- `integration = ["dynamo-runtime/integration"]` (already present) now also
covers the KV publisher's NATS tests (`#[cfg(all(test, feature =
"integration"))]` in kv_publisher/tests.rs).
Dependencies (all `{ workspace = true }` unless noted):
- backend-common gains blake3, bytes, rand, rmp-serde, thiserror, uuid, and
arc-swap = "1.9.1" (pinned alongside dynamo-llm); dynamo-kv-router gains
features ["metrics", "runtime-protocols", "standalone-selection"];
dev-dependencies gain dynamo-runtime/testing, tokio/test-util, and
tokio-stream. dynamo-llm-worker is dropped. cargo machete reports nothing
unused in backend-common or runtime, only the pre-existing bs62 in llm.
- dynamo-llm: dynamo-llm-worker -> dynamo-backend-common (normal), and
dynamo-backend-common with features = ["testing"] in dev-dependencies.
- dynamo-sglang-sidecar: dynamo-llm-worker dropped; headless.rs imports
dynamo_backend_common::{kv_publisher, runtime_configs,
kv_source_membership}.
- Workspace: lib/llm-worker removed from [workspace] members and
[workspace.dependencies]; the directory is deleted.
Direction invariants hold: dynamo-backend-common has no path to dynamo-llm
and no direct dependency on dynamo-http-server (reached only through
dynamo-runtime, dynamo-kv-router, and dynamo-llm-protocol); dynamo-runtime
has no path to dynamo-kv-router, dynamo-llm-protocol, or
dynamo-backend-common; dynamo-vllm-sidecar has no path to dynamo-llm.
Shims in dynamo-llm, each a one-line retarget:
- src/lib.rs: `pub use dynamo_backend_common::{admission_gate, first_token}`
- src/kv_router.rs: `pub use dynamo_backend_common::kv_publisher as
publisher`, `pub use dynamo_backend_common::sequence`, and the four
constants
- src/discovery.rs: `pub use dynamo_backend_common::kv_source_membership`
and `pub(crate) use dynamo_backend_common::runtime_configs`
- src/discovery/kv_state_agent.rs: `pub use dynamo_backend_common::
kv_state::*`
- src/kv_router/metrics.rs: the registration test imports
dynamo_backend_common::kv_publisher::metrics::KvPublisherMetrics
Housekeeping:
- CODEOWNERS is generated: areas.yaml drops lib/llm-worker/ from the router
area and adds a `shared` row for lib/backend-common/src/kv_publisher/
owned by runtime (the lib/backend-common/ owner) plus router, so the
router team keeps co-ownership of the publisher. build_codeowners.py
--strict passes; emit_codeowners.py regenerated CODEOWNERS and left
CONTRIBUTORS.md identical.
- Root AGENTS.md repository map: the llm-worker entry is gone and
backend-common is described as the worker library. lib/backend-common/
README.md (layout tree) and AGENTS.md (Key Files) list the new modules.
- The two component comments that named lib/llm-worker/src/admission_gate.rs
and lib/llm-worker/src/publisher/ now name lib/backend-common/src/
admission_gate.rs and lib/backend-common/src/kv_publisher/.
- Cargo.lock and lib/bindings/python/Cargo.lock regenerated (the
dynamo-llm-worker package entry is gone); the kvbm and runtime/examples
lockfiles are unchanged.
Validation:
- cargo fmt --all -- --check
- cargo check -p dynamo-backend-common --all-targets
- cargo check -p dynamo-backend-common --all-targets --features testing
- cargo check -p dynamo-llm -p dynamo-sglang-sidecar --all-targets
- cargo check --workspace --all-targets
- cargo clippy --workspace --all-targets -- -D warnings
- cargo test -p dynamo-runtime --lib -- transports::zmq (25 passed)
- cargo test -p dynamo-runtime --lib (862 passed, 3 ignored)
- cargo test -p dynamo-backend-common (386 passed, 4 ignored)
- cargo test -p dynamo-backend-common --features testing (386 passed,
5 ignored)
- cargo test -p dynamo-llm --lib (2533 passed, 12 ignored)
- cargo test -p dynamo-sglang-sidecar -p dynamo-vllm-sidecar
-p dynamo-ext-proc (all passed)
- cargo machete lib/backend-common lib/llm lib/runtime
- cargo tree -p dynamo-backend-common -e normal -i dynamo-llm (no match)
- cargo tree -p dynamo-runtime -e normal -i dynamo-kv-router (no match)
- cargo tree -p dynamo-runtime -e normal -i dynamo-llm-protocol (no match)
- cargo tree -p dynamo-runtime -e normal -i dynamo-backend-common (no match)
- cargo tree -p dynamo-vllm-sidecar -e normal -i dynamo-llm (no match)
- cargo tree -p dynamo-backend-common -e normal -i dynamo-http-server
(transitive only, via dynamo-runtime / dynamo-kv-router /
dynamo-llm-protocol)
- PYO3_PYTHON=.venv/bin/python cargo check --manifest-path
lib/bindings/python/Cargo.toml
- cargo metadata --offline --locked for Cargo.toml, lib/bindings/python,
lib/bindings/kvbm, lib/runtime/examples
- python .github/codeowners/build_codeowners.py --areas
.github/codeowners/areas.yaml --repo . --strict
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
had a problem deploying
to
external_collaborator
October 23, 2026 12:40 — with
GitHub Actions
Failure
GitHub Actions
Failure
Owner
Author
|
Closing: superseded by the router re-architecture plan (stage crates with a shared types leaf, SelectionCore/SelectionService behind dynamo-scheduler, HTTP routes as features on dynamo-http-server). The http-server extraction landed separately via ai-dynamo#17178. Branch left in place for reference. |
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Modularizes
dynamo-llmalong the seam that ai-dynamo#11281 first proposed, redone as 48 small commits intended to be sliced into separate PRs (see "Slicing" below). Behavior is unchanged: every moved item is re-exported fromdynamo-llmat its old path, so the Python/C bindings, mock servers, and test kit compile without edits.Seven new crates, leaf to consumer, plus two existing crates that grew a role:
dynamo-runtime-protocolslib/runtime-protocolsRouterMode,Slugdynamo-metricslib/metricsdynamo-http-serverlib/http-serverdynamo-llm-protocollib/llm-protocoldynamo-local-modellib/local-modelLocalModeland builder, model card loader, HF hub, LoRA download and cache, multimodal routing specs, frontend configdynamo-kvbmlib/kvbmdynamo-kv-dc-relaylib/kv-dc-relaydynamo-llm)dynamo-backend-common(existing)lib/backend-commondynamo-runtime(existing)lib/runtimedynamo-http-serverDependency direction, verified with
cargo tree -e normal -i <dep>:dynamo-http-serveranddynamo-llm-protocolhave no path todynamo-runtime.dynamo-local-model,dynamo-backend-common, anddynamo-kvbmhave no path todynamo-llm.dynamo-sidecar-common, the SGLang sidecar, and the vLLM sidecar have no path todynamo-llmin any feature configuration.dynamo-runtimehas no path todynamo-kv-router,dynamo-llm-protocol, ordynamo-backend-common.dynamo-llmis the composition root; it depends ondynamo-backend-commonfor its in-process worker entrypoint and mocker.Also folded in: default-on feature gates for the text REPL, S3 LoRA source, ModelExpress client, and Swagger UI; the HTTP
Metricssplit into request-level (dynamo_http_server::metrics::RequestMetrics) and LLM families; four pre-existing test compile errors or lints fixed in their own commits (kvbmResultalias, NIXL media loader tests, kvbm all-features clippy). An intermediatedynamo-llm-workercrate (commits 36-38) is dissolved again in commits 47-48 after review: its modules belong indynamo-backend-common.Refs ai-dynamo#11281.
Validation
On the full branch (every step passes):
cargo fmt --all -- --checkcargo check --workspace --all-targets,cargo clippy --workspace --all-targets -- -D warningscargo clippy -p dynamo-kvbm --all-features --all-targets -- -D warningscargo clippy -p dynamo-llm --features media-ffmpeg,mm-routing,testing-nixl --all-targets -- -D warnings(FFmpeg 9 on PATH)dynamo-local-modelalso with--features lora-s3,mm-routing,testing),dynamo-backend-common(386, with and without--features testing),dynamo-kv-router(1104),dynamo-runtime --lib(862),dynamo-llm --lib(2533) plus 8 integration binaries,dynamo-sglang-sidecar,dynamo-vllm-sidecar(also--no-default-features),dynamo-ext-proccargo test -p dynamo-llm --features media-ffmpeg,mm-routing,testing-nixl --lib -- preprocessor::media(95 passed against an FFmpeg built with the container's VP8/VP9-only configure flags)cargo test -p dynamo-kvbm --all-features: 603 of 610 non-GDS tests pass with the NIXL wheel runtime; the GDS_MT-dependent tests need thenvidia_fskernel module, which the test host lacks (the same tests do not compile at all on the base commit)PYO3_PYTHON=... cargo check --manifest-path lib/bindings/python/Cargo.tomlwith and without--features mm-routingcargo metadata --lockedfor the four lockfiles;.github/codeowners/build_codeowners.py --strictTwo tracing-capture tests in the
dynamo-llmlib binary flake under parallel execution (discovery::model_manager::tests::disaggregated_pair_sharing_an_alias_reports_one_claim_per_role,kv_router::prefill_router::admission::tests::bootstrap_cancellation_logging_distinguishes_expected_abort). Running those groups concurrently with themigration::tests, which install a global subscriber, fails 6 of 50 runs on the unmodified base commit versus 1 of 50 here; the files are byte-identical to the base.Slicing
Each commit is one logical change with its own validation list in the message. Suggested PR order: commits 1-3 (W0 feature gates and deletions, independent), 4-14 (W2 in-crate preparation), 15-17 (W1 leaf crates), 18-26 (W3 http-server and llm-protocol), 27-30 (kvbm), 31-33 (status server, relay), 34-35 (metrics split), 36-38 and 47-48 together (worker-side modules into backend-common; squash the intermediate llm-worker crate away when slicing), 39-43 (local-model), 44-46 (multimodal routing, kvbm lints). The branch base is 2470d51; rebase onto main before slicing.