fix(backend): harden administrative route lifecycle

[connorcarpenter15]

Overview:

Harden Backend SDK administrative routes so controls and updates cannot race worker startup, discovery registration, or shutdown.

This is PR 1 of 4 in the vLLM RL-control stack.

Details:

• Gate engine control, engine update, and model-taint routes on the worker serving lifecycle.
• Serialize administrative mutations with discovery transitions.
• Validate control payloads before unregistering healthy capacity.
• Keep partially awakened workers out of discovery.
• Cancel in-flight controls before final shutdown unregister.
• Add narrowly scoped regressions for malformed pause, lifecycle gating, partial wake, and shutdown/resume races.

Where should the reviewer start?

Start with EngineRouteLifecycle and wrap_engine_control_callback in lib/backend-common/src/worker.rs, then review the four integration regressions in the same file.

Validation:

• cargo fmt --all -- --check
• cargo test --locked -p dynamo-backend-common — 137 passed
• cargo clippy --locked -p dynamo-backend-common --all-targets --features integration -- -D warnings
• cargo test --locked -p dynamo-backend-common --features integration — all four new process-local lifecycle regressions passed; five unrelated existing tests require a live NATS server

Stack:

1. #13259 — fix(backend): harden administrative route lifecycle (this PR)
2. #13260 — feat(backend): add RL route discovery endpoint
3. #13261 — feat(vllm): sync RL control protocol
4. #13066 — feat(vllm): route RL controls through sidecar

Linear: DIS-2671

Related Issues

🚫 This PR is NOT linked to a GitHub issue:

• Confirmed — no related GitHub issue

Summary by CodeRabbit

• New Features

  • Added validation for engine control requests before they are processed.
  • Added lifecycle management for engine routes during startup, operation, and shutdown.
  • Added support for pausing and resuming generation through discovery policies.
  • Engine routes now become available only after successful serving registration.

• Bug Fixes

  • Improved handling of partial wake responses, cancellation, registration failures, and shutdown.
  • Prevented invalid or malformed control requests from affecting engine state.

[coderabbitai]

Walkthrough

Changes

Engine route lifecycle

Layer / File(s)	Summary
Startup registration and lifecycle state lib/backend-common/src/engine.rs, lib/backend-common/src/worker.rs	The public LLMEngine trait now supports control validation. The worker gates routes through Starting, Running, and ShuttingDown states, activates routes after registration, and waits for callbacks during shutdown.
Administrative route validation and execution lib/backend-common/src/worker.rs	Administrative routes validate request bodies and engine controls, apply pause and resume discovery policies, reject unavailable lifecycle states, and cancel resume operations during shutdown.
Lifecycle policy and integration coverage lib/backend-common/src/worker.rs	Tests and fixtures cover registration ordering, lifecycle rejection, malformed controls, partial wake responses, and cancellation of in-flight resume callbacks.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: ⚪ Minimal · up to 75d96

The PR hardens administrative route lifecycle behavior and reports passing validation; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: hardening backend administrative route lifecycle handling.
Description check	✅ Passed	The description covers the required overview, details, reviewer starting point, and related-issues section, with useful validation results.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🔇 Additional comments (7)

lib/backend-common/src/engine.rs (1)

337-348: LGTM!

lib/backend-common/src/worker.rs (6)

261-267: LGTM!

Also applies to: 346-356, 407-412, 439-442, 746-755, 1402-1407, 1423-1456, 2147-2162

---

1052-1127: LGTM!

---

1501-1701: LGTM!

---

3277-3290: 📐 Maintainability & Code Quality

⚠️ Unverified finding
Sandbox verification was unavailable.

Verify the local-runtime fixture APIs and the feature gate.

test_local_endpoint avoids an external NATS server. The enclosing module is still gated behind the integration feature, so the four new lifecycle tests do not run in a default cargo test. Confirm that this gating is intended. If the lifecycle tests need no external service, move them into the always-compiled tests module so they run in CI.

Also confirm the dynamo-runtime API shapes used here.

---

3348-3402: LGTM!

---

3524-3703: LGTM!

Also applies to: 3754-3754

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2d5cb9d7-23ea-412c-8865-3bdf7be83569

📥 Commits

Reviewing files that changed from the base of the PR and between c7aa0bf and 75d9641.

📒 Files selected for processing (2)

• lib/backend-common/src/engine.rs
• lib/backend-common/src/worker.rs

[devin-ai-integration]

Devin Review found 3 potential issues.