fix: make shell hotfix startup atomic and fail closed

[plotarmordev]

Problem

The enabled shell hotfix train could fail open in two ways:

• the seven multi-hunk DSV4 backports wrote directly to the installed vLLM tree one hunk at a time, so a later anchor/commit failure could leave an earlier target partially modified;
• Compose skipped missing scripts and suppressed nonzero exits with || true, allowing vllm serve to start after an incomplete hotfix train.

Design

This change makes startup integrity transactional and fail-closed without changing model execution behavior:

• each of the seven DSV4 shell backports validates every target and all 35 anchors against an in-memory staged view before the first write;
• publication uses same-directory temporary files plus atomic rename;
• original bytes and file modes are retained for the whole transaction;
• committed bytes and modes are verified;
• commit, verification, read, corruption, and interrupt failures roll published targets back in reverse order, byte-for-byte and mode-for-mode;
• idempotent reruns detect the fully applied state and make no writes;
• enabled issue22, spin-wait, and all seven DSV4 scripts must exist and exit 0 before vllm serve; a missing or failing script aborts startup;
• the existing DSPARK_SKIP_ISSUE22_HOTFIX, DSPARK_SKIP_SPIN_WAIT_HOTFIX, and DSPARK_SKIP_HOTFIX switches retain their existing scopes;
• PR feat(auth): multi-key API auth via DSPARK_API_KEYS (default off) #89's API-key log-redaction apply/--status gate remains separate from the optional seven-script loop and remains fail-closed.

Changed files

• docker-compose.dspark.yml
• patches/hotfix-dsv4-mtp-buffer-50312.sh
• patches/hotfix-dsv4-adaptive-topk-50004.sh
• patches/hotfix-dsv4-skip-topk-49486.sh
• patches/hotfix-dsv4-dense-prefill-indexer-48407.sh
• patches/hotfix-dsv4-skip-empty-c128-48957.sh
• patches/hotfix-dsv4-flashmla-workspace-50298.sh
• patches/hotfix-dsv4-grammar-advance.sh
• scripts/test-hotfix-atomic-transaction.py
• scripts/ci-validate.sh
• CHANGELOG.md

Explicit non-goals

• no model math, weights, kernels, prompts, sampling, scheduling policy, or performance tuning changes;
• no throughput, quality, speculative-acceptance, memory, or long-context benchmark claims;
• no owner-held .env/credentials/model-data changes;
• no change to the existing hotfix skip-switch semantics.

Pinned image

• tag: ghcr.io/anemll/dspark-vllm-gx10:0.1.1
• image ID: sha256:3430d6614a8e2925f34d059af6caf05aff42387326db4d05639a60f10f2654d8
• manifest digest: sha256:a83948492cf13df455170fb42885f5ef4db54fefe0feff0f841ecbff464ac9d8

Moving-main integration

Current main advanced through PR #89 before publication. The feature branch was rebased onto c1d7b75d3cadd5b39f4786e70475713038176115.

Conflicts were resolved additively:

• docker-compose.dspark.yml: retained PR feat(auth): multi-key API auth via DSPARK_API_KEYS (default off) #89's independent fail-closed API-key redaction gate and added fail-closed missing/nonzero handling for issue22, spin-wait, and the seven transactional scripts;
• scripts/ci-validate.sh: retained auth/redaction suites and added the complete atomic-transaction suite;
• CHANGELOG.md: retained both feature entries under separate Added/Changed sections.

CPU/static validation

Exact candidate head: b73346bef4e991e2af425b5754f453d9763ca17c

bash scripts/ci-validate.sh
git diff --check origin/main...HEAD
python3 -m py_compile scripts/test-hotfix-atomic-transaction.py
python3 scripts/test-hotfix-atomic-transaction.py -v
# Each of the seven modified hotfix scripts was independently checked with bash -n.

Results:

• full repository validation: PASS;
• diff whitespace check: PASS;
• all seven independent shell syntax checks: PASS;
• atomic test module compile: PASS;
• complete atomic transaction suite: 23/23 PASS;
• suite coverage includes all 35 hunks, zero-write validation failures, byte/mode preservation, idempotence, normal caller order, commit/post-rename/verification faults, corruption/read failures, reverse rollback, rollback failure, and interrupt rollback;
• worktree clean after validation.

Disposable pinned-image validation

A disposable container was created from the exact pinned digest with the candidate patches/ mounted read-only.

First ordered train:

1. issue22;
2. spin-wait;
3. MTP buffer;
4. adaptive top-k;
5. short-context top-k skip;
6. dense-prefill indexer;
7. empty-C128 skip;
8. FlashMLA workspace;
9. grammar advance.

Result: all nine scripts exited 0, every verification reported applied, all seven transactional scripts committed their complete staged hunk sets, no anchor mismatch/rollback/partial warning appeared, and no transaction temporary file remained.

Second run in the same container: all nine scripts exited 0 and reported already-applied/idempotent state; no writes or transaction leftovers. The disposable container was removed.

Real local TP=2 startup validation

The exact candidate runtime files were hash-matched into an isolated local-cluster staging directory. The owner-held profile was referenced in place and never edited or copied into git.

Command shape (owner-local profile path redacted from this public description):

ENV_FILE=<owner-held-local-profile> ./start-deepseek-v4-flash-dspark.sh

First cold/recreated start:

• start command exit: 0;
• both ranks used the exact pinned image ID;
• all nine shell hotfixes appeared in the required order before the vLLM launch banner;
• every script applied and verified successfully;
• no missing script, nonzero exit, anchor mismatch, rollback, ambiguous-anchor, partial-application, traceback, or transaction temporary file;
• vllm serve started and application startup completed;
• GET /health: HTTP 200;
• GET /v1/models: HTTP 200 and expected served model deepseek-v4-flash-abliterated;
• deterministic minimal chat request (temperature=0, max_tokens=4): HTTP 200 with a valid chat-completion object and no server error.

Restart validation used the documented stop/start procedure. Containers were recreated from the pinned image, so all nine scripts applied cleanly again rather than observing a preserved writable layer. The second start exited 0; ordered hotfix, health, model-listing, and minimal inference gates all passed again.

The local stack was restored to its exact initial operational state afterward: original worker payload hashes and owner profile hash restored, original model serving on both ranks, GPU utilization 0%, and no established API connections.

Scans and scope gate

• outgoing range contains exactly the expected 11 files listed above;
• no .env, model data, logs, benchmark output, archive, image, or unrelated file is present;
• credential/private-key scan: no real secrets;
• private identifier scan: no internal IP, hostname, username, email, or home path introduced;
• owner-held inputs remained byte-identical.

Independent reviews

• Fable final rebased review: PASS — /tmp/fable-hf13-final-rebased.md, SHA-256 7dfcfefc8355e2dccfb5cffdcbbbb06af54145e3c6a443d9c0b63f988ccc7550
• Daybreak correctness/security review: PASS — /tmp/daybreak-hf13-correctness-final.md, SHA-256 fbf7127f43ba10080be55dc4361013bfa4c59c23aae28541a0f219fdbe61b3f0
• Daybreak maintainability/scope review: PASS, no blockers — /tmp/daybreak-hf13-maintainability-final.md, SHA-256 ae7b2bff02d3968b52b1e8bacc89011978ba04f8eeebb675d8f86081183685aa

All three final reviews are bound to exact candidate head b73346bef4e991e2af425b5754f453d9763ca17c and base c1d7b75d3cadd5b39f4786e70475713038176115.

Why no GPU performance/quality A/B

This change modifies startup publication and failure handling, not model execution behavior. The appropriate GPU gate is real boot, ordered hotfix execution, health, model listing, minimal inference, restart, and restoration. Throughput, quality, acceptance-rate, memory, and long-context A/B benchmarks were intentionally not run.