Make crates.io publishes retryable and resumable - #695
Merged
i386 merged 1 commit intoMay 26, 2026
Conversation
Validation * Validation tier: Tier 4 - release publish verification tooling and crates.io partial-publish recovery for issue Mesh-LLM#691. * git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at 5c805ea. * git diff --check: PASS, no output * git diff --cached --check: PASS, no output * bash -n scripts/publish-crates.sh: PASS * scripts/publish-crates.sh --allow-dirty: PASS, rejected because --allow-dirty requires --dry-run * python3 -m unittest scripts.tests.test_publish_crates: PASS, 7 passed * python3 -m unittest discover -s scripts/tests: PASS, 27 passed * cargo run -p xtask -- repo-consistency release-targets: PASS * CARGO_TARGET_DIR=$(mktemp -d /tmp/mesh-llm-publish-target.XXXXXX) scripts/publish-crates.sh --dry-run --allow-dirty: PASS; dry-run verified publishable crates through model-artifact and deferred downstream crates whose 0.66.0 registry dependencies are not published yet. * Ledger: not applicable - not required for selected validation tier/change family. * Version: not applicable - release publish tooling/docs only; no release version sync required. * Not run: actionlint - not installed locally and workflow file was not changed. * Not run: real cargo publish / just release - intentionally avoided for PR validation. Rollback * git revert HEAD
Collaborator
|
LGTM |
IvGolovach
added a commit
that referenced
this pull request
May 26, 2026
Summary Adds a repeatable KV/tool-loop stability certification harness for direct-model agent/tool-call pressure runs, including plan output, manifest evidence, transcript handling, native-log checkpoint scanning, docs, tests, and a repo-local agent skill. Validation * Validation tier: Tier 4 - verification tooling and testing documentation for KV/tool-loop stability certification; no runtime, protocol, workflow, or CI gate change. * git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at 12aa611. * git rebase origin/main: PASS, rebased cleanly after #695/#696. * git diff --check origin/main...HEAD: PASS, no output * git diff --check: PASS, no output * git diff --cached --check: PASS, no output * python3 -m unittest scripts.tests.test_qa_kv_tool_loop_stability: PASS, 16 passed * python3 -m unittest discover -s scripts/tests: PASS, 43 passed * python3 -m py_compile scripts/qa-kv-tool-loop-stability.py scripts/tests/test_qa_kv_tool_loop_stability.py: PASS * mkdir -p target/kv-tool-loop-stability && python3 scripts/qa-kv-tool-loop-stability.py --models auto,mesh --attempts 1 --pressure-turns 2 --timeout 30 --min-cached-tokens 128 --suffix-prefill-limit 64 --output-dir target/kv-tool-loop-stability/review-smoke --print-plan > /tmp/kv-tool-loop-plan.json && python3 -m json.tool /tmp/kv-tool-loop-plan.json >/dev/null: PASS * Remote PR checks after rebase: PASS, changes + summary green; non-applicable build/test matrices skipped by path filters. * Review/merge state: APPROVED, MERGEABLE, CLEAN. * Ledger: not applicable - not required for selected validation tier/change family. * Version: not applicable - verification tooling/docs only; no release/version sync required. * Not run: cargo check/tests - not required for selected validation tier; no Rust/runtime code changed. * Not run: live direct-model Skippy certification run - no local loaded direct-model endpoint was available; deterministic unit and plan-smoke coverage proves the harness behavior. Rollback * Revert this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the crates.io release publish chain resumable after partial uploads and resilient to crates.io new-crate rate limiting.
The publish script now checks whether each
crate@workspace_versionis already visible on crates.io before real publishing. Already-published crate versions are skipped, missing crate versions are published, and HTTP 429 rate-limit responses are retried with a bounded retry loop using the crates.io retry timestamp when one is provided.Why
Issue #691 showed the v0.66.0 release could publish GitHub artifacts successfully while the crates.io chain failed partway through after several crates had already uploaded. A plain rerun was not safe enough because the script did not distinguish already-published crates from missing crates, and it treated crates.io 429 responses as permanent publish failures.
This keeps the GitHub release/tag surface separate from crates.io recovery: operators can rerun the publish chain for the same release tag, skip crate versions that already landed, and continue with the missing downstream crates.
Diff scope
published,missing, andunknowncrate versions.cargo publishattempts.Please try again after ...timestamp from crates.io when available, with Linux/macOS date parsing support.CARGO_REGISTRY_TOKENfrom captured cargo diagnostics before replaying output.cargo,curl,sleep, anddatefor retry/resume/error behavior.mesh-llm-api-clientcrate and partial-publish recovery guidance.Compatibility
Release tooling/docs only. No runtime behavior, crate API, mesh protocol, protobuf, Skippy ABI, plugin protocol, or release version change.
Prerelease behavior is unchanged: crates.io publishing remains gated to non-prerelease release tags by the existing workflow.
Branch integrity
mainorigin/main@5c805ead3449c54de631ed4d60fd00ab89d57c0f0 behind / 1 ahead602d21e3908eeafcc62297f0bf1ab1404ca14e71Commit integrity
One logical commit:
602d21e3908eeafcc62297f0bf1ab1404ca14e71- Harden crates.io publish retriesFinal diff contains only release docs, the crates.io publish script, and targeted script tests.
Validation
git fetch --no-tags origin main:refs/remotes/origin/main: PASS,origin/mainat5c805ead3449c54de631ed4d60fd00ab89d57c0f.git diff --check: PASS, no outputgit diff --cached --check: PASS, no outputbash -n scripts/publish-crates.sh: PASSscripts/publish-crates.sh --allow-dirty: PASS, rejected because--allow-dirtyrequires--dry-run.python3 -m unittest scripts.tests.test_publish_crates: PASS, 7 passedpython3 -m unittest discover -s scripts/tests: PASS, 27 passedcargo run -p xtask -- repo-consistency release-targets: PASSCARGO_TARGET_DIR=$(mktemp -d) scripts/publish-crates.sh --dry-run --allow-dirty: PASS; dry-run verified publishable crates throughmodel-artifactand deferred downstream crates whose 0.66.0 registry dependencies are not published yet.cargo publish/just release- intentionally avoided for PR validation.Runtime safety
Not applicable - no runtime path changed.
CI context confirmation
No workflow file or CI context names changed. Required PR CI will run after PR creation/update.
Rollback
Rollback: revert this PR.
DB downgrade: not applicable.
Data repair: not applicable.
Operational caveats: none known.
Known residual risks
This PR cannot prove a real crates.io upload in PR validation because it intentionally does not use a registry token or publish crates. The first production rerun should verify all 11 crate versions on crates.io after the release publish job completes.
crates.io can still enforce account-level new-crate rate limits; this PR makes the release job resumable and retry-aware, but it does not replace requesting a crates.io rate-limit increase if the publish chain continues to grow.