Skip to content

Make crates.io publishes retryable and resumable - #695

Merged
i386 merged 1 commit into
Mesh-LLM:mainfrom
IvGolovach:codex/publish-crates-rate-limit-retry
May 26, 2026
Merged

Make crates.io publishes retryable and resumable#695
i386 merged 1 commit into
Mesh-LLM:mainfrom
IvGolovach:codex/publish-crates-rate-limit-retry

Conversation

@IvGolovach

Copy link
Copy Markdown
Collaborator

Summary

Makes the crates.io release publish chain resumable after partial uploads and resilient to crates.io new-crate rate limiting.

The publish script now checks whether each crate@workspace_version is already visible on crates.io before real publishing. Already-published crate versions are skipped, missing crate versions are published, and HTTP 429 rate-limit responses are retried with a bounded retry loop using the crates.io retry timestamp when one is provided.

Why

Issue #691 showed the v0.66.0 release could publish GitHub artifacts successfully while the crates.io chain failed partway through after several crates had already uploaded. A plain rerun was not safe enough because the script did not distinguish already-published crates from missing crates, and it treated crates.io 429 responses as permanent publish failures.

This keeps the GitHub release/tag surface separate from crates.io recovery: operators can rerun the publish chain for the same release tag, skip crate versions that already landed, and continue with the missing downstream crates.

Diff scope

  • Adds crates.io registry status classification for published, missing, and unknown crate versions.
  • Skips already-published crate versions before real cargo publish attempts.
  • Retries only crates.io rate-limit failures, with configurable max attempts and fallback backoff.
  • Parses the Please try again after ... timestamp from crates.io when available, with Linux/macOS date parsing support.
  • Fails closed if the registry status cannot be verified before a real publish.
  • Redacts CARGO_REGISTRY_TOKEN from captured cargo diagnostics before replaying output.
  • Adds process-level script tests with mocked cargo, curl, sleep, and date for retry/resume/error behavior.
  • Updates release docs with the missing mesh-llm-api-client crate and partial-publish recovery guidance.

Compatibility

Release tooling/docs only. No runtime behavior, crate API, mesh protocol, protobuf, Skippy ABI, plugin protocol, or release version change.

Prerelease behavior is unchanged: crates.io publishing remains gated to non-prerelease release tags by the existing workflow.

Branch integrity

  • Base branch: main
  • Validated base: origin/main@5c805ead3449c54de631ed4d60fd00ab89d57c0f
  • Branch state at local validation: 0 behind / 1 ahead
  • Head commit: 602d21e3908eeafcc62297f0bf1ab1404ca14e71

Commit integrity

One logical commit:

  • 602d21e3908eeafcc62297f0bf1ab1404ca14e71 - Harden crates.io publish retries

Final diff contains only release docs, the crates.io publish script, and targeted script tests.

Validation

  • Validation tier: Tier 4 - release publish verification tooling and crates.io partial-publish recovery for issue Release crates.io publish chain breaks on HTTP 429 (new-crate rate limit) #691.
  • git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at 5c805ead3449c54de631ed4d60fd00ab89d57c0f.
  • git diff --check: PASS, no output
  • git diff --cached --check: PASS, no output
  • bash -n scripts/publish-crates.sh: PASS
  • scripts/publish-crates.sh --allow-dirty: PASS, rejected because --allow-dirty requires --dry-run.
  • python3 -m unittest scripts.tests.test_publish_crates: PASS, 7 passed
  • python3 -m unittest discover -s scripts/tests: PASS, 27 passed
  • cargo run -p xtask -- repo-consistency release-targets: PASS
  • CARGO_TARGET_DIR=$(mktemp -d) scripts/publish-crates.sh --dry-run --allow-dirty: PASS; dry-run verified publishable crates through model-artifact and deferred downstream crates whose 0.66.0 registry dependencies are not published yet.
  • Ledger: not applicable - not required for selected validation tier/change family.
  • Version: not applicable - release publish tooling/docs only; no release version sync required.
  • Not run: actionlint - not installed locally and workflow file was not changed.
  • Not run: real cargo publish / just release - intentionally avoided for PR validation.

Runtime safety

Not applicable - no runtime path changed.

CI context confirmation

No workflow file or CI context names changed. Required PR CI will run after PR creation/update.

Rollback

Rollback: revert this PR.

DB downgrade: not applicable.
Data repair: not applicable.
Operational caveats: none known.

Known residual risks

This PR cannot prove a real crates.io upload in PR validation because it intentionally does not use a registry token or publish crates. The first production rerun should verify all 11 crate versions on crates.io after the release publish job completes.

crates.io can still enforce account-level new-crate rate limits; this PR makes the release job resumable and retry-aware, but it does not replace requesting a crates.io rate-limit increase if the publish chain continues to grow.

Validation
* Validation tier: Tier 4 - release publish verification tooling and crates.io partial-publish recovery for issue Mesh-LLM#691.
* git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at 5c805ea.
* git diff --check: PASS, no output
* git diff --cached --check: PASS, no output
* bash -n scripts/publish-crates.sh: PASS
* scripts/publish-crates.sh --allow-dirty: PASS, rejected because --allow-dirty requires --dry-run
* python3 -m unittest scripts.tests.test_publish_crates: PASS, 7 passed
* python3 -m unittest discover -s scripts/tests: PASS, 27 passed
* cargo run -p xtask -- repo-consistency release-targets: PASS
* CARGO_TARGET_DIR=$(mktemp -d /tmp/mesh-llm-publish-target.XXXXXX) scripts/publish-crates.sh --dry-run --allow-dirty: PASS; dry-run verified publishable crates through model-artifact and deferred downstream crates whose 0.66.0 registry dependencies are not published yet.
* Ledger: not applicable - not required for selected validation tier/change family.
* Version: not applicable - release publish tooling/docs only; no release version sync required.
* Not run: actionlint - not installed locally and workflow file was not changed.
* Not run: real cargo publish / just release - intentionally avoided for PR validation.

Rollback
* git revert HEAD
@i386

i386 commented May 26, 2026

Copy link
Copy Markdown
Collaborator

LGTM

@i386
i386 merged commit 12aa611 into Mesh-LLM:main May 26, 2026
16 checks passed
IvGolovach added a commit that referenced this pull request May 26, 2026
Summary
Adds a repeatable KV/tool-loop stability certification harness for direct-model agent/tool-call pressure runs, including plan output, manifest evidence, transcript handling, native-log checkpoint scanning, docs, tests, and a repo-local agent skill.

Validation
* Validation tier: Tier 4 - verification tooling and testing documentation for KV/tool-loop stability certification; no runtime, protocol, workflow, or CI gate change.
* git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at 12aa611.
* git rebase origin/main: PASS, rebased cleanly after #695/#696.
* git diff --check origin/main...HEAD: PASS, no output
* git diff --check: PASS, no output
* git diff --cached --check: PASS, no output
* python3 -m unittest scripts.tests.test_qa_kv_tool_loop_stability: PASS, 16 passed
* python3 -m unittest discover -s scripts/tests: PASS, 43 passed
* python3 -m py_compile scripts/qa-kv-tool-loop-stability.py scripts/tests/test_qa_kv_tool_loop_stability.py: PASS
* mkdir -p target/kv-tool-loop-stability && python3 scripts/qa-kv-tool-loop-stability.py --models auto,mesh --attempts 1 --pressure-turns 2 --timeout 30 --min-cached-tokens 128 --suffix-prefill-limit 64 --output-dir target/kv-tool-loop-stability/review-smoke --print-plan > /tmp/kv-tool-loop-plan.json && python3 -m json.tool /tmp/kv-tool-loop-plan.json >/dev/null: PASS
* Remote PR checks after rebase: PASS, changes + summary green; non-applicable build/test matrices skipped by path filters.
* Review/merge state: APPROVED, MERGEABLE, CLEAN.
* Ledger: not applicable - not required for selected validation tier/change family.
* Version: not applicable - verification tooling/docs only; no release/version sync required.
* Not run: cargo check/tests - not required for selected validation tier; no Rust/runtime code changed.
* Not run: live direct-model Skippy certification run - no local loaded direct-model endpoint was available; deterministic unit and plan-smoke coverage proves the harness behavior.

Rollback
* Revert this PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants