Skip to content

fix(cargo): restore sparse index version discovery - #11011

Merged
jdx merged 2 commits into
jdx:mainfrom
Turbo87:cargo-sparse-index
Jul 16, 2026
Merged

jdx merged 2 commits into
jdx:mainfrom
Turbo87:cargo-sparse-index

Conversation

@Turbo87

@Turbo87 Turbo87 commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

PR #7295 added VersionInfo.created_at and moved Cargo version discovery from the sparse index to the crates.io versions API because index records did not expose publication timestamps.

Sparse index records include a pubtime field these days which resolves the issue. This restores index lookups and maps that field into VersionInfo.created_at. Missing timestamps remain absent and yanked versions remain filtered.

This removes unnecessary load from the crates.io API servers and improves stability by serving version discovery through the cacheable index.crates.io infrastructure backed by CDNs and S3.

Summary by CodeRabbit

  • Bug Fixes
    • Improved Cargo version discovery by switching to crates.io’s sparse index format.
    • Excluded yanked crate versions from reported results.
    • Displayed Cargo version timing using the crate’s publication time.
    • Updated cargo install error output to match the new crates.io URL path.
  • Tests
    • Updated end-to-end error display expectations for the new URL behavior.
    • Added unit test coverage for sparse index URL generation and parsing.

@coderabbitai

coderabbitai Bot commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 367657c9-8508-46a3-939e-03476a721a49

📥 Commits

Reviewing files that changed from the base of the PR and between ea00de0 and 4fb9928.

📒 Files selected for processing (1)
  • src/backend/cargo.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/backend/cargo.rs

📝 Walkthrough

Walkthrough

Cargo version discovery now uses crates.io’s sparse index, parses newline-delimited version records, excludes yanked releases, and maps pubtime to creation timestamps. Error expectations and unit tests reflect the new index URL and parsing behavior.

Changes

Cargo sparse index migration

Layer / File(s) Summary
Sparse index URL and version parsing
src/backend/cargo.rs
Cargo derives sparse index paths, parses newline-delimited records with pubtime, excludes yanked versions, and tests URL mapping and parsing.
Cargo discovery integration and expectations
src/backend/cargo.rs, e2e/cli/test_error_display
Remote version discovery uses the sparse index parser, while friendly and detailed error assertions expect the new crates.io index URL.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CargoBackend
  participant CratesIoSparseIndex
  participant parse_crate_versions
  CargoBackend->>CratesIoSparseIndex: request crate index URL
  CratesIoSparseIndex-->>CargoBackend: return newline-delimited records
  CargoBackend->>parse_crate_versions: parse response text
  parse_crate_versions-->>CargoBackend: return non-yanked versions
Loading

Poem

A rabbit hops through crates.io,
Finds sparse paths in rows,
Skips the yanked, keeps the bright,
Maps pubtime just right,
And updates errors in delight.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately and concisely summarizes the main change: restoring Cargo version discovery via the sparse index.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

Replaces the crates.io REST API (/api/v1/crates/{name}/versions) with the cacheable sparse index (index.crates.io) for Cargo version discovery, using the pubtime field now available in index records to populate VersionInfo.created_at.

  • get_crate_url builds the correct sparse index URL per the 1/2/3/4+ char naming convention, with ASCII validation before byte slicing.
  • parse_crate_versions streams NDJSON via serde_json::Deserializer, filtering yanked entries; the old .rev() is dropped because sparse index entries are already in oldest-first order.
  • The e2e error-display test is updated to match the new 404 URL, and four new unit tests cover URL generation, input validation, and NDJSON parsing.

Confidence Score: 5/5

Safe to merge — the change is a straightforward backend swap with well-bounded scope and good test coverage.

The sparse index URL construction is correct for all name-length cases, ASCII validation makes the byte slicing safe, and the NDJSON streaming parser handles missing pubtime and yanked entries cleanly. The e2e test expectation is updated consistently with the new endpoint. No regressions introduced.

No files require special attention.

Important Files Changed

Filename Overview
src/backend/cargo.rs Replaces crates.io REST API version lookup with sparse index (index.crates.io) NDJSON parsing; adds get_crate_url, parse_crate_versions, CrateVersion struct, and unit tests covering URL layout, validation, and parsing.
e2e/cli/test_error_display Updates expected 404 error URL for the cargo backend test from the crates.io REST endpoint to the sparse index URL, matching the new lookup path.

Reviews (3): Last reviewed commit: "fix(cargo): reject invalid crate names" | Re-trigger Greptile

Comment thread src/backend/cargo.rs

fn parse_crate_versions(response: &str) -> eyre::Result<Vec<VersionInfo>> {
let mut versions = vec![];
for version in Deserializer::from_str(response).into_iter::<CrateVersion>() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The sparse index format guarantees that version entries are stored in publication order (oldest first), which is why .rev() from the old API path was dropped. A short comment here would make that implicit assumption explicit and prevent a future reader from "restoring" the reversal by mistake.

Suggested change
for version in Deserializer::from_str(response).into_iter::<CrateVersion>() {
// Sparse index records are stored oldest-first (unlike the crates.io REST API).
for version in Deserializer::from_str(response).into_iter::<CrateVersion>() {

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/backend/cargo.rs`:
- Around line 417-429: Validate the normalized name at the start of
get_crate_url, rejecting empty and non-ASCII crate names before any byte
slicing; preserve the existing sparse-index URL construction for valid names.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: f76f3519-6807-42db-9938-6eabc0f6215d

📥 Commits

Reviewing files that changed from the base of the PR and between d2276c3 and 0ecf803.

📒 Files selected for processing (2)
  • e2e/cli/test_error_display
  • src/backend/cargo.rs

Comment thread src/backend/cargo.rs Resolved
PR jdx#7295 added `VersionInfo.created_at` and moved Cargo version discovery from the sparse index to the crates.io versions API because index records did not expose publication timestamps.

Sparse index records include a `pubtime` field these days which resolves the issue. This restores index lookups and maps that field into `VersionInfo.created_at`. Missing timestamps remain absent and yanked versions remain filtered.

This removes unnecessary load from the crates.io API servers and improves stability by serving version discovery through the cacheable `index.crates.io` infrastructure backed by CDNs and S3.
@Turbo87
Turbo87 force-pushed the cargo-sparse-index branch from 0ecf803 to ea00de0 Compare July 16, 2026 07:29
This rejects empty and non-ASCII Cargo crate names before constructing sparse index paths.

It prevents byte-slice panics and returns a normal backend error for invalid input.
@jdx
jdx merged commit 5508db1 into jdx:main Jul 16, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants