Skip to content

volumes: read a held volume live in ls/du/cat/get (ENG-7113) - #1113

Merged
cabrinha merged 2 commits into
mainfrom
feature/volumes-live-reads
Oct 29, 2026
Merged

cabrinha merged 2 commits into
mainfrom
feature/volumes-live-reads

Conversation

@cabrinha

Copy link
Copy Markdown
Contributor

What

ENG-7113: while a run (or a read-write session) holds a volume, prime volumes ls, du, cat and get read the volume as the writer has it right now, so a run's checkpoints can be listed and downloaded mid-run. Before this change those commands read R2, which catches up only when the run ends.

This needs platform#7011. That PR owns the source decision; the CLI only asks for it.

How

  • ls/du/cat (via _volume_reads) and get (via _transfer) call the transfer route with live: true.
    • Writer live: the platform answers via: session with a read-only live view that mounts the writer's PVC. It starts the view or reuses an existing one, and the CLI waits for it the same way ssh does.
    • Otherwise: the platform answers via: r2 as before. A note, for example "the run X is syncing its changes back …", is printed to stderr.
  • Over the live session:
    • ls: one find … -exec stat -c '%s|%F|%Y|%n' (BusyBox in the session image, which is Alpine). The output is rendered in exactly the storage shapes: plain, -l/-h and -R, with directories as dir/ and --output json unchanged. runs/.sessions/ is hidden, as before.
    • du: one whole-volume find that counts files. A live volume has no directory markers.
    • cat: streams cat over ssh to stdout as bytes.
    • get: the existing session transfer path (rsync, else scp), given the view.
  • Session progress and live/sync notes go to stderr whenever stdout carries data. _open_session and the wait helpers take a ui console for this. ssh --read-only also says when it is a live view.
  • Puts never ask for a live route.
  • Help text: "A live read-write session or run's changes appear once it ends." is replaced on ls/du/cat, and the get help and README are updated.

Compatibility: older backends ignore live and keep routing to R2. A via: session read route used to be refused ("a session started on the volume; rerun"). It is now served through that session.

Tests

New tests/test_volume_live_reads.py:

  • Real BusyBox: the remote commands run in alpine:3.22 (the volume-ssh image's base) over a temp dir mounted at /volume, in place of ssh. These tests skip without Docker. They cover:
    • ls json/plain/-l/-R over a run's checkpoint tree, with internal sync state hidden and notes on stderr only
    • missing paths for ls/du/cat, and a file named with a trailing /
    • du subtree and whole-volume counts
    • binary cat
  • Hermetic:
    • stat-line parsing, including a | in a name and newline tails
    • get routing through the live view, with live=True and the route's session reused
    • puts never send live
    • the R2 route's note printed to stderr
    • the client wire contract (live sent only when asked; note, liveViewOf)

Existing tests: the routing assertions now include live: True, and fake routes accept live.

cd packages/prime
uv run --extra dev pytest -q tests/
4 failed, 1754 passed, 3 skipped

The 4 failures are test_put_refuses_a_symlink_source[*]. They fail the same way on a clean origin/main checkout here: the macOS temp path wraps "points to" across a line. They are unrelated to this PR. ruff check, ruff format (changed files) and ty check src/ are clean.

🤖 Generated with Claude Code

While a run or read-write session holds a volume, ls, du, cat and get ask
the platform for a live read and go through the read-only SSH view it
starts or reuses, running the listing remotely in the same output formats.
Otherwise, and while a finished writer still syncs back, they read storage
and print why. (ENG-7113)
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T17:29:13.525479Z e87405b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e87405bce5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

A filename may contain newlines, so a line can never bound a live stat
record: the remote loop now prints one NUL-terminated record per path
(BusyBox stat has no NUL escape, so the shell prints the terminators).
du only matches a file operand exactly when it has no trailing slash,
like the storage implementation.
@cabrinha
cabrinha merged commit 7502f12 into main Oct 29, 2026
26 checks passed
@cabrinha
cabrinha deleted the feature/volumes-live-reads branch October 29, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants