docs(ops): JuiceFS mount credential via file-mounted secret; measured cross-node + SSH findings - #2577
Conversation
|
Important Review available on request
Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c68a12ab84
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Secret lives in a 0600 file owned by the invoking user; never committed. | ||
| # /home/<user>/.pmoves-secrets/jfs_meta_pw (mode 600) | ||
| # Mount it read-only and read it at exec time: | ||
| docker run -d --name juicefs-mount --restart unless-stopped \ |
There was a problem hiding this comment.
Operators following this runbook are told to execute a raw docker run, but this repository's damage-control hooks block raw Docker operations unless the user explicitly directs them; consequently the documented remediation cannot be followed through the normal agent/operator workflow and bypasses the canonical restart and environment-injection path. Put the file-mounted-secret implementation behind an appropriate Make target and document that invocation instead.
AGENTS.md reference: AGENTS.md:L22-L35
Useful? React with 👍 / 👎.
| The JuiceFS mount's metadata credential is supplied by a **file-mounted secret** | ||
| read at container start — the `*_FILE` indirection the fleet standardises on | ||
| (`#2492 § 8`, following the `#1901` precedent). This is the pattern to copy for | ||
| any node that mounts JuiceFS. |
There was a problem hiding this comment.
When an operator follows the cross-node runbook for the 4090 or 5090, pmoves/scripts/juicefs-cross-node-setup.sh:99-105 still supplies META_PASSWORD="$DB_PASS" with -e META_PASSWORD, which persists the expanded Supabase admin password in the container's .Config.Env. Thus the new statement that this is the pattern for “any node” and the handoff's RESOLVED status are false for the documented remote-mount path; update that target to consume a mounted secret before generalizing the sign-off.
AGENTS.md reference: AGENTS.md:L171-L177
Useful? React with 👍 / 👎.
| cross-node setup script **refuses** it unless you set `ALLOW_FILE_STORAGE=1` | ||
| (don't, for the shared mount). Verify before mounting: | ||
| | **`pmoves`** | **minio** (tailnet MinIO) | Postgres (`juicefs_meta`) | ✅ yes | | ||
| | **`pmoves-media`** | **minio** — *reformatted; was `file`* | Postgres (`juicefs_meta`) | ✅ yes (as of 2026-08-15) | |
There was a problem hiding this comment.
If an operator follows the related blocker handoff after reading this new pmoves-media status, that handoff still says the live volume is Storage: file and leaves inventory, reformat, and juicefs sync as unchecked next actions (juicefs-cross-node-storage-blocker-2026-08-04.md:3-6,109-113). These mutually exclusive current-state instructions can lead someone to repeat a destructive migration that this line says has already completed; update or archive the stale measured-state and checklist entries in the same change.
Useful? React with 👍 / 👎.
|
|
||
| > **UPDATED 2026-08-15 — the `pmoves-media` file-backend blocker is RESOLVED.** | ||
| > B850's live mount reports its backend as `minio://…/juicefs/pmoves-media`, i.e. | ||
| > the reformat in `JUICEFS_MEDIA_MINIO_REFORMAT_RUNBOOK.md` has been executed. |
There was a problem hiding this comment.
The newly cited JUICEFS_MEDIA_MINIO_REFORMAT_RUNBOOK.md is absent from the repository: a repo-wide filename search and rg find only references to it, and MERGE_ORDER_BRIEF_2026-08-10.md:52 explicitly records that it is absent. Operators therefore cannot inspect the procedure, validation, or rollback behind the newly asserted storage migration; land the runbook or link to the actual retained evidence.
Useful? React with 👍 / 👎.
…s" (#2601) * fix(fleet): knuckles is reachable at pmoves-b850-ai-top, not "knuckles" The capacity registry gave knuckles `reach: "knuckles"`. Its actual Tailscale hostname is pmoves-b850-ai-top (operator-confirmed). `reach` is documented in that file as "Tailscale hostname (NEVER a raw 100.x IP)", so anything resolving knuckles through the registry was pointed at a name that does not resolve — independently of any CI check. Also registers the Jetson edge node pmoves-nano-1. There are three Jetsons; only this one's reach is confirmed, and the other two are deliberately left out. An invented hostname would be worse than an absent one: it would satisfy the anchor check for docs naming a machine nobody can reach. Why the anchor check was flagging correct documentation ------------------------------------------------------- validate_command_anchors.known_hosts() collects `pmoves-[a-z0-9-]+` tokens out of config/fleet-map.yaml and config/operator_nodes.yaml. pmoves-laptop, pmoves-5090, pmoves-spark and pmoves-z890 are known because they appear there as `reach` values. The KVMs and Jetsons are absent from the registry entirely, and knuckles' reach was unprefixed — so every doc using the REAL hostname tripped UNKNOWN_HOST and got recorded as a gap. The baseline had accumulated 12 such entries, all of them real machines: pmoves-kvm2, pmoves-kvm4-1, pmoves-kvm4-2, pmoves-b850, pmoves-b850-ai-top, pmoves-nano, pmoves-nano-1, pmoves-9850x3d-r9700, pmoves-pve-01. The gate was absorbing correct docs as defects because the registry, not the docs, was wrong. This fixes the two with confirmed hostnames and drops their now-stale baseline entries (507 -> 505 findings; ratchet reports 0 new, 0 stale). The remaining ten need their reach values confirmed before the same correction can be applied — they are left recorded rather than guessed. Unblocks #2577, whose two NEW findings were exactly these two hosts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(creator-operator): a registered node is not automatically a schedulable one Addresses the P2 on this PR from chatgpt-codex-connector. Verified against source rather than accepted: the finding is correct. creator-operator/config.py:16 loads pmoves/config/operator_nodes.yaml as NODES_PATH, and router.select_node() picks the LOWEST-VRAM node satisfying min_vram_gb and caps. So adding nano-1 (8 GB, cuda) did not just record a hostname — it made nano-1 WIN every work order with needs ["cuda"] and min_vram_gb <= 8, dispatching those jobs to a host that is not compose-bootstrapped, has no /opt/pmoves and no arm64 worker images. Nothing could consume them. The registry has to name every machine: the anchor validator derives known hostnames from this file, so a node absent from it makes correct documentation look like a typo (that is what this PR set out to fix). But "this box exists" and "a worker can consume a job on it" are different claims, and the registry could previously only make the second one. select_node() now skips entries with schedulable: false. It defaults to True, so every existing entry keeps its behaviour. 4 tests. One deliberately flips the flag back to true and asserts nano-1 then DOES win — without that, the exclusion test would pass even if the filter did nothing. Router suite 23 passed; anchors ratchet still 0 new / 0 stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…easured cross-node + SSH findings Rewrites the mount-credential guidance from a standing "here is what is wrong on this host" description into the resolved pattern plus its verification, and folds in two measured operational findings. Credential handling (B850_BRINGBACK_RUNBOOK.md §6, storage-blocker handoff): the meta DSN carries no credential; the value is supplied by a file-mounted secret read at container start — the *_FILE indirection the fleet standardises on (#2492 §8, #1901 precedent). Documents WHY the file and not `-e VAR`: docker run -e VAR (value-less) persists the expanded value into the container's stored config, so reading from a mounted file is what keeps it out of both argv and stored config. Includes the four-surface verification and the rotation blast radius (~27 consumers of the DB password on B850 — funnel + restart in one window or the data tier drops). Verification gotcha recorded: `ps aux | grep -F "$SEC"` matches the grep's own argv and reports a false positive; use `pgrep -a juicefs` (greps pgrep's output). Cross-node runbook — measured 2026-08-15: * B850 Postgres is docker-internal only; 5432 unreachable from BOTH z890 and nano-1, so no 4090/5090/jetson mount can be established yet. Flagged as a security decision (bind to the tailnet interface, scoped) rather than a mechanical port-publish — the whole data tier is behind that port and pmoves_data is internal:true on purpose. Same class as the NATS leafnode gap. * pmoves-media is now MinIO-backed (live mount reports minio://.../pmoves-media), so the file-backend single-node blocker is RESOLVED. Volume table corrected; the historical warning is kept because storage is fixed at format time and is per-volume — re-check rather than assume. SSH correction (B850 §5): "the ACL denies z890" was WRONG. From a tagged workstation `ssh <host>` defaults to the LOCAL username, and the tailnet error names the USER, not the node — easy to misread as an ACL block. The rule tag:pmoves -> tag:pmoves (autogroup:nonroot) already permits it and z890 carries tag:pmoves; no ACL change was needed. Pass the per-node account (pmoves@b850, pmovesnvme@nano-1). Also records that the account has docker-group but no passwordless sudo, and that mount binds live under another user's home. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5a42189 to
f5d6440
Compare
1 participant
Rewrites the mount-credential guidance from a standing description of what was wrong on a host into the resolved pattern plus its verification, and folds in two measured operational findings.
Credential handling — now the pattern, not a status report
The meta DSN carries no credential; the value comes from a file-mounted secret read at container start — the
*_FILEindirection the fleet standardises on (#2492 §8, #1901 precedent). Applied and verified on B850.Documents why the file and not
-e VAR:docker run -e VAR(value-less) makes Docker persist the expanded value into the container's stored config, so only reading from a mounted file keeps it out of both argv and stored config.Includes the four-surface verification, and the rotation blast radius: ~27 containers on B850 carry the DB password — funnel + restart in one window or the data tier drops.
Cross-node runbook — measured 2026-08-15
5432unreachable from both z890 and nano-1 → no 4090/5090/jetson mount can be established yet. Flagged as a security decision (bind to the tailnet interface, scoped) rather than a mechanical port-publish: the whole data tier sits behind that port andpmoves_dataisinternal:trueon purpose. Same class as the NATS leafnode gap (feat(nats): proven hub leafnode{7422} + EDGE jetson leaf configs; fix 4 latent conf bugs #2576).pmoves-mediais now MinIO-backed (live mount reportsminio://…/pmoves-media) — the file-backend single-node blocker is resolved. Volume table corrected; the historical warning is kept because storage is fixed at format time and is per-volume, so re-check rather than assume.SSH correction — I was wrong, recorded so it isn't repeated
"The ACL denies z890" was wrong. From a tagged workstation
ssh <host>defaults to your local username; the tailnet error names the USER, not the node:The rule
tag:pmoves → tag:pmoves (autogroup:nonroot)already permits it and z890 carriestag:pmoves— no ACL change was needed. Pass the per-node account (pmoves@…b850,pmovesnvme@…nano-1). Also records that the account has docker-group but no passwordless sudo, and that the mount's binds live under a different user's home.🤖 Generated with Claude Code