Skip to content

docs(ops): the metadata credential has an owner and a vendor-sourced procedure - #2709

Merged
POWERFULMOVES merged 2 commits into
mainfrom
docs/juicefs-meta-credential-runbook
Aug 24, 2026
Merged

POWERFULMOVES merged 2 commits into
mainfrom
docs/juicefs-meta-credential-runbook

Conversation

@POWERFULMOVES

Copy link
Copy Markdown
Owner

"Someone creates the GitHub Prod secret" isn't an assignment. Provisioning the JuiceFS metadata credential gates every node's ability to mount pmoves-media, so it belongs to Z890 (Infrastructure Coordinator) — not to whichever node notices it's missing. B850 holds the metadata DB and shouldn't become the ambient owner of a secret it merely consumes.

Written against JuiceFS community docs rather than our own habits. The comparison found four things:

finding
✅ ALIGNED Dedicated user + network-scoped pg_hba is the vendor's own recommended shape. #2702 landed it scoped to the tailnet. Worth recording that this was vendor practice, not local invention — and that the doc pairs the role with the hba rule, which is the pairing the original lane handoff was missing.
⚠️ PARTIAL META_PASSWORD is documented and we use it. META_PASSWORD_FILE is also documented, and B850 already bind-mounts the secret as a file before reading it back through a shell. Adopt on next recreate.
⚠️ DIVERGENT sslmode=disable. Unremarkable container-to-container; not unremarkable once :5432 is on the tailnet. WireGuard covers the transport, but the session has no TLS of its own. Flagged as a decision to make at Step 4 rather than inherit.
❌ GAP The vendor asks for restore tests, not just backups: "do some tests to restore the data in an experimental environment to confirm that the backup is valid." B850 dumps metadata hourly to MinIO and has never restored one. An untested backup is a hypothesis. Z890 task, tracked separately.

Plus a constraint worth writing down before someone proposes the obvious fix:

"PostgreSQL does not yet support Multi-Shard (Distributed) transactions, do not use a multi-server distributed architecture for the JuiceFS metadata."

B850 is an SPOF for metadata by vendor design. Availability work belongs in restore, not replication.

Also corrects the mount runbook

It still told operators DB_PASS was "the Supabase DB password". That predates the scoped-role cutover and is now actively wrong — pg_hba rejects supabase_admin from the tailnet, so following it produces an auth error that reads like a bad secret rather than a rejected role. It now passes META_ROLE=juicefs_meta explicitly, since the script still defaults to supabase_admin for back-compat.

Note on the vendor quote

Their pg_hba example carries a private LAN CIDR, so it's paraphrased rather than quoted. The no-LAN-IPs guard blocked my first draft and was right to — the rule holds even when the address is someone else's documentation.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz

…procedure

"Someone creates the GitHub Prod secret" is not an assignment. Provisioning the
JuiceFS metadata credential is fleet infrastructure — it gates every node's
ability to mount pmoves-media — so it belongs to Z890 (Infrastructure
Coordinator), not to whichever node happens to notice it is missing. B850 holds
the metadata DB and should not become the ambient owner of a secret it merely
consumes.

Written against JuiceFS community documentation rather than our own habits, and
the comparison found four things:

  ALIGNED    Dedicated user + network-scoped pg_hba is the vendor's own
             recommended shape. PR #2702 landed it scoped to the tailnet. Worth
             recording that this was vendor practice and not local invention --
             and that the doc pairs the role WITH the hba rule, which is the
             pairing the original lane handoff was missing.

  PARTIAL    META_PASSWORD is documented and we use it. META_PASSWORD_FILE is
             also documented, and B850 already bind-mounts the secret as a file
             before reading it back through a shell. Adopt on next recreate.

  DIVERGENT  sslmode=disable. Unremarkable container-to-container; not
             unremarkable once :5432 is on the tailnet. WireGuard covers the
             transport, but the session has no TLS of its own. Flagged as a
             decision to make AT Step 4 rather than inherit.

  GAP        The vendor asks for restore TESTS, not just backups. B850 dumps
             metadata hourly to MinIO and has never restored one. An untested
             backup is a hypothesis. Z890 task, tracked separately.

Plus a constraint worth writing down before someone proposes the obvious fix:
PostgreSQL metadata must stay single-server per vendor guidance, so B850 is an
SPOF by design and availability work belongs in restore, not replication.

Also corrects the mount runbook, which still told operators DB_PASS was "the
Supabase DB password". That predates the scoped-role cutover and is now actively
wrong: pg_hba REJECTS supabase_admin from the tailnet, so following it produces
an auth error that reads like a bad secret rather than a rejected role. It now
passes META_ROLE explicitly, since the script still defaults to supabase_admin
for back-compat.

The vendor's pg_hba example is paraphrased rather than quoted -- it carries a
private LAN CIDR, and the no-LAN-IPs guard is right to block that in a public
repo even when the address is someone else's documentation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz
@github-actions github-actions Bot added the docs Documentation label Aug 24, 2026
@coderabbitai

coderabbitai Bot commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8082ac5f-a583-441f-8fe9-dd292042d7f8


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b5b1c7332

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pmoves/docs/operations/JUICEFS_CROSS_NODE_MOUNT_RUNBOOK.md
Comment thread pmoves/docs/operations/JUICEFS_META_CREDENTIAL_RUNBOOK.md
Comment thread pmoves/docs/operations/JUICEFS_META_CREDENTIAL_RUNBOOK.md Outdated
…tecting

Two Codex P1s, both correct.

1. ROTATION CONTRADICTED ITSELF. Section 2.1 says never paste the value on a
   CLI, then 2.2-alt handed over a `psql -c "ALTER ROLE … PASSWORD '<new>'"`
   that puts it in shell history AND in the container's argv. `psql -v` is no
   better — same argv. Replaced with a stdin form: `read -rs` (no echo, no
   history) piped through a shell builtin, so the value never becomes another
   process's argv. Validated against the live DB on a throwaway role: ALTER ROLE
   applied and the role authenticated with the piped value over a real scram
   path. Also noted where it belongs long-term — behind a Make target reading
   stdin, next to secrets-rotate.

2. DELIVERY ORDER WAS SILENTLY WRONG FOR RUNNERLESS NODES. `secrets-funnel-from-prod`
   calls pull_chit_bundle.sh, which takes the newest ALREADY-successful run:

       gh run list --workflow "$WORKFLOW" --status success --limit 1

   That run can predate the Prod secret you just created. The pull succeeds, the
   funnel reports success, and JUICEFS_META_PASSWORD is absent or stale — on the
   5090, which is exactly the runnerless node this slot exists for. The runbook
   now requires a producer run FIRST, an explicit wait, and a createdAt
   comparison against when the secret was set. That timestamp check is the whole
   guarantee.

The third finding is a merge-order dependency, not a doc defect: the mount
snippet's `META_ROLE=juicefs_meta` with no DB_PASS needs the target change in
#2708. Merge #2708 first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz
@POWERFULMOVES

Copy link
Copy Markdown
Owner Author

All three verified; two were doc defects and one is a merge-order dependency.

1. Rotation contradicted itself — fixed. Section 2.1 says never paste the value on a CLI, then 2.2-alt handed over a psql -c "ALTER ROLE … PASSWORD '<new>'". You're right on both counts, and psql -v is no better — same argv. Replaced with a stdin form (read -rs → shell builtin printf → docker exec -i psql), so the value never becomes another process's argv. Validated on a throwaway role against the live DB: ALTER ROLE applied, role authenticated with the piped value over a real scram path, juicefs_meta untouched. Also recorded where it belongs long-term — behind a Make target reading stdin, next to secrets-rotate.

2. Delivery order — fixed, and this was the dangerous one. Confirmed pull_chit_bundle.sh:45-46 takes the newest already-successful run:

gh run list --workflow "$WORKFLOW" --status success --limit 1

which can predate the Prod secret. Pull succeeds, funnel reports success, key absent or stale — on the 5090, the exact runnerless node this slot exists for. The runbook now requires a producer run first, an explicit wait, and a createdAt comparison against when the secret was set. That timestamp check is the whole guarantee.

3. Mount snippet — merge-order dependency, not a doc defect. META_ROLE=juicefs_meta with no DB_PASS needs the target change in #2708, which now resolves the funnel password through scripts/with-env.sh. Merging #2708 first, after which this snippet is accurate as written.

@POWERFULMOVES
POWERFULMOVES merged commit 137fecf into main Aug 24, 2026
22 checks passed
@POWERFULMOVES
POWERFULMOVES deleted the docs/juicefs-meta-credential-runbook branch August 24, 2026 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant