Skip to content

docs(pmoves-yt): service runbook — two deployment paths run different yt-dlp builds - #2801

Merged
POWERFULMOVES merged 4 commits into
mainfrom
docs/pmoves-yt-runbook
Aug 28, 2026
Merged

docs(pmoves-yt): service runbook — two deployment paths run different yt-dlp builds#2801
POWERFULMOVES merged 4 commits into
mainfrom
docs/pmoves-yt-runbook

Conversation

@POWERFULMOVES

Copy link
Copy Markdown
Owner

The pmoves-yt lane had README.md and UPDATE_NOTES.md but no operations doc. This adds
pmoves/docs/services/pmoves-yt/RUNBOOK.md (652 lines) covering bring-up (all six variants), health
verification, the cookie chain, log locations, failure triage by operator-visible symptom, and
teardown.

Every command in it was run on B850/Knuckles and its real output pasted. Claims that could not be
verified from this node are marked UNVERIFIED rather than written as plausible text — §9 lists
all four.

It leads with the discriminator

Because it makes every other answer right or wrong:

up-yt up-yt-published
Builds from ../PMOVES.YT — the fork's vendored yt-dlp pmoves/services/pmoves-yt/Dockerfile
yt-dlp CHIT signing, NATS publish, SoundCloud fixes, phase9c multi-client pip install yt-dlp[default]stock PyPI, without those commits

The published image is built by integrations-ghcr.matrix.json from the path README.md itself
calls "a compatibility mirror/shim, not the source of truth".

Verification sharpened that in two ways

1. The image name is a hint, not the fact. docker-compose.integrations.images.yml overrides
only image: — the base service's build: {context: ../PMOVES.YT} survives the merge,
confirmed with docker compose config on both chains. A node with the submodule checked out can end
up running a locally-built fork image wearing the GHCR name. The runbook therefore gives three
commands, not one, and names /healthz's yt_dlp_version as authoritative.

This refines YTDLP_CURRENCY.md §1b (merged as #2793 mid-session), which maps
ghcr.io/... → "published image (stock PyPI yt-dlp)". The runbook says so explicitly rather than
silently diverging. Both docs agree on the conclusion that matters: when the layers disagree, the
running container wins.

2. RepoDigests does NOT discriminate build-vs-pull. This node's locally-built image carries
one (sha256:bf62dae4...). Checked before documenting it, not after.

Two undocumented failure modes found while verifying

Both in the "healthy but lying" class:

  • pmoves-yt on this node runs cookie-less. YT_COOKIES reads the base-compose placeholder
    darkxside.youtube.cookies.txt, that file does not exist (the dir holds only .gitignore), and no
    yt-cookies-vol is mounted. Reading YT_COOKIES is now the documented one-command check for "is
    the cookie chain actually wired in". Stated plainly: up-yt-published and up-yt-hardened are
    cookie-less as written
    — both use $(DC) and neither passes --profile yt-cookies. Expected on
    those paths, not a fault.
  • yt-cookie-writer reports healthy with a dead NATS subscription. Its healthcheck asserts
    only that /tmp/healthy exists — touched once at first subscribe, never removed. Measured: a DNS
    failure at 2026-08-27T14:10:15Z sitting behind a marker file dated Aug 15, FailingStreak: 0,
    RestartCount: 0.

Commands actually run

yt-docs-catalog-smoke (1751 extractors) · yt-jellyfin-smoke (extraction PASSed on
dQw4w9WgXcQ; the FAIL is the Jellyfin bridge on :8093, absent here) · POST /yt/info · bgutil
/ping1.2.2 · make -n up-yt-amd (2 amd.yml refs vs 0 for up-yt) · docker compose config
on both chains.

Also documented

  • The git submodule update --remote downgrade hazard, with its reproduction: the pin
    ebd39b7f sits on the unmerged feat/upstream-sync-2026.07 (yt-dlp 2026.07.04), while
    .gitmodules declares PMOVES.AI-Edition-Hardened (2026.02.04) — --remote moves it back five
    months.
  • AMD/ROCm nodes need the -amd targets (fix(compose): the AMD overlay no Known Road could reach — retroactive review of 6a7499dfc #2798), with the could not select device driver "nvidia"
    symptom and the DeviceRequests check.
  • No down-yt Known Road exists (grep -c → 0), so teardown uses docker compose ... stop
    permitted, since the enforcer's regex blocks only up|restart|down.
  • make seed-data destroys pmoves_chunks_qwen3, flagged loudly.
  • yt-cookies-status fails from the host on in-network SUPABASE_URL (papercut, not a pipeline fault).

Scope

Version/currency questions are deferred to YTDLP_CURRENCY.md, not restated; §2.1 was compressed
to a pointer once #2793 merged. No Makefile or compose file touched. Register resolved as UNION —
git diff origin/main --numstat shows 8 insertions, 0 deletions on AGNOTE4482PHI.t1.md.

CHIT trail signed: hmac BkvRVrRYkKYV1GD8JtqA4ZUfCxOIGcPduZe9fiITnAE=, kid chit-signing-v01.

🤖 Generated with Claude Code

POWERFULMOVES and others added 2 commits August 27, 2026 21:40
… yt-dlp builds

The pmoves-yt lane had README.md and UPDATE_NOTES.md but no operations doc.
This adds RUNBOOK.md: bring-up, health verification, the cookie chain, logs,
failure triage by operator-visible symptom, and teardown. Every command was run
on B850/Knuckles and its real output pasted; unverifiable claims are marked
UNVERIFIED rather than written as plausible text.

Leads with the discriminator, because it makes every other answer right or wrong:

  up-yt           builds ../PMOVES.YT      -> the fork's vendored yt-dlp
  up-yt-published ghcr.io/.../pmoves-yt    -> built from the services/pmoves-yt
                                              shim, which pip installs STOCK
                                              yt-dlp[default] from PyPI

Verification sharpened that in two ways:

- The image name is a hint, not the fact. docker-compose.integrations.images.yml
  overrides only `image:`; the base `build: {context: ../PMOVES.YT}` survives the
  merge (confirmed via `docker compose config` on both chains). A node with the
  submodule checked out can run a locally-built fork image wearing the GHCR name.
  So: check the image for intent, /healthz's yt_dlp_version for fact.
- RepoDigests does NOT discriminate build-vs-pull — this node's locally-built
  image carries one. Checked before documenting it.

Two undocumented failure modes found while verifying, both "healthy but lying":

- pmoves-yt here runs cookie-less: YT_COOKIES reads the base-compose placeholder
  darkxside.youtube.cookies.txt, that file does not exist, and no yt-cookies-vol
  is mounted. Reading YT_COOKIES is now the one-command wiring check. Note
  up-yt-published and up-yt-hardened are cookie-less as written — both use $(DC)
  and neither passes --profile yt-cookies.
- yt-cookie-writer reports healthy with a dead NATS subscription: its healthcheck
  only asserts /tmp/healthy exists, touched once at first subscribe and never
  removed. Measured a DNS failure at 14:10:15Z behind a marker dated Aug 15,
  FailingStreak 0, RestartCount 0.

Also documented: no down-yt Known Road exists, so teardown uses
`docker compose ... stop` (the enforcer blocks only up|restart|down); the
--remote submodule downgrade hazard with its reproduction; AMD -amd targets;
and that `make seed-data` destroys pmoves_chunks_qwen3.

Version/currency questions are deferred to YTDLP_CURRENCY.md (#2793), not
restated here. No Makefile or compose file touched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz
… refinement

#2793 merged mid-session, so YTDLP_CURRENCY.md now exists on main. Two changes:

- §2.1 duplicated its §1a (pin-reading mechanism). Compressed to the two
  commands an operator needs mid-incident plus a pointer, per 'link, do not
  restate'.
- §1.2 refines its §1b rather than silently diverging. That section maps
  ghcr.io/... -> 'published image (stock PyPI yt-dlp)'. True for a genuinely
  pulled image, but not guaranteed by the name: the images overlay overrides
  only `image:`, leaving `build: {context: ../PMOVES.YT}` intact, so a node
  with the submodule checked out can build the fork and tag it with the GHCR
  name. Both docs agree on the conclusion — the running container wins.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz
@coderabbitai

coderabbitai Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 5d1aaffd-efa2-47e3-b93d-964aeb9645fe


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added docs Documentation governance AGNOTE register / agent definitions / damage-control hooks labels Aug 28, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31b1c1b8bc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread pmoves/docs/services/pmoves-yt/RUNBOOK.md Outdated
Comment thread pmoves/docs/services/pmoves-yt/RUNBOOK.md Outdated
Comment thread pmoves/docs/services/pmoves-yt/RUNBOOK.md Outdated
Comment thread pmoves/docs/services/pmoves-yt/RUNBOOK.md Outdated
Comment thread pmoves/docs/services/pmoves-yt/RUNBOOK.md Outdated
POWERFULMOVES and others added 2 commits August 27, 2026 22:05
…e recipes

Codex review of #2801 flagged five claims. All five reproduced or refuted on
B850 / Knuckles; every command below was executed and its output pasted.

P1 — the published image is not "stock PyPI yt-dlp", it is unrunnable (§1.3).
`/app/yt.py:9` does `parents[3]`, written for the repo layout; `WORKDIR /app` +
`COPY . .` flattens those three levels, so `.parents` is length 2 and the import
raises `IndexError: 3` before any shim fallback. Correcting the depth would not
help: the GHCR build context excludes the repo-root `PMOVES.YT` submodule.
Publishing is inherited — the matrix entry predates the shim by four months
(#699 2026-02-23 vs #883 2026-03-12) and no Dockerfile revision ever used a
layout where the shim resolved. CI cannot catch it (runtime import, not build)
and nothing pulls it (the images overlay leaves `build:` intact, so
up-yt-published rebuilds the fork under the GHCR name). Recorded as a docs
finding plus follow-up issue #2802; no code change — this is a docs branch.

P2 §2.2 — `git checkout -- PMOVES.YT` does not undo `--remote`. Verified on a
throwaway two-repo fixture: it restores the gitlink `--remote` never moved and
leaves the submodule HEAD downgraded (` M SUB` throughout). Replaced with
`git submodule update --init --force`, with the plain-`--init` distinction shown
(it aborts at exit 1 on local edits).

P2 §5.4 — the `supabase-kong` diagnosis was wrong and would mask the real cause.
`yt_oauth_flow.py::_supabase_url()` already rewrites that host to localhost; from
the primary tree the target exits 0. The observed failure came from a linked
worktree (untracked env files absent) falling through to an ambient
`SUPA_REST_URL=http://host.docker.internal:...`, a Docker-Desktop-only name.
Errno is the tell: -2 for that host vs -3 for supabase-kong. Replaced with an
ordered triage list.

P2 §6/§8 — refuted as stated, then fixed for the real defect. Compose selects by
project label on running containers, so these work from any cwd with no compose
file; the cookie pair resolves fine. The actual hazard is that unmatched names
are skipped silently at exit 0 — which is exactly what happens to
bgutil-pot-provider, unlabelled on this node. Added a coverage check and a
direct `docker stop`/`docker logs` step for it.

Also: §7 triage rows corrected, §9 item 1 reframed (registry access is the only
open question; artifact behaviour is settled), and a fifth honest UNVERIFIED
added for whether bgutil is Compose-unowned beyond this node.

Refs #2802
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz
Three `###`-prefixed labels inside the §2.2 verification fence rendered as code
but were picked up by heading extraction. Switched to `---` prefixes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz
@POWERFULMOVES
POWERFULMOVES merged commit cb676a9 into main Aug 28, 2026
22 checks passed
@POWERFULMOVES
POWERFULMOVES deleted the docs/pmoves-yt-runbook branch August 28, 2026 02:18
@POWERFULMOVES

Copy link
Copy Markdown
Owner Author

Severity note — this is the delivery artifact, not an edge case

Recording the deployment framing alongside §1.3, because the runbook states the defect correctly but understates who it reaches.

The operator corrected my framing while this PR was in review: this is an infra node on a private AI lab, not a dev node. The repo is public, the nodes are private, and the delivery is public-good — education, community programs, creator collectives. The platform vision states the brand promise as "Local-first autonomy, reproducible provisioning, and self-improving research loops."

Under that structure:

  • A community, school, or partner install has no PMOVES.YT submodule to build from.
  • So it takes the published path — and pulls the artifact §1.3 documents as unable to start.
  • The private mesh works because build: survives the images overlay and it compiles the fork locally.

So up-yt-published is not an edge case. It is the path for everyone outside the lab, and it is the one that fails.

Why no node can catch this

#2800 tagged all ten fleet profiles deployment_class: private-mesh, with esp32/orin/nano/hermes deliberately unset. Every classed node in the fleet takes the build path. There is no node whose normal operation exercises the published image, which is why a real service becoming a shim in #883 went unnoticed for five months while CI stayed green.

That makes this structurally invisible rather than merely missed. It cannot be found by a node noticing; it needs a gate.

Two follow-ups this implies

  1. Consider removing pmoves-yt from integrations-ghcr.matrix.json until the shim is fixed. Publishing a green-but-unrunnable image is worse than publishing none — a consumer's failure mode is an import crash rather than an honest absence.
  2. Nothing verifies a published image starts. The matrix builds and pushes; no step pulls the result and health-checks it. A docker run --rm <image> + /healthz probe would have caught this in February and would catch the next one.

Point 2 is the generalisable half. Six findings landed today and every one was the same shape — a thing that reports success while doing nothing: a workflow triggering on a tree it never collected, an overlay reachable only via a blocked path, an MCP roster pointing at an unreachable host, a provisioner with no Known Road, a bump workflow that discards its own image, and this. In five of six the only symptom was an absence.

Operator decision on both. Documented here rather than acted on, since removing a published image affects consumers I cannot enumerate from this node.

POWERFULMOVES added a commit that referenced this pull request Aug 28, 2026
…tro-claim #2806

Three b850-claude lanes (docs/ytdlp-currency-reconcile #2793,
docs/pmoves-yt-runbook #2801, fix/pmoves-yt-published-image #2803) were
merged but still parsed as open by claim-collision-pre.py's own
open_claims_in(). One RELEASE for docs/pmoves-yt-runbook already existed
at line 2289 but the parser never saw it close: that RELEASE's prose
backticks an unrelated branch name (`feat/upstream-sync-2026.07`, cited
only to describe the PMOVES.YT submodule's pin) which the lane-matching
regex picks up, switching the release from "closes everything" to
"closes only named lanes" — narrowing itself without the author
intending it. Documented as a durable grammar trap and closed with a
supplementary RELEASE row rather than editing historical entries or the
hook.

Also verified the #2803 republish more strongly than the merged PR
claimed possible: the merge-triggered integrations-ghcr.yml run's
"Verify pushed image starts (runtime gate)" step succeeded against the
real GHCR digest (still unreachable via `docker manifest inspect`/`gh
api` from this node, but CI's own gate is authoritative). Also surfaced
a separate, pre-existing gap: the same run is red end-to-end from 42
unaddressed HIGH-severity Trivy CVE findings, unrelated to and not
caused by #2803 (baselined against the prior merge's identical
failure).

Retroactively CLAIMs PR #2806 (open, in flight via a delegated
delivery-agent) since it and its merged sibling #2805 were pushed with
no register entry — the unclaimed-work failure the steward role exists
to catch.

Register conflict resolved as UNION: git diff origin/main --numstat
shows 15 insertions, 0 deletions on this file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz
POWERFULMOVES added a commit that referenced this pull request Aug 28, 2026
…tro-claim #2806 (#2808)

Three b850-claude lanes (docs/ytdlp-currency-reconcile #2793,
docs/pmoves-yt-runbook #2801, fix/pmoves-yt-published-image #2803) were
merged but still parsed as open by claim-collision-pre.py's own
open_claims_in(). One RELEASE for docs/pmoves-yt-runbook already existed
at line 2289 but the parser never saw it close: that RELEASE's prose
backticks an unrelated branch name (`feat/upstream-sync-2026.07`, cited
only to describe the PMOVES.YT submodule's pin) which the lane-matching
regex picks up, switching the release from "closes everything" to
"closes only named lanes" — narrowing itself without the author
intending it. Documented as a durable grammar trap and closed with a
supplementary RELEASE row rather than editing historical entries or the
hook.

Also verified the #2803 republish more strongly than the merged PR
claimed possible: the merge-triggered integrations-ghcr.yml run's
"Verify pushed image starts (runtime gate)" step succeeded against the
real GHCR digest (still unreachable via `docker manifest inspect`/`gh
api` from this node, but CI's own gate is authoritative). Also surfaced
a separate, pre-existing gap: the same run is red end-to-end from 42
unaddressed HIGH-severity Trivy CVE findings, unrelated to and not
caused by #2803 (baselined against the prior merge's identical
failure).

Retroactively CLAIMs PR #2806 (open, in flight via a delegated
delivery-agent) since it and its merged sibling #2805 were pushed with
no register entry — the unclaimed-work failure the steward role exists
to catch.

Register conflict resolved as UNION: git diff origin/main --numstat
shows 15 insertions, 0 deletions on this file.


Claude-Session: https://claude.ai/code/session_01FkwiW3VY1xWmahTAtVioxz

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation governance AGNOTE register / agent definitions / damage-control hooks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant