Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 32 additions & 5 deletions .github/workflows/yt-dlp-bump.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,14 +80,41 @@ jobs:
set -euo pipefail
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
branch="chore/bump-yt-dlp-${VERSION}"
branch="docs/yt-dlp-upstream-${VERSION}"
git checkout -b "$branch"
# Record version in tracker to surface provenance
sed -i "1i- $(date +%Y-%m-%d): bumped yt-dlp to ${VERSION} (scheduled)" docs/hardening/PMOVES-hardening-tracker.md
# This workflow moves NO pin. It resolves the latest upstream release,
# build-validates pmoves/services/pmoves-yt/Dockerfile -- which is NOT
# what the local-build path (`up-yt`) uses, though it IS what GHCR
# publishes for `up-yt-published` -- discards the image (push: false),
# and records a note. Saying "bumped" here
# produced a git log of chore(...) subjects that read as version bumps
# and were not -- see #2713 and pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md.
# The note is worded to be self-describing because its position in the
# tracker is not stable (each run prepends above the previous one).
sed -i "1i- $(date +%Y-%m-%d): upstream yt-dlp ${VERSION} released -- build-validated only, NO pin moved (see pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md)" docs/hardening/PMOVES-hardening-tracker.md
git add docs/hardening/PMOVES-hardening-tracker.md
git commit -m "chore(pmoves-yt): bump yt-dlp to ${VERSION}"
git commit -m "docs(pmoves-yt): note upstream yt-dlp ${VERSION} release (no pin moved)"
git push origin "$branch"
# No `|| true`: if the PR cannot be opened, this workflow has done
# nothing useful and must say so. `set -e` is already on.
gh pr create --title "chore(pmoves-yt): bump yt-dlp to ${VERSION}" --body "Automated weekly bump to yt-dlp ${VERSION}."
# Body written via a quoted heredoc: markdown with blank lines cannot sit
# at column 0 inside a YAML block scalar, and backticks must not reach
# the shell. ${VERSION} is substituted explicitly after the heredoc.
cat > /tmp/pr-body.md <<'BODY'
Automated weekly **notification**, not a bump.

Upstream yt-dlp `__VERSION__` is available on PyPI. This PR changes **one line of documentation** and moves **no pin**.

What ran: the latest upstream release was resolved from PyPI, and `pmoves/services/pmoves-yt` was built against it multi-arch with `push: false`. The image was then discarded.

What did **not** run: nothing upgraded the `PMOVES.YT` submodule, which is the yt-dlp fork the *locally built* image (`make -C pmoves up-yt`) installs from source. Upgrading it means a fork-sync plus a gitlink promotion, which this workflow does not do.

Note that the Dockerfile validated above is also the one GHCR publishes as `ghcr.io/powerfulmoves/pmoves-yt:pmoves-latest` for the `up-yt-published` path -- and that published image installs **stock PyPI yt-dlp**, not the fork. This workflow does not change which version it gets either, because the real GHCR build passes no `YTDLP_VERSION`.

To find out which yt-dlp version PMOVES actually runs, and what an upgrade would involve, see `pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md`.
BODY
sed -i "s/__VERSION__/${VERSION}/g" /tmp/pr-body.md
gh pr create \
--title "docs(pmoves-yt): note upstream yt-dlp ${VERSION} release (no pin moved)" \
--body-file /tmp/pr-body.md

2 changes: 1 addition & 1 deletion docs/PMOVES.AI Services and Integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ If a detail here disagrees with those files, prefer the compose + env pins and u

## Media Ingestion & Processing Services

**PMOVES.YT** – *YouTube ingestion service.* PMOVES.YT handles downloading and processing YouTube content, including videos and transcripts, and feeding them into the system’s storage and indexes[\[68\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L5-L13). It integrates with **MinIO** (for storing video files and assets) and Supabase (for logging metadata/events)[\[68\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L5-L13). **Path:** pmoves/services/pmoves-yt (submodule **PMOVES.YT**). **Image:** built from services/pmoves-yt/Dockerfile (optionally specifying a yt-dlp version) and published as needed. **Ports:** **8077** for HTTP API[\[69\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L8-L11). The API provides endpoints such as POST /yt/ingest (used by Channel Monitor to queue new video downloads) and possibly a root or health page (GET / returns an HTML snippet per the smoke test)[\[70\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L23-L30). **Env:** MINIO\_ENDPOINT, MINIO\_ACCESS\_KEY, MINIO\_SECRET\_KEY (to store fetched media in MinIO)[\[71\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L14-L22). YT\_BUCKET (MinIO bucket name for videos, default “assets”)[\[71\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L14-L22). INDEXER\_NAMESPACE (e.g. “pmoves” – label for indexing)[\[71\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L14-L22). Supabase integration via SUPA\_REST\_URL (for writing back to PostgREST, default internal URL)[\[72\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L16-L20). NATS\_URL (for publishing events, e.g. when transcripts are ready)[\[72\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L16-L20). HIRAG\_URL (Hi-RAG gateway base URL to possibly trigger indexing or use for analysis). It also supports YT\_RATE\_LIMIT to throttle batch ingests[\[73\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L20-L22), and uses an optional **Brainicism bgutil** component for YouTube (to retrieve transcripts via the YouTube frontend API); this is enabled via BGUTIL\_POT\_PROVIDER container and related env (BGUTIL\_HTTP\_BASE\_URL)[\[74\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docker-compose.yml#L354-L362). **Depends on:** minio (for output storage) and optionally bgutil-pot-provider (a helper service for YouTube API). It runs under profiles “orchestration, workers, agents” (meaning it’s used in most deployments)[\[75\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L8-L12).
**PMOVES.YT** – *YouTube ingestion service.* PMOVES.YT handles downloading and processing YouTube content, including videos and transcripts, and feeding them into the system’s storage and indexes[\[68\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L5-L13). It integrates with **MinIO** (for storing video files and assets) and Supabase (for logging metadata/events)[\[68\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L5-L13). **Path:** pmoves/services/pmoves-yt (submodule **PMOVES.YT**). **Image:** built by compose from the `PMOVES.YT` submodule using `pmoves_yt_service/Dockerfile`, which installs the fork **from source** — so the running yt-dlp version is whatever the recorded submodule gitlink carries. `pmoves/services/pmoves-yt/Dockerfile` is a separate, **non-deployed** copy that installs stock yt-dlp from PyPI (tracked in `pmoves/configs/dockerfiles/_known_orphans.yaml`); the CI image workflows build that one. Build args `YTDLP_VERSION` / `YTDLP_PIP_URL` override the packaged fork rather than upgrading it. Measured versions and how to re-derive them: pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md. **Ports:** **8077** for HTTP API[\[69\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L8-L11). The API provides endpoints such as POST /yt/ingest (used by Channel Monitor to queue new video downloads) and possibly a root or health page (GET / returns an HTML snippet per the smoke test)[\[70\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L23-L30). **Env:** MINIO\_ENDPOINT, MINIO\_ACCESS\_KEY, MINIO\_SECRET\_KEY (to store fetched media in MinIO)[\[71\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L14-L22). YT\_BUCKET (MinIO bucket name for videos, default “assets”)[\[71\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L14-L22). INDEXER\_NAMESPACE (e.g. “pmoves” – label for indexing)[\[71\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L14-L22). Supabase integration via SUPA\_REST\_URL (for writing back to PostgREST, default internal URL)[\[72\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L16-L20). NATS\_URL (for publishing events, e.g. when transcripts are ready)[\[72\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L16-L20). HIRAG\_URL (Hi-RAG gateway base URL to possibly trigger indexing or use for analysis). It also supports YT\_RATE\_LIMIT to throttle batch ingests[\[73\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L20-L22), and uses an optional **Brainicism bgutil** component for YouTube (to retrieve transcripts via the YouTube frontend API); this is enabled via BGUTIL\_POT\_PROVIDER container and related env (BGUTIL\_HTTP\_BASE\_URL)[\[74\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docker-compose.yml#L354-L362). **Depends on:** minio (for output storage) and optionally bgutil-pot-provider (a helper service for YouTube API). It runs under profiles “orchestration, workers, agents” (meaning it’s used in most deployments)[\[75\]](https://github.com/POWERFULMOVES/PMOVES.AI/blob/45caf6783bb14f1f4e357c5465d7bfeb76730fb1/pmoves/docs/services/pmoves-yt/README.md#L8-L12).

**Flute Gateway** – *Multimodal voice gateway (TTS/STT).* Flute (`pmoves/services/flute-gateway`) provides `/v1/voice/recognize` (STT) and `/v1/voice/synthesize` (TTS) plus a `/healthz` probe on port **8055**. STT is implemented via the `ffmpeg-whisper` service; ad-hoc STT requests use `POST /transcribe_file` (multipart) on ffmpeg-whisper. TTS is typically provided by VibeVoice Realtime running outside Docker (Pinokio/host) and reached via `VIBEVOICE_URL` (for example `http://host.docker.internal:3000`). The n8n Voice Agent router flow (`pmoves/n8n/flows/voice_platform_router.json`) accepts webhook requests at `POST http://localhost:5678/webhook/voice-agent/ingest`, calls Hi‑RAG v2 for context, calls TensorZero for LLM response generation, logs to Supabase, and publishes `voice.agent.response.v1` to NATS.

Expand Down
2 changes: 1 addition & 1 deletion docs/PMOVES.AI-Edition-Hardened-Summary.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ _Note: PMOVES-DoX tracks `PMOVES.AI-Edition-Hardened-DoX` branch (contains neste
| `deploy-gateway-agent.yml` | 3 runner types | RED | Requires self-hosted runners |
| `sync-secrets-local.yml` | `self-hosted, ai-lab` | FIXED | Switched from cleartext to base64 encoding |
| `webhook-smoke.yml` | `self-hosted, vps` | YELLOW | Manual only |
| `yt-dlp-bump.yml` | `self-hosted, vps` | RED | Scheduled, fails silently when offline |
| `yt-dlp-bump.yml` | `ubuntu-latest` | YELLOW | Runs, but cannot do what its name implies — it build-validates a non-deployed Dockerfile and files a tracker note without moving any pin (pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md). Runner label corrected 2026-08-27: this row read `self-hosted, vps`; the workflow has `runs-on: ubuntu-latest`. |

### CI Improvements Made (2026-02-13)

Expand Down
25 changes: 23 additions & 2 deletions docs/hardening/PMOVES-hardening-tracker.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,20 @@
<!-- yt-dlp currency notes. `yt-dlp-bump.yml` prepends a dated line to this file
each week; those lines are UPSTREAM-RELEASE NOTIFICATIONS, not bumps. The
workflow moves no pin. Historical lines are preserved verbatim for provenance
and annotated in place rather than rewritten. This file is NOT a source of
truth for what yt-dlp version PMOVES runs -- do not read one off it, and do
not add one to it. Measured state and the commands that re-derive it live in
pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md. -->

- 2026-08-24: bumped yt-dlp to 2026.8.19 (scheduled)
- **2026-08-27 correction — this note is a note, not a bump.** The PR it landed
in (#2713, `47c0c0d3b`) changed exactly one file — this tracker,
`1 insertion(+)`. No pin moved. `2026.8.19` is what PyPI reported as the latest
*upstream* release that morning, not what PMOVES runs. yt-dlp here is not a pip
dependency to bump: it is the `PMOVES.YT` submodule, which is itself a yt-dlp
fork, and the deployed image installs that fork from source. Changing the
version means a fork-sync plus a gitlink promotion — which `yt-dlp-bump.yml`
does not do and cannot do. See pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md.
# PMOVES Hardening Tracker v4.0

Comprehensive hardening posture, CI/CD build infrastructure, and service runtime status for the PMOVES.AI platform.
Expand Down Expand Up @@ -126,7 +142,7 @@ Tracked in `pmoves/docs/security/P2_SUBMODULE_TRACKER.md`.
| `env-preflight.yml` | Windows env validation | PR + manual |
| `sync-secrets-local.yml` | CGP/env secret sync | Manual |
| `webhook-smoke.yml` | Render webhook smoke test | Manual |
| `yt-dlp-bump.yml` | Weekly yt-dlp dependency bump | Schedule (Mon 08:00) |
| `yt-dlp-bump.yml` | Weekly upstream-release *check*. Build-validates the non-deployed `pmoves/services/pmoves-yt/Dockerfile` against the latest PyPI yt-dlp and files a tracker note; moves no pin (see pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md) | Schedule (Mon 08:00) |
| `python-images-toolchain-canary.yml` | Weekly pinned Python image toolchain canary (build + Trivy + PR) | Schedule (Mon 09:00) + manual |

### Build Matrix (`pmoves/images.yaml` -- 16 services)
Expand Down Expand Up @@ -167,7 +183,12 @@ Use two documentation sinks on purpose:
- Code scanning:
- `codeql.yml` provides recurring code-level CVE/security signal.
- Dependency freshness:
- `yt-dlp-bump.yml` keeps the extractor lane current on a weekly cadence.
- `yt-dlp-bump.yml` reports weekly on upstream yt-dlp releases. It does **not**
keep the extractor lane current, and nothing else does either: it builds a
throwaway image (`push: false`) from a Dockerfile the deployed stack does not
use, then commits a prose line. The extractor code that ships is the
`PMOVES.YT` submodule at its recorded gitlink. To learn how current that is,
run the commands in pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md — do not read a version off this tracker.
- `python-images-toolchain-canary.yml` tests pinned Python image candidates weekly with a Trivy gate before opening a PR.
- Image / runtime release gates:
- `make -C pmoves ghcr-prepublish-inrepo`
Expand Down
17 changes: 12 additions & 5 deletions pmoves/configs/dockerfiles/_known_orphans.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -97,11 +97,18 @@ pmoves/services/model-fitness-bridge/Dockerfile: service with Dockerfile but no
# host for low-latency). Not a compose service today.
pmoves/services/owner-presence/Dockerfile: owner-presence service run as a side-channel process (not compose); Dockerfile kept as the source for the standalone binary.

# PMOVES.YT is built from the sibling `PMOVES.YT` submodule
# in compose (the `pmoves-yt` service). This Dockerfile is the
# source for that build but isn't called directly by any compose
# stanza.
pmoves/services/pmoves-yt/Dockerfile: pmoves-yt image source; compose builds from the sibling PMOVES.YT submodule. Kept as the upstream source.
# Corrected 2026-08-27. The previous reason line said this file was
# "the source for" the compose build. It is not: compose's `pmoves-yt`
# service builds `../PMOVES.YT` with `pmoves_yt_service/Dockerfile`
# (the fork, from source), which is a DIFFERENT Dockerfile installing a
# DIFFERENT yt-dlp. This file is orphaned only in this registry's sense
# -- no compose `build:` stanza names it -- and is otherwise very much
# live: `.github/workflows/integrations-ghcr.yml` builds and PUSHES it
# (push: true) as ghcr.io/powerfulmoves/pmoves-yt:pmoves-latest, which
# `docker-compose.integrations.images.yml` consumes on the
# `make -C pmoves up-yt-published` path. Do not delete it, and do not
# read "orphan" here as "unused".
pmoves/services/pmoves-yt/Dockerfile: published-image source, built+pushed to GHCR by integrations-ghcr.yml and consumed by docker-compose.integrations.images.yml (up-yt-published). No compose `build` stanza names it, hence listed here. NOT the same Dockerfile compose builds -- that is PMOVES.YT/pmoves_yt_service/Dockerfile.

# The `publisher-discord` service in compose uses
# `./services/publisher-discord/Dockerfile` (note the `-discord`
Expand Down
12 changes: 12 additions & 0 deletions pmoves/configs/living_docs_registry.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,18 @@ tracked:
severity: P2
description: "Archon mint-contract current state — archon.mint.* implementation + registration gaps; authoritative over the mint-agent command doc"

# ── yt-dlp currency ────────────────────────────────────────────────────
# Enrolled 2026-08-27. This doc exists because #2713 landed a `chore(...): bump
# yt-dlp to 2026.8.19` subject on a docs-only change, and no tracked doc could
# contradict it. Section 1 is a mechanism and does not rot; section 2 is a dated
# measurement and will. Enrolling it makes that rot visible instead of silent —
# a stale number here is the exact failure mode being corrected, so it must be
# something CI can see.
- path: pmoves/docs/services/pmoves-yt/YTDLP_CURRENCY.md
freshness_days: 60
severity: P2
description: "yt-dlp currency — measured pin/runtime/upstream versions, the commands that re-derive them, and the fork-sync gap"

# ── Cross-service health-endpoint truth ────────────────────────────────
# The doc that would have caught the Archon blackbox probe pointing at a dead
# port and path (#2448). Untracked until now.
Expand Down
Loading
Loading