From ee18dbb42a234481fa982fd308f646c228e4c41c Mon Sep 17 00:00:00 2001 From: POWERFULMOVES <142271328+POWERFULMOVES@users.noreply.github.com> Date: Thu, 6 Aug 2026 20:18:43 -0400 Subject: [PATCH 1/2] docs(longbow): correct a factually false deprecation rationale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PMOVES_LONGBOW_DEPRECATED.md asserted on main: "No PMOVES-Longbow submodule exists in `.gitmodules`. No code was ever written." Both clauses are false. Verified against git: 294ea52f7 2026-07-15 feat(submodule): add PMOVES--longbow (Arrow-Flight vector cache) 23af4df50 2026-07-15 feat(longbow): Phase D — compose stanza + profile data wiring Both authored by PMOVES-AGENT-ZERO-SPARK on origin/fix/cipher-search-memoryid-followup, which is 345 commits behind main with no open PR. The work was never merged, so there is no removal commit either — which is exactly why searching main alone finds nothing. The cited justification was `TAC_CIPHER.md:402` ("LongBow is documentation-only — never integrated"), dated 2026-07-13 — ONE DAY before that integration work landed. True when written, stale when cited on 2026-07-30. This corrects the FACTS, not the DECISION. Not adopting Longbow may well be right: Cipher (Qdrant dense + BM25 sparse + RRF) and Hi-RAG v2 (dense + Meilisearch lexical + graph boost) genuinely do deliver the hybrid retrieval it promised. The decision now rests on superseded-by grounds rather than on a false premise, and the unmerged work is findable instead of denied. Also documents the naming collision the original conflated: "Longbow" is both an Arrow-Flight vector cache (real, benchmarked, unmerged — positioned as an L1 cache IN FRONT OF Qdrant, not a replacement) and an unrelated contextual-bandit model router (never built, correctly abandoned). The router was rightly dropped; the vector cache was deprecated by proxy. Note for whoever owns launch-readiness: this doc satisfies the `stage-4.longbow-or-deprecated` gate (pmoves-launch-readiness.tac.yaml:253). The gate remains satisfied — but it was passed on a premise that was not true, which may warrant re-examination rather than a silent patch. The generalizable finding: an automated gate-closure that reads only `main` will miss unmerged work and can write "never existed" into the permanent record as fact. Co-Authored-By: Claude Opus 5 (1M context) --- .../architecture/PMOVES_LONGBOW_DEPRECATED.md | 29 +++++++++++++++++-- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/pmoves/docs/architecture/PMOVES_LONGBOW_DEPRECATED.md b/pmoves/docs/architecture/PMOVES_LONGBOW_DEPRECATED.md index ed1cd34ff6..7801a4afcd 100644 --- a/pmoves/docs/architecture/PMOVES_LONGBOW_DEPRECATED.md +++ b/pmoves/docs/architecture/PMOVES_LONGBOW_DEPRECATED.md @@ -2,7 +2,8 @@ **Status:** Deprecated **Date:** 2026-07-30 -**Decision:** Longbow is documentation-only, never integrated. Qdrant + Meilisearch serve as the de facto vector + lexical layers. +**Decision:** Longbow is **not adopted**. Qdrant + Meilisearch serve as the de facto vector + lexical layers. +**Corrected:** 2026-08-06 — the original rationale asserted that no Longbow code was ever written. That was factually wrong; see "Correction" below. The decision stands on the superseded-by-Cipher/Hi-RAG grounds, not on the false premise. ## Context @@ -10,14 +11,36 @@ Longbow was an external Go-based vector database (HNSW + BM25 + hybrid + GraphRA ## Why Deprecated -Per `pmoves/docs/TAC/TAC_CIPHER.md:402`: *"LongBow is documentation-only — never integrated. Qdrant (:6333) is the de facto vector layer today."* +The original rationale cited `pmoves/docs/TAC/TAC_CIPHER.md:402`: *"LongBow is documentation-only — never integrated. Qdrant (:6333) is the de facto vector layer today."* That line is dated **2026-07-13** — one day BEFORE the integration work described in "Correction" below. It was true when written and stale by the time it was cited. The BM25/hybrid retrieval capability that Longbow was meant to provide is already delivered through two production paths: 1. **Cipher Memory** — Qdrant collection `pmoves_cipher_memory` with dense (2560d COSINE) + BM25 sparse (`modifier: 'idf'`) vectors, fused via RRF (`Pmoves-cipher/src/pmoves/embedding.ts`). 2. **Hi-RAG Gateway v2** — Qdrant dense kNN + Meilisearch full-text lexical + optional Neo4j graph boost, fused via convex-combine `hybrid_score(alpha)` (`pmoves/services/hi-rag-gateway-v2/`). -No PMOVES-Longbow submodule exists in `.gitmodules`. No code was ever written. +## Correction (2026-08-06) — code WAS written + +The statement previously here — *"No PMOVES-Longbow submodule exists in `.gitmodules`. No code was ever written."* — is **false**. Verified against git: + +| Commit | Date | Author | What | +|---|---|---|---| +| `294ea52f7` | 2026-07-15 | PMOVES-AGENT-ZERO-SPARK | `feat(submodule): add PMOVES--longbow (Arrow-Flight vector cache)` — real submodule, fork of `23skdu/longbow` (Go, Arrow-Flight, HNSW) | +| `23af4df50` | 2026-07-15 | PMOVES-AGENT-ZERO-SPARK | `feat(longbow): Phase D — compose stanza + profile data wiring` — ports 3100/3101/9190, healthcheck, resource limits, networks, volume | + +Both live on `origin/fix/cipher-search-memoryid-followup`, which is **345 commits behind main** with no open PR. The work was never merged — so there is no removal commit either, which is why a search of `main` alone finds nothing. + +That branch also carries a benchmarked multi-tier embedding spec (`high` Qwen3-8B 4096d / `medium` BGE-M3 1024d / `low` nomic-embed 768d, plus `bm25`/`bge_sparse` sparse fields) with recall@10 and p50/p99 measured on SPARK GB10 and 4090 hardware. + +**How the error happened, and why it matters beyond Longbow:** the closing pass checked `main`'s current state plus one doc line. It did not check git history or unmerged branches. Any automated gate-closure that reads only `main` will miss unmerged work and can write "never existed" into the permanent record. That is a process failure mode, not a Longbow-specific one. + +### Naming collision + +Two unrelated things share the name, and the original deprecation conflated them: + +- **Longbow (vector cache)** — the fork above, positioned as an **L1 hot-path cache in front of Qdrant**, not a replacement for it. Real, benchmarked, unmerged. +- **Longbow (model router)** — a contextual-bandit router between agents and models, described in `pmoves/docs/architecture/LONGBOW_INTEGRATION.md`. Never built; correctly abandoned. + +The router was rightly dropped. The vector cache was deprecated by proxy. ## Resolution From 581236046cb79aa7c35983f2785a0d10745609af Mon Sep 17 00:00:00 2001 From: POWERFULMOVES <142271328+POWERFULMOVES@users.noreply.github.com> Date: Thu, 6 Aug 2026 20:20:23 -0400 Subject: [PATCH 2/2] fix(yt-egress): header documented the inverse of which IP class YouTube flags MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit docker-compose.yt-egress.yml:7-12 claimed this overlay routes YouTube traffic through the KVM4-1 datacenter exit node "to bypass YouTube's residential-IP anti-bot 403s", asserting that "home residential fingerprints are invisible to YouTube once every yt-dlp request ... resolves through the datacenter egress." That is backwards. Per the 2026-07-28 discovery in mk/egress.mk:7-10: "YouTube datacenter IPs get FLAGGED by YouTube's bot detection. Residential IPs work. ... the DEFAULT for YT download traffic should be host-direct (residential)." Two files stated opposite truths and the wrong one had a make target attached: `make up-yt-egress` still routes through the datacenter — the IP class now known to be bot-gated — while its own header advertised that as the fix. Running it today makes ingestion worse, not better. Kept rather than deleted: datacenter egress is still a legitimate fallback for nodes with no residential path, and the Tailscale plumbing here is correct. Only the claim about which IP class YouTube flags was wrong. The header now leads with the correction, marks the overlay FALLBACK-ONLY, points at `make yt-direct` / `make yt-egress-check`, and retains the original text labelled as history. Verified: YAML parses, all 5 services intact (tailscale-yt-egress, pmoves-yt, bgutil-pot-provider, invidious-companion, invidious); validate-composes clean; validate-dockerfile-paths OK. Found while establishing why the Keith D / Coin Bureau corpus was never ingested. Two live blockers, this being the second: a host-side Tailscale exit node is currently ACTIVE on the 4090, so container egress resolves to datacenter and gets bot-gated. The first is ffmpeg-whisper running CPU-only at 0.24x realtime, which would make 81 minutes of audio take ~5.6 hours instead of ~30 minutes. Known Road: compose:pr:2456 Co-Authored-By: Claude Opus 5 (1M context) --- pmoves/docker-compose.yt-egress.yml | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/pmoves/docker-compose.yt-egress.yml b/pmoves/docker-compose.yt-egress.yml index dd13317d94..da67a8ad19 100644 --- a/pmoves/docker-compose.yt-egress.yml +++ b/pmoves/docker-compose.yt-egress.yml @@ -2,14 +2,33 @@ # ============================================================================ # Phase 9Q (2026-04-16) # -# Purpose +# ⚠️ NOT THE DEFAULT — FALLBACK ONLY (corrected 2026-08-06) +# --------------------------------------------------------------------------- +# The Purpose text below originally claimed this overlay bypasses "YouTube's +# residential-IP anti-bot 403s" by routing through the datacenter. That is +# BACKWARDS. Per the 2026-07-28 discovery recorded in mk/egress.mk:7-10: +# +# "YouTube datacenter IPs get FLAGGED by YouTube's bot detection. +# Residential IPs work. ... the DEFAULT for YT download traffic should be +# host-direct (residential)." +# +# So this overlay routes YouTube traffic through the IP class that is KNOWN to +# be bot-gated. Bringing it up today makes ingestion WORSE, not better, while +# the old header presented it as the fix. +# +# Want working YouTube ingest? -> make -C pmoves yt-direct +# Check which mode you are in? -> make -C pmoves yt-egress-check +# +# Kept (not deleted) because datacenter egress remains a legitimate fallback +# for nodes with no residential path, and because the Tailscale plumbing here +# is still correct — only the claim about which IP class YouTube flags was wrong. +# +# Purpose (original text, retained for history — see correction above) # ------- # Routes all YouTube-facing services (pmoves-yt, bgutil-pot-provider, # invidious-companion, invidious) through the KVM4-1 Tailscale exit node -# (pmoves-kvm4-1, Hostinger datacenter) to bypass YouTube's residential-IP -# anti-bot 403s. Home residential fingerprints are invisible to YouTube once -# every yt-dlp request + PO token fetch + companion stream resolves through -# the datacenter egress. +# (pmoves-kvm4-1, Hostinger datacenter). The original rationale — that this +# bypasses residential-IP 403s — is inverted; see the correction above. # # KVM4-1 was approved as a Tailscale exit node on 2026-04-12 by 4090-CLAUDE # (see pmoves/docs/AGENTS/AGNOTE4482PHI.t1.md, 2026-04-12 RELEASE entry).