Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 24 additions & 5 deletions pmoves/docker-compose.yt-egress.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,33 @@
# ============================================================================
# Phase 9Q (2026-04-16)
#
# Purpose
# ⚠️ NOT THE DEFAULT — FALLBACK ONLY (corrected 2026-08-06)
# ---------------------------------------------------------------------------
# The Purpose text below originally claimed this overlay bypasses "YouTube's
# residential-IP anti-bot 403s" by routing through the datacenter. That is
# BACKWARDS. Per the 2026-07-28 discovery recorded in mk/egress.mk:7-10:
#
# "YouTube datacenter IPs get FLAGGED by YouTube's bot detection.
# Residential IPs work. ... the DEFAULT for YT download traffic should be
# host-direct (residential)."
#
# So this overlay routes YouTube traffic through the IP class that is KNOWN to
# be bot-gated. Bringing it up today makes ingestion WORSE, not better, while
# the old header presented it as the fix.
#
# Want working YouTube ingest? -> make -C pmoves yt-direct
# Check which mode you are in? -> make -C pmoves yt-egress-check
#
# Kept (not deleted) because datacenter egress remains a legitimate fallback
# for nodes with no residential path, and because the Tailscale plumbing here
# is still correct — only the claim about which IP class YouTube flags was wrong.
#
# Purpose (original text, retained for history — see correction above)
# -------
# Routes all YouTube-facing services (pmoves-yt, bgutil-pot-provider,
# invidious-companion, invidious) through the KVM4-1 Tailscale exit node
# (pmoves-kvm4-1, Hostinger datacenter) to bypass YouTube's residential-IP
# anti-bot 403s. Home residential fingerprints are invisible to YouTube once
# every yt-dlp request + PO token fetch + companion stream resolves through
# the datacenter egress.
# (pmoves-kvm4-1, Hostinger datacenter). The original rationale — that this
# bypasses residential-IP 403s — is inverted; see the correction above.
#
# KVM4-1 was approved as a Tailscale exit node on 2026-04-12 by 4090-CLAUDE
# (see pmoves/docs/AGENTS/AGNOTE4482PHI.t1.md, 2026-04-12 RELEASE entry).
Expand Down
29 changes: 26 additions & 3 deletions pmoves/docs/architecture/PMOVES_LONGBOW_DEPRECATED.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,45 @@

**Status:** Deprecated
**Date:** 2026-07-30
**Decision:** Longbow is documentation-only, never integrated. Qdrant + Meilisearch serve as the de facto vector + lexical layers.
**Decision:** Longbow is **not adopted**. Qdrant + Meilisearch serve as the de facto vector + lexical layers.
**Corrected:** 2026-08-06 — the original rationale asserted that no Longbow code was ever written. That was factually wrong; see "Correction" below. The decision stands on the superseded-by-Cipher/Hi-RAG grounds, not on the false premise.

## Context

Longbow was an external Go-based vector database (HNSW + BM25 + hybrid + GraphRAG, Arrow Flight, SlabArena) evaluated in `research/LONGBOW_COMPARATIVE_ANALYSIS.md` as a candidate for PMOVES's "vector memory layer."

## Why Deprecated

Per `pmoves/docs/TAC/TAC_CIPHER.md:402`: *"LongBow is documentation-only — never integrated. Qdrant (:6333) is the de facto vector layer today."*
The original rationale cited `pmoves/docs/TAC/TAC_CIPHER.md:402`: *"LongBow is documentation-only — never integrated. Qdrant (:6333) is the de facto vector layer today."* That line is dated **2026-07-13** — one day BEFORE the integration work described in "Correction" below. It was true when written and stale by the time it was cited.

The BM25/hybrid retrieval capability that Longbow was meant to provide is already delivered through two production paths:

1. **Cipher Memory** — Qdrant collection `pmoves_cipher_memory` with dense (2560d COSINE) + BM25 sparse (`modifier: 'idf'`) vectors, fused via RRF (`Pmoves-cipher/src/pmoves/embedding.ts`).
2. **Hi-RAG Gateway v2** — Qdrant dense kNN + Meilisearch full-text lexical + optional Neo4j graph boost, fused via convex-combine `hybrid_score(alpha)` (`pmoves/services/hi-rag-gateway-v2/`).

No PMOVES-Longbow submodule exists in `.gitmodules`. No code was ever written.
## Correction (2026-08-06) — code WAS written

The statement previously here — *"No PMOVES-Longbow submodule exists in `.gitmodules`. No code was ever written."* — is **false**. Verified against git:

| Commit | Date | Author | What |
|---|---|---|---|
| `294ea52f7` | 2026-07-15 | PMOVES-AGENT-ZERO-SPARK | `feat(submodule): add PMOVES--longbow (Arrow-Flight vector cache)` — real submodule, fork of `23skdu/longbow` (Go, Arrow-Flight, HNSW) |
| `23af4df50` | 2026-07-15 | PMOVES-AGENT-ZERO-SPARK | `feat(longbow): Phase D — compose stanza + profile data wiring` — ports 3100/3101/9190, healthcheck, resource limits, networks, volume |

Both live on `origin/fix/cipher-search-memoryid-followup`, which is **345 commits behind main** with no open PR. The work was never merged — so there is no removal commit either, which is why a search of `main` alone finds nothing.

That branch also carries a benchmarked multi-tier embedding spec (`high` Qwen3-8B 4096d / `medium` BGE-M3 1024d / `low` nomic-embed 768d, plus `bm25`/`bge_sparse` sparse fields) with recall@10 and p50/p99 measured on SPARK GB10 and 4090 hardware.

**How the error happened, and why it matters beyond Longbow:** the closing pass checked `main`'s current state plus one doc line. It did not check git history or unmerged branches. Any automated gate-closure that reads only `main` will miss unmerged work and can write "never existed" into the permanent record. That is a process failure mode, not a Longbow-specific one.

### Naming collision

Two unrelated things share the name, and the original deprecation conflated them:

- **Longbow (vector cache)** — the fork above, positioned as an **L1 hot-path cache in front of Qdrant**, not a replacement for it. Real, benchmarked, unmerged.
- **Longbow (model router)** — a contextual-bandit router between agents and models, described in `pmoves/docs/architecture/LONGBOW_INTEGRATION.md`. Never built; correctly abandoned.

The router was rightly dropped. The vector cache was deprecated by proxy.

## Resolution

Expand Down
Loading