Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions .claude/commands/search/ingest-content.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Ingest Content to Hi-RAG

Ingest text content into Hi-RAG v2 for knowledge retrieval.

## Pipeline Paths

### Path 1: YouTube → PMOVES.YT → Extract Worker → Qdrant
```bash
# Step 1: Ingest video (downloads + transcribes)
curl -X POST http://localhost:8077/yt/ingest \
-H "Content-Type: application/json" \
-d '{"url": "https://www.youtube.com/watch?v=VIDEO_ID"}'

# Step 2: Push transcript to extract-worker (chunks + embeds + indexes)
curl -X POST http://localhost:8083/ingest \
-H "Content-Type: application/json" \
-d '{"chunks": [{"text": "...", "chunk_id": "unique-id"}]}'

# Step 3: Query via Hi-RAG
curl -X POST http://localhost:8086/hirag/query \
-H "Content-Type: application/json" \
-d '{"query": "search terms", "top_k": 5}'
```

### Path 2: Direct text → Extract Worker
```bash
curl -X POST http://localhost:8083/ingest \
-H "Content-Type: application/json" \
-d '{"chunks": [{"text": "content here", "chunk_id": "doc-1"}]}'
```

## Known Issues

- Extract-worker requires TensorZero + Ollama for Qwen3 embeddings
- If TensorZero embedding fails (500), fall back to `EXTRACT_WORKER_EMBEDDING_BACKEND=sentence-transformers`
- `QDRANT_COLLECTION` must match between extract-worker and Hi-RAG (both should be `pmoves_chunks_qwen3`)
- PMOVES.YT downloads + transcribes but does NOT auto-trigger extract-worker — manual POST required

## Dependencies

| Service | Port | Role |
|---------|------|------|
| PMOVES.YT | 8077 | YouTube download + transcribe |
| Extract Worker | 8083 | Chunk + embed + index |
| Hi-RAG v2 | 8086 | Query (hybrid search) |
| Qdrant | 6333 | Vector storage (Docker internal) |
| Meilisearch | 7700 | Full-text search (Docker internal) |
| TensorZero | 3030 | Embedding model routing |
| Ollama | 11434 | Local embedding model (qwen3-embedding:4b) |

## HuggingFace Alternatives (No TensorZero dependency)

For environments without TensorZero/Ollama, use direct HuggingFace models:
- `BAAI/bge-large-en-v1.5` (1024d, high quality)
- `sentence-transformers/all-MiniLM-L6-v2` (384d, fast)
- `Alibaba-NLP/gte-Qwen2-1.5B-instruct` (1536d, multilingual)

Set `EXTRACT_WORKER_EMBEDDING_BACKEND=sentence-transformers` in env to bypass TensorZero.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the env var extract-worker actually reads

The fallback instruction recommends EXTRACT_WORKER_EMBEDDING_BACKEND=sentence-transformers, but extract-worker reads EMBEDDING_BACKEND in pmoves/services/extract-worker/worker.py (line 31). In the documented TensorZero-failure scenario, setting the variable shown here does not switch backends, so ingestion continues failing instead of recovering.

Useful? React with 👍 / 👎.


## Make Targets

```bash
make -C pmoves up-hirag # Start Hi-RAG v2
make -C pmoves up-workers # Start extract-worker + media workers

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add PMOVES.YT bring-up to Path 1 prerequisites

Path 1 starts with POST http://localhost:8077/yt/ingest, but the Make Targets section omits make -C pmoves up-yt; up-workers does not include pmoves-yt (see WORKER_SERVICES in pmoves/Makefile) and PMOVES.YT is started by a separate up-yt target. If operators follow this command verbatim, port 8077 is typically not running and Step 1 fails before ingestion begins.

Useful? React with 👍 / 👎.

make -C pmoves brand-defaults # Set QDRANT_COLLECTION default
```
Loading