chore: commit deployed runtime (*_local modules, migrations, units, config example) — reproducible install

[davidgut1982]

chore: commit deployed runtime (*_local modules, migrations, units, config example) — reproducible install

What this is

Captures the entire deployed runtime for vector-indexer-mcp that was previously untracked, making the setup reproducible from a clean clone. This indexer backs the embedding reranker benchmark in NousResearch/hermes-agent#35457 and the eval suite in davidgut1982/hermes-agent#1.

---

Problem

The deployed production runtime (*_local.py modules, pgvector schema, systemd units) existed entirely in the local install but was absent from the repository. Anyone cloning the repo to reproduce the reranker benchmark — including the 194-tool embedding pass and the 98-query Recall@K suite — had no path to a working install. Four concrete gaps:

1. Core modules (server_local.py, sse_server_local.py, all daemon/*_local.py) not committed
2. pgvector schema (migrations/020_vector_search_indexer.sql) not committed
3. systemd user service units not committed
4. asyncpg>=0.29.0 missing from requirements.txt (required by all *_local.py modules; fails at import)
5. No config.yaml.example — no template for a fresh install

---

Approach

Committed all untracked runtime files using explicit paths (no git add -A). Added config.yaml.example with all credentials replaced by CHANGE_ME and machine-specific paths replaced with <YOURPATH> placeholders. Added SETUP.md documenting the full install path: PostgreSQL + pgvector, role/db creation, migration, venv + deps, config, systemd units, and the MCP stdio entry. Updated .gitignore to permanently exclude config.yaml, *.bak*, and embed_cache.json.

---

PROOF — backend was operational before this PR

The pgvector backend (PostgreSQL + pgvector, asyncpg, SentenceTransformer embeddings) was verified operational before this PR was created:

# Verified operational:
- 194 tool docs indexed and searchable
- 98-query benchmark (30 labeled scenarios) executed end-to-end
  Result: R@5 overall 0.810 (see NousResearch/hermes-agent#35457)
- stdio MCP transport (src/server_local.py) confirmed running
- watcher + worker systemd units confirmed running and surviving session restart

Secret scan result: zero matches for passwords, private keys, or API credentials across all committed files. config.yaml and *.bak* remain gitignored and were not staged.

Reproduce from a clean clone:

git clone https://github.com/davidgut1982/vector-indexer-mcp
cd vector-indexer-mcp
# Follow SETUP.md:

# 1. PostgreSQL + pgvector
sudo apt install postgresql postgresql-contrib
sudo -u postgres psql -c "CREATE ROLE vectoruser LOGIN PASSWORD 'CHANGE_ME';"
sudo -u postgres psql -c "CREATE DATABASE vectorindex OWNER vectoruser;"
sudo -u postgres psql -d vectorindex -c "CREATE EXTENSION IF NOT EXISTS vector;"

# 2. Run migration
sudo -u postgres psql -d vectorindex < migrations/020_vector_search_indexer.sql

# 3. Python environment
python3 -m venv venv
venv/bin/pip install -r requirements.txt  # now includes asyncpg>=0.29.0

# 4. Config
cp config.yaml.example config.yaml
# Edit: fill in db password, embedding model, watcher.paths, etc.

# 5. Systemd units
cp systemd/*.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now vector-indexer-worker.service vector-indexer-watcher.service

# 6. MCP stdio entry (add to your MCP client config):
# {
#   "vector-indexer-mcp": {
#     "command": "/path/to/vector-indexer-mcp/venv/bin/python",
#     "args": ["-m", "src.server_local"],
#     "cwd": "/path/to/vector-indexer-mcp"
#   }
# }

Watcher path validation — critical behavior note:

reindex_path(path) enqueues files, but the worker validates each path against watcher.paths in config.yaml and silently discards anything outside the allowed roots. Files outside watcher.paths will never persist regardless of how often you reindex. To index a new tree: add its root to watcher.paths, restart the worker, then reindex. This is documented in SETUP.md.

---

What was untracked and is now committed

File	Role
src/server_local.py	asyncpg-based MCP server (production replacement for the Supabase server.py)
sse_server_local.py	HTTP/SSE transport wrapper
daemon/worker_local.py	Queue processor + embedding writer
daemon/watcher_local.py	File system watcher → index_queue
daemon/chunker_local.py	Token-aware text chunker
daemon/embedder_local.py	SentenceTransformer wrapper
migrations/020_vector_search_indexer.sql	Full pgvector schema + stored functions
systemd/*.service	Three user service units (worker, watcher, HTTP/SSE)
config.yaml.example	Redacted config template (all secrets replaced)
SETUP.md	Full install walkthrough

---

Known limitations (documented in SETUP.md; not fixed in this PR)

Honest known-bad items captured so follow-up PRs have a clear list:

1. CONFIG_PATH is hardcoded — server reads a fixed path; should read VECTOR_INDEXER_CONFIG env var with a fallback
2. pyproject.toml scripts mismatch — scripts point at old Supabase modules, not *_local modules
3. Dead dependencies — supabase, postgrest-py, tiktoken are in requirements but unused by the local runtime
4. asyncpg only in requirements.txt — not in pyproject.toml dependencies (this PR adds it to requirements.txt; pyproject.toml is a follow-up)

---

Recommended follow-ups (explicit; not in this PR)

1. De-hardcode CONFIG_PATH — read from VECTOR_INDEXER_CONFIG env var with fallback
2. Fix pyproject.toml scripts to point at *_local modules
3. Prune dead dependencies (supabase, postgrest-py, tiktoken) from requirements
4. Add asyncpg to pyproject.toml dependencies

---

Related PRs

• NousResearch/hermes-agent#35457 — reranker + progressive disclosure (uses the indexer-backed embeddings for the 98-query benchmark)
• davidgut1982/hermes-agent#1 — eval suite (the harness that ran the benchmark against the indexed tools)
• NousResearch/hermes-agent#35526 — delegation security hardening