Finance reconciliation: indexer (closes unimatrix27/ideas#21)

[unimatrix27]

Summary

Mode-B receipt ingest half of the finance reconciliation feature (parent: unimatrix27/ideas#27). One indexer, two entry points, no LLM:

• CLI for cron: python -m finance.indexer [--mailbox X] [--folder F] [--since YYYY-MM-DD] [--limit N]
• Library for Terminal backend fanout pr2 - PENDING NousResearch/hermes-agent#23's run_indexer() wrapper: finance.indexer.run(config=…, adapter=…, fetcher=…)

Both reach the same _run_async function. Closes unimatrix27/ideas#21. Do not merge — needs review per the issue's acceptance criteria.

Depends on #1 (schema), #2 (fixtures), #3 (parsers). All three merge commits are in this branch; rebase them out once they land on main.

Files

Path	Purpose
finance/indexer.py	CLI + library entrypoint; IndexerAdapter Protocol + InMemoryIndexerAdapter + PostgresIndexerAdapter; DelegatedTokenProvider that plugs into tools/microsoft_graph_client.py; LocalBlobBackend (S3-style backend left as the documented pluggable extension point); pymupdf default + marker-pdf opt-in via FINANCE_INDEXER_ALLOW_MARKER
finance/migrations/002_indexer_state.up.sql / .down.sql	Adds bank.indexer_state keyed by mailbox (composite mailbox::folder text PK)
finance/migrate.py	Rewritten to discover every 00N_*.{up,down}.sql in order; optional prefix arg applies just one migration
finance/tests/test_indexer.py	10 offline tests against InMemoryIndexerAdapter + FakeGraphClient — one per acceptance bullet
~/.hermes/scripts/finance_indexer.sh	Cron wrapper (silent on no-op ticks; surfaces the run summary otherwise)
~/.hermes/cron/jobs.json	Adds job id=691b68f89b0a (no_agent=True, script=finance_indexer.sh, schedule "0 9-18 * * 1-5" Europe/Berlin, deliver telegram) — second cron entry on this host

Spec note — Graph folder-scoped delta

The first live attempt against /users/{mb}/messages/delta returned

400 BadRequest: Change tracking is not supported against 'microsoft.graph.message'

Graph's delta tracking is folder-scoped only. The indexer therefore walks /users/{mb}/mailFolders/{folder}/messages/delta, with folder defaulting to the well-known "inbox" ID (which resolves to Posteingang in rechnung@ and Inbox in marketing@, regardless of locale). Neither mailbox currently exposes a Belege subfolder (verified live), so the default folder list is ["inbox"]; operators can pass --folder to add others. bank.indexer_state is keyed by the composite mailbox::folder string so the existing schema doesn't need a second column.

Live smoke-run summaries

Run 1 (canonical --limit 10, both mailboxes)

{"scanned": 10, "new": 10, "dedup_skipped": 0, "portal_required": 0, "parse_failed": 6}

The 10 new candidates from rechnung@: 4 Sipgate PDFs parsed cleanly (parse_status='ok', vendor='sipgate', invoice numbers B4373121 / B4411208 / B4459838 / B4500117), 6 vendors not in NousResearch#22's parser list (Finovia, Captrader×2, Shine, DKB, GS-90159) honestly land as parse_status='failed' with the blob and SHA captured for later re-parse. marketing@'s delta was already empty after Run-3 below ran first; the very first end-to-end live run scanned all 10 from that mailbox too (0 new — no PDFs / portal hits in the first 10 messages there).

Run 2 (immediate re-run, same params — idempotency)

{"scanned": 0, "new": 0, "dedup_skipped": 0, "portal_required": 0, "parse_failed": 0}

SELECT count(*) FROM bank.receipt_candidates before run = 81, after run = 81. Zero writes, as required.

Run 3 (extended walk to surface a Vodafone notification)

python -m finance.indexer --mailbox rechnung@lineo.finance --since 2026-04-01 --limit 50

{"scanned": 50, "new": 30, "dedup_skipped": 10, "portal_required": 1, "parse_failed": 29}

The dedup_skipped: 10 is real SHA-256 dedupe — those are the 10 Run-1 attachments re-encountered after deleting only the indexer_state row (the candidates were still there). The Vodafone portal_required row landed exactly as specified:

id=106, parse_status='portal_required', source_system='graph',
from_email='nicht.antworten@kundenservice.vodafone.com',
subject='Ihre Mobilfunk-Rechnung vom 14.04.2026 steht im Internet bereit.',
extracted_json.vendor='vodafone',
extracted_text starts: "Deine Mobilfunk-Rechnung ist da … Deine Rechnung vom 14.04.2026 findest Du in Deinem persönlichen Service-Portal MeinVodafone…"

Acceptance criteria

• First run on fresh DB → N rows inserted; second immediate run → 0 new rows.
  Run 1 above: count_before=71, count_after=81 (10 inserted). Run 2 above: count_before=81, count_after=81 (0 inserted). Final idempotency tick after Run 3 also flat at 111.
• A PDF attached twice (forwarded thread) produces exactly one receipt_candidates row.
  test_attachment_sha_dedupe_across_runs: same PDF bytes arrive under two different internetMessageIds; second run reports dedup_skipped=1, new=0. Live Run 3 also dedupe-skipped 10 attachments by SHA-256 against the candidates from Run 1.
• Vodafone notification (no PDF) lands as parse_status='portal_required', not failed.
  Live: Run 3 surfaced 1 portal_required row (id=106), wired through from_email=nicht.antworten@kundenservice.vodafone.com, vendor='vodafone', body anchored on the real "Deine Rechnung … findest Du in Deinem persönlichen Service-Portal MeinVodafone" phrasing. Test: test_vodafone_portal_required_path (loads the real PR Finance reconciliation: fixture pack (closes unimatrix27/ideas#24) #2 fixture text + meta).
• Indexer survives Graph 429/5xx via existing client retry — no custom retry loop.
  Reuses tools.microsoft_graph_client.MicrosoftGraphClient's built-in retry/backoff. The indexer adds only delta-token-expiry recovery (a distinct concern — 410/syncStateNotFound is a permanent rejection, not a transient retry).
• Delta-token expiry recovery exercised by a test that simulates a 410 / invalid-token response.
  test_delta_token_expiry_recovery: pre-seeds a stale token + already-indexed candidate, the FakeGraphClient raises a 410-shaped error on the first call, the indexer falls back to a fresh delta, re-walks the same message, dedupes via internet_message_id, and persists the new token. Also test_is_delta_token_expired_classifier for the 410 / 400-with-syncStateNotFound classifier.
• Run summary written.
  Per-mailbox RunSummary{scanned, new, dedup_skipped, portal_required, parse_failed, delta_reset, delta_reset_reason, error} persisted to bank.indexer_state.last_summary as jsonb. The CLI prints the aggregate to stdout (see Run 1 / 2 / 3 above).
• Invokable both via CLI/cron AND as the run_indexer() tool from Terminal backend fanout pr2 - PENDING NousResearch/hermes-agent#23, with identical behavior.
  main() builds the same IndexerConfig + PostgresIndexerAdapter + GraphFetcher and calls run(...). Terminal backend fanout pr2 - PENDING NousResearch/hermes-agent#23's wrapper would call finance.indexer.run(adapter=…, fetcher=…) directly. One code path.

Hard constraints honored

Constraint	Where
Zero mutations to bank.transactions, bank.belege_sent, bank.belege_to_send	PostgresIndexerAdapter only SELECTs from those three tables (dedupe lookups), no INSERT/UPDATE/DELETE against them anywhere in the code
No LLM calls anywhere	grep for anthropic/openai in finance/indexer.py returns nothing
No invocation of the matcher (NousResearch#22)	finance.indexer.run never imports finance.matcher
Survive Graph 429/5xx via existing client retry	MicrosoftGraphClient._should_retry handles 429 + 5xx; no custom retry loop in indexer.py
marker-pdf not in requirements unconditionally	Gated behind FINANCE_INDEXER_ALLOW_MARKER env flag; import happens inside the conditional branch (extract_pdf_text body)

Migration cycle

$ SUPABASE_DB_URL=...@:5432  python3 finance/migrate.py up 002
Migration 'up' applied from 002_indexer_state.up.sql
$ SUPABASE_DB_URL=...@:5432  python3 finance/migrate.py down 002
Migration 'down' applied from 002_indexer_state.down.sql
# indexer_state count → 0
$ SUPABASE_DB_URL=...@:5432  python3 finance/migrate.py up 002
Migration 'up' applied from 002_indexer_state.up.sql
# indexer_state count → 1 (table back)

Tests

$ python -m pytest finance/tests/ -q -o addopts=''
..........................................................               [100%]
58 passed in 0.30s

48 prior (PRs #1/#2/#3) + 10 new indexer tests. Test file: finance/tests/test_indexer.py.

Coexistence with outlook_auto_rule

PostgresIndexerAdapter.belege_sent_has_message short-circuits the indexer when an inbound message's outlook_message_id or internet_message_id already lives in bank.belege_sent. Verified by test_coexistence_with_belege_sent_skips_indexing (no download triggered, no candidate row written). The 163 legacy server-side-forward rows therefore cannot produce a duplicate candidate.

Backfill vs indexer row-shape asymmetry — intentional

Acknowledged in the spec and preserved here: PR #1's backfill produces one receipt_candidates row per legacy belege_sent row (historical, no SHA available). The live indexer produces one row per attachment, deduped by attachment_sha256. The two shapes coexist by design; no rewriting of the 71 backfilled rows.

Spec note worth a moment of review (Graph auth)

The brief said to reuse tools/microsoft_graph_client.py. The upstream client expects app-only client_credentials (MSGRAPH_TENANT_ID / MSGRAPH_CLIENT_SECRET), but the only credentials available on-host are delegated (refresh-token grant against LINEO_MS_CLIENT_ID, bundle at ~/.hermes/lineo-ms-tokens/sebastian.json). PR #2 documented the same gap and solved it with a standalone urllib script — I went the other way: DelegatedTokenProvider is shape-compatible with MicrosoftGraphTokenProvider, so it plugs directly into MicrosoftGraphClient(token_provider=…) and the indexer keeps the upstream pagination / streaming / retry/backoff machinery intact. Happy to migrate PR #2's script to the same provider in a follow-up.

Configuration knobs

Knob	Default	How to override
Mailboxes	rechnung@lineo.finance, marketing@lineo.finance	--mailbox (repeatable)
Folder filter	["inbox"] (well-known ID)	--folder (repeatable)
Backfill window	6 months	--since YYYY-MM-DD
Blob backend	LocalBlobBackend(~/.hermes/finance/blobs)	--blob-root <path> for local; programmatic via BlobBackend Protocol for S3
Portal-required senders	Vodafone (kundenservice.vodafone.com, vodafone.com, vodafone.de)	IndexerConfig.portal_required_senders
marker-pdf fallback	gated off	FINANCE_INDEXER_ALLOW_MARKER=1
Token bundle	~/.hermes/lineo-ms-tokens/sebastian.json	--token-file <path>

Cron entry

{
  "id": "691b68f89b0a",
  "name": "Finance indexer (Mode-B receipt ingest)",
  "script": "finance_indexer.sh",
  "no_agent": true,
  "schedule": {"kind": "cron", "expr": "0 9-18 * * 1-5"},
  "enabled_toolsets": ["terminal"],
  "deliver": "telegram",
  "workdir": "/home/hermes/work/hermes-agent"
}

Wrapper at ~/.hermes/scripts/finance_indexer.sh loads ~/.hermes/.env, runs the indexer with the project's venv Python (the only one with httpx+pymupdf+psycopg2 together), and is silent when there's no work to report so the channel doesn't get an hourly empty ping.

Out of scope (intentionally not in this PR)

• Parsers (Env robustness: context-safe prompting + tool arg normalization NousResearch/hermes-agent#22 — shipped in PR Finance reconciliation: vendor parsers + deterministic matcher (closes unimatrix27/ideas#22) #3; indexer calls finance.parsers.parse(...))
• Matcher invocation (Terminal backend fanout pr2 - PENDING NousResearch/hermes-agent#23's run_matcher() wrapper)
• Toolbox / send / approve / reject (Terminal backend fanout pr2 - PENDING NousResearch/hermes-agent#23)
• Agent skill (Nous Portal subscription integration NousResearch/hermes-agent#25)
• Hunter subagent (Receiving API errors when messaging through Telegram and using Nous Portal auth NousResearch/hermes-agent#28)
• New vendors beyond the four in Env robustness: context-safe prompting + tool arg normalization NousResearch/hermes-agent#22

Fixture-named PDFs in the live ingest

None of the fixture-named PDFs (Sipgate B4373121 / Notion ZWLWGPDN-0002 / Lucky Penny / Vodafone 122203440401) showed up in Run 1's 10-message window — they're outside the most-recent first-10. They'd land naturally as the cron runs over time, or when the 6-month default backfill kicks in on a fresh state. Per the brief, the goal is realistic ingest behaviour rather than fixture-driven completeness; not extending the window to force a match.

🤖 Generated with Claude Code

[github-actions]

🔎 Lint report: feat/finance-indexer vs origin/main

ruff

Total: 2 on HEAD, 0 on base (🆕 +2)

🆕 New issues (2):

Rule	Count
PLW1514	2

First entries

finance/scripts/build_fixtures.py:138: [PLW1514] `pathlib.Path(...).read_text` without explicit `encoding` argument
finance/indexer.py:1324: [PLW1514] `pathlib.Path(...).read_text` without explicit `encoding` argument

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 7994 on HEAD, 7967 on base (🆕 +27)

🆕 New issues (26):

Rule	Count
unresolved-import	19
invalid-argument-type	5
invalid-assignment	1
possibly-missing-submodule	1

First entries

finance/indexer.py:1075: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2.extras`
finance/scripts/build_fixtures.py:51: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2.extras`
finance/matcher.py:88: [invalid-argument-type] invalid-argument-type: Argument is incorrect: Expected `date`, found `(Any & ~str & ~datetime) | None | date`
finance/verify_backfill.py:12: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2`
finance/indexer.py:1367: [invalid-argument-type] invalid-argument-type: Argument to `MicrosoftGraphClient.__init__` is incorrect: Expected `MicrosoftGraphTokenProvider`, found `DelegatedTokenProvider`
finance/migrate.py:19: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2`
finance/backfill_receipts.py:30: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2.extras`
finance/parsers/vodafone.py:124: [invalid-assignment] invalid-assignment: Invalid subscript assignment with key of type `Literal["gross_amount"]` and value of type `int | float` on object of type `dict[str, str]`
finance/indexer.py:1400: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2`
finance/tests/test_indexer.py:27: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
finance/scripts/build_fixtures.py:50: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2`
finance/indexer.py:876: [invalid-argument-type] invalid-argument-type: Argument to bound method `GraphFetcher.fetch_message` is incorrect: Expected `str`, found `Any | None`
finance/tests/test_fixture_pack.py:11: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
finance/backfill_receipts.py:29: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2`
finance/indexer.py:704: [invalid-argument-type] invalid-argument-type: Argument to bound method `GraphFetcher.list_attachments` is incorrect: Expected `str`, found `Any | None`
finance/indexer.py:389: [unresolved-import] unresolved-import: Cannot resolve imported module `marker.models`
finance/matcher.py:597: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2.extras`
finance/scripts/build_fixtures.py:218: [possibly-missing-submodule] possibly-missing-submodule: Submodule `error` might not have been imported
finance/indexer.py:783: [invalid-argument-type] invalid-argument-type: Argument to bound method `GraphFetcher.download_attachment` is incorrect: Expected `str`, found `Any | None`
finance/tests/test_parsers.py:12: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
finance/indexer.py:388: [unresolved-import] unresolved-import: Cannot resolve imported module `marker.convert`
finance/indexer.py:368: [unresolved-import] unresolved-import: Cannot resolve imported module `pymupdf`
finance/scripts/build_fixtures.py:52: [unresolved-import] unresolved-import: Cannot resolve imported module `pymupdf`
finance/tests/test_matcher.py:12: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
finance/verify_backfill.py:13: [unresolved-import] unresolved-import: Cannot resolve imported module `psycopg2.extras`
... and 1 more

✅ Fixed issues: none

Unchanged: 4210 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.