feat(knowledge-graph): structured dataset ingestion via CSV import

docs/CHANGELOG.md

@@ -18,6 +18,132 @@ entry. See `CONTRIBUTING.md` § Releases & changelog.
 
 ## [Unreleased]
 
+### Added — structured dataset ingestion (CSV import) for the Knowledge Graph (#430)
+
+- New `KnowledgeGraph` surface (`ingestDataset`, `listDatasets`, `getDataset`,
+  `queryDatasetRows`, `deleteDataset`) backed by a relational sidecar —
+  `datasets` + `dataset_rows` tables (migration `0029_datasets.sql`) — NOT a
+  graph-node explosion: individual rows never become graph nodes, only one
+  `Dataset` node (`PluginEntity`, `system='dataset'`) is created per dataset
+  for recall/citation linking. Implemented in both `@omadia/knowledge-graph-neon`
+  (real SQL, parameterized JSONB filters/aggregates) and
+  `@omadia/knowledge-graph-inmemory` (full parity, not a stub).
+- `POST /api/v1/datasets` (multipart CSV upload), `GET /api/v1/datasets`,
+  `GET /api/v1/datasets/:id`, `GET /api/v1/datasets/:id/rows`,
+  `DELETE /api/v1/datasets/:id` — ACL pattern mirrors `/api/v1/memory`
+  (session-derived owner, no anonymous access).
+- CSV attachments in chat now import as a queryable dataset instead of being
+  silently truncated at the existing 20,000-char text cap
+  (`attachmentExtract.ts`'s `MAX_TEXT_CHARS`).
+- New `query_dataset` native tool: `list_datasets` / `get_schema` /
+  `query_rows` (a constrained filter+aggregate DSL — never raw SQL from the
+  model), always paginated/aggregated server-side.
+- Every imported row runs through the existing C0 regex PII-detector
+  baseline (`@omadia/plugin-privacy-guard`) before being persisted — the
+  same masking pipeline that already protects free-text user prompts.
+- Admin UI (upload/schema/delete page under `web-ui/app/admin/`) is
+  intentionally NOT part of this change — see the PR description.
+- Fixup: `inferColumnType` (`datasetImport.ts`) no longer types a column as
+  `'number'` when any value has a leading zero (`'0301234567'`, `'01234'`) —
+  such columns are zero-padded identifiers (phone numbers, postal codes),
+  not numbers. Previously `Number()` silently dropped the leading zero
+  (data corruption) AND the column skipped the mandatory C0 privacy scan
+  because number-typed columns are assumed to have no free-text surface.
+  Both bugs are fixed by keeping such columns `'string'`-typed, which
+  restores the scan and preserves the value verbatim.
+- Fixup (round 2, adversarial cross-vendor review): the chat-attachment CSV
+  auto-ingest path (`orchestrator.ts`'s `ingestAttachments`) was writing
+  `ownerOmadiaUserId` from the turn's raw channel-native id (Teams AAD oid,
+  …) instead of the canonical `omadiaUserId` uuid the KG's ACL routes
+  filter on. `ChatTurnInput` gains an optional `channelIdentity` field
+  (`{ channelKind, channelUserId }`, populated only by
+  `createOrchestratorDispatcher` for channel kinds the KG model has a
+  mapping for); the CSV-import call site now resolves it via
+  `KnowledgeGraph.resolveOrCreateChannelIdentity` before using it as the
+  dataset owner, and declines the KG-import branch (falling back to the
+  plain-text attachment path) rather than guess for a channel it can't map.
+- Fixup (round 2): per-cell CSV truncation (`MAX_CELL_CHARS` in
+  `datasetImport.ts`) is still applied but is no longer silent —
+  `parseCsv`/`buildDatasetFromCsv`/`importCsvDataset` now return a
+  `truncation: { truncatedCellCount, truncatedColumns }` alongside
+  `privacyScan`, surfaced in the `POST /api/v1/datasets` response and in the
+  chat-ingest tool-result note.
+- Fixup (round 2): `NeonKnowledgeGraph`'s `contains` dataset filter now
+  escapes `%`, `_`, and `\` in the filter value before wrapping it for
+  `ILIKE ... ESCAPE '\'`, so a literal `%`/`_` in the value matches literally
+  instead of being treated as a SQL wildcard — matching the in-memory
+  backend's literal substring `.includes()` semantics.
+- Fixup (round 2): `InMemoryKnowledgeGraph`'s grouped dataset query now caps
+  results at 200 groups (sorted by aggregate value descending, nulls last),
+  matching `NeonKnowledgeGraph`'s existing `LIMIT 200` — an unbounded
+  group-by could otherwise blow the turn token budget through the
+  in-memory backend only.
+- Scope correction: this change addresses #430's CSV import/query path.
+  #430's own triage acceptance criteria also call for an admin
+  upload/schema/delete UI, which is deliberately not part of this change —
+  see Phase 14 in `docs/middleware-agent-handoff.md` §13 for the tracked
+  follow-up.
+- Fixup (round 3, adversarial review): `InMemoryKnowledgeGraph`'s
+  `matchesDatasetFilter` compared `eq`/`neq`/`contains` filter values with
+  no type coercion (`value === filter.value`), while
+  `NeonKnowledgeGraph`'s `buildDatasetFilterClause` already coerced
+  `filter.value` to the target column's declared type
+  (`::numeric`/`::text`) before comparing. Concrete failing case: a
+  `number` column `amount` storing `250` (a JS number) with
+  `query_dataset` filter `{column:'amount', op:'eq', value:'250'}` (a JSON
+  string — the tool's Zod schema permits this regardless of column type or
+  op) matched on Neon but silently returned `totalMatched: 0` on the
+  in-memory backend for the identical logical query. Fixed by coercing
+  `filter.value` against the row value using the column's schema-declared
+  type, mirroring Neon's cast choice exactly (`Number(...)` for a
+  `number` column, `String(...)` otherwise; `contains` now also coerces a
+  non-string `filter.value` to a string before the substring check instead
+  of rejecting it). Regression test added in
+  `middleware/test/inMemoryKnowledgeGraph.test.ts` reproducing the exact
+  case above plus the `neq`/`contains` mirrors.
+- Fixup (round 5, adversarial review): round 2's channel-identity fix only
+  covered the IMPORT path (`ingestAttachments`) — `QueryDatasetTool.handle`
+  still resolved the viewer as `turnContext.current()?.userId`, the RAW
+  channel-native id, never the canonical `omadiaUserId` a channel turn's
+  dataset was actually stored under. Net effect: a dataset imported via
+  Teams/Slack/Telegram chat could never be found again by `list_datasets` /
+  `get_schema` / `query_rows` from that same chat — the exact "query
+  ingested datasets" requirement #430 exists for. Fixed by resolving the
+  canonical id ONCE per turn (`resolveTurnOwnerIdentity`, new
+  `TurnContextValue.resolvedOmadiaUserId`) in both `runTurn` and
+  `chatStream` (the latter is what channel adapters actually call —
+  previously it never populated any per-turn user identity at all for the
+  `query_dataset`/dataset-ACL purpose), and pointing both `QueryDatasetTool`
+  and `ingestAttachments` at that single shared value instead of each
+  re-deriving it. Regression test in `queryDatasetTool.test.ts` simulates a
+  channel turn's raw-id-at-write-vs-read mismatch end-to-end.
+- Fixup (round 6, adversarial review): round 1's `LEADING_ZERO_RE` fix only
+  matched an UNSIGNED leading zero (`/^0\d/`), so a signed zero-padded value
+  like `-0123`/`-0456` still passed `NUMBER_RE` (which allows an optional
+  leading `-`) without tripping the leading-zero guard — the exact same
+  corruption-plus-scan-bypass defect as round 1, just missed for the signed
+  case. Fixed by widening the pattern to `/^-?0\d/`, which still correctly
+  excludes a bare `0`/`-0` or a `0.x`/`-0.x` decimal (those are followed by
+  nothing or a `.`, not another digit). Regression test added in
+  `datasetImport.test.ts` with signed zero-padded values proving the column
+  types as `'string'`, the value round-trips with sign and leading zero
+  intact, and the privacy scan runs on it.
+- Fixup (round 7, adversarial review): `POST /api/v1/datasets` was the only
+  one of the five dataset route handlers (`middleware/src/routes/datasets.ts`)
+  with no `try/catch` around its core call (`importCsvDataset`). Since
+  Express 5 auto-forwards async rejections to its default error handler and
+  this app registers no global JSON error middleware, an unexpected THROWN
+  error during import (e.g. a transient Postgres error inside
+  `NeonKnowledgeGraph.ingestDataset`) fell through to Express's default
+  handler and returned an HTML error page instead of the `{code, message}`
+  JSON envelope every other dataset endpoint already returns via
+  `mapErrorToHttp`. Fixed by wrapping the handler's `importCsvDataset` call
+  in the same `try/catch` + `mapErrorToHttp` pattern the other four
+  handlers use — the existing, already-handled `{ok: false, reason}`
+  not-ok/privacy-rejection return path is unchanged. Regression test added
+  in `datasetsRoute.test.ts` with a graph whose `ingestDataset` throws,
+  asserting the route returns a JSON `{code, message}` body.
+
 ### Fixed — orchestrator no longer offers or invokes a not-yet-authenticated plugin's tools (#474)
 
 - A native plugin (`ctx.tools.register` from `activate()`) whose own

docs/middleware-agent-handoff.md

@@ -671,6 +671,47 @@ durch `runTurn({ ..., viewer })`. Tests: `test/conductorBuilder.test.ts`
 (Digest-Sichtbarkeit inkl. pending/fremd-privat, Proposal-Vetting,
 Malformed-Blocks, No-Proposal-Regression).
 
+### Dataset-Routen + `query_dataset`-Tool (#430)
+
+Neue REST-Oberfläche `src/routes/datasets.ts`, gemountet unter
+`/api/v1/datasets` (ACL-Pattern wie `/api/v1/memory` —
+`req.session.omadia_user_id`, kein anonymer Zugriff):
+
+- `POST /api/v1/datasets` — multipart CSV-Upload (`multer`, ein File pro
+  Request, `MAX_UPLOAD_BYTES` = 25 MB).
+- `GET /api/v1/datasets` — Liste der eigenen Datasets.
+- `GET /api/v1/datasets/:id` — Schema + Metadaten eines Datasets.
+- `GET /api/v1/datasets/:id/rows` — paginierte Roh-Zeilen.
+- `DELETE /api/v1/datasets/:id` — Dataset löschen.
+
+Dieselbe Pipeline (`importCsvDataset` aus
+`harness-orchestrator/src/datasetImport.ts`) läuft auch automatisch beim
+CSV-Chat-Attachment-Pfad in `orchestrator.ts`'s `ingestAttachments` (ersetzt
+dort den bisherigen 20.000-Zeichen-Text-Cutoff für CSVs) — siehe §7 für die
+Knowledge-Graph-seitige Implementierung.
+
+Neues natives Tool **`query_dataset`** (`tools/queryDatasetTool.ts`),
+registriert wie die übrigen Orchestrator-Tools in §3's Orchestrator-Setup:
+`list_datasets` / `get_schema` / `query_rows` gegen eine eingeschränkte
+Filter/Aggregat-DSL (nie rohes SQL vom Modell), Ergebnisse immer
+server-seitig paginiert/aggregiert bzw. auf 200 Gruppen gecappt.
+
+**Identity-Resolution (Fixup Runde 5):** für einen Channel-Turn (Teams/
+Slack/Telegram) ist `ChatTurnInput.userId` die RAW channel-native id, NICHT
+die kanonische `omadiaUserId` uuid. `resolveTurnOwnerIdentity`
+(`resolveTurnOwnerIdentity.ts`) löst sie EINMAL pro Turn auf (via
+`KnowledgeGraph.resolveOrCreateChannelIdentity`, wenn `input.channelIdentity`
+gesetzt ist — sonst fällt sie auf `input.userId` zurück, das für HTTP/CLI-
+Turns bereits kanonisch ist) und legt sie in
+`TurnContextValue.resolvedOmadiaUserId` ab — einmal in `runTurn` (non-
+streaming) und einmal in `chatStream` (der Pfad, den
+`createOrchestratorDispatcher` für Channel-Turns tatsächlich aufruft).
+`QueryDatasetTool` und `ingestAttachments` lesen beide ausschließlich dieses
+Feld für die Dataset-ACL (niemals das rohe `TurnContextValue.userId`) — vorher
+schrieb der Import-Pfad unter der kanonischen id, während der Query-Pfad die
+rohe id las, sodass ein Channel-User sein eigenes gerade importiertes Dataset
+nie wiederfinden konnte.
+
 ---
 
 ## 4. Migration Managed Agents → Lokal
@@ -883,6 +924,35 @@ Wird vom Orchestrator aufgerufen, wenn der User auf prior art verweist.
 End-to-End verifiziert: der Orchestrator nutzt das Tool von selbst, ohne
 dass man ihn zwingt.
 
+### Structured Datasets — CSV Import (#430)
+
+Separate Ablage neben dem eigentlichen Graph — bewusst KEINE Graph-Node-
+Explosion pro Zeile (Node-Properties sind GIN-indexiert, siehe
+`ingestEntities`-Doku). Relationale Sidecar-Tabellen `datasets` +
+`dataset_rows` (Migration `packages/harness-knowledge-graph-neon/src/
+migrations/0029_datasets.sql`); pro Dataset genau EIN `Dataset`-Graph-Node
+(`PluginEntity`, `system='dataset'`) für Recall/Zitation.
+
+- **Interface:** `KnowledgeGraph.{ingestDataset,listDatasets,getDataset,
+  queryDatasetRows,deleteDataset}` (`plugin-api/src/knowledgeGraph.ts`),
+  implementiert in `@omadia/knowledge-graph-neon` (echtes SQL) UND
+  `@omadia/knowledge-graph-inmemory` (volle Parität, kein Stub).
+- **Import:** `POST /api/v1/datasets` (multipart CSV, `src/routes/
+  datasets.ts`) sowie automatisch bei CSV-Chat-Attachments
+  (`attachmentExtract.ts`'s `isCsvAttachment` branch in `orchestrator.ts`'s
+  `ingestAttachments` — ersetzt den bisherigen 20.000-Zeichen-Text-Cutoff
+  für CSVs).
+- **Privacy:** jede importierte Zeile läuft vor dem Schreiben durch den
+  bestehenden C0-Regex-Baseline-Detector (`@omadia/plugin-privacy-guard`'s
+  `createBaselineDetector`/`maskPrompt`) — dieselbe Pipeline, die
+  Freitext-User-Prompts schützt. Nur `string`/`date`-Spalten werden
+  gescannt (Details + Kosten-Hinweis in `datasetImport.ts`'s Modul-Doc).
+- **Query:** `query_dataset`-Tool (`tools/queryDatasetTool.ts`) — eine
+  eingeschränkte Filter/Aggregat-DSL (nie rohes SQL vom Modell), immer
+  server-seitig paginiert/aggregiert.
+- **Admin-UI:** bewusst NICHT Teil dieser Änderung — siehe PR-Beschreibung
+  von #430 für die Begründung; offener Folge-Task.
+
 ---
 
 ## 8. Skills
@@ -920,6 +990,14 @@ Migration. Statt dessen überschreibt der Preamble in
 Funktioniert in der Praxis. Falls ein Sub-Agent dennoch curl-Muster
 produziert, Skill selbst anpassen.
 
+### Cross-Referenz: `query_dataset` (#430) ist kein Skill
+
+AGENTS.md's Doku-Regel ordnet "Neue Route / Tool / Sub-Agent" §3 **und**
+§8 zu. #430's `query_dataset`-Tool ist ein natives Orchestrator-Tool ohne
+eigenen `skills/<name>/SKILL.md`-Ordner — es gehört also inhaltlich nicht
+in "Aktuelle Skills" oben. Referenz statt Duplikat: volle Doku in §3
+("Dataset-Routen + `query_dataset`-Tool") und §7 (Knowledge-Graph-Schicht).
+
 ---
 
 ## 9. Tests (63 Stück, alle grün)
@@ -1302,6 +1380,18 @@ gekettet, weil `requires` beim Boot enforced wird): docs-RFC (diese PR)
 omadia-ui-Orchestrator-Consumer. Details + per-PR-Doc-Pflichten in §15
 des RFC.
 
+### Phase 14 — Admin-UI für Dataset-Upload/Schema/Delete (#430 Follow-up)
+
+Der #430-Scope (CSV-Import + `query_dataset`-Tool, siehe §3 und §7) deckt
+absichtlich **keine** Admin-UI ab — Upload/Schema-Browse/Delete bleibt
+API-only (`POST/GET/DELETE /api/v1/datasets*`, siehe §3). #430's eigene
+Triage-Acceptance-Criteria verlangen aber genau diese UI; der Branch
+schließt das Issue deshalb NICHT, sondern "addresses" es — ein
+Folge-Issue für die Admin-UI-Seite (`web-ui/app/admin/datasets/` o.ä.,
+Upload-Dropzone + Schema-Tabelle + Zeilen-Preview + Delete-Bestätigung,
+Pattern analog zur bestehenden Package-Upload-Seite) ist offen zu
+erfassen.
+
 ---
 
 ## 14. Commands (vom `middleware/`-Dir aus)