fix(bamboohr): stop the field-history build running out of memory

[aleksdotbar]

Problem

bamboohr__employees_fields_history diffs two per-version Map(String, String) payloads and reads both back after an ARRAY JOIN over the union of their keys. ClickHouse must therefore keep both maps alive across every replicated row, so one block costs (rows x keys x payload). The build is stopped by the server memory tracker with Code: 241 ... While executing ArrayJoinTransform, failing the sync after extraction has already committed its records and leaving the target table stale.

The BambooHR employee stream is the worst case for this by design: it collects every non-sensitive field the account defines, so the key count per row is large on purpose.

Fix

Reshape fields_history_raw as GROUP BY aggregation over a per-field timeline: explode raw_data to (key, value) rows before any wide payload exists, group each (entity, field) into its value-per-version timeline, and derive the transitions with array functions.

The point is not just fewer bytes — it is which operator holds the state. Aggregation state spills to disk past max_bytes_before_external_group_by; ARRAY JOIN and window-sort state do not spill at all, which is why the spill knobs were inert on the previous shape (and, per #1921, on the one before that).

max_memory_usage is deliberately not set on the model. Testing it showed a self-imposed cap converting a build the server could still afford into a hard failure — the same trap as #1921.

Verification

Local ClickHouse 25.7.5, synthetic fixtures only. Output byte-identical to the previous implementation on:

• an edge fixture covering field removed mid-history, removed at the last version, gap-then-reappears, multi-version gap, first appearance after v1, empty-string value, value-to-empty, single-version entity, and excluded-key churn
• a 301-row fixture
• a 217,000-row set: same row count, same byte total, same cityHash64 checksum

snapshot rows	before	after
10,000	1.94 GiB / 48 s	294 MiB / 1.0 s
40,000	fails at 3.60 GiB	328 MiB / 3.5 s

Peak memory no longer tracks snapshot size: 4x the data costs ~12% more memory. Spill to disk was confirmed active via ExternalAggregationWritePart.

dbt parse and dbt compile clean.

Known limit

This raises the ceiling but does not remove it. The model is materialized='table' over an append-only snapshot, so every sync recomputes the entire history — millions of exploded rows to emit a few hundred thousand. At a snapshot roughly an order of magnitude larger than the sizes above, the build becomes impractically slow even though it no longer exhausts memory.

Computed transitions are immutable, so the durable answer is incremental materialization behind a watermark. That needs a design decision about the watermark and is left as follow-up rather than bundled here.

Scope

fields_history_raw has one caller. The named-field fields_history used by the other nine connectors has no ARRAY JOIN and carries only scalar columns; it is untouched and unaffected.

Closes #2491
Refs #1804, #1921

Summary by CodeRabbit

• Bug Fixes

  • Improved field history tracking across entity versions.
  • Correctly records initial values, updated values, and fields that become absent.
  • Groups field changes more consistently for accurate historical results.

• Performance

  • Improved processing of large history datasets by limiting resource usage and allowing intermediate aggregation results to spill to disk when needed.

• Documentation

  • Added guidance describing memory usage and spill behavior during history processing.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 248f8a2e-cb35-442c-b57c-a68eded9cf68

📥 Commits

Reviewing files that changed from the base of the PR and between a4e5d05 and 76d6bc9.

📒 Files selected for processing (2)

• src/ingestion/connectors/hr-directory/bamboohr/dbt/bamboohr__employees_fields_history.sql
• src/ingestion/dbt/macros/fields_history.sql

---

📝 Walkthrough

Walkthrough

The fields history macro now reconstructs per-field timelines with spillable grouped aggregation and emits changes from adjacent values. The BambooHR model adds ClickHouse settings for block size, thread count, and external group-by spilling.

Changes

Fields history processing

Layer / File(s)	Summary
Timeline reconstruction and change emission src/ingestion/dbt/macros/fields_history.sql	The macro groups entity versions and field values into timelines. Missing fields resolve to empty strings. Output rows now compare adjacent timeline values to emit changes.
BambooHR ClickHouse execution settings src/ingestion/connectors/hr-directory/bamboohr/dbt/bamboohr__employees_fields_history.sql	The model sets group-by spill thresholds, an 8192-row block size, and a four-thread limit. A comment documents the memory behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Mergeability Score: ⚪ Minimal · up to 76d6b

The change is localized and no actionable merge-blocking risk remains after normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant RawData
  participant EntityVersions
  participant FieldTimelines
  participant HistoryOutput
  RawData->>EntityVersions: Provide entity version timestamps
  RawData->>FieldTimelines: Provide entity and field value pairs
  EntityVersions->>FieldTimelines: Map values across all versions
  FieldTimelines->>HistoryOutput: Emit changes between adjacent values

Loading

Possibly related PRs

• constructorfabric/insight#2263: Uses the shared fields_history-based HR snapshot history processing.
• constructorfabric/insight#2287: Modifies the same BambooHR model and uses fields_history_raw.

Suggested reviewers: cyberantonz

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the BambooHR field-history memory failure addressed by the pull request.
Linked Issues check	✅ Passed	The changes address issue #2491 by reducing field-history memory use and enabling ClickHouse spill behavior during BambooHR transformation.
Out of Scope Changes check	✅ Passed	The changes remain within scope by modifying the BambooHR model settings and the shared fields-history implementation required to resolve the memory failure.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/investigate-issue-2491-075341

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}