feat(etl): add JSONL/NDJSON support alongside CSV#2471
Merged
Conversation
Create json-utils.ts with isJsonlFile() and mapJsonRowToItem() helpers, then branch both the queue-path (processCatalogEtl) and workflow-path (catalog-etl-workflow) processors to stream JSONL when the object key ends in .jsonl/.ndjson. CSV path is unchanged. Also backports relax_quotes + on_skip to the workflow CSV parser to match the queue path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
WalkthroughThe PR adds JSONL/NDJSON support to the catalog ETL pipeline. New utilities detect file format by extension and map JSON rows to catalog items. The service and workflow layers branch on format to stream JSONL line-by-line or parse CSV, with tests validating mapping and edge cases. ChangesJSONL/NDJSON Catalog ETL
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint skipped: no ESLint configuration detected in root package.json. To enable, add Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Coverage Report for Expo Unit Tests Coverage (./apps/expo)
File CoverageNo changed files found. |
Contributor
Coverage Report for API Unit Tests Coverage (./packages/api)
File Coverage
|
||||||||||||||||||||||||||||||||||||||
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the catalog ETL ingestion pipeline to support JSONL/NDJSON inputs in addition to CSV, auto-detecting the format via the R2 object key extension and updating both the legacy queue-based ETL path and the Cloudflare Workflows-based ETL path.
Changes:
- Add format detection (
.jsonl/.ndjson) and a streaming JSONL parsing path in both ETL engines. - Introduce
json-utils.tsto map JSONL row objects intoNewCatalogItem-shaped partials (mirroring CSV mappings but with JSON-native types). - Align the workflow CSV parsing options with the queue path (
relax_quotes,skip_records_with_error,on_skip).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| packages/api/src/workflows/catalog-etl-workflow.ts | Adds JSONL streaming support and updates CSV parser options in the workflow chunk processor. |
| packages/api/src/utils/json-utils.ts | New utilities to detect JSONL/NDJSON keys and map parsed JSON objects into catalog items. |
| packages/api/src/services/etl/processCatalogEtl.ts | Adds JSONL streaming support to the legacy queue-based ETL processor. |
Comments suppressed due to low confidence (1)
packages/api/src/workflows/catalog-etl-workflow.ts:232
csv-parseparse errors handled byon_skipare added toinvalidItemsBatch, but they do not increment the chunk’s processed counter (rowIndex). Since the workflow later writestotalProcessedfromrowsProcessed: rowIndex, any skipped/malformed CSV rows will be excluded fromtotalProcessed, making the final aggregate totals inconsistent (oftentotalValid + totalInvalid > totalProcessed).
Suggested fix: track rowsProcessed separately from rowIndex and increment it inside on_skip (or increment rowIndex on skip), then return that value as rowsProcessed.
const parser = parse({
relax_column_count: true,
relax_quotes: true,
skip_empty_lines: true,
skip_records_with_error: true,
on_skip: (err: Error) => {
const parserLine = (err as { lines?: number }).lines ?? rowIndex;
invalidItemsBatch.push({
jobId,
errors: [{ field: 'csv_parse', reason: err.message }],
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+117
to
+118
| const skipPartialLine = isNonFirstChunk; | ||
| let firstLineSkipped = !skipPartialLine; |
Comment on lines
+85
to
+103
| // --- JSONL streaming path --- | ||
| // No csv-parse, no header injection. Each line is a JSON object. | ||
| let buffer = ''; | ||
| const skipPartialLine = byteStart !== undefined && byteStart > 0; | ||
| let firstLineSkipped = !skipPartialLine; | ||
|
|
||
| for await (const chunk of streamToText(r2Object.body)) { | ||
| let text = chunk; | ||
|
|
||
| if (skipPartialRow) { | ||
| // Discard bytes up to and including the first newline — those bytes are | ||
| // the tail of the row that the previous chunk already processed. | ||
| const nl = text.indexOf('\n'); | ||
| if (nl === -1) continue; // entire buffer is still the partial row tail | ||
| text = text.slice(nl + 1); | ||
| skipPartialRow = false; | ||
| if (!text) continue; | ||
| } | ||
| buffer += chunk; | ||
| const lines = buffer.split('\n'); | ||
| buffer = lines.pop() ?? ''; | ||
|
|
||
| // Respect backpressure: if the parser buffer is full, wait for drain before | ||
| // pushing more data. Without this, R2 fills the parser buffer for the entire | ||
| // file (up to 600 MB) before the main loop processes any rows → Worker OOM. | ||
| const ok = parser.write(text); | ||
| if (!ok) await new Promise<void>((resolve) => parser.once('drain', resolve)); | ||
| } | ||
| parser.end(); | ||
| })(); | ||
|
|
||
| for await (const record of parser) { | ||
| if (rowIndex % 100 === 0) await new Promise((resolve) => setTimeout(resolve, 1)); // Yield every 100 rows for GC; per-row yield hits the CF Worker wall-clock limit on large files | ||
| const row = record as string[]; | ||
| if (!isHeaderProcessed) { | ||
| fieldMap = row.reduce<Record<string, number>>((acc, header, idx) => { | ||
| acc[header.trim()] = idx; | ||
| return acc; | ||
| }, {}); | ||
| isHeaderProcessed = true; | ||
| console.log( | ||
| `🔍 [TRACE] Header processed - fields: ${Object.keys(fieldMap).length}, mapping:`, | ||
| Object.keys(fieldMap), | ||
| ); | ||
| continue; | ||
| } | ||
| for (const line of lines) { | ||
| const trimmed = line.trim(); | ||
| if (!trimmed) continue; | ||
|
|
||
| const item = mapCsvRowToItem({ values: row, fieldMap }); | ||
| if (!firstLineSkipped) { | ||
| firstLineSkipped = true; | ||
| continue; // discard partial row at chunk boundary | ||
| } |
Comment on lines
+16
to
+24
| /** | ||
| * Maps a parsed JSON object (one line from a JSONL file) to a partial catalog item. | ||
| * Uses `unknown` with proper type narrowing — no `any`. | ||
| */ | ||
| export function mapJsonRowToItem(obj: Record<string, unknown>): Partial<NewCatalogItem> | null { | ||
| const item: Partial<NewCatalogItem> = {}; | ||
|
|
||
| // --- String scalar fields --- | ||
| const rawName = obj.name; |
- Replace raw typeof checks with isString/isNumber/isObject from @packrat/guards - Fixes custom lint rule violation (no-raw-typeof CI check) - Add json-utils.test.ts with 30+ tests covering all branches - Brings json-utils.ts line/statement coverage above 95% threshold Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use toRecord() for JSON.parse results (catalog-etl-workflow, processCatalogEtl) - Use toStringRecord() for techs narrowing (json-utils) - Fix availability test value (in_stock not InStock) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/api/src/services/etl/processCatalogEtl.ts`:
- Around line 194-205: The on_skip handler currently only pushes parse errors
into invalidItemsBatch (in processCatalogEtl.ts) and doesn't trigger a flush,
which can let invalidItemsBatch grow unbounded when many rows are malformed;
also the CSV writer launched via the streamToText -> parser.write(...) ->
parser.end() path runs in an unawaited async IIFE so its failures can become
unhandled rejections. Fix by making on_skip an async function that (a)
increments/uses the same batching logic as the main consumer (the for await
(const record of parser) flush threshold) and calls the same flush function when
invalidItemsBatch reaches the batch size, and (b) awaits the shared CSV writer
promise instead of letting its IIFE run detached (capture the promise returned
by the writer task created around streamToText/parser.write/parser.end and await
it before completing the outer try/catch), ensuring any writer errors propagate
and the batch is flushed timely from on_skip.
- Around line 209-237: The CSV pump is started as an un-awaited async IIFE which
can reject out of the outer try/catch and leave the parser hanging; change the
IIFE to a captured promise (e.g., const pump = (async () => { ... })()), wrap
the pump body in try/catch/finally so parser.end() is always called in finally,
and on any pump error call parser.destroy(err) (or parser.destroy()) before
rethrowing so the parser loop won't hang; after the for-await-of parser loop,
await pump to surface any pump errors to the caller. Use the existing
identifiers parser, streamToText(r2Object.body), injectedHeader and byteStart to
locate and modify the code.
In `@packages/api/src/utils/__tests__/json-utils.test.ts`:
- Line 2: The test imports helpers via a relative path; change the import to use
the repo TypeScript path alias instead (so the test imports isJsonlFile and
mapJsonRowToItem from the `@packrat/api` alias rather than ../json-utils). Locate
the import line that currently reads "import { isJsonlFile, mapJsonRowToItem }
from '../json-utils';" in the json-utils.test.ts and replace it with the
corresponding aliased module import (e.g., import { isJsonlFile,
mapJsonRowToItem } from '`@packrat/api/`...') so the test follows the project's
`@packrat/api/`* import convention.
In `@packages/api/src/utils/json-utils.ts`:
- Line 4: The import in json-utils.ts currently uses a relative path
('./csv-utils') for functions like parseFaqs, parsePrice, parseWeight, and
safeJsonParse; update that import to use the repository path alias (replace
'./csv-utils' with the corresponding '`@packrat/api/`...' alias) so it conforms to
the root tsconfig path aliases and keeps imports consistent across the repo.
- Around line 98-109: The JSON-parse branch assigns JSON.parse(val) directly to
item.categories which can introduce non-string entries (e.g., numbers); update
the block that handles val.startsWith('[') to validate and normalize the parsed
value to an array of non-empty strings (like the native-array branch): call
JSON.parse(val), ensure the result is an array, map each element to String(...)
or filter by typeof === 'string' and .trim(), then filter(Boolean) before
assigning to item.categories; keep the existing catch fallback to [val]
unchanged.
- Around line 175-180: The branch that handles stringified techs currently
assigns parsed output directly to item.techs, which can allow primitives,
arrays, or non-string values; instead, after safeJsonParse(readable) check that
parsed is a plain object (typeof parsed === 'object' && parsed !== null &&
!Array.isArray(parsed)), then pass it through toStringRecord(parsed) and assign
that result to item.techs; if parsed is an array or a primitive fall back to {};
keep the existing catch to set item.techs = {} on parse errors.
In `@packages/api/src/workflows/catalog-etl-workflow.ts`:
- Around line 240-249: In writerPromise, when isNonFirstChunk is true you must
discard bytes up to and including the first newline before feeding the chunk to
parser to avoid prepending the tail of the previous row; implement a one-time
strip on the first yielded text from streamToText(obj.body) (e.g., track a local
boolean like didStrip) and if isNonFirstChunk && !didStrip remove everything
through the first '\n' from that text chunk (then set didStrip) before calling
parser.write (and still honor parser.once('drain')); update logic around
injectedHeader, parser.write and streamToText(obj.body) handling to perform this
strip only for the first chunk of non-initial payloads.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 4bc811a8-f8ac-47b5-b622-dcaea37faeef
📒 Files selected for processing (4)
packages/api/src/services/etl/processCatalogEtl.tspackages/api/src/utils/__tests__/json-utils.test.tspackages/api/src/utils/json-utils.tspackages/api/src/workflows/catalog-etl-workflow.ts
Comment on lines
+194
to
+205
| on_skip: (err: Error) => { | ||
| const parserLine = (err as { lines?: number }).lines ?? rowIndex; | ||
| const parseErrorLog: NewInvalidItemLog = { | ||
| jobId, | ||
| errors: [{ field: 'csv_parse', reason: err.message }], | ||
| rawData: { parseError: err.message }, | ||
| rowIndex: parserLine, | ||
| }; | ||
| invalidItemsBatch.push(parseErrorLog); | ||
| console.warn( | ||
| `[ETL] Skipped malformed CSV row at parser line ${parserLine}: ${err.message}`, | ||
| ); |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="packages/api/src/services/etl/processCatalogEtl.ts"
echo "== File =="
ls -l "$file"
echo "== Snippet around on_skip (lines ~170-240) =="
nl -ba "$file" | sed -n '150,260p'
echo "== Snippet around invalidItemsBatch flush points (search) =="
rg -n "invalidItemsBatch|push\\(parseErrorLog\\)|flush|EOF|for await \\(const record of parser\\)" "$file" || true
echo "== Find unawaited async IIFE / writer task =="
rg -n "IIFE|\\(async \\(\\) =>|void \\(async|async \\(\\) =>|new Promise|setImmediate|writer|append|writeStream|createWrite|stream|pipeline|then\\(" "$file" || true
echo "== Snippet around parser loop (around for-await) =="
lnum=$(rg -n "for await \\(const record of parser\\)" "$file" | head -n1 | cut -d: -f1 || true)
if [ -n "${lnum:-}" ]; then
start=$((lnum-60)); end=$((lnum+80))
nl -ba "$file" | sed -n "${start},${end}p"
fiRepository: PackRat-AI/PackRat
Length of output: 281
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="packages/api/src/services/etl/processCatalogEtl.ts"
echo "== rg locations =="
rg -n "on_skip" "$file" || true
rg -n "invalidItemsBatch" "$file" || true
rg -n "for await \\(const record of parser\\)" "$file" || true
rg -n "IIFE|void \\(async|\\(async \\(\\) =>|new Promise|then\\(|append\\(|writeStream|createWrite|pipeline|stream\\.write|writer" "$file" || true
rg -n "flush|EOF|end-of|close\\(|finish\\(" "$file" || true
echo "== Show on_skip block (±40 lines) =="
on_skip_line=$(rg -n "on_skip" "$file" | head -n1 | cut -d: -f1)
if [ -n "${on_skip_line:-}" ]; then
start=$((on_skip_line-60)); end=$((on_skip_line+60))
awk -v s="$start" -v e="$end" 'NR>=s && NR<=e {printf "%5d\t%s\n", NR, $0}' "$file"
fi
echo "== Show invalidItemsBatch section (±80 lines around first occurrence) =="
inv_line=$(rg -n "invalidItemsBatch" "$file" | head -n1 | cut -d: -f1)
if [ -n "${inv_line:-}" ]; then
start=$((inv_line-80)); end=$((inv_line+160))
awk -v s="$start" -v e="$end" 'NR>=s && NR<=e {printf "%5d\t%s\n", NR, $0}' "$file"
fi
echo "== Show parser for-await loop section (±80 lines) =="
loop_line=$(rg -n "for await \\(const record of parser\\)" "$file" | head -n1 | cut -d: -f1 || true)
if [ -n "${loop_line:-}" ]; then
start=$((loop_line-80)); end=$((loop_line+140))
awk -v s="$start" -v e="$end" 'NR>=s && NR<=e {printf "%5d\t%s\n", NR, $0}' "$file"
fiRepository: PackRat-AI/PackRat
Length of output: 24727
Bound on_skip buffering and await the CSV writer task.
on_skiponly pushes intoinvalidItemsBatchand never flushes; flushing happens only while consumingfor await (const record of parser)and at the final post-loop flush, so a chunk/file with mostly or entirely malformed rows can growinvalidItemsBatchto total-row scale (risking OOM).- The CSV writer runs in an unawaited async IIFE (the
streamToText(...)->parser.write(...)->parser.end()path), so failures there can bypass the outertry/catchand surface as unhandled promise rejections.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/api/src/services/etl/processCatalogEtl.ts` around lines 194 - 205,
The on_skip handler currently only pushes parse errors into invalidItemsBatch
(in processCatalogEtl.ts) and doesn't trigger a flush, which can let
invalidItemsBatch grow unbounded when many rows are malformed; also the CSV
writer launched via the streamToText -> parser.write(...) -> parser.end() path
runs in an unawaited async IIFE so its failures can become unhandled rejections.
Fix by making on_skip an async function that (a) increments/uses the same
batching logic as the main consumer (the for await (const record of parser)
flush threshold) and calls the same flush function when invalidItemsBatch
reaches the batch size, and (b) awaits the shared CSV writer promise instead of
letting its IIFE run detached (capture the promise returned by the writer task
created around streamToText/parser.write/parser.end and await it before
completing the outer try/catch), ensuring any writer errors propagate and the
batch is flushed timely from on_skip.
| @@ -0,0 +1,218 @@ | |||
| import { describe, expect, it } from 'vitest'; | |||
| import { isJsonlFile, mapJsonRowToItem } from '../json-utils'; | |||
Contributor
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win
Use the API path alias in the test import.
Please import this helper through @packrat/api/* instead of a relative path to match the repo rule for TypeScript imports.
As per coding guidelines, "Use path aliases defined in root tsconfig.json: @packrat/api/*, @packrat/ui/*, expo-app/*, guides-app/*, landing-app/*, and nativewindui/*."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/api/src/utils/__tests__/json-utils.test.ts` at line 2, The test
imports helpers via a relative path; change the import to use the repo
TypeScript path alias instead (so the test imports isJsonlFile and
mapJsonRowToItem from the `@packrat/api` alias rather than ../json-utils). Locate
the import line that currently reads "import { isJsonlFile, mapJsonRowToItem }
from '../json-utils';" in the json-utils.test.ts and replace it with the
corresponding aliased module import (e.g., import { isJsonlFile,
mapJsonRowToItem } from '`@packrat/api/`...') so the test follows the project's
`@packrat/api/`* import convention.
- Remove incorrect JSONL partial-line skip (chunker guarantees clean boundaries; skipPartialLine was dropping the first valid record per non-first chunk) - Fix @packrat/guards import order for Biome organizeImports (after @packrat/db) - Use @packrat/api/utils/csv-utils alias instead of relative path in json-utils - Filter non-strings from JSON-parsed categories array (parity with native branch) - Apply toStringRecord to safeJsonParse techs result (parity with native branch) - Add test for JSON array categories non-string filtering Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@packrat/api/* → @packrat/db → @packrat/guards → @packrat/schemas/* Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…olution csv-parse infers the correct CsvError type; explicit Error annotation caused the Options overload to be rejected, resolving to Callback<string[]> instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
csv-parse types on_skip as (err: CsvError | undefined, ...) => void Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Avoids TS18048 — `err` is possibly undefined at line 206; the `message` variable is already safely computed with optional chaining and a fallback. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The string fits within the 100-char line width; collapsing it removes the only Biome format error in the checks job. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/api/src/workflows/catalog-etl-workflow.ts (1)
225-239:⚠️ Potential issue | 🟠 Major | ⚡ Quick winBound
invalidItemsBatchinsideon_skip.Skipped CSV rows bypass the
for await (const record of parser)flush path. On a malformed chunk this can accumulate one log per bad row until EOF and blow the worker's memory.Suggested fix
+ let invalidLogsFlush = Promise.resolve(); + const flushSkippedLogsIfNeeded = () => { + if (invalidItemsBatch.length < BATCH_SIZE) return; + const batch = invalidItemsBatch.splice(0, invalidItemsBatch.length); + invalidLogsFlush = invalidLogsFlush.then(() => + processLogsBatch({ jobId, logs: batch, env }), + ); + }; + const parser = parse({ relax_column_count: true, relax_quotes: true, skip_empty_lines: true, skip_records_with_error: true, on_skip: (err) => { const parserLine = (err as { lines?: number } | undefined)?.lines ?? rowIndex; const message = err?.message ?? 'unknown parse error'; invalidItemsBatch.push({ jobId, errors: [{ field: 'csv_parse', reason: message }], rawData: { parseError: message }, rowIndex: parserLine, }); + flushSkippedLogsIfNeeded(); }, }); ... await writerPromise; + await invalidLogsFlush;Also applies to: 302-302
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/api/src/workflows/catalog-etl-workflow.ts` around lines 225 - 239, The on_skip parser callback currently pushes skipped-row entries directly into the in-memory invalidItemsBatch, which can grow without bound because skipped rows never hit the downstream for-await (const record of parser) flush path; fix by replacing the direct push with a helper that appends the invalid item and triggers an immediate flush when the buffer exceeds a small threshold (e.g., 50–100) or always persists immediately for skips: implement a function like addInvalidItemAndMaybeFlush(invalidItemsBatch, item) that pushes the item and calls the existing flushInvalidItemsBatch()/persistInvalidItems() routine (or creates one) to persist and clear the buffer, and call that helper from parser's on_skip (and at the other skip site around line ~302) so skipped rows do not accumulate in memory.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@packages/api/src/workflows/catalog-etl-workflow.ts`:
- Around line 225-239: The on_skip parser callback currently pushes skipped-row
entries directly into the in-memory invalidItemsBatch, which can grow without
bound because skipped rows never hit the downstream for-await (const record of
parser) flush path; fix by replacing the direct push with a helper that appends
the invalid item and triggers an immediate flush when the buffer exceeds a small
threshold (e.g., 50–100) or always persists immediately for skips: implement a
function like addInvalidItemAndMaybeFlush(invalidItemsBatch, item) that pushes
the item and calls the existing flushInvalidItemsBatch()/persistInvalidItems()
routine (or creates one) to persist and clear the buffer, and call that helper
from parser's on_skip (and at the other skip site around line ~302) so skipped
rows do not accumulate in memory.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 129b2939-bf6c-4269-b043-73b291d04546
📒 Files selected for processing (4)
packages/api/src/services/etl/processCatalogEtl.tspackages/api/src/utils/__tests__/json-utils.test.tspackages/api/src/utils/json-utils.tspackages/api/src/workflows/catalog-etl-workflow.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.jsonl/.ndjson→ JSONL path, otherwise CSV)json-utils.tswithisJsonlFile+mapJsonRowToItem— mirrors all CSV field mappings with JSON-native types (arrays pass through, numbers pass through, no string deserialization)relax_quotes/skip_records_with_error/on_skipfrom the prior PR (was missing from workflow path)Why JSONL over CSV
JSON.parse(line)per row, no library dependencyTest plan
.jsonlfile via the ETL endpoint and verify items load.csvupload still works🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Tests