Skip to content

ESQL: reapply "NDJSON datasource"#142855

Merged
swallez merged 4 commits intoelastic:mainfrom
swallez:esql/reapply-ndjson
Feb 23, 2026
Merged

ESQL: reapply "NDJSON datasource"#142855
swallez merged 4 commits intoelastic:mainfrom
swallez:esql/reapply-ndjson

Conversation

@swallez
Copy link
Copy Markdown
Contributor

@swallez swallez commented Feb 23, 2026

Reapply #142700 which was reverted in commit e6aaa8b

Also update to the recent changes in the ESQL Datasource SPI and implementations, which is what was causing the build failures.

Adds an NDJSON ESQL datasource. The schema is inferred from the first 100 lines.
Supported types are INTEGER, DOUBLE, LONG, BOOLEAN, DATETIME (strings in ISO format) and KEYWORD.

Multi-values are supported, nested objects are flattened by joining property names with a '.'

In case of parsing failure, the stream is consumed until the next end of line to recover.

swallez and others added 2 commits February 23, 2026 14:21
Adds an NDJSON ESQL datasource. The schema is inferred from the first 100 lines.
Supported types are INTEGER, DOUBLE, LONG, BOOLEAN, DATETIME (strings in ISO format) and KEYWORD.

Multi-values are supported, nested objects are flattened by joining property names with a '.'

In case of parsing failure, the stream is consumed until the next end of line to recover.
@swallez swallez requested review from bpintea and costin February 23, 2026 13:32
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Feb 23, 2026
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @swallez, I've created a changelog YAML for you.

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

Copy link
Copy Markdown
Contributor

@bpintea bpintea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPI adaptations LGTM

@swallez swallez merged commit c55f9e9 into elastic:main Feb 23, 2026
35 checks passed
@swallez swallez deleted the esql/reapply-ndjson branch February 23, 2026 16:12
szybia added a commit to szybia/elasticsearch that referenced this pull request Feb 23, 2026
…on-sliced-reindex

* upstream/main: (110 commits)
  Add search task watchdog to log hot threads on slow search (elastic#142746)
  Fix return_intermediate_results query param on get async search results (elastic#142875)
  Mute org.elasticsearch.compute.operator.exchange.BatchDriverTests testSinglePageSingleBatch elastic#142895
  Cancel reindex body always has status (elastic#142766)
  Fix built-in roles sync losing updates (elastic#142433)
  ESQL: Clarify docs and add csv test for WHERE in STATS (elastic#133629)
  Fix and unmute ReindexResumeIT (elastic#142788)
  Fix broken release notes
  Mute org.elasticsearch.benchmark.vector.scorer.VectorScorerOSQBenchmarkTests testSingleScalarVsVectorized {p0=384 p1=4 p2=NIO p3=COSINE} elastic#142883
  ES|QL: fix Generative tests for commands that don't change the output schema (elastic#142864)
  Mute org.elasticsearch.benchmark.vector.scorer.VectorScorerOSQBenchmarkTests testSingleScalarVsVectorized {p0=1024 p1=1 p2=NIO p3=DOT_PRODUCT} elastic#142881
  SQL: Fix QlIllegalArgumentException with non-foldable date range queries (elastic#142386)
  Add more errors to the allowed_errors with github issue links (elastic#142862)
  ESQL: reapply "NDJSON datasource" (elastic#142855)
  Add implementation to update service settings method for Alibaba Cloud Search service (elastic#142738)
  Mute org.elasticsearch.snapshots.SnapshotShutdownIT testStartRemoveNodeButDoNotComplete elastic#142871
  Mute org.elasticsearch.snapshots.SnapshotShutdownIT testDeleteSnapshotWithPausedShardSnapshots elastic#142870
  Mute org.elasticsearch.snapshots.SnapshotShutdownIT testAbortSnapshotWhileRemovingNode elastic#142869
  Mute org.elasticsearch.snapshots.SnapshotShutdownIT testRemoveNodeDuringSnapshot elastic#142868
  ES|QL: Guard exponential_histogram TO_STRING against too large inputs (elastic#140718)
  ...
jdconrad pushed a commit to jdconrad/elasticsearch that referenced this pull request Feb 24, 2026
Adds an NDJSON ESQL datasource. The schema is inferred from the first 100 lines.
Supported types are INTEGER, DOUBLE, LONG, BOOLEAN, DATETIME (strings in ISO format) and KEYWORD.

Multi-values are supported, nested objects are flattened by joining property names with a '.'

In case of parsing failure, the stream is consumed until the next end of line to recover.
sidosera pushed a commit to sidosera/elasticsearch that referenced this pull request Feb 24, 2026
Adds an NDJSON ESQL datasource. The schema is inferred from the first 100 lines.
Supported types are INTEGER, DOUBLE, LONG, BOOLEAN, DATETIME (strings in ISO format) and KEYWORD.

Multi-values are supported, nested objects are flattened by joining property names with a '.'

In case of parsing failure, the stream is consumed until the next end of line to recover.
@tylerperk tylerperk added the ES|QL|DS ES|QL datasources label Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL ES|QL|DS ES|QL datasources >feature Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants