From a34de994243ef2a7f0ff65dc1939bf28d3d35af6 Mon Sep 17 00:00:00 2001 From: Andre Manoel Date: Tue, 14 Apr 2026 08:40:38 +0000 Subject: [PATCH 1/3] fix: text-to-sql devnote date, images, and publish-devnotes nav - Update post date from 2026-03-11 to 2026-04-14 so it appears as the newest post on the devnotes page. - Replace raw tags with markdown image syntax so mkdocs rewrites relative paths correctly for the blog plugin's slug-based URLs. - Overlay mkdocs.yml from HEAD in publish-devnotes workflow so new nav entries are included in devnotes-only rebuilds. --- .github/workflows/publish-devnotes.yml | 9 +++++++++ docs/devnotes/posts/text-to-sql.md | 15 ++++----------- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/.github/workflows/publish-devnotes.yml b/.github/workflows/publish-devnotes.yml index 474a2bf5d..5f830a9b4 100644 --- a/.github/workflows/publish-devnotes.yml +++ b/.github/workflows/publish-devnotes.yml @@ -39,6 +39,15 @@ jobs: run: | git checkout ${{ env.SOURCE_SHA }} git checkout ${{ github.sha }} -- docs/devnotes/ + + # Patch the "Dev Notes" nav section from HEAD's mkdocs.yml into the + # old source's mkdocs.yml. This keeps nav entries for new devnotes + # without pulling in entries for non-devnotes pages that may not + # exist in the old source checkout. + pip install -q yq + DEVNOTES_NAV=$(git show ${{ github.sha }}:mkdocs.yml \ + | yq '.nav[] | select(has("Dev Notes")) | .["Dev Notes"]') + yq -y -i "(.nav[] | select(has(\"Dev Notes\")))[\"Dev Notes\"] = ${DEVNOTES_NAV}" mkdocs.yml - name: Install uv uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7 with: diff --git a/docs/devnotes/posts/text-to-sql.md b/docs/devnotes/posts/text-to-sql.md index 00bf1df27..bef426bb7 100644 --- a/docs/devnotes/posts/text-to-sql.md +++ b/docs/devnotes/posts/text-to-sql.md @@ -1,5 +1,5 @@ --- -date: 2026-03-11 +date: 2026-04-14 authors: - dnathawani - ymeyer @@ -8,9 +8,7 @@ authors: # **Engineering an Enterprise-Grade Text-to-SQL Dataset with NeMo Data Designer** -Text-to-SQL Synthetic Data Pipeline - -
+![Text-to-SQL Synthetic Data Pipeline](assets/text-to-sql/text-to-sql-pipeline.jpg){ width=800 } While LLMs have mastered generic coding, Text-to-SQL remains one of the most challenging frontiers in enterprise AI. In many ways this is due to (i) SQL tasks relying on both code and data and (ii) real-world data and databases being quite messy. Focusing on careful data design that accounts for real-world diversity and complexity, we built a [NeMo Data Designer](https://github.com/NVIDIA-NeMo/DataDesigner) pipeline that includes conditional sampling, three-stage LLM generation, code validators, and multi-dimensional judge scoring to generate reasoning-heavy text-to-SQL samples across PostgreSQL, MySQL, and SQLite, and automatically filter down to the highest quality 96.5k records. Each sample pairs a natural-language prompt and a fully synthetic database schema context with a target SQL query. To improve robustness and mimic the messiness of production databases, the pipeline injects distractor tables and columns into the schema context, forcing the model to learn to ignore irrelevant schema elements. The final dataset is validated and filtered through per-dialect syntax validators and five LLM-as-a-critic judges. @@ -426,9 +424,7 @@ The high rejection rate is a feature, not a bug. By generating 3x more data than This dataset was shipped in the SFT stage of **Nemotron Super v3**. On the [BIRD SQL benchmark](https://bird-bench.github.io/) (1,534 dev samples, 5-run average), Nemotron Super achieves **41.80% EX** (execution accuracy) --- outperforming GPT-OSS-120B at 38.25%. Including our synthetic dataset in the SFT blend raised Nemotron Super's EX on BIRD by **15 points**, from 26.77% to 41.80%. -BIRD SQL Benchmark Results — Nemotron Super EX improves from 26.77% to 41.80% - -
+![BIRD SQL Benchmark Results - Nemotron Super EX improves from 26.77% to 41.80%](assets/text-to-sql/bird-benchmark-results.jpg){ width=800 } | Model | BIRD EX (%) | |-------|-------------| @@ -465,7 +461,7 @@ This dataset was shipped in the SFT stage of **Nemotron Super v3**. On the [BIRD - **Code Sandbox for semantic correctness.** The current Quality Waterfall validates syntax and assesses quality (LLM judges), but it doesn't verify whether the query actually returns the right results. A natural next step would be adding Code Sandbox support to Data Designer --- executing generated SQL against a ground-truth database and comparing results to enable execution-based filtering, end-to-end verification, and hard negative mining for preference training. - **RL on BIRD.** Run reinforcement learning experiments using the [NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) RL environment for BIRD, training models to improve execution accuracy through reward signals from actual query execution. - **Schema representation.** Improve how schemas are represented in prompts to close the gap with SOTA approaches that use richer structural encodings (e.g., foreign key graphs, column descriptions, value examples). -- **More benchmarks.** Incorporate additional SQL benchmarks --- [Spider 2.0](https://spider2-sql.github.io/), [LiveSQLBench](https://livesqlbench.github.io/) --- to evaluate generalization beyond BIRD and drive the next iteration of the pipeline. +- **More benchmarks.** Incorporate additional SQL benchmarks --- [Spider 2.0](https://spider2-sql.github.io/), [LiveSQLBench](https://livesqlbench.ai/) --- to evaluate generalization beyond BIRD and drive the next iteration of the pipeline. --- @@ -590,9 +586,6 @@ Because this pipeline is encapsulated in Data Designer, the configuration can be - **NeMo Data Designer:** [github.com/NVIDIA-NeMo/DataDesigner](https://github.com/NVIDIA-NeMo/DataDesigner) - **BIRD Benchmark:** [bird-bench.github.io](https://bird-bench.github.io/) - **Spider 2.0 Benchmark:** [spider2-sql.github.io](https://spider2-sql.github.io/) -- **Structured Outputs Dev Note** (related pipeline): [Structured Outputs for Nemotron](structured-outputs-from-nemotron.md) -- **RQA Dev Note** (reasoning data with Data Designer): [Graduate-Level Science Reasoning Data](rqa.md) - --- *Want to learn more about NeMo Data Designer? Check out our [documentation](https://github.com/NVIDIA-NeMo/DataDesigner) and start building your own high-fidelity synthetic datasets today.* From bbf9d1312ad66d0e7e9a3ea7912a1f08fd0c86eb Mon Sep 17 00:00:00 2001 From: Andre Manoel Date: Tue, 14 Apr 2026 08:54:49 +0000 Subject: [PATCH 2/3] fix(ci): replace yq with surgical Python nav patching in publish-devnotes The yq JSON roundtrip was mangling the entire mkdocs.yml file (indentation, quoting, comments), causing mike deploy to fail. Replace with a Python script that extracts only the Dev Notes nav block from HEAD's mkdocs.yml and splices it into the old source's copy, leaving all other content byte-identical. --- .github/workflows/publish-devnotes.yml | 39 +++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 4 deletions(-) diff --git a/.github/workflows/publish-devnotes.yml b/.github/workflows/publish-devnotes.yml index 5f830a9b4..775fa432e 100644 --- a/.github/workflows/publish-devnotes.yml +++ b/.github/workflows/publish-devnotes.yml @@ -44,10 +44,41 @@ jobs: # old source's mkdocs.yml. This keeps nav entries for new devnotes # without pulling in entries for non-devnotes pages that may not # exist in the old source checkout. - pip install -q yq - DEVNOTES_NAV=$(git show ${{ github.sha }}:mkdocs.yml \ - | yq '.nav[] | select(has("Dev Notes")) | .["Dev Notes"]') - yq -y -i "(.nav[] | select(has(\"Dev Notes\")))[\"Dev Notes\"] = ${DEVNOTES_NAV}" mkdocs.yml + git show ${{ github.sha }}:mkdocs.yml > /tmp/mkdocs-head.yml + python3 - <<'PYEOF' + import re + + def extract_devnotes_block(text): + """Extract the ' - Dev Notes:' block including all indented children.""" + lines = text.splitlines(keepends=True) + start = None + for i, line in enumerate(lines): + if re.match(r'^ - Dev Notes:', line): + start = i + break + if start is None: + raise SystemExit("Dev Notes nav section not found") + end = start + 1 + while end < len(lines): + # Stop at next top-level nav entry (2-space indent) or non-nav section + if lines[end].strip() and not lines[end].startswith(' ') and not lines[end].startswith(' #'): + break + end += 1 + return start, end, lines + + with open("/tmp/mkdocs-head.yml") as f: + head_start, head_end, head_lines = extract_devnotes_block(f.read()) + head_block = head_lines[head_start:head_end] + + with open("mkdocs.yml") as f: + old_text = f.read() + old_start, old_end, old_lines = extract_devnotes_block(old_text) + new_lines = old_lines[:old_start] + head_block + old_lines[old_end:] + + with open("mkdocs.yml", "w") as f: + f.writelines(new_lines) + print(f"Patched Dev Notes nav: replaced lines {old_start+1}-{old_end} with {len(head_block)} lines from HEAD") + PYEOF - name: Install uv uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7 with: From 3512e2db7a20edaa5893112834925e69df804f8d Mon Sep 17 00:00:00 2001 From: Andre Manoel Date: Tue, 14 Apr 2026 08:57:09 +0000 Subject: [PATCH 3/3] refactor(ci): extract nav patching into standalone script Move the inline Python heredoc from publish-devnotes.yml into .github/scripts/patch-devnotes-nav.py. The workflow checks out the script from HEAD alongside the devnotes content. --- .github/scripts/patch-devnotes-nav.py | 57 ++++++++++++++++++++++++++ .github/workflows/publish-devnotes.yml | 37 +---------------- 2 files changed, 59 insertions(+), 35 deletions(-) create mode 100644 .github/scripts/patch-devnotes-nav.py diff --git a/.github/scripts/patch-devnotes-nav.py b/.github/scripts/patch-devnotes-nav.py new file mode 100644 index 000000000..f35e69f69 --- /dev/null +++ b/.github/scripts/patch-devnotes-nav.py @@ -0,0 +1,57 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Patch the Dev Notes nav block in mkdocs.yml. + +Used by publish-devnotes.yml to splice HEAD's Dev Notes nav entries into an +older source checkout without touching the rest of the file. + +Usage: python patch-devnotes-nav.py +""" + +from __future__ import annotations + +import re +import sys + + +def extract_devnotes_block(text: str) -> tuple[int, int, list[str]]: + """Return (start, end, lines) for the ' - Dev Notes:' nav block.""" + lines = text.splitlines(keepends=True) + start = None + for i, line in enumerate(lines): + if re.match(r"^ - Dev Notes:", line): + start = i + break + if start is None: + raise SystemExit("Dev Notes nav section not found") + end = start + 1 + while end < len(lines): + # Stop at next top-level nav entry (2-space indent) or non-nav section + if lines[end].strip() and not lines[end].startswith(" ") and not lines[end].startswith(" #"): + break + end += 1 + return start, end, lines + + +def main() -> None: + if len(sys.argv) != 3: + raise SystemExit(f"Usage: {sys.argv[0]} ") + + head_path, target_path = sys.argv[1], sys.argv[2] + + with open(head_path) as f: + head_start, head_end, head_lines = extract_devnotes_block(f.read()) + head_block = head_lines[head_start:head_end] + + with open(target_path) as f: + old_start, old_end, old_lines = extract_devnotes_block(f.read()) + new_lines = old_lines[:old_start] + head_block + old_lines[old_end:] + + with open(target_path, "w") as f: + f.writelines(new_lines) + print(f"Patched Dev Notes nav: replaced lines {old_start + 1}-{old_end} with {len(head_block)} lines from HEAD") + + +if __name__ == "__main__": + main() diff --git a/.github/workflows/publish-devnotes.yml b/.github/workflows/publish-devnotes.yml index 775fa432e..613942e6e 100644 --- a/.github/workflows/publish-devnotes.yml +++ b/.github/workflows/publish-devnotes.yml @@ -38,47 +38,14 @@ jobs: - name: Checkout docs source and overlay devnotes run: | git checkout ${{ env.SOURCE_SHA }} - git checkout ${{ github.sha }} -- docs/devnotes/ + git checkout ${{ github.sha }} -- docs/devnotes/ .github/scripts/patch-devnotes-nav.py # Patch the "Dev Notes" nav section from HEAD's mkdocs.yml into the # old source's mkdocs.yml. This keeps nav entries for new devnotes # without pulling in entries for non-devnotes pages that may not # exist in the old source checkout. git show ${{ github.sha }}:mkdocs.yml > /tmp/mkdocs-head.yml - python3 - <<'PYEOF' - import re - - def extract_devnotes_block(text): - """Extract the ' - Dev Notes:' block including all indented children.""" - lines = text.splitlines(keepends=True) - start = None - for i, line in enumerate(lines): - if re.match(r'^ - Dev Notes:', line): - start = i - break - if start is None: - raise SystemExit("Dev Notes nav section not found") - end = start + 1 - while end < len(lines): - # Stop at next top-level nav entry (2-space indent) or non-nav section - if lines[end].strip() and not lines[end].startswith(' ') and not lines[end].startswith(' #'): - break - end += 1 - return start, end, lines - - with open("/tmp/mkdocs-head.yml") as f: - head_start, head_end, head_lines = extract_devnotes_block(f.read()) - head_block = head_lines[head_start:head_end] - - with open("mkdocs.yml") as f: - old_text = f.read() - old_start, old_end, old_lines = extract_devnotes_block(old_text) - new_lines = old_lines[:old_start] + head_block + old_lines[old_end:] - - with open("mkdocs.yml", "w") as f: - f.writelines(new_lines) - print(f"Patched Dev Notes nav: replaced lines {old_start+1}-{old_end} with {len(head_block)} lines from HEAD") - PYEOF + python3 .github/scripts/patch-devnotes-nav.py /tmp/mkdocs-head.yml mkdocs.yml - name: Install uv uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7 with: