Skip to content

feat(datasets): dataset profiler engine #658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
albcui wants to merge 44 commits into
base: main
Choose a base branch
from
+5,693 −133
Open

feat(datasets): dataset profiler engine #658

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
ee7c315
feat(datasets): scaffold nemo-datasets plugin
albcui Jul 13, 2026
981bf9d
feat(datasets): add file-source seam and parquet/jsonl readers
albcui Jul 13, 2026
c2dbd92
feat(datasets): add profiling pipeline (partitions, splits, digest, e…
albcui Jul 13, 2026
84d4bc6
feat(datasets): derive row schema (features) from parquet and jsonl
albcui Jul 13, 2026
5b190b7
feat(datasets): compute per-column stats
albcui Jul 13, 2026
d1dd3df
feat(datasets): classify roles, format/prompt-form axes, and dataset …
albcui Jul 13, 2026
1f194d0
feat(datasets): detect verifiability and implicit-prompt probes
albcui Jul 13, 2026
261174a
fix: harden the profiler and wire up the profile CLI
albcui Jul 13, 2026
d99e2a9
fix(datasets): gate verifiability on a minimum coverage
albcui Jul 14, 2026
3ff7bb0
Make `nemo datasets profile` a real command
albcui Jul 14, 2026
1110c44
resync uv.lock
albcui Jul 16, 2026
0ada117
fix(datasets): drop non-finite floats from numeric column stats
albcui Jul 24, 2026
58a2b1a
fix(datasets): profile mixed-format directories as separate partitions
albcui Jul 24, 2026
8faf743
feat(datasets): record per-file read errors in the profile contract
albcui Jul 31, 2026
d045d49
fix(datasets): keep reading a JSONL file past a corrupt line
albcui Jul 31, 2026
96a4470
fix(datasets): measure chat rows and column cardinality correctly
albcui Jul 31, 2026
8a0f8cc
fix(datasets): require rank to have something to rank, accept integer…
albcui Jul 31, 2026
0a305e8
fix(datasets): correct partition, split and sampling structure
albcui Jul 31, 2026
118b103
fix(datasets): measure content probes independently of role assignment
albcui Aug 4, 2026
0893185
refactor(files): drop the profile content digest rather than repair it
albcui Aug 4, 2026
8c78dde
refactor(datasets): make the directory a partition's identity, not it…
albcui Aug 5, 2026
046ecb2
refactor(datasets): split coverage into the numbers it was collapsing
albcui Aug 5, 2026
51375f8
fix(datasets): gate quoting a column's values on its role, not its ca…
albcui Aug 5, 2026
c292a8b
feat(datasets): report every dataset type the roles satisfy, and acce…
albcui Aug 5, 2026
9b63e4e
perf(datasets): budget rows per partition instead of capping them per…
albcui Aug 5, 2026
8f5f858
refactor(datasets): run the profiler as a job task, not a `nemo` CLI …
albcui Aug 5, 2026
96cd5ca
docs(files): say why the profile contract lives in the shared package
albcui Aug 6, 2026
4ff3bf2
refactor(datasets): identify a partition by one name, not a label plu…
albcui Aug 6, 2026
61956c4
refactor(datasets): enumerate the files that failed, count the ones t…
albcui Aug 6, 2026
377c2be
refactor(datasets): drop SamplingInfo.seed and FeatureSchema.fixed_le…
albcui Aug 6, 2026
db0df00
feat(datasets): report how much a dataset weighs
albcui Aug 7, 2026
fb2ca48
docs(datasets): correct the pipeline module docstring
albcui Aug 7, 2026
be6130f
feat(datasets): infer a glob that selects each split's files
albcui Aug 7, 2026
37f3ab2
feat(datasets): count distinct values only while a column is a vocabu…
albcui Aug 7, 2026
4036b5a
feat(datasets): bound and cheapen the per-character quality scan
albcui Aug 7, 2026
fff3fad
feat(datasets): isolate each column's measurement from its neighbours
albcui Aug 7, 2026
7a5cded
refactor(datasets): measure each column with an accumulator
albcui Aug 7, 2026
f369b32
feat(datasets): read quantiles off counters instead of retained lengths
albcui Aug 7, 2026
1c97875
refactor(datasets): take rows out of the measure stage
albcui Aug 7, 2026
a375d6d
feat(datasets): fold parquet partitions instead of materialising them
albcui Aug 7, 2026
65acf68
fix(datasets): bound the two things row content could grow without limit
albcui Aug 7, 2026
85bf215
feat(datasets): fold partitions that declare no schema
albcui Aug 8, 2026
6b94f31
feat(datasets): read everything by default, and say what completeness…
albcui Aug 8, 2026
c195822
fix(datasets): sample quality in blocks, and count a partial read for…
albcui Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
353 changes: 281 additions & 72 deletions packages/nemo_platform_plugin/src/nemo_platform_plugin/files/dataset_profile.py
View file
Open in desktop

Large diffs are not rendered by default.

194 changes: 133 additions & 61 deletions packages/nemo_platform_plugin/tests/files/test_dataset_profile.py
View file
Open in desktop

Large diffs are not rendered by default.

30 changes: 30 additions & 0 deletions plugins/nemo-datasets/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
[project]
name = "nemo-datasets-plugin"
description = "Dataset profiler for NeMo Platform filesets."
requires-python = ">=3.11,<3.15"
dependencies = [
"nemo-platform-plugin",
"nemo-platform-sdk",
"pyarrow>=19.0.1",
"pydantic>=2.10.3",
]
version = "0.1.0"

[tool.uv.sources]
nemo-platform-plugin = { workspace = true }
nemo-platform-sdk = { workspace = true }

# Deliberately contributes no `nemo.cli` entry point. The profiler runs as a job task
# (`python -m nemo_datasets_plugin.tasks.profile`), which is invoked by the platform rather than
# typed by a user, so its inputs can keep moving while the feature is new.
Comment on lines +17 to +19

@coderabbitai coderabbitai Bot Aug 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Expose the required CLI command.

Lines 17-19 explicitly omit the nemo.cli entry point. The PR objective requires nemo datasets profile. The documented command cannot run. Add the CLI registration and an integration test.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-datasets/pyproject.toml` around lines 17 - 19, Update the plugin
configuration in pyproject.toml to register the required nemo datasets profile
command through the nemo.cli entry point, replacing the deliberate omission.
Ensure the registration invokes the existing profiling task or CLI handler, and
add an integration test that executes nemo datasets profile and verifies it
reaches the expected profile behavior.


[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/nemo_datasets_plugin"]

[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src"]
Loading