Skip to content

feat(data): task_data schemas for all 108 resources servers - #2800

Merged
kajalj22 merged 1 commit into
mainfrom
data-prep/task-data-schemas-b2
Aug 28, 2026
Merged

feat(data): task_data schemas for all 108 resources servers#2800
kajalj22 merged 1 commit into
mainfrom
data-prep/task-data-schemas-b2

Conversation

@adil-a

@adil-a adil-a commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Problem

PR #2795 ships the task-data schema machinery with only the six example servers covered. The other 108 resources servers still have no machine-checkable definition of their dataset rows.

Change

One task_data.py per resources server. All 108. Each schema was generated from a field-level survey of the server: its committed data shapes, what verify() actually consumes, and its existing wire request models. Each batch was then adversarially reviewed against app.py and the data. Confirmed findings were fixed before this PR.

The conventions, applied uniformly:

  • Required-ness mirrors the wire. A field the server's request model requires stays required, even when verify() never reads it. A wire-Optional field stays Optional. Existing datasets keep validating and 422 behavior is unchanged.
  • Servers with several row shapes get one model per shape. The field that names the shape picks the model at validation time: ruler2 keys on eval_type, format_verification on verifier.type, asr_with_pc on task_type. Always a field the server already reads, never one invented for the schema.
  • Open payloads stay open. reasoning_gym metadata, gymnasium extra fields, per-domain injection environments, and dynamic reference columns are typed as envelopes: the stable keys are declared, the rest stays dict. No false precision.
  • Families share models. math_with_judge's heirs, mcqa's heirs, tavily_search's heir, the gymnasium family, and the SWE-instance core import the parent's task_data instead of redefining it.
  • Annotations for future tooling. Every field carries a consumed_by tag (verify, metrics, prompt, or provenance). The 14 servers that still use verifier_metadata carry legacy_location markers.
  • Pointer and stateless servers (aviary, toolsandbox, code_fim, evalplus, and friends) ship minimal schemas. Their docstrings name where the real task payload lives outside the row.

How this was tested

  • Every schema was validated against every committed dataset row that routes to its server, repo-wide. That includes benchmark and environment data living outside the server's own directory. 153 files, 1003 rows, 0 errors, 0 ambiguous duplicate keys. This sweep was run independently after generation. The generators' own reports were not trusted.
  • Six servers have no config-declared committed data (their configs point at generated files). Their local tracked example data was validated directly. All clean.
  • The CI enforcement sweeps from feat(data): per-server task_data schemas, gym env schema, collate validation #2795 now cover 114 schema files: the dependency-light import check and the extra="ignore" ban. 256 tests pass.
  • gym env schema smoke-tested across plain models, unions, and envelope schemas.
  • One test fix rides along: the enforcement test now unwraps Annotated[Union[...]] aliases when collecting models.

Stack

PR 2 of the task-data-schema series. Base: #2795 (the machinery). Next: per-server flips of task_data_validation to error, and docs teaching the schema convention.

🤖 Generated with Claude Code

@adil-a

adil-a commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 86d1888

@copy-pr-bot

copy-pr-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@adil-a adil-a changed the title feat(data): task_data schemas for all 108 resources servers (RFC pillar 2) feat(data): task_data schemas for all 108 resources servers Aug 26, 2026
@adil-a
adil-a force-pushed the data-prep/task-data-schemas-b2 branch from 86d1888 to c6cb251 Compare August 26, 2026 19:25
@adil-a
adil-a force-pushed the data-prep/task-data-schemas branch from 3e44b4c to 176039d Compare August 26, 2026 19:25
@adil-a

adil-a commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test c6cb251

@adil-a

adil-a commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 191e19c

@adil-a
adil-a force-pushed the data-prep/task-data-schemas-b2 branch from c6cb251 to 191e19c Compare August 26, 2026 20:36
@adil-a
adil-a force-pushed the data-prep/task-data-schemas branch from 7486f9d to 43b3b50 Compare August 26, 2026 21:10
@adil-a
adil-a force-pushed the data-prep/task-data-schemas-b2 branch from 191e19c to 45c72e4 Compare August 26, 2026 21:10
@adil-a

adil-a commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 45c72e4

@adil-a
adil-a force-pushed the data-prep/task-data-schemas-b2 branch from 47ad660 to b12f47e Compare August 28, 2026 09:18
@adil-a

adil-a commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b12f47e

@adil-a

adil-a commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4f83404

@adil-a

adil-a commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 3a75506

ananthsub
ananthsub previously approved these changes Aug 28, 2026
One task_data.py per resources server, generated from a field-level
survey of every server (committed data shapes, verify() consumption,
existing wire request models) and adversarially reviewed per batch.
Every schema was validated against every committed dataset row that
routes to its server (153 files, 1003 rows, 0 errors), including
benchmark and environment data living outside the server's own dir.

Conventions applied throughout: required-ness mirrors each server's
wire request models; JSON-encoded-string fields stay str; servers with
several row shapes use one model per shape keyed on a field the server
already reads; genuinely open payloads (reasoning_gym metadata,
gymnasium extras, dynamic columns) stay typed as envelopes; families
import the parent's models (math_with_judge heirs, mcqa heirs,
tavily_search heir, gymnasium family, and friends); fields carry
consumed_by tags and, for the 14 verifier_metadata servers,
legacy_location markers for the future row-format migration.

Also fixes the enforcement test's model extraction to unwrap
Annotated[Union[...]] discriminated-union aliases.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Adil Asif <adasif@nvidia.com>
@adil-a

adil-a commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 1188068

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants