feat(data): task_data schemas for all 108 resources servers - #2800
Merged
Conversation
Contributor
Author
|
/ok to test 86d1888 |
adil-a
force-pushed
the
data-prep/task-data-schemas-b2
branch
from
August 26, 2026 19:25
86d1888 to
c6cb251
Compare
adil-a
force-pushed
the
data-prep/task-data-schemas
branch
from
August 26, 2026 19:25
3e44b4c to
176039d
Compare
Contributor
Author
|
/ok to test c6cb251 |
adil-a
force-pushed
the
data-prep/task-data-schemas
branch
from
August 26, 2026 20:36
176039d to
7486f9d
Compare
Contributor
Author
|
/ok to test 191e19c |
adil-a
force-pushed
the
data-prep/task-data-schemas-b2
branch
from
August 26, 2026 20:36
c6cb251 to
191e19c
Compare
adil-a
force-pushed
the
data-prep/task-data-schemas
branch
from
August 26, 2026 21:10
7486f9d to
43b3b50
Compare
adil-a
force-pushed
the
data-prep/task-data-schemas-b2
branch
from
August 26, 2026 21:10
191e19c to
45c72e4
Compare
Contributor
Author
|
/ok to test 45c72e4 |
adil-a
force-pushed
the
data-prep/task-data-schemas
branch
from
August 26, 2026 21:16
43b3b50 to
bea683e
Compare
adil-a
force-pushed
the
data-prep/task-data-schemas
branch
from
August 27, 2026 08:22
12a9154 to
ecc19b8
Compare
adil-a
force-pushed
the
data-prep/task-data-schemas-b2
branch
from
August 27, 2026 08:22
6df9253 to
47ad660
Compare
ananthsub
approved these changes
Aug 27, 2026
adil-a
force-pushed
the
data-prep/task-data-schemas-b2
branch
from
August 28, 2026 09:18
47ad660 to
b12f47e
Compare
Contributor
Author
|
/ok to test b12f47e |
adil-a
force-pushed
the
data-prep/task-data-schemas
branch
from
August 28, 2026 09:29
ac66f17 to
b5cf00d
Compare
adil-a
force-pushed
the
data-prep/task-data-schemas-b2
branch
from
August 28, 2026 09:29
b12f47e to
4f83404
Compare
Contributor
Author
|
/ok to test 4f83404 |
Contributor
Author
|
/ok to test 3a75506 |
ananthsub
previously approved these changes
Aug 28, 2026
One task_data.py per resources server, generated from a field-level survey of every server (committed data shapes, verify() consumption, existing wire request models) and adversarially reviewed per batch. Every schema was validated against every committed dataset row that routes to its server (153 files, 1003 rows, 0 errors), including benchmark and environment data living outside the server's own dir. Conventions applied throughout: required-ness mirrors each server's wire request models; JSON-encoded-string fields stay str; servers with several row shapes use one model per shape keyed on a field the server already reads; genuinely open payloads (reasoning_gym metadata, gymnasium extras, dynamic columns) stay typed as envelopes; families import the parent's models (math_with_judge heirs, mcqa heirs, tavily_search heir, gymnasium family, and friends); fields carry consumed_by tags and, for the 14 verifier_metadata servers, legacy_location markers for the future row-format migration. Also fixes the enforcement test's model extraction to unwrap Annotated[Union[...]] discriminated-union aliases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Adil Asif <adasif@nvidia.com>
Contributor
Author
|
/ok to test 1188068 |
kajalj22
approved these changes
Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PR #2795 ships the task-data schema machinery with only the six example servers covered. The other 108 resources servers still have no machine-checkable definition of their dataset rows.
Change
One
task_data.pyper resources server. All 108. Each schema was generated from a field-level survey of the server: its committed data shapes, whatverify()actually consumes, and its existing wire request models. Each batch was then adversarially reviewed againstapp.pyand the data. Confirmed findings were fixed before this PR.The conventions, applied uniformly:
verify()never reads it. A wire-Optional field stays Optional. Existing datasets keep validating and 422 behavior is unchanged.eval_type, format_verification onverifier.type, asr_with_pc ontask_type. Always a field the server already reads, never one invented for the schema.dict. No false precision.task_datainstead of redefining it.consumed_bytag (verify, metrics, prompt, or provenance). The 14 servers that still useverifier_metadatacarrylegacy_locationmarkers.How this was tested
extra="ignore"ban. 256 tests pass.gym env schemasmoke-tested across plain models, unions, and envelope schemas.Annotated[Union[...]]aliases when collecting models.Stack
PR 2 of the task-data-schema series. Base: #2795 (the machinery). Next: per-server flips of
task_data_validationtoerror, and docs teaching the schema convention.🤖 Generated with Claude Code