Skip to content

[GDPVal] Add per-provider judge transports with deterministic eligibility replay - #2896

Merged
Kh4L merged 4 commits into
spanev/gdpval-production-hardeningfrom
spanev/gdpval-true3-transport
Aug 31, 2026
Merged

[GDPVal] Add per-provider judge transports with deterministic eligibility replay#2896
Kh4L merged 4 commits into
spanev/gdpval-production-hardeningfrom
spanev/gdpval-true3-transport

Conversation

@Kh4L

@Kh4L Kh4L commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Ports the per-provider judge transport layer, executed and validated in a full 220-task GDPVal evaluation campaign, onto the production-hardening branch.

Changes

  • Per-judge media transport modes: native_pdf, images_and_text, and native_pdf_overflow_images (native PDFs kept; over-cap PDFs rasterized at a pinned DPI with every page retained).
  • Provider-specific request limits as judge-panel config: serialized-request bytes, native-PDF page/document/byte caps (aggregate and per-document), video count, raster DPI tiers.
  • Per-document native-PDF cap (max_native_pdf_bytes_per_document) forces lossless full rasterization of any PDF above the provider's per-file limit; page-count overflow rasterizes only the required prefix, keeping the suffix native.
  • Deterministic transport eligibility preflight before any billable call, with seeded schedule replay; exclusions recorded per row (media_routing_exclusions, transport_by_judge).
  • Overflow eligibility also enforces the judge's aggregate document/byte caps against the post-overflow native remainder, and a render page cap so a plan the renderer cannot satisfy is excluded at preflight instead of failing at dispatch.
  • New transport_assignment.py: provider-free task/reference reassignment repair using exact artifact footprints against provider caps.
  • transport_assignment_repair settings enter the multistage fingerprint, so changing them invalidates journals instead of silently replaying stale plans on resume.
  • Final verdict reminder block appended to judge messages; the text budget's framing reserve includes its serialized upper bound so the request-size invariant holds at any configured cap.

Validation

  • resources_servers/gdpval/tests: 379 passed, 3 skipped, including 16 new tests in test_transport_modes.py (overflow planning/application, eligibility receipts, seeded replay, assignment solver capacity contract, fingerprint invalidation).
  • ruff check / ruff format clean.
  • Same code path that produced a strict 220/220-task, 880/880-trial judging PASS in the validating campaign, rebased onto this branch.

First of four planned child PRs (media caps/tiling, provider timeout/retry hardening, transport receipts follow). Draft until the example three-provider judge panel config lands.

Kh4L added 4 commits August 28, 2026 22:39
…lity replay

Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
Signed-off-by: Serge Panev <spanev@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Kh4L
Kh4L marked this pull request as ready for review August 31, 2026 20:03
@Kh4L
Kh4L merged commit bf9be54 into spanev/gdpval-production-hardening Aug 31, 2026
13 checks passed
@Kh4L
Kh4L deleted the spanev/gdpval-true3-transport branch August 31, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants