feat(evals): flat cross-category shard pool for unified evals - #4745
Merged
Conversation
Install one PEP 508 spec per line so an unreleased Harbor build can override both harbor and harbor-langsmith at once. Newline is the only safe delimiter since a spec contains spaces (name @ git+url); specs are collected into an array and passed as separate args, preserving the existing env-passed, no-eval injection safety.
Add an `agent_impl` choice input (dcode|bare) to the unified workflow, threaded through unified_prep.py. It swaps the harness for the code categories (autonomous, context) only; conversation stays on tau3. Lets a run use the bare create_deep_agent graph, which avoids deepagents-code's reasoning_effort injection (that breaks function tools on models gated to the Responses API). Defaults to dcode; invalid values fail closed.
The harbor langgraph agent builds models with init_chat_model directly, bypassing the deep-agents OpenAI provider profile. OpenAI's model profile defaults reasoning_effort for gpt-5.x, and OpenAI gates reasoning_effort + function tools to /v1/responses, so every trial 400'd on /v1/chat/completions (all harnesses: dcode, bare, tau3). Set use_responses_api=True for openai: models in a shared _build_model helper; a caller-supplied model_kwargs wins.
harbor-index verifiers are OpenAI LLM judges but only received OPENAI_BASE_URL, so they exited without writing a reward file (RewardFileNotFoundError) on both docker and langsmith sandboxes. Forward OPENAI_API_KEY to the verifier env (mirroring the always-on base URL), and grant the key to the job for harbor-index runs so non-openai agent models can still be judged.
harbor-index verifiers run native_judge, which requires JUDGE_PROVIDER, JUDGE_MODELS, JUDGE_REPEATS, JUDGE_CONCURRENCY set 'by the job yaml' or it raises JudgeConfigurationError and writes no reward file. Wire an OpenAI judge into verifier_env_args (provider=openai, repeats/concurrency=1) with a configurable judge_models input (default gpt-5.6-luna). Prefer an independent grader, not the model under test, for real scorecards.
harbor-index tasks with large base images + capsule downloads (e.g. bix*) exceed their environment.build_timeout_sec on the first cold snapshot build and hit EnvironmentStartTimeoutError. Plumb --environment-build-timeout-multiplier as an input (default 1.0). Raising it for a one-time cold build lets it finish; the content-addressed (force_build=false) snapshot then persists so subsequent runs cache-hit at 1.0.
Heavy harbor-index images exhaust the LangSmith snapshot filesystem (harbor's 32 GiB floor) during the layer-copy build and fail with a masked 'Snapshot creation failed. Please retry.'. Plumb harbor's --override-storage-mb as an input (0 = default) so heavy tasks can build with e.g. 64 GiB. (Distinct from env_build_timeout_multiplier, which only extends harbor's client-side wait, not the server build.)
… sandbox The LangSmith sandbox + Client should authenticate with the standard LANGSMITH_API_KEY, not a separate LANGSMITH_SANDBOX_API_KEY (which routed to a different tenant). Remove it from deepagents_harbor.resolve_langsmith_api_key precedence and from the workflow env + credential check; LANGSMITH_API_KEY (always set) covers both. Update tests accordingly.
unified_prep minted a per-model LangSmith dataset (harbor-index__<model>-<hash>) per run, duplicating examples and preventing cross-model comparison — and the per-model override name broke the context (local-dataset) experiment attach (dataset synced, 0 sessions). Pass empty langsmith_dataset so the leaf derives the canonical shared dataset per category (harbor-index/harbor-index-1.0, tau3-subset, local/datasets-context-retrieval-evals); each model is then an experiment on the one dataset. Removes now-dead ls_dataset/slugify/_short_hash + updates tests.
profile=lite runs a frozen high-signal task subset per category (lite_tasks.py: autonomous 15, conversation 11, context 8) at FULL rollouts — biased to the difficulty frontier (partial-pass / hard-but-solvable on a weaker model), excluding saturated + verifier-unstable tasks. unified_prep sets per-category include_tasks + right-sizes n_shards; unified_evals threads include_tasks to each eval job; the tau3-subset step now respects a caller include_tasks (lite wins).
The two bix* tasks pull the ~6 GB chenzizhao/bixbench image, which exhausts the Docker-sandbox runner disk when co-scheduled in one shard (and fails the LangSmith builder). Replace with light python-slim tasks (build-word2vec-pipeline, arcagi2-grid-transform-8b7b) to keep the frontier size at 15 without the disk blowout.
The lite profile was right-sizing shards to ~5 tasks each (min(default, ceil(n/5))), so a 15-task autonomous slice ran only 3 shards => 12 concurrent trials of a 40/model budget (~30% utilization). Fan out to ~2 tasks/shard capped at shard_parallel so every shard runs at once: autonomous 3->8 shards (32 concurrent), conversation 3->6, context 2->4. Also scatters heavy task images one-per-runner instead of concentrating them in a fat shard.
Cap-at-shard_parallel meant n_shards never exceeded max-parallel, so there was no queue: all shards started at once with a static task partition, and a runner that drew fast tasks went idle while the long-pole shard ran. Set n_shards = n_tasks so shard_parallel is purely the concurrent-slot count and GitHub's matrix scheduler pulls the next single-task shard the instant one finishes. The long pole becomes the slowest single task, and each runner builds exactly one task image (no heavy-image concentration).
One-task-per-shard left --concurrency idle: a shard ran only its `rollouts` trials (3 of 4). Size each shard to ceil(concurrency/rollouts) tasks so its tasks*rollouts trial count fills concurrency and every active runner runs a full 4-wide. Rollout-aware: rollouts=1 -> 4 tasks/shard instead of wasting slots. Threads concurrency+rollouts through prep (new UNIFIED_ROLLOUTS env). Full-profile sharding is unchanged.
…t-aware)" This reverts commit 7f6abae.
ruff-format two evals files (hand-wrapped lines the repo's ruff collapses), and mirror the multi-spec harbor_package_override description into harbor.yml so the both-surfaces consistency test passes.
_build_model had no return annotation, so ty inferred object and rejected it at the create_cli_agent / create_deep_agent call sites (which want str | BaseChatModel[ | None]). Annotate the return as BaseChatModel (what init_chat_model returns); TYPE_CHECKING-only import, no runtime change.
Released harbor-langsmith==0.1.1 suffixes every shard's LangSmith experiment
with -{job.id[:8]}, so a model fragments into one experiment per shard on the
shared dataset. harbor main honors the explicit experiment_name the leaf sets
and reuses the session across shards; document that the harbor_package_override
(commit af2e862) is required until that ships in a release.
Append a short sha256 hash of the raw category (mirroring LEAF_SLUG's technique) to HARBOR_CATEGORY_SAFE and the aggregate job's cat-slug, so two raw categories that sanitize alike (e.g. auto.test vs auto_test) no longer collide on the same shard-artifact glob and silently merge into one aggregation.
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Nick Hollon (nick-hollon-lc)
temporarily deployed
to
evals
July 15, 2026 15:56 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
# Conflicts: # .github/scripts/test_unified_prep.py # .github/scripts/unified_prep.py # .github/workflows/_harbor_run.yml # .github/workflows/unified_evals.yml
- unified_prep.py: total_job_guard counted the raw pre-pack task count, but build_flat_matrix packs each model to at most MAX_SHARDS (200) shards. A valid 2-model full run (e.g. 2x280 tasks -> 2x200=400 jobs, within budget) was rejected on a 2x280=560 miscount. Guard on the packed shard count. - _harbor_run.yml: the per-category aggregate legs all received the single global needs.harbor.result, so one failed shard marked every category's summary incomplete. On the flat multi-category path, rely on the per-category --expected-shards count and drop the global result; the single-dataset path (one category) still passes it.
The prior fix bounded jobs/model by min(est_tasks, MAX_SHARDS), but build_flat_matrix's ceil-division packing usually emits fewer entries than MAX_SHARDS (e.g. 120+120+40 tasks -> 140, not 200). The loose bound still wrongly rejected valid runs (3 models x 260 tasks packs to 3x130=390 <= 400, yet 3x200=600 tripped the guard). Measure the real per-model entry count via build_flat_matrix (model-independent) and guard on that. Regression test pins the discriminating 3-model case.
A trial that ran but produced no usable verifier reward (timeout, OOM, exception) is already scored 0 by build_summary; it must not also flag the run incomplete. In the flat pool needs.harbor.result spans every category, so one errored autonomous shard was flagging fully-complete conversation/context as incomplete, and the combiner then voided the entire cross-model scorecard. aggregate_shards.py: incomplete now reflects only genuine coverage/integrity gaps (shard shortfall vs expected_shards, unreadable/non-numeric results); harbor_result and off-K rollout counts are recorded as diagnostics, not incompleteness. aggregate_unified.py: emit a warning and still write the scorecard instead of exiting 1 when every model is incomplete.
Full-profile results (run 29430259116), re-aggregated from the run's existing artifacts after the incomplete-flag fix; autonomous's 14 errored trials count as failures and one shard was recovered from the retry attempt. No tasks re-run.
![open-swe[bot]](https://avatars.githubusercontent.com/in/1395372?s=60&v=4){kind=small}
Open SWE review flagged that dropping shard_failure lets a partial shard (some tasks absent) publish as complete. Verified: the flat-pool void-everything bug is already fixed at the workflow layer -- _harbor_run.yml skips --harbor-result on the flat path (`if [ -z "$FLAT_MATRIX" ]`), since that result spans every category. So dropping shard_failure was unnecessary for the flat path and harmful to the single-dataset (harbor.yml) path, where harbor_result maps to the lone category and is a valid completeness signal. Revert aggregate_shards.py + its tests to the original semantics; keep only the aggregate_unified.py change so a genuinely-incomplete model still yields a scorecard (with warnings) instead of failing the combine job.
Lite full-profile subset (run 29451389710): macro pass@3 0.440 / avg@3 0.269. Add approximate first-attempt wall time to both terra entries (full ~4h, lite ~1h).
replicationbench-find-galactic-vz-peaks (naive big-data script hits the 1h per-command timeout; ~65m observed) and usaco-assign-cows-to-barns (~27m) defeated lite's low-cost goal. Replaced with gpqadiamond-cope-rearrangement- products and swesmith-fix-oauth1-header-params (both <5m, partial-pass frontier on gpt-5.6-terra). Count stays 15.
GH-hosted runners intermittently have the Docker daemon not-yet-ready at job start, hard-failing a shard with 'Docker daemon is not running'. Add a docker-sandbox-only preflight that polls 'docker info' for up to ~120s so a startup race no longer loses an otherwise-good shard. Wait-only (no sudo/start, per the no-privilege-escalation-in-workflows policy).
Re-ran lite (new autonomous subset) for both models. terra lite macro pass@3 0.584 (autonomous 0.400, conversation 0.727, context 0.625); luna lite macro 0.369. terra conversation recovered from a transient Docker-daemon shard failure by merging the run's artifacts (no tasks re-run); luna clean. Wall ~25m / ~50m.
Member
Review: a few things to harden before mergeNice refactor — the packing math and drift guard are solid. Four things I'd want addressed, roughly in priority order. The through-line: #2 and #3 create ways to reach an empty/garbage-data state without any red step, and #1 then turns that state into a green run — so #1 is the highest-value fix. 1.
|
…nified-evals-perf # Conflicts: # .github/scripts/test_unified_prep.py # .github/scripts/test_validate_harbor_limits.py # .github/scripts/unified_prep.py
Mason Daugherty (mdrxy)
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replace the serialized per-provider / per-category eval jobs with one dynamically-drained per-model pool spanning all three categories. A shard is a 1-task unit carrying its own
(category, dataset, agent_impl, task);unified_prepemits a per-model flat matrix and oneevaljob drains it.max_parallel/model_parallelare derived fromconcurrency+rollouts;MAX_SHARDS64→200 with proportional per-category packing above the cap. Per-category aggregation keeps the tier-2 scorecard (aggregate_unified.py) unchanged.harbor.yml's single-dataset path decouplesshard_parallelfromn_shardsfor dynamic dispatch (no-op at the defaultn_shards=10).Validated (CI, docker sandbox, bare harness)
gpt-5.6-terra): complete scorecard across all 3 categories, macro pass@3 0.479 (context 0.0 → 0.625 after the fixes below).gpt-5.6-terra+anthropic:claude-sonnet-5): both per-model pools run concurrently undermodel_parallel, total-job guard holds (68 shards), and the non-OpenAI agent runs with the OpenAI judge.Fixes folded in (found during the dry-runs)
OPENAI_API_KEYnow read the per-shardmatrix.*values (they were gated on the leaf inputs, which the flat design leaves empty).Dependency
Needs the harbor build-lock fix (harbor-framework/harbor#2340) via
harbor_package_overrideuntil it's released — pinnick-hollon-lc/harbor@27a6eac8meanwhile.