Add dynamo.source and pin source revs to commits at submit - #400
Closed
ishandhanani wants to merge 1 commit into
Closed
ishandhanani wants to merge 1 commit into
ishandhanani wants to merge 1 commit into
Conversation
ishandhanani
marked this pull request as ready for review
September 7, 2026 19:16
ishandhanani
requested review from
alec-flowers,
csahithi,
hjjq,
kedarpotdar-nv,
kyleliang-nv,
nlevin-ui and
qiching
as code owners
September 7, 2026 19:17
Closed
One shape for "this code, from git", shared by services[].source and the
new dynamo.source:
dynamo:
source:
git: https://github.com/ai-dynamo/dynamo # default when only rev is set
rev: refs/pull/14000/head # commit, tag, or PR head
dynamo.source takes exactly one of git+rev (with optional patches, the
legacy cargo_patches), pypi (the legacy version), or wheel. It maps onto the
legacy fields in DynamoConfig.__post_init__, so the install code and every
downstream consumer keep reading hash / version / wheel unchanged, and v1
recipes still load. Combining source with a legacy field is rejected.
Testing an unmerged Dynamo PR was impossible before: hash needed a SHA,
and a plain clone does not carry PR refs. The cached install now clones
the configured repo (forks work) and fetches a non-commit ref by name.
A ref like refs/pull/N/head moves, so srtctl apply resolves every unpinned
source.rev with git ls-remote (peeled tags preferred) and records the
commit as source.sha in the submitted config.yaml, comments preserved and
the recipe on disk untouched. The job builds exactly the commit the
lockfile names and the /configs/dynamo-wheels cache is keyed by it; two
runs of one recipe cannot silently build different code because the PR
was pushed to while the job queued. dry-run and preflight never touch the
network. A resolution failure warns and leaves the ref unpinned rather
than blocking the submit; --json lists what was pinned as pinned_sources.
Adds examples/features/dynamo-source.yaml (Dynamo built from the v1.4.2
tag), docs, dry-run output, tests for the shape, resolution, pinning, the
Dynamo mapping, and the submit path, and the regenerated schema reference.
ishandhanani
force-pushed
the
idhanani/srt2-12-dynamo-source
branch
from
September 8, 2026 03:53
9ea0ca5 to
5cf6886
Compare
Collaborator
Author
|
Consolidated into #407, which carries these same commits as one PR against main (this description is reproduced there as one of its parts). Closing to keep review in one place; the branch stays for per-step reference via stack #398. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #399 (Track 2 of #385). Draft until the stack below it merges.
What
One shape for "this code, from git", shared by
services[].source(#399) and the newdynamo.source:dynamo.sourcetakes exactly one ofgit+rev(with optionalpatches, the oldcargo_patches),pypi(the oldversion), orwheel. It maps onto the legacyDynamoConfigfields at load, so the install code and every downstream consumer keep readinghash/version/wheelunchanged and v1 recipes still load. Combiningsourcewith a legacy field is rejected.Testing an unmerged Dynamo PR was not possible before:
hashneeded a commit SHA, and a plain clone does not carry PR refs. The cached source install now clones the configured repository and fetches a non-commit ref by name before checkout.Pinning at submit
A ref like
refs/pull/14000/headmoves.srtctl applyresolves every unpinnedsource.rev(Dynamo and services alike) withgit ls-remote, preferring the peeled commit for tags, and records it assource.shain the submittedconfig.yaml. Comments are preserved and the recipe on disk is untouched. The job builds exactly the commit the lockfile names, and the/configs/dynamo-wheelscache is keyed by that commit, so two runs of one recipe cannot silently build different code because the PR was pushed to while the job sat in the queue.dry-run,preflight, andresolve-overridenever touch the network.--jsonoutput lists what was pinned underpinned_sources;dry-runprints the Dynamo source and whether it is pinned.Also
examples/features/dynamo-source.yaml: Dynamo built from thev1.4.2tag (the same release the PyPI examples install).--set dynamo.source.rev=refs/pull/<n>/headturns it into a PR test.## dynamoindocs/config-reference.md, the pinning note indocs/services.md, regenerateddocs/schema-reference.md.tests/test_source.py(shape validation,git ls-remoteresolution incl. peeled tags and failures, pinning across plain and override-format documents with comments kept, the Dynamo mapping and install script for unpinned refs, pinned SHAs, forks, and patches, and the submit-pathmaterialize_config_pathbehaviour) plus dry-run cases.Validation
git ls-remoteresolution ofv1.4.2andrefs/pull/14000/headagainst ai-dynamo/dynamo from this branch.srtctl applypinnedv1.4.2to2ecbdfdfand wrote theshainto the submittedconfig.yaml(--jsonlisted it underpinned_sources); the workers did a cold source build at that commit inside the SGLang container (rustup, cargo, maturin), populated/configs/dynamo-wheels/2ecbdfdf..., and the aggregated Dynamo benchmark completed. Whole job, build included: 7m45s.Dynamo installed from source (2ecbdfdf...)) and the benchmark completed. Whole job: 3m06s.