Add cluster-wide git_http_version config for intermittent git smart-HTTP failures - #376
Merged
Merged
Conversation
…git smart-HTTP failures Some clusters see intermittent git smart-HTTP failures negotiating HTTP/2 against github.com -- stalls, or truncated responses git misreports as "could not read Username" auth-prompt failures -- on certain network paths. Observed on NVIDIA/InferenceMAX PR #271 on both a login host and its compute nodes, across three separate git operations (clone, fetch, and checkout of a partial clone, which does its own lazy blob fetch). Add srtctl.core.config.git_clone_command_prefix(), reading a new git_http_version srtslurm.yaml setting (e.g. "HTTP/1.1"), and wire it into every git clone/fetch srtctl performs on the real sbatch path: the dynamo-from-source install scripts (core/schema.py, the highest-impact site -- used by any recipe with dynamo.install: true) and the srtlog post-processing clone (cli/mixins/postprocess_stage.py). --bash dev-mode (render/direct_stages/) can't read srtslurm.yaml (deliberately stdlib-only, no srtctl.core imports) -- applies the same http.version=HTTP/1.1 guard unconditionally instead, rather than leaving it unfixed. Default (git_http_version unset) is unchanged: plain `git`, no behavior change for clusters that don't need this.
ishandhanani
requested review from
alec-flowers,
csahithi and
nlevin-ui
as code owners
September 3, 2026 00:50
2 tasks
ishandhanani
added a commit
that referenced
this pull request
Sep 3, 2026
…eclared) (#377) #376 added git_clone_command_prefix() reading git_http_version via get_srtslurm_setting(), but never declared the key on ClusterConfig. srtslurm.yaml is schema-validated, and an unknown top-level key fails that validation and silently drops every cluster default for the whole config, not just the unrecognized key -- confirmed live on NVIDIA/InferenceMAX PR #271: "Failed to load or validate srtslurm.yaml: {'git_http_version': ['Unknown field.']}", which also broke model_paths resolution as collateral damage (FileNotFoundError on a literal "deepseek-v4-pro" path instead of the aliased /scratch/... path). Same lesson TestHostSetup.test_cluster_schema_accepts_the_key already captured for an earlier field -- #376 needed the same test and didn't have it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Some clusters see intermittent git smart-HTTP failures negotiating HTTP/2 against github.com -- stalls, or truncated responses git misreports as
could not read Usernameauth-prompt failures -- on certain network paths. Observed live on NVIDIA/InferenceMAX PR #271, on both a login host and its compute nodes, across three separate git operations (clone, fetch, and checkout of a partial clone, which does its own lazy blob fetch for missing objects).srtctl.core.config.git_clone_command_prefix(), reading an optionalgit_http_versionsetting fromsrtslurm.yaml(e.g."HTTP/1.1").core/schema.py(highest-impact site -- used by any recipe withdynamo.install: true, not just one feature), and the srtlog post-processing clone (cli/mixins/postprocess_stage.py).--bashdev-mode (render/direct_stages/) can't readsrtslurm.yaml(deliberately stdlib-only, nosrtctl.coreimports) -- applieshttp.version=HTTP/1.1unconditionally there instead of leaving it unfixed.git_http_versionunset) is unchanged -- plaingit, no behavior change for clusters that don't need this.Test plan
1544 passed, 2 skipped-- the 2 unrelated failures (test_fingerprint.py,test_apply_mock.py) are pre-existing macOS-only platform gaps (os.sched_getaffinitydoesn't exist on macOS), confirmed by reproducing the same failures on a cleanorigin/maincheckout.