Skip to content

Add sglang and Megatron-LM as git submodules under third_party/ - #2493

Open
Shi-Dong wants to merge 1 commit into
mainfrom
shi/add-third-party-submodules
Open

Add sglang and Megatron-LM as git submodules under third_party/#2493
Shi-Dong wants to merge 1 commit into
mainfrom
shi/add-third-party-submodules

Conversation

@Shi-Dong

Copy link
Copy Markdown
Contributor

Summary

  • Add .gitmodules plus two submodule pointers under third_party/:
    • third_party/sglangsgl-project/sglang @ branch sglang-miles, pinned at cb05a44f3
    • third_party/Megatron-LMradixark/Megatron-LM @ branch miles-main, pinned at 37f91617b
  • Nothing else changes. No source file, no Dockerfile line, no workflow.

Why

Right now a miles commit does not record which sglang and which Megatron-LM it is built
against. docker/Dockerfile resolves both by branch name at image-build time — it runs
git fetch origin sglang-miles against the sglang checkout in the base image, and
git clone -b miles-main https://github.com/radixark/Megatron-LM.git. So whoever pushed to
those branches most recently decides what any given miles commit builds against, and after
the fact there is no way to look at a miles commit and answer "which sglang was that?".

A git submodule is exactly that missing record: the miles tree stores an immutable commit id
per dependency. Bumping a dependency becomes an ordinary reviewable commit ("bump sglang from
X to Y") instead of an invisible side effect of someone else's push.

This is also the prerequisite for building the miles-rl wheel reproducibly — the wheel needs
the patched sglang and Megatron-LM source, and a submodule is how that source gets into
the build. That part is a separate follow-up PR; this one deliberately does nothing but record
the pointers.

Why this is safe to land on its own

  • No workflow in this repo passes submodules: to actions/checkout, so CI never populates
    these directories and clone times are unchanged.
  • The Docker image is unaffected. docker/Dockerfile still installs sglang and Megatron-LM
    the way it does today; it does not read third_party/. It also clones miles without
    --recursive, so the directories stay empty inside the image.
  • Both URLs are https://, so unauthenticated clones and GitHub-hosted runners can fetch them
    without SSH credentials.
  • Existing clones are unaffected until someone opts in with git submodule update --init.

Test plan

  • git submodule update --init --depth 1 populates both submodules and checks out exactly the
    pinned commits (verified locally: cb05a44f3 and 37f91617b).
  • Both pins are the current tips of their tracking branches, so no drift at merge time.
  • git diff --name-status main...HEAD shows only the three added entries.
  • CI on this PR exercises the unchanged build/test path; a green run is the evidence that
    adding the pointers is inert.

Record the exact sglang and Megatron-LM revisions miles is built against
as git submodules, so a single miles commit pins the whole dependency
triple.

Today the two are resolved at image-build time by branch name:
docker/Dockerfile does `git fetch origin sglang-miles` and
`git clone -b miles-main radixark/Megatron-LM`. Whoever last pushed to
those branches therefore decides what a given miles commit builds
against, and nothing in the miles tree records the answer after the fact.

This commit changes no build or runtime behaviour. docker/Dockerfile and
every CI workflow keep resolving sglang/Megatron-LM exactly as before -
no workflow in this repo passes `submodules:` to actions/checkout, so the
new directories are never even populated in CI. The submodules are
inert on-disk pointers whose only job for now is to be the source the
miles-rl wheel is built from (follow-up PR).

Pins match the current branch tips:
  third_party/sglang        sgl-project/sglang @ sglang-miles  cb05a44f3
  third_party/Megatron-LM   radixark/Megatron-LM @ miles-main  37f91617b

Both use https:// URLs so unauthenticated clones and GitHub-hosted
runners can fetch them without SSH credentials.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant