Skip to content

Add git submodule update to build_backend.py #3190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aleozlx merged 2 commits into from
Apr 30, 2026
Merged

Add git submodule update to build_backend.py #3190

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0512c41
add git submodule update
kahyunnam Apr 27, 2026
e7d78d6
Merge branch 'main' into knam/cccl-submodules-update-fix
aleozlx Apr 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions flashinfer-jit-cache/build_backend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,23 +79,32 @@ def _create_build_metadata():

def _compile_jit_cache(output_dir: Path, verbose: bool = True):
"""Compile AOT modules using flashinfer.aot functions directly."""
# Get the project root directory
project_root = Path(__file__).parent.parent

# Ensure 3rdparty submodules are populated (may be empty in CI Docker images)
import subprocess

subprocess.run(
["git", "submodule", "update", "--init", "--recursive"],
cwd=str(project_root),
check=True,
)
Comment on lines +85 to +92
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Running git submodule update unconditionally will cause the build to fail when the source is not a git repository (e.g., when installing from a source distribution or a zip archive). It is important to check for the existence of the .git directory before executing git commands. Additionally, wrapping the call in a try-except block ensures that the build doesn't crash if the git executable is missing from the environment.

Suggested change
# Ensure 3rdparty submodules are populated (may be empty in CI Docker images)
import subprocess
subprocess.run(
["git", "submodule", "update", "--init", "--recursive"],
cwd=str(project_root),
check=True,
)
# Ensure 3rdparty submodules are populated (may be empty in CI Docker images)
if (project_root / ".git").exists():
import subprocess
try:
subprocess.run(
["git", "submodule", "update", "--init", "--recursive"],
cwd=str(project_root),
check=True,
)
except (subprocess.CalledProcessError, FileNotFoundError) as e:
print(f"Warning: Failed to update git submodules: {e}")

Comment on lines +85 to +92
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "1) Verify current code uses PATH-only git invocation"
rg -n 'subprocess\.run\(\s*\["git",\s*"submodule"' flashinfer-jit-cache/build_backend.py -n -C2 || true

echo
echo "2) Demonstrate git submodule fails outside a git repo"
tmpdir="$(mktemp -d)"
if git -C "$tmpdir" submodule update --init --recursive >"$tmpdir/out.log" 2>&1; then
  echo "Unexpected: command succeeded outside a git repo"
  exit 1
else
  echo "Expected failure:"
  sed -n '1,3p' "$tmpdir/out.log"
fi

Repository: flashinfer-ai/flashinfer

Length of output: 263

🏁 Script executed:

cat -n flashinfer-jit-cache/build_backend.py | sed -n '80,95p'

Repository: flashinfer-ai/flashinfer

Length of output: 801

🏁 Script executed:

# Check where _compile_jit_cache is called
rg -n '_compile_jit_cache' flashinfer-jit-cache/ --type py -B2 -A2

# Check if there are any other references to git in the build
echo "---"
echo "Other git references in build files:"
rg -n 'git' flashinfer-jit-cache/build_backend.py --type py

Repository: flashinfer-ai/flashinfer

Length of output: 1547

🏁 Script executed:

# Check how get_git_version handles non-git scenarios
rg -n 'def get_git_version' --type py -A10

# Check if 3rdparty is included in sdist
ls -la flashinfer-jit-cache/3rdparty/ 2>/dev/null | head -20 || echo "3rdparty dir check failed"

# Check the actual directory structure
find flashinfer-jit-cache -maxdepth 2 -type d -name "3rdparty" -o -name ".gitmodules"

Repository: flashinfer-ai/flashinfer

Length of output: 540

🏁 Script executed:

# Read full get_git_version function
cat build_utils.py | sed -n '24,50p'

# Check .gitmodules
if [ -f .gitmodules ]; then
  echo ".gitmodules exists:"
  cat .gitmodules
else
  echo "No .gitmodules found"
fi

# Check 3rdparty structure
ls -la 3rdparty/ 2>/dev/null | head -20 || echo "3rdparty not found at root"

Repository: flashinfer-ai/flashinfer

Length of output: 1254

Guard submodule update for non-git sources and avoid PATH-only executable lookup.

At lines 88–92, git submodule update ... is unconditional and uses a partial executable path ("git"). This will fail:

  • In sdist builds (non-git checkouts)
  • When git is not in PATH (flagged by Ruff S607)

The codebase already detects git repo existence at lines 57–59. Apply the same pattern here:

Suggested fix 
 def _compile_jit_cache(output_dir: Path, verbose: bool = True):
     """Compile AOT modules using flashinfer.aot functions directly."""
     # Get the project root directory
     project_root = Path(__file__).parent.parent
 
     # Ensure 3rdparty submodules are populated (may be empty in CI Docker images)
+    import shutil
     import subprocess
 
-    subprocess.run(
-        ["git", "submodule", "update", "--init", "--recursive"],
-        cwd=str(project_root),
-        check=True,
-    )
+    git_dir = project_root / ".git"
+    if git_dir.exists():
+        git_bin = shutil.which("git")
+        if git_bin is None:
+            raise RuntimeError("git is required to initialize submodules in a git checkout")
+        subprocess.run(
+            [git_bin, "submodule", "update", "--init", "--recursive"],
+            cwd=str(project_root),
+            check=True,
+        )
+    else:
+        required = [
+            project_root / "3rdparty" / "cutlass",
+            project_root / "3rdparty" / "spdlog",
+            project_root / "3rdparty" / "cccl",
+        ]
+        missing = [str(p) for p in required if not p.exists()]
+        if missing:
+            raise RuntimeError(
+                "Missing required 3rdparty dependencies outside git checkout: "
+                + ", ".join(missing)
+            )
🧰 Tools
🪛 Ruff (0.15.12)

[error] 89-89: Starting a process with a partial executable path

(S607)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@flashinfer-jit-cache/build_backend.py` around lines 85 - 92, The
unconditional subprocess.run(["git", ...]) call should be guarded and should not
rely on a PATH-only "git" executable; reuse the repository existence check
already used earlier (the same test that determines if this source is a git
checkout) to skip the submodule update for non-git/sdist sources, and resolve
the git binary with shutil.which before invoking subprocess.run to avoid Ruff
S607; specifically, replace the direct call to subprocess.run with logic that
(1) checks the existing is-git-repo condition or project_root/.git presence and
returns/skip if false, (2) locates the git executable via shutil.which("git")
(or raises/logs a clear error if not found), and (3) calls subprocess.run using
the absolute git path variable instead of the literal "git" string for the
subprocess invocation that updates submodules (refer to the subprocess.run
invocation and the project_root variable in build_backend.py).


# Ensure flashinfer/data/ symlinks exist (normally created by the main
# package's build_backend, but jit-cache builds may not install the main
# package first). Use importlib to avoid name collision with this file.
import importlib.util

spec = importlib.util.spec_from_file_location(
"main_build_backend", Path(__file__).parent.parent / "build_backend.py"
"main_build_backend", project_root / "build_backend.py"
)
main_build_backend = importlib.util.module_from_spec(spec)
spec.loader.exec_module(main_build_backend)
main_build_backend._create_data_dir(use_symlinks=True)

from flashinfer import aot

# Get the project root directory
project_root = Path(__file__).parent.parent

# Set up build directory
build_dir = project_root / "build" / "aot"

Expand Down
Loading