sccache: integrate with Velox build
#52
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| #!/bin/bash | ||
| set -euo pipefail | ||
|
|
||
| # Check for required auth files | ||
| if [[ ! -f /sccache_auth/github_token ]]; then | ||
| echo "ERROR: GitHub token not found at /sccache_auth/github_token" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [[ ! -f /sccache_auth/aws_credentials ]]; then | ||
| echo "ERROR: AWS credentials not found at /sccache_auth/aws_credentials" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Set up directories | ||
| mkdir -p ~/.config/sccache ~/.aws | ||
|
|
||
| # Install AWS credentials (safe in Docker container environment) | ||
| cp /sccache_auth/aws_credentials ~/.aws/credentials | ||
|
|
||
| # Read GitHub token | ||
| GITHUB_TOKEN=$(cat /sccache_auth/github_token | tr -d '\n\r ') | ||
|
|
||
| # Create sccache config | ||
| SCCACHE_ARCH=$(uname -m | sed 's/x86_64/amd64/') | ||
|
|
||
| # Check if we should disable distributed compilation (disabled by default) | ||
| if [[ "${SCCACHE_DISABLE_DIST:-ON}" == "ON" ]]; then | ||
| cat > ~/.config/sccache/config << SCCACHE_EOF | ||
| [cache.disk] | ||
| size = 107374182400 | ||
|
|
||
| [cache.disk.preprocessor_cache_mode] | ||
| use_preprocessor_cache_mode = true | ||
|
|
||
| [cache.s3] | ||
| bucket = "rapids-sccache-devs" | ||
| region = "us-east-2" | ||
| no_credentials = false | ||
|
|
||
| # No [dist] section -> disables distributed compilation | ||
| SCCACHE_EOF | ||
| else | ||
| cat > ~/.config/sccache/config << SCCACHE_EOF | ||
| [cache.disk] | ||
| size = 107374182400 | ||
|
|
||
| [cache.disk.preprocessor_cache_mode] | ||
| use_preprocessor_cache_mode = true | ||
|
|
||
| [cache.s3] | ||
| bucket = "rapids-sccache-devs" | ||
| region = "us-east-2" | ||
| no_credentials = false | ||
|
|
||
| [dist] | ||
| scheduler_url = "https://${SCCACHE_ARCH}.linux.sccache.rapids.nvidia.com" | ||
|
|
||
| [dist.auth] | ||
| type = "token" | ||
| token = "${GITHUB_TOKEN}" | ||
| SCCACHE_EOF | ||
| fi | ||
|
|
||
| # Configure sccache for high parallelism | ||
| # Increase file descriptor limit for high parallelism (if possible) | ||
| ulimit -n $(ulimit -Hn) || echo "Could not increase file descriptor limit" | ||
|
|
||
| # Start sccache server | ||
| sccache --start-server | ||
|
|
||
| # Test sccache | ||
| sccache --show-stats | ||
|
|
||
| # Testing distributed compilation status (only if enabled) | ||
| if [[ "${SCCACHE_DISABLE_DIST:-ON}" == "ON" ]]; then | ||
| echo "Distributed compilation is DISABLED by default - using local compilation with remote S3 caching" | ||
| else | ||
| if sccache --dist-status; then | ||
| echo "Distributed compilation is available" | ||
| else | ||
| echo "Error: Distributed compilation not available, check connectivity" | ||
| exit 1 | ||
| fi | ||
| fi |
|
Contributor
There was a problem hiding this comment. Is this dockerfile based on an existing dockerfile or documentation?
Contributor
Author
There was a problem hiding this comment. This dockerfile is largely based on documentation in a slack channel which I can link offline. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| FROM ubuntu:22.04 | ||
|
|
||
| # Prevent interactive prompts during package installation | ||
| ENV DEBIAN_FRONTEND=noninteractive | ||
|
|
||
| # Install basic dependencies | ||
| RUN <<EOF | ||
| apt-get update && apt-get install -y \ | ||
| curl \ | ||
| wget \ | ||
| ca-certificates \ | ||
| gnupg \ | ||
| lsb-release \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
| EOF | ||
|
|
||
| # Install GitHub CLI | ||
| RUN <<EOF | ||
| curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg | ||
| chmod go+r /usr/share/keyrings/githubcli-archive-keyring.gpg | ||
| echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null | ||
| apt-get update | ||
| apt-get install gh -y | ||
| rm -rf /var/lib/apt/lists/* | ||
| EOF | ||
|
|
||
| # Install gh-nv-gha-aws plugin manually | ||
| RUN <<EOF | ||
| NV_GHA_AWS_VERSION="0.1.1" | ||
| ARCH=$(dpkg --print-architecture) | ||
| if [ "$ARCH" = "amd64" ]; then ARCH="amd64"; elif [ "$ARCH" = "arm64" ]; then ARCH="arm64"; fi | ||
| mkdir -p /root/.local/share/gh/extensions/gh-nv-gha-aws | ||
| wget --no-hsts -q -O /root/.local/share/gh/extensions/gh-nv-gha-aws/gh-nv-gha-aws \ | ||
| "https://github.com/nv-gha-runners/gh-nv-gha-aws/releases/download/v${NV_GHA_AWS_VERSION}/gh-nv-gha-aws_v${NV_GHA_AWS_VERSION}_linux-${ARCH}" | ||
| chmod 0755 /root/.local/share/gh/extensions/gh-nv-gha-aws/gh-nv-gha-aws | ||
| EOF | ||
|
|
||
| # Create plugin manifest | ||
| RUN <<EOF | ||
| cat > /root/.local/share/gh/extensions/gh-nv-gha-aws/manifest.yml << 'MANIFEST' | ||
| owner: nv-gha-runners | ||
| name: gh-nv-gha-aws | ||
| host: github.com | ||
| tag: v0.1.1 | ||
| ispinned: false | ||
| path: $HOME/.local/share/gh/extensions/gh-nv-gha-aws/gh-nv-gha-aws | ||
| MANIFEST | ||
| EOF | ||
|
|
||
| # Create output directory for credentials | ||
| RUN mkdir -p /output |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Running this script results in the following error:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh oops I think I forgot to include a dockerfile in this PR, my bad.