`sccache`: integrate with Velox build by mattgara · Pull Request #52 · rapidsai/velox-testing

mattgara · 2025-09-19T22:11:09Z

Add sccache integration with build_velox.sh
Document how to setup and use sccache

* Add `sccache` integration with `build_velox.sh` * Document how to setup and use `sccache`

Remove non-existent temporary directory removal.

paul-aiyedun · 2025-09-23T19:07:09Z

README.md

+First, set up authentication credentials:
+```bash
+cd velox-testing/velox/scripts
+./setup_sccache_auth.sh


Running this script results in the following error:

ERROR: failed to build: failed to solve: failed to read dockerfile: open sccache_auth.dockerfile: no such file or directory

Oh oops I think I forgot to include a dockerfile in this PR, my bad.

paul-aiyedun · 2025-09-23T19:14:49Z

velox/scripts/build_velox.sh

+sccache Usage:
+  The --sccache option enables distributed compilation caching using the RAPIDS sccache infrastructure.
+
+  Setup authentication first:
+    ./setup_sccache_auth.sh [output_dir]
+
+  Then use with build:
+    --sccache --sccache-auth-dir ~/.sccache-auth
+
+  Or set environment variable:
+    export SCCACHE_AUTH_DIR=~/.sccache-auth
+    ./build_velox.sh --sccache


Nit: Is this duplicating information in the README?

paul-aiyedun · 2025-09-23T19:15:48Z

velox/scripts/build_velox.sh

+  $(basename "$0") --sccache --sccache-github-token ghp_xxx  # Full distributed compilation
+  $(basename "$0") --sccache --sccache-show-stats            # Build with sccache and show stats


Are --sccache-github-token and --sccache-show-stats supported options?

Sorry these were vestigial option I had at one point and forgot to remove from docs.

paul-aiyedun · 2025-09-23T19:18:12Z

velox/scripts/build_velox.sh

+# Add sccache build arguments
+if [[ "$ENABLE_SCCACHE" == true ]]; then
+  DOCKER_BUILD_OPTS+=(--build-arg ENABLE_SCCACHE="ON")
+  # Copy auth files to build context


Should this be cleaned up later?

Sure, I put in a trap.

paul-aiyedun · 2025-09-23T19:33:56Z

velox/docker/sccache/sccache_setup.sh

+mkdir -p ~/.config/sccache ~/.aws
+
+# Install AWS credentials
+cp /sccache_auth/aws_credentials ~/.aws/credentials


Will this override existing credential setup?

Yes, there is a warning put in place to notify the user that this will happen here: https://github.com/rapidsai/velox-testing/pull/52/files/2cb983596eb88b719f86819ee4e6b88fc55dd1f2#diff-d2d6851dcf7cc303d86898bf66d55ab402ed49efa26304f2940a8d1218519026R28-R34

This also only affects the ~/.aws/credentials in the docker container that is used to build velox, it does not affect the host.

paul-aiyedun · 2025-09-23T19:42:08Z

velox/docker/sccache/sccache_setup.sh

+else
+    echo "Error: Distributed compilation not available, check connectivity"
+    exit 1
+fi 


Nit: Add newline to the end of file.

paul-aiyedun · 2025-09-23T19:50:14Z

velox/scripts/setup_sccache_auth.sh

+echo "   ./build_velox.sh --sccache"
+echo
+
+echo -e "${GREEN}Setup complete!${NC}" 


Nit: Add newline to the end of file.

* Added missing dockerfile which was omitted in first commit

paul-aiyedun · 2025-09-24T18:46:02Z

velox/docker/sccache/sccache_setup.sh

+echo "AWS credentials file preview:"
+head -3 ~/.aws/credentials


Should we be printing credentials?

Similarly, should this script be enabling tracing (i.e. set -x) by default?

Hmm I'm not sure why this is showing in GH, this section was deleted a while ago, please double check.

I've disabled set -x for this script now.

paul-aiyedun · 2025-09-24T18:55:56Z

velox/scripts/build_velox.sh

  -j|--num-threads            NUM Number of threads to use for building (default: 3/4 of CPU cores).
  --benchmarks true|false     Enable benchmarks and nsys profiling tools (default: true).
+  --sccache                   Enable sccache distributed compilation caching.
+  --sccache-auth-dir DIR      Directory containing sccache authentication files (github_token, aws_credentials).


Does this have to be an argument given that this directory is created by the setup script?

I can remove this option and we can just rely on the relevant environment variable.

paul-aiyedun · 2025-09-24T18:58:33Z

README.md

+
+Then build Velox with sccache enabled:
+```bash
+./build_velox.sh --sccache --sccache-auth-dir ~/.sccache-auth


Running this results in the following error:

=> [stage-0 9/9] RUN --mount=type=bind,source=velox,target=/workspace/velox,ro set -euxo pipefail && if [ "ON" = "ON" ]; then 542.7s => => # ed-qualifiers -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-comment -Wno-int-in-bool-context => => # -Wno-redundant-move -Wno-array-bounds -Wno-maybe-uninitialized -Wno-unused-result -Wno-for => => # mat-overflow -Wno-strict-aliasing -Wno-restrict -Werror -O3 -DNDEBUG -std=gnu++20 -fPIC -fdiagnostics-color=always -ffp-contr => => # act=off -fPIC -MD -MT velox/buffer/CMakeFiles/velox.dir/__/dwio/dwrf/writer/ColumnWriter.cpp.o -MF velox/buffer/CMakeFiles/velox.dir/_ => => # _/dwio/dwrf/writer/ColumnWriter.cpp.o.d -o velox/buffer/CMakeFiles/velox.dir/__/dwio/dwrf/writer/ColumnWriter.cpp.o -c /workspace/velo => => # x/velox/dwio/dwrf/writer/ColumnWriter.cpp failed to execute bake: exit status 1

This looks like a compilation error unrelated to this PR, can you post more of the logs to confirm?

Build logs have been attached. This is using this velox commit: facebookincubator/velox@8875b4d. Note that the build succeeds when sccache is not used.

./build_velox.sh --no-cache --log velox_build.log
velox_build.log

./build_velox.sh --no-cache --sccache --sccache-auth-dir ~/.sccache-auth --log velox_build_sccache.log
velox_build_sccache.log

@paul-aiyedun Hmm, I've been able to build that commit (facebookincubator/velox@8875b4d) using the command

./build_velox.sh --no-cache --sccache --sccache-auth-dir ~/.sccache-auth --log velox_build_sccache.log successfully.

Build log:
velox_build_sccache.log

Could you please retry with a clean checkout/env to further debug this?

After several attempts I was able to reproduce the issue above, and it looks like when distributed compilation is enabled with sccache it can compile with warnings that otherwise would not be hit. This causes the build to fail due to treating warnings as errors.

To address this, I've made distributed compilation an opt in flag, and provide a warning that observable behaviour of the compilers may differ if enabled (resulting in failed compilation.)

Do not enable tracing (`set -x`).

…sting into mattgara/sccache-velox

Avinash-Raj · 2025-09-26T08:55:32Z

velox/scripts/build_velox.sh

+          exit 1
+        fi
+        SCCACHE_AUTH_DIR="$2"
+        shift 2


Oops this looks like a merge error by the automated merge in Cursor/VS Code.

Add missing ;; lost in automated merge of conflict.

paul-aiyedun · 2025-10-01T17:30:34Z

velox/docker/sccache_auth.dockerfile

Is this dockerfile based on an existing dockerfile or documentation?

This dockerfile is largely based on documentation in a slack channel which I can link offline.

paul-aiyedun · 2025-10-01T17:37:26Z

velox/scripts/build_velox.sh

+# Cleanup function to remove copied sccache auth files
+cleanup_sccache_auth() {
+    if [[ "$ENABLE_SCCACHE" == true && -d "../docker/sccache/sccache_auth/" ]]; then
+        rm -f ../docker/sccache/sccache_auth/github_token ../docker/sccache/sccache_auth/aws_credentials


Nit: Delete the ../docker/sccache/sccache_auth directory?

* Add flag to enable it. * PR revision

* PR revision

paul-aiyedun

Looks good to me.

Avinash-Raj · 2025-10-22T10:50:03Z

velox/scripts/build_velox.sh

 TREAT_WARNINGS_AS_ERRORS="${TREAT_WARNINGS_AS_ERRORS:-1}"
 LOGFILE="./build_velox.log"
+ENABLE_SCCACHE=false
+SCCACHE_AUTH_DIR="$HOME/.sccache-auth"


Make this to accept value from SCCACHE_AUTH_DIR env var or fallback to default "$HOME/.sccache-auth"

Good point, thanks for the catch.

Avinash-Raj · 2025-10-22T11:24:58Z

.github/workflows/velox-test.yml

+          cat > $HOME/.sccache-auth/aws_credentials << EOF
+          [default]
+          aws_access_key_id = $SCCACHE_AWS_ACCESS_KEY_ID
+          aws_secret_access_key = $SCCACHE_AWS_SECRET_ACCESS_KEY


Uses the default AWS credentials available through GHA secrets instead of generating them every time. The same applies for the GitHub token.

I don't think we want to make these changes in this PR, enabling sccache for CI is a separate work item which will have a follow-up PR.

Avinash-Raj · 2025-10-22T11:43:52Z

Relevant CI run., can we fix the CI before merging?

mattgara · 2025-10-22T17:26:41Z

Relevant CI run., can we fix the CI before merging?

See above. In summary, I think this is a separate work item that should not block this PR.

This reverts commit 700d2ee.

This reverts commit 7cdbd64.

sccache: Integrate with Velox build

02c005f

* Add `sccache` integration with `build_velox.sh` * Document how to setup and use `sccache`

mattgara requested review from Avinash-Raj, misiugodfrey and paul-aiyedun September 19, 2025 22:11

Create dir if missing

398dae0

Avinash-Raj requested a review from simoneves September 22, 2025 09:53

mattgara changed the title ~~sccache: Integrate with Velox build~~ sccache: integrate with Velox build Sep 22, 2025

Update build_velox.sh

2cb9835

Remove non-existent temporary directory removal.

paul-aiyedun reviewed Sep 23, 2025

View reviewed changes

PR revision

3d3c7a7

* Added missing dockerfile which was omitted in first commit

mattgara requested a review from paul-aiyedun September 23, 2025 20:22

Add line about this requiring access to GH NVIDIA organization

0c690f5

paul-aiyedun reviewed Sep 24, 2025

View reviewed changes

mattgara added 4 commits September 24, 2025 15:11

Update sccache_setup.sh

f412028

Do not enable tracing (`set -x`).

Merge remote-tracking branch 'origin/main' into mattgara/sccache-velox

eadaefa

Merge branch 'mattgara/sccache-velox' of github.com:rapidsai/velox-te…

4350283

…sting into mattgara/sccache-velox

Fixup minor syntax issue

188fdaf

Avinash-Raj reviewed Sep 26, 2025

View reviewed changes

Update build_velox.sh

f7eb417

Add missing ;; lost in automated merge of conflict.

paul-aiyedun reviewed Oct 1, 2025

View reviewed changes

mattgara force-pushed the mattgara/sccache-velox branch from 082ca6e to f7eb417 Compare October 9, 2025 22:02

mattgara added 4 commits October 10, 2025 14:18

Turn off distributed compilation by default

a458d0d

* Add flag to enable it. * PR revision

README update

48d0224

* PR revision

Keep consistent default auth dir

5a27fe2

Go back to overriding ~/.aws/credentials

251e566

paul-aiyedun approved these changes Oct 10, 2025

View reviewed changes

Merge branch 'main' into mattgara/sccache-velox

3df2a85

Avinash-Raj reviewed Oct 22, 2025

View reviewed changes

Added setup sccache step on Velox Test CI Worflow

7cdbd64

Avinash-Raj reviewed Oct 22, 2025

View reviewed changes

fixed incorrect env var name on CI

700d2ee

mattgara added 3 commits October 22, 2025 12:30

Revert "fixed incorrect env var name on CI"

cf288c4

This reverts commit 700d2ee.

Revert "Added setup sccache step on Velox Test CI Worflow"

9a74c25

This reverts commit 7cdbd64.

Use env variable for SCCACHE_AUTH_DIR if set

165ad9e

mattgara merged commit 8d52c99 into main Oct 22, 2025

mattgara deleted the mattgara/sccache-velox branch October 22, 2025 19:35

simoneves mentioned this pull request Oct 22, 2025

Simplify Velox build changes #90

Merged

		$(basename "$0") --sccache --sccache-github-token ghp_xxx # Full distributed compilation
		$(basename "$0") --sccache --sccache-show-stats # Build with sccache and show stats

		echo "AWS credentials file preview:"
		head -3 ~/.aws/credentials

Conversation

mattgara commented Sep 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattgara Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattgara Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattgara Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paul-aiyedun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Avinash-Raj commented Oct 22, 2025

Uh oh!

mattgara commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

mattgara Sep 23, 2025 •

edited

Loading

mattgara Sep 24, 2025 •

edited

Loading

mattgara Oct 7, 2025 •

edited

Loading