Skip to content

fix(ci): free more disk space and add nix fallback for cache mismatches #1929

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shunkakinoki merged 1 commit into from
Jun 19, 2026
Merged

fix(ci): free more disk space and add nix fallback for cache mismatches #1929

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .github/workflows/docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,12 @@ jobs:
if: runner.os == 'Linux'
uses: jlumbroso/free-disk-space@v1.3.1
with:
tool-cache: false
tool-cache: true
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true

@indent-zero indent-zero Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing swap increases OOM risk under the new fallback behavior: swap-storage: true deletes the runner's swap file. Combined with the new fallback = true (in e2e.yml extra_nix_config, Dockerfile nix.conf, and --fallback in the Makefile), heavyweight derivations that would previously be fetched from a substituter may now build from source and spike RSS. Without swap, a memory spike turns directly into an OOM kill — especially on the arm64 builder (ubuntu-24.04-arm) which has less headroom than ubuntu-latest. Note this matches the previous default (swap-storage defaults to true in this action), so the explicit value isn't a behavior change on its own — it's only worth surfacing because the fallback change makes from-source builds materially more common.

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Login to GHCR
Expand Down
8 changes: 7 additions & 1 deletion .github/workflows/e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,12 @@ jobs:
if: runner.os == 'Linux'
uses: jlumbroso/free-disk-space@v1.3.1
with:
tool-cache: false
tool-cache: true
android: true
dotnet: true
haskell: true
large-packages: true
swap-storage: true
- name: KVM Linux Virtualization
if: runner.os == 'Linux'
run: |
Expand All @@ -50,6 +55,7 @@ jobs:
system-features = kvm
max-jobs = auto
cores = 0
fallback = true
- name: Prepare System Files
if: runner.os == 'macOS'
run: |
Expand Down
1 change: 1 addition & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
ARG USER_UID=1001
ARG USER_GID=$USER_UID
ARG COMMIT_SHA=main
ARG GITHUB_TOKEN

Check warning on line 70 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-build-push (linux/amd64, amd64, ubuntu-latest)

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "GITHUB_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 70 in Dockerfile

View workflow job for this annotation

GitHub Actions / docker-build-push (linux/arm64, arm64, ubuntu-24.04-arm)

Sensitive data should not be used in the ARG or ENV commands 

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "GITHUB_TOKEN")
More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
ARG GITHUB_PR
ENV GITHUB_PR=${GITHUB_PR}

Expand Down Expand Up @@ -102,6 +102,7 @@
echo "experimental-features = nix-command flakes" >> /etc/nix/nix.conf && \
echo "filter-syscalls = false" >> /etc/nix/nix.conf && \
echo "sandbox = true" >> /etc/nix/nix.conf && \
echo "fallback = true" >> /etc/nix/nix.conf && \

@indent-zero indent-zero Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image-level fallback affects downstream consumers: Adding fallback = true to /etc/nix/nix.conf persists into the published ghcr.io/.../dotfiles image, so any user who runs nix build/nix run inside the image inherits this behavior. Where they previously got an immediate error on a substituter miss (the upstream default), they will now silently build from source — potentially long jobs for things like Rust toolchains or Chromium. If the intent is only to harden CI for this repo, consider scoping the change to Makefile + the e2e workflow and leaving the image's nix.conf at defaults; if the broader behavior is intended, calling it out in the image docs would save downstream debugging.

if [ -n "$GITHUB_TOKEN" ]; then \
echo "access-tokens = github.com=$GITHUB_TOKEN" >> /etc/nix/nix.conf ; \
fi
Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ NIX_USERNAME := $(shell \
echo "$(shell whoami)"; \
fi)
NIX_ENV := $(shell . ~/.nix-profile/etc/profile.d/nix.sh 2>/dev/null || . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh 2>/dev/null || command -v nix >/dev/null 2>&1 || echo "not_found")
NIX_FLAGS := --extra-experimental-features 'flakes nix-command' --no-pure-eval --impure
NIX_FLAGS := --extra-experimental-features 'flakes nix-command' --no-pure-eval --impure --fallback

@gemini-code-assist gemini-code-assist Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Adding --fallback globally to NIX_FLAGS affects local development environments. If a binary cache is temporarily unreachable or returns a hash mismatch locally, Nix will silently fall back to building packages from source. For large packages, this can cause extremely long build times and high resource consumption on developer machines.

It is safer to enable --fallback only in CI or Docker environments where automated builds need to be resilient to transient cache issues.

NIX_FLAGS := --extra-experimental-features 'flakes nix-command' --no-pure-eval --impure
ifneq ($(filter true,$(CI) $(IN_DOCKER)),)
NIX_FLAGS += --fallback
endif

@indent-zero indent-zero Bot Jun 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--fallback is now applied to local dev runs too: Unlike the workflow- and image-scoped changes, this flag is global to every $(NIX_FLAGS) invocation, including a developer running make on their laptop. If a substituter is temporarily unreachable or returns a hash mismatch, Nix will silently start a from-source build (e.g., rustc, GHC, Chromium) rather than erroring — producing long, surprising builds. The conditional substituter block immediately below (ifeq ($(OS),Darwin) / else ifdef CI etc.) already demonstrates the pattern for gating; one option:

NIX_FLAGS := --extra-experimental-features 'flakes nix-command' --no-pure-eval --impure
ifneq ($(filter true,$(CI) $(IN_DOCKER)),)
NIX_FLAGS += --fallback
endif

This keeps the CI-resilience benefit while preserving fast-failure semantics locally.

# Only add cache options when user is trusted or on Darwin/CI (avoids "ignoring untrusted substituter" warnings)
ifeq ($(OS),Darwin)
NIX_FLAGS += --option substituters "$(NIX_SUBSTITUTERS)" --option trusted-public-keys "$(NIX_TRUSTED_KEYS)"
Expand Down
Loading