Skip to content

ci : make cache bucket public #27728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
CISC merged 2 commits into from
Aug 26, 2026
Merged

ci : make cache bucket public #27728

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c82532f
check for hf token
CISC Aug 26, 2026
a7620db
make bucket public
CISC Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 27 additions & 23 deletions .github/actions/ccache-buckets/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,34 +58,38 @@ runs:
if: ${{ inputs.save == 'true' }}
shell: bash
run: |
set +e -uo pipefail
source .venv-hf/bin/activate
CCACHE_DIR=$(ccache -k cache_dir)
if [[ -d "$CCACHE_DIR" ]]; then
ccache -s
if [[ -n "${{ inputs.evict-old-files }}" ]]; then
ccache --evict-older-than "${{ inputs.evict-old-files }}"
fi
DATESTAMP=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
CACHEFILE="${{ inputs.key }}-$DATESTAMP.tar.gz"
if tar -czf ccache_bucket.tar.gz -C "$CCACHE_DIR" .; then
hf buckets cp ccache_bucket.tar.gz "hf://buckets/${{ inputs.hf_bucket }}/${{ inputs.folder }}/$CACHEFILE"
if [[ -n "$HF_TOKEN" ]]; then
set +e -uo pipefail
source .venv-hf/bin/activate
CCACHE_DIR=$(ccache -k cache_dir)
if [[ -d "$CCACHE_DIR" ]]; then
ccache -s
if [[ -n "${{ inputs.evict-old-files }}" ]]; then
ccache --evict-older-than "${{ inputs.evict-old-files }}"
fi
DATESTAMP=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
CACHEFILE="${{ inputs.key }}-$DATESTAMP.tar.gz"
if tar -czf ccache_bucket.tar.gz -C "$CCACHE_DIR" .; then
hf buckets cp ccache_bucket.tar.gz "hf://buckets/${{ inputs.hf_bucket }}/${{ inputs.folder }}/$CACHEFILE"
fi
rm ccache_bucket.tar.gz
else
echo "'$CCACHE_DIR' not found."
fi
rm ccache_bucket.tar.gz
else
echo "'$CCACHE_DIR' not found."
fi

- name: Remove old ccache files from buckets
if: ${{ inputs.save == 'true' }}
shell: bash
run: |
set +e -uo pipefail
source .venv-hf/bin/activate
CACHE_FILES=$(hf buckets list "hf://buckets/${{ inputs.hf_bucket }}/${{ inputs.folder }}" --json | jq -r '[.[] | select(.type == "file") | select((.uploaded_at | .[:19]+"Z" | fromdateiso8601) < (now - 5 * 60)) | select(.path | startswith("${{ inputs.folder }}/${{ inputs.key }}") and endswith(".tar.gz"))] | sort_by(.path)[:-1] | .[] | [.path // ""] | @tsv')
if [[ -n "$CACHE_FILES" ]]; then
echo "Removing old ccache files..."
while IFS=$'\t' read -r CACHE_PATH; do
hf buckets rm "hf://buckets/${{ inputs.hf_bucket }}/$CACHE_PATH" -y
done <<< "$CACHE_FILES"
if [[ -n "$HF_TOKEN" ]]; then
set +e -uo pipefail
source .venv-hf/bin/activate
CACHE_FILES=$(hf buckets list "hf://buckets/${{ inputs.hf_bucket }}/${{ inputs.folder }}" --json | jq -r '[.[] | select(.type == "file") | select((.uploaded_at | .[:19]+"Z" | fromdateiso8601) < (now - 5 * 60)) | select(.path | startswith("${{ inputs.folder }}/${{ inputs.key }}") and endswith(".tar.gz"))] | sort_by(.path)[:-1] | .[] | [.path // ""] | @tsv')
if [[ -n "$CACHE_FILES" ]]; then
echo "Removing old ccache files..."
while IFS=$'\t' read -r CACHE_PATH; do
hf buckets rm "hf://buckets/${{ inputs.hf_bucket }}/$CACHE_PATH" -y
done <<< "$CACHE_FILES"
fi
fi
12 changes: 6 additions & 6 deletions .github/workflows/build-cuda-ubuntu.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ jobs:
with:
key: cuda-ubuntu-24.04-cuda
folder: llama.cpp
hf_bucket: ${{ vars.HF_BUCKET_CACHE_OUTPUT }}
hf_bucket: ggml-org/cache

- name: Build with CMake
# TODO: Remove GGML_CUDA_CUB_3DOT2 flag once CCCL 3.2 is bundled within CTK and that CTK version is used in this project
Expand All @@ -89,7 +89,7 @@ jobs:
key: cuda-ubuntu-24.04-cuda
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ${{ vars.HF_BUCKET_CACHE_OUTPUT }}
hf_bucket: ggml-org/cache
save: true

hip:
Expand Down Expand Up @@ -120,7 +120,7 @@ jobs:
with:
key: cuda-ubuntu-22.04-hip
folder: llama.cpp
hf_bucket: ${{ vars.HF_BUCKET_CACHE_OUTPUT }}
hf_bucket: ggml-org/cache

- name: Build with native CMake HIP support
id: cmake_build
Expand All @@ -140,7 +140,7 @@ jobs:
key: cuda-ubuntu-22.04-hip
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ${{ vars.HF_BUCKET_CACHE_OUTPUT }}
hf_bucket: ggml-org/cache
save: true

musa:
Expand Down Expand Up @@ -171,7 +171,7 @@ jobs:
with:
key: cuda-ubuntu-22.04-musa
folder: llama.cpp
hf_bucket: ${{ vars.HF_BUCKET_CACHE_OUTPUT }}
hf_bucket: ggml-org/cache

- name: Build with native CMake MUSA support
id: cmake_build
Expand All @@ -189,5 +189,5 @@ jobs:
key: cuda-ubuntu-22.04-musa
folder: llama.cpp
evict-old-files: 1d
hf_bucket: ${{ vars.HF_BUCKET_CACHE_OUTPUT }}
hf_bucket: ggml-org/cache
save: true