ci : refactor

[ggerganov]

Overview

• Split CUDA workflows in 2: build-cuda-ubuntu.yml and build-cuda-windows.yml
• build-cuda-windows.yml is now manual
• The entire release.yml workflow is now executed sequentially in a queue. This allows to have non-timestamped caches for the release jobs, which should reduce the cache usage of the repo
• Make ccache names more consistent by prefixing with the workflow name
• Disable android ccache - does not seem to improve build time and at the same time takes a lot of space

Additional information

The main goal of this change is to avoid running the heavy CUDA Windows jobs too often. To do that, PRs now will run only the CUDA Ubuntu jobs (includes hip and musa) in build-cuda-ubuntu.yml automatically. The CUDA Windows jobs can only be started manually for a PR with the new build-cuda-windows.yml workflow.

A problem on master was that both the build.yml and release.yml workflows would start the same CUDA Windows jobs. When we miss a hit for the ccache, this results in 2 parallel jobs that run for 3 hours and produce the same ccache twice. With multiple commits to master, this can scale even more. To fix that, all release workflows now run sequentially in a queue:

# note: run this workflow one at a time for better cache reuse
concurrency:
  group: release
  queue: max

I think this should result in better utilization of the runners and the ccache.

The release jobs also have a new ccache policy:

      - name: ccache
        uses: ggml-org/ccache-action@v1.2.21
        with:
          key: release-${{ matrix.os }}-${{ matrix.arch }}
          append-timestamp: false # note: use this only with non-concurrent jobs!

This prevents this from happening:

Now we will have a single cache entry per release job (note there is not timestamp suffix):

We can do that because the jobs do not run concurrently.

Next PRs

• Deduplicate the jobs in build-cuda-windows.yml and release.yml

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[ggerganov]

There is non-negligible chance that this breaks something - will monitor closely.

[ggerganov]

A positive effect of the change is that we can now cancel intermediate releases when it makes sense. For example, at the moment we have:

So the 3111 release is ongoing and the next 3 releases are waiting for it to finish. We can cancel 3112 and 3113. This way we jump directly to 3114 saving some CI runs.

This is the release queue: https://github.com/ggml-org/llama.cpp/actions/workflows/release.yml

[CISC]

The release jobs also have a new ccache policy:

      - name: ccache
        uses: ggml-org/ccache-action@v1.2.21
        with:
          key: release-${{ matrix.os }}-${{ matrix.arch }}
          append-timestamp: false # note: use this only with non-concurrent jobs!

This doesn't work as intended, it causes the ccache to never update, leaving outdated caches with currently only 4% hits:

Failed to save: Unable to reserve cache with key ccache-release-windows-2022-x64-cuda-13.3-, another job may be creating this cache.

[CISC]

I've deleted the ccaches without timestamp now so the next 6+ releases should be swifter.

[ggerganov]

Prepared a fix here: #23895