Skip to content

Cache registries per configuration file#3826

Merged
suzuki-shunsuke merged 12 commits into
mainfrom
perf-cache-registry
May 2, 2025
Merged

Cache registries per configuration file#3826
suzuki-shunsuke merged 12 commits into
mainfrom
perf-cache-registry

Conversation

@suzuki-shunsuke

@suzuki-shunsuke suzuki-shunsuke commented Apr 29, 2025

Copy link
Copy Markdown
Member

This pull request improves the performance of aqua which command.

aqua which read registries, but the standard registry is very large (about 2MB) so it makes the performance worse.
Furthermore, the standard registry gets larger every time we improve the standard registry.

To solve the problem, this pull request introduces the cache mechanism of packages by aqua.yaml.
aqua creates a cache file per aqua.yaml into $(aqua root-dir)/registry-cache/<base64 encoded aqua.yaml absolute path>.json.

The structure of cache files is:

{
  "absolute path of registry.yaml": {
    "package name": {
      "type": "github_release",
      // ...
    }
  }
}

If a registry type is local, a cache file isn't created.
Cache files are much smaller than the standard registry, so aqua can read them much faster.
If packages aren't found in cache files, aqua reads the original registries and updates cache files.
aqua removes unused records from cache.

Bench mark

$ hyperfine --warmup 3 "/Users/shunsukesuzuki/go/bin/aqua which golangci-lint" "/Users/shunsukesuzuki/.local/share/aquaproj-aqua/internal/pkgs/github_release/github.com/aquaproj/aqua/v2.50.1-1/aqua_darwin_arm64.tar.gz/aqua which golangci-lint" "/Users/shunsukesuzuki/.local/share/aquaproj-aqua/internal/pkgs/github_release/github.com/aquaproj/aqua/v2.50.0/aqua_darwin_arm64.tar.gz/aqua which golangci-lint"
Benchmark 1: /Users/shunsukesuzuki/go/bin/aqua which golangci-lint
  Time (mean ± σ):      25.0 ms ±   3.5 ms    [User: 21.4 ms, System: 3.7 ms]
  Range (min … max):    21.6 ms …  45.4 ms    77 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark 2: /Users/shunsukesuzuki/.local/share/aquaproj-aqua/internal/pkgs/github_release/github.com/aquaproj/aqua/v2.50.1-1/aqua_darwin_arm64.tar.gz/aqua which golangci-lint
  Time (mean ± σ):      41.7 ms ±   4.5 ms    [User: 41.1 ms, System: 5.3 ms]
  Range (min … max):    37.2 ms …  62.3 ms    55 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark 3: /Users/shunsukesuzuki/.local/share/aquaproj-aqua/internal/pkgs/github_release/github.com/aquaproj/aqua/v2.50.0/aqua_darwin_arm64.tar.gz/aqua which golangci-lint
  Time (mean ± σ):      42.8 ms ±   4.0 ms    [User: 41.9 ms, System: 5.6 ms]
  Range (min … max):    39.4 ms …  55.4 ms    43 runs
 
  Warning: The first benchmarking run for this command was significantly slower than the rest (54.4 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You are already using the '--warmup' option which helps to fill these caches before the actual benchmark. You can either try to increase the warmup count further or re-run this benchmark on a quiet system in case it was a random outlier. Alternatively, consider using the '--prepare' option to clear the caches before each timing run.
 
Summary
  /Users/shunsukesuzuki/go/bin/aqua which golangci-lint ran
    1.67 ± 0.29 times faster than /Users/shunsukesuzuki/.local/share/aquaproj-aqua/internal/pkgs/github_release/github.com/aquaproj/aqua/v2.50.1-1/aqua_darwin_arm64.tar.gz/aqua which golangci-lint
    1.71 ± 0.29 times faster than /Users/shunsukesuzuki/.local/share/aquaproj-aqua/internal/pkgs/github_release/github.com/aquaproj/aqua/v2.50.0/aqua_darwin_arm64.tar.gz/aqua which golangci-lint

@suzuki-shunsuke suzuki-shunsuke changed the title Perf cache registry Cache registries per configuration file Apr 30, 2025
@suzuki-shunsuke suzuki-shunsuke marked this pull request as ready for review May 2, 2025 22:53
@suzuki-shunsuke suzuki-shunsuke merged commit 3714ee8 into main May 2, 2025
25 checks passed
@suzuki-shunsuke suzuki-shunsuke deleted the perf-cache-registry branch May 2, 2025 22:54
@github-project-automation github-project-automation Bot moved this to Done in main May 2, 2025
@suzuki-shunsuke suzuki-shunsuke added this to the v2.50.1 milestone May 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant