CircleCI caching doesn't like wildcards in filenames/directory names

[tlaurion]

Resulting in partial caches and limited gain in reusing those caches.

Let's remember that currently, 3 caches are created/reused by CirlceCI if hashes of modules+patches are matching.

• Full cache (works. But any change in any patches will not reuse that cache and reuse less complete cache below)
• Coreboot (each version crossgcc prebuilt environement) + Heads musl-cross-make. (If coreboot changed, use less complete cache below)
• Heads musl-cross-make (if musl-cross-make module or patches changed, rebuilt everything from scratch)

The cache are kept to be reused by CircleCI for 30 days. So this caching system is useful for projects with fast paces commits and merges. Locally for developers and when pushed upstream in project to reuse past existing caches to gain compilation time when building all CircleCI defines boards.

Heads musl-cross-make (cache name: heads-musl-cross) doesn't cache built musl-cross-VERSION, only installed result into crossgcc directory:

Creating cache archive...
Warning: could not archive /root/project/build/musl-cross-* - Not found
Uploading cache archive...
Stored Cache to heads-musl-cross-ECVtMDg0M5AAlQlMGT+8nQCEdK8++89gnFIUMXqmg9E=test
  * /root/project/crossgcc
  * /root/project/build/musl-cross-*
Total size uploaded: 666 MiB

Coreboot cache + Heads musl-cross-make (cache name: heads-coreboot-musl-cross) is caching only installed musl-cross-make install in crossgcc directory:

Creating cache archive...
Warning: could not archive /root/project/build/coreboot-* - Not found
Warning: could not archive /root/project/build/musl-cross-* - Not found
Uploading cache archive...
Stored Cache to heads-coreboot-musl-cross-L+USf6UrFvhSoK1Qz61xoU4q3lbmqsTKjRKR9jRatUw=test
  * /root/project/build/coreboot-*
  * /root/project/crossgcc
  * /root/project/build/musl-cross-*
Total size uploaded: 666 MiB

Full cache works (cache name: heads-modules-and-patches):

Creating cache archive...
Uploading cache archive...
Stored Cache to heads-modules-and-patches-QNuALb1_4qZg8Nbbrat2H2Bdf30xopv5NAah0S5QVzg=test
  * /root/project/packages
  * /root/project/crossgcc
  * /root/project/build
  * /root/project/install
Total size uploaded: 4.6 GiB

Solution:

• Programmatically list and store directories in variable and cache them without using wildcards in cache statements.

[tlaurion]

Posted on CircleCI doc: circleci/circleci-docs#5804

[tlaurion]

Programmatically list and store directories in variable and cache them without using wildcards in cache statements.

Which also doesn't work.

[tlaurion]

Pointed the doc issue here and tagged assigned person to look here at circleci/circleci-docs#5804

[tlaurion]

#1015 merged.