In CI skip installing into local repository

[nineinchnick]

Description

I noticed the size of the cache package with the local Maven repo is about 7.5GB, near the 10GB limit. I think installing Trino packages into the local repository is not necessary, and might both save some time on copying and reduce the size of the cache. Downloading and extracting such a big cache takes about 1.5 minutes.

Is this change a fix, improvement, new feature, refactoring, or other?
improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)
ci

How would you describe this change to a non-technical end user or system administrator?
n/a

Related issues, pull requests, and links

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

(x) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

[nineinchnick]

@hashhar this is now ready for review. I changed install to package in a few places where we're only supposed to build the project and we run mvn test right after that. I changed it to verify in the Maven Checks job and I also added changes in the error-prone-checks job that @findepi suggested in #13257

I checked that with these changes the size of the local Maven repo is ~1GB and shouldn't grow much. Currently, it's ~7GB which is not sustainable.

[findepi]

I also added changes in the error-prone-checks job that @findepi suggested in #13257

Do you want me to merge or close that PR?

[nineinchnick]

Do you want me to merge or close that PR?

You can close it and merge this one.

[findepi]

Thanks. just FYI, it's red now.

[nineinchnick]

I made some wrong assumptions before and had to reevaluate my approach here. When we run mvn test there have to be artifacts from other modules in the local repository, because the tested module has to be built. If I skip the build phase, it tries to run the artifact from the local repository, not the build directory.

I reverted replacing install with package in most jobs, except for maven-checks, which should be the only job where the cache can be saved. I added clearing the local repo in other jobs to ensure that. I didn't find any different way to prevent storing the cache.

[nineinchnick]

@hashhar @findepi this is ready to go

[findepi]

I don't know how our CI caching works. @hashhar can you PTAL?

[hashhar]

How much time does our cache save us. If it's not a very large difference I'd probably opt to drop the cache instead.

[hashhar]

This is the non-controversial part and we can merge it right away if extracted.

[nineinchnick]

How much time does our cache save us. If it's not a very large difference I'd probably opt to drop the cache instead.

Based on highly scientific tests, that is a single run of this branch

with cache:

without cache:

Note that this PR attempts to fix the cache to be under 1gb, instead of the 7gb we have now. Downloading that 7gb cache takes 2-5 minutes. I hope to get this under 30 seconds. Then it'll be beneficial, since it saves 50 seconds.

Note that not installing artifacts in a local maven repo should not be related to caching it or not. If we don't have to copy files around, we shouldn't.

[nineinchnick]

Oh and I think using the cache should also increase resiliency against Maven Central being flaky sometimes. But I don't have data about how often this happens.

[hashhar]

I still see large cache being used - https://github.com/trinodb/trino/runs/7672132855?check_suite_focus=true#step:4:81

Cache Size: ~6914 MB (7249763541 B)
/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/10163cbd-356c-43ed-98d8-a3350f756b3e/cache.tzst -P -C /home/runner/work/trino/trino
Cache restored successfully

[nineinchnick]

I added a temporary commit with a change in the main pom to get a new cache key. When the run completes I'll trigger another one and that will demonstrate the changes.

[nineinchnick]

I give up, I don't know what criteria Github uses when restoring the cache. It's supposed to check for a hash of all pom files, but I modified it and it still uses the cache from the previous workflow run:

Cache restored from key: setup-java-Linux-maven-0a53bc970763b2f9f964aba9ebd2fde72497ff9e8e884aac25d25409d362496b

where I computed the hash and its:

d14eb4b7c95b06aa930bc26818f8be71fb1aba18c6c758adbc7894f6c544a349

Maybe it's trying to be smart and getting some cache when there's no hit. Anyway, I used this action to verify the hash:

      - uses: nineinchnick/github-script@0a9984563e6199a44285d325a9921f1bff9517b2
        with:
          script: |
            const hash = await glob.hashFiles("**/pom.xml", {followSymbolicLinks: false}, true)
            core.info(hash)

[findepi]

Add a code comment?

The commit changes some install into package, so worth noting why it's still worthwhile to delete ~/.m2/repository.

Also, why ~/.m2/repository and not eg ~/.m2/repository/io/trino ?

[nineinchnick]

This removes the whole local repo to avoid creating a cache entry from this job since it might not be the one with the most dependencies.

[findepi]

Is it worth a code comment?

[nineinchnick]

#13796