Skip to content

Download more dependencies to decrease the chance of network errors #15797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
findepi merged 6 commits into from
Feb 9, 2023
Merged

Download more dependencies to decrease the chance of network errors #15797

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3235ebb
Download dynamic build dependencies with retries
MiguelWeezardo Jan 24, 2023
6abe73e
Enable Maven cache in `build-test-matrix` job of `ci.yml`
MiguelWeezardo Jan 31, 2023
025d13b
Disable revapi error triggered in offline mode
MiguelWeezardo Jan 25, 2023
abeefa1
Add Maven --offline flag as a default option
MiguelWeezardo Jan 20, 2023
84bb376
Remove retries on Maven calls
MiguelWeezardo Feb 2, 2023
8e54b75
Add documentation regarding Maven offline mode
MiguelWeezardo Feb 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/DEVELOPMENT.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ be found in the [docs module](../docs).
* [Code style](#code-style)
* [Additional IDE configuration](#additional-ide-configuration)
* [Building the Web UI](#building-the-web-ui)
* [CI pipeline](#ci-pipeline)

## Commits and pull requests

Expand Down Expand Up @@ -259,3 +260,17 @@ Once it is time to release, the release process is kicked off. A code freeze is
announced on the Trino Slack in the #releases channel, and then a maintainer
utilizes the [release scripts](https://github.com/trinodb/release-scripts) to
update Trino to the next version.

## CI pipeline

### Maven offline mode

The CI pipeline runs most Maven commands in offline mode to avoid dependency on Maven Central.

Before calling Maven a job should execute the `./.github/actions/setup` compound action to
download all the dependencies it will require.

It uses two complementary tools to detect and download Maven dependencies,
but some plugin dependencies are not expressed in the tree hierarchy.

These dynamic dependencies must be listed as `<DynamicDependency>` entries in the root `pom.xml`.
4 changes: 2 additions & 2 deletions .github/actions/compile-commit/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ runs:
# It's important that these values are NOT passed as parameters, because then their values would always be taken from PR HEAD
# -------
# allow overriding Maven command
MAVEN="./mvnw"
MAVEN="./mvnw --offline"
# maven.wagon.rto is in millis, defaults to 30m
MAVEN_INSTALL_OPTS="-Xmx3G -XX:+ExitOnOutOfMemoryError -Dmaven.wagon.rto=60000"
MAVEN_COMPILE_COMMITS="-B --strict-checksums --quiet -T C1 -DskipTests -Dmaven.source.skip=true -Dair.check.skip-all=true -Dmaven.javadoc.skip=true --no-snapshot-updates --no-transfer-progress -pl !:trino-server-rpm"
Expand All @@ -27,7 +27,7 @@ runs:

# For building with Maven we need MAVEN_OPTS to equal MAVEN_INSTALL_OPTS
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
$RETRY $MAVEN dependency:go-offline
./.github/bin/download-maven-dependencies.sh
$MAVEN package \
${MAVEN_COMPILE_COMMITS} `# defaults, kept in sync with ci.yml` \
-Dair.check.skip-all=false -Dair.check.skip-basic=true -Dair.check.skip-extended=true -Dair.check.skip-checkstyle=false \
Expand Down
3 changes: 3 additions & 0 deletions .github/actions/setup/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ runs:
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-
- name: Fetch any missing dependencies
shell: bash
run: ./.github/bin/download-maven-dependencies.sh
- name: Configure Problem Matchers
if: ${{ inputs.java-version != '' }}
shell: bash
Expand Down
21 changes: 21 additions & 0 deletions .github/bin/download-maven-dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/usr/bin/env bash

set -exuo pipefail

RETRY=".github/bin/retry"
MAVEN_ONLINE="${MAVEN//--offline/}"

# Run download tools without any profiles to use active-by-default profiles
$RETRY $MAVEN_ONLINE -B dependency:go-offline
$RETRY $MAVEN_ONLINE -B de.qaware.maven:go-offline-maven-plugin:resolve-dependencies

# Downloading dependencies is used to populate the maven cache shared between PRs, so PR-specific GIB state needs to be ignored

# Enable common profiles to make sure their plugin dependencies are downloaded as well
# GIB should be disabled even though it's profile is active, to make sure it doesn't skip any submodules
$RETRY $MAVEN_ONLINE -B -P ci,errorprone-compiler ${MAVEN_GIB} -Dgib.disable dependency:go-offline
$RETRY $MAVEN_ONLINE -B -P ci,errorprone-compiler ${MAVEN_GIB} -Dgib.disable de.qaware.maven:go-offline-maven-plugin:resolve-dependencies

# TODO: Remove next step once https://github.com/qaware/go-offline-maven-plugin/issues/28 is fixed
# trino-pinot overrides some common dependency versions, focus on it to make sure those overrides are downloaded as well
$RETRY $MAVEN_ONLINE -B -P ci,errorprone-compiler ${MAVEN_GIB} -Dgib.disable de.qaware.maven:go-offline-maven-plugin:resolve-dependencies -pl ':trino-pinot'
24 changes: 9 additions & 15 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,14 @@ env:
# An envar that signals to tests we are executing in the CI environment
CONTINUOUS_INTEGRATION: true
# allow overriding Maven command
MAVEN: ./mvnw
MAVEN: ./mvnw --offline
Comment thread
MiguelWeezardo marked this conversation as resolved.
Outdated
# maven.wagon.rto is in millis, defaults to 30m
MAVEN_OPTS: "-Xmx512M -XX:+ExitOnOutOfMemoryError -Dmaven.wagon.rto=60000"
MAVEN_INSTALL_OPTS: "-Xmx3G -XX:+ExitOnOutOfMemoryError -Dmaven.wagon.rto=60000"
MAVEN_FAST_INSTALL: "-B --strict-checksums -V --quiet -T C1 -DskipTests -Dmaven.source.skip=true -Dair.check.skip-all"
MAVEN_COMPILE_COMMITS: "-B --strict-checksums --quiet -T C1 -DskipTests -Dmaven.source.skip=true -Dair.check.skip-all=true -Dmaven.javadoc.skip=true --no-snapshot-updates --no-transfer-progress -pl '!:trino-server-rpm'"
MAVEN_GIB: "-P gib -Dgib.referenceBranch=refs/remotes/origin/${{ github.event_name == 'pull_request' && github.event.pull_request.base.ref || github.event.repository.default_branch }}"
MAVEN_TEST: "-B --strict-checksums -Dmaven.source.skip=true -Dair.check.skip-all --fail-at-end -P gib -Dgib.referenceBranch=refs/remotes/origin/${{ github.event_name == 'pull_request' && github.event.pull_request.base.ref || github.event.repository.default_branch }}"
RETRY: .github/bin/retry
# Testcontainers kills image pulls if they don't make progress for > 30s and retries for 2m before failing. This means
# that if an image doesn't download all it's layers within ~2m then any other concurrent pull will be killed because
# the Docker daemon only downloads 3 layers concurrently which prevents the other pull from making any progress.
Expand Down Expand Up @@ -69,9 +68,6 @@ jobs:
- uses: ./.github/actions/setup
with:
java-version: ${{ matrix.java-version }}
- name: Download all Trino dependencies to make sure cache is complete
run: |
$RETRY $MAVEN dependency:go-offline
- name: Maven Checks
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
Expand Down Expand Up @@ -160,12 +156,12 @@ jobs:
- name: Maven Package
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
$RETRY $MAVEN clean package ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -pl '!:trino-docs,!:trino-server,!:trino-server-rpm'
$MAVEN clean package ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -pl '!:trino-docs,!:trino-server,!:trino-server-rpm'
- name: Error Prone Checks
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
# Run Error Prone on one module with a retry to ensure all runtime dependencies are fetched
$RETRY $MAVEN ${MAVEN_TEST} -T C1 clean verify -DskipTests -P gib,errorprone-compiler -am -pl ':trino-spi'
$MAVEN ${MAVEN_TEST} -T C1 clean verify -DskipTests -P gib,errorprone-compiler -am -pl ':trino-spi'
# The main Error Prone run
$MAVEN ${MAVEN_TEST} -T C1 clean verify -DskipTests -P gib,errorprone-compiler \
-pl '!:trino-docs,!:trino-server,!:trino-server-rpm'
Expand Down Expand Up @@ -205,7 +201,7 @@ jobs:
- name: Maven Install
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
$RETRY $MAVEN clean install ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -Dgib.logImpactedTo=gib-impacted.log -pl '!:trino-docs,!:trino-server,!:trino-server-rpm'
$MAVEN clean install ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -Dgib.logImpactedTo=gib-impacted.log -pl '!:trino-docs,!:trino-server,!:trino-server-rpm'
- name: Test old JDBC vs current server
run: |
if [ ! -f gib-impacted.log ] || grep -q testing/trino-test-jdbc-compatibility-old-driver gib-impacted.log; then
Expand Down Expand Up @@ -274,7 +270,7 @@ jobs:
- name: Install Hive Module
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
$RETRY $MAVEN clean install ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -Dgib.logImpactedTo=gib-impacted.log -am -pl :trino-hive-hadoop2
$MAVEN clean install ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -Dgib.logImpactedTo=gib-impacted.log -am -pl :trino-hive-hadoop2
- name: Run Hive Tests
run: |
source plugin/trino-hive-hadoop2/conf/hive-tests-${{ matrix.config }}.sh &&
Expand Down Expand Up @@ -419,7 +415,7 @@ jobs:
- name: Maven Install
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
$RETRY $MAVEN clean install ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -pl '!:trino-docs,!:trino-server,!:trino-server-rpm'
$MAVEN clean install ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -pl '!:trino-docs,!:trino-server,!:trino-server-rpm'
- name: Maven Tests
run: |
$MAVEN test ${MAVEN_TEST} -pl '
Expand Down Expand Up @@ -499,12 +495,10 @@ jobs:
github.event.client_payload.pull_request.head.sha == github.event.client_payload.slash_command.args.named.sha &&
format('refs/pull/{0}/head', github.event.client_payload.pull_request.number) || '' }}
- uses: ./.github/actions/setup
with:
cache: false
Comment thread
findepi marked this conversation as resolved.
Outdated
Comment thread
MiguelWeezardo marked this conversation as resolved.
Outdated
- name: Maven validate
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
$RETRY $MAVEN validate ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -Dgib.logImpactedTo=gib-impacted.log -P disable-check-spi-dependencies -pl '!:trino-docs'
$MAVEN validate ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -Dgib.logImpactedTo=gib-impacted.log -P disable-check-spi-dependencies -pl '!:trino-docs'
- name: Set matrix
id: set-matrix
run: |
Expand Down Expand Up @@ -593,7 +587,7 @@ jobs:
- name: Maven Install
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
$RETRY $MAVEN clean install ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -am -pl "${{ matrix.modules }}"
$MAVEN clean install ${MAVEN_FAST_INSTALL} ${MAVEN_GIB} -am -pl "${{ matrix.modules }}"
- name: Maven Tests
if: >-
matrix.modules != 'plugin/trino-singlestore'
Expand Down Expand Up @@ -759,7 +753,7 @@ jobs:
- name: Maven Install
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
$RETRY $MAVEN clean install ${MAVEN_FAST_INSTALL} -pl '!:trino-docs,!:trino-server-rpm'
$MAVEN clean install ${MAVEN_FAST_INSTALL} -pl '!:trino-docs,!:trino-server-rpm'
- name: Map impacted plugins to features
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
Expand Down
27 changes: 27 additions & 0 deletions core/trino-spi/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,33 @@
<match>@java.lang.Deprecated(*) ^*;</match>
</old>
</item>
<!-- Revapi tries to compare any API with its previous version -->
<!-- If it can't download the old version, if falls back to comparing with an empty file -->
Comment thread
MiguelWeezardo marked this conversation as resolved.
Outdated
<!-- This can lead to errors which are not shown in daily development where local Maven cache has the old version -->
<item>
<ignore>true</ignore>
<code>java.class.externalClassExposedInAPI</code>
<new>class io.airlift.slice.BasicSliceInput</new>
<justification>Trino SPI depends on Airlift</justification>
</item>
<item>
<ignore>true</ignore>
<code>java.class.externalClassExposedInAPI</code>
<new>class io.airlift.slice.Slice</new>
<justification>Trino SPI depends on Airlift</justification>
</item>
<item>
<ignore>true</ignore>
<code>java.class.externalClassExposedInAPI</code>
<new>class io.airlift.slice.SliceInput</new>
<justification>Trino SPI depends on Airlift</justification>
</item>
<item>
<ignore>true</ignore>
<code>java.class.externalClassExposedInAPI</code>
<new>class io.airlift.slice.SliceOutput</new>
<justification>Trino SPI depends on Airlift</justification>
</item>
<!-- Backwards incompatible changes since the previous release -->
<!-- Any exclusions below can be deleted after each release -->
</differences>
Expand Down
104 changes: 104 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2114,6 +2114,110 @@
</exceptions>
</configuration>
</plugin>

<plugin>
<groupId>de.qaware.maven</groupId>
<artifactId>go-offline-maven-plugin</artifactId>
<version>1.2.8</version>
<configuration>
<dynamicDependencies>
<DynamicDependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>3.2.0</version>
<repositoryType>PLUGIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.apache.yetus</groupId>
<artifactId>audience-annotations</artifactId>
Comment thread
MiguelWeezardo marked this conversation as resolved.
Outdated
<version>0.8.0</version>
<repositoryType>PLUGIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>io.airlift.drift</groupId>
<artifactId>drift-javadoc</artifactId>
<version>${dep.drift.version}</version>
<repositoryType>PLUGIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>com.google.errorprone</groupId>
<artifactId>error_prone_core</artifactId>
<version>${dep.errorprone.version}</version>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.3.2</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>pom</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>5.3.2</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>pom</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>2.28.2</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>pom</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-reflect</artifactId>
Comment thread
MiguelWeezardo marked this conversation as resolved.
Outdated
<version>2.0.5</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>pom</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>pom</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>5.10</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>pom</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.9.1</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>pom</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-library</artifactId>
<version>1.3</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>pom</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.easytesting</groupId>
<artifactId>fest-assert</artifactId>
<version>1.4</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>pom</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
<DynamicDependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-testng</artifactId>
<version>2.22.2</version> <!-- legacy version which Surefire seems to need for some reason -->
<type>jar</type>
<repositoryType>MAIN</repositoryType>
</DynamicDependency>
</dynamicDependencies>
</configuration>
</plugin>
</plugins>
</pluginManagement>

Expand Down