Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Maven build reproducible #2543

Merged

Conversation

Marcono1234
Copy link
Collaborator

Purpose

Makes the Maven build reproducible

Description

Makes the Maven build reproducible, see https://maven.apache.org/guides/mini/guide-reproducible-builds.html, and also google/guava#6322. However, it might still be necessary to use the same OS and JDK version to actually be able to reproduce a build.

The changes also format the pom.xml files in the way the Maven Release Plugin would.

Checklist

  • New code follows the Google Java Style Guide
    This is automatically checked by mvn verify, but can also be checked on its own using mvn spotless:check.
    Style violations can be fixed using mvn spotless:apply; this can be done in a separate commit to verify that it did not cause undesired changes.
  • If necessary, new public API validates arguments, for example rejects null
  • New public API has Javadoc
    • Javadoc uses @since $next-version$
      ($next-version$ is a special placeholder which is automatically replaced during release)
  • If necessary, new unit tests have been added
    • Assertions in unit tests use Truth, see existing tests
    • No JUnit 3 features are used (such as extending class TestCase)
    • If this pull request fixes a bug, a new test was added for a situation which failed previously and is now fixed
  • mvn clean verify javadoc:jar passes without errors

@Marcono1234 Marcono1234 force-pushed the marcono1234/maven-reproducible-builds branch from 56ab8f6 to ee53c4d Compare November 18, 2023 14:45
However, it might still be necessary to use the same OS and JDK version
to actually be able to create identical artifacts.

This commit also formats the `pom.xml` files in the way the Maven Release
Plugin would.
@Marcono1234 Marcono1234 force-pushed the marcono1234/maven-reproducible-builds branch from ee53c4d to 4ab8f6a Compare November 18, 2023 16:11
Copy link
Collaborator Author

@Marcono1234 Marcono1234 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested the release workflow locally and it seems to work as expected, but I excluded the GPG plugin for my tests because I don't have it set up locally.

The output of mvn help:effective-pom -Prelease suggests it will be run after the buildinfo goal of the maven-artifact-plugin, as desired. And if I understand it correctly the GPG plugin does not modify the JARs but just attaches additional .asc signature files, so this should not be a problem (I hope).

Comment on lines +47 to +48
verify-reproducible-build:
name: "Verify reproducible build"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have included this as separate job (instead of including it in the steps of the build above) because it requires running mvn ... install which is normally not needed for a regular build, and also to keep it independent from the 'real' build setup above.

However, this makes the GitHub workflow slightly slower though.

Comment on lines +40 to +44
## Testing Maven release workflow locally

The following describes how to perform the steps of the release locally to verify that they work as desired.

**Warning:** Be careful with this, these steps might be outdated or incomplete. Doublecheck that you are working on a copy of your local Gson Git repository and make sure you have followed all steps. To be safe you can also temporarily turn off your internet connection to avoid accidentally pushing changes to the real remote Git or Maven repository.\
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This setup is a bit hacky and I am not sure if there is a proper better alternative to this, therefore I added this "Warning". But it does allow you to test the behavior of the release workflow quite accurately, which is why I added it.

Comment on lines +186 to +194
<!-- Attaches a `.buildinfo` file which contains information for reproducing the build,
such as OS, JDK version, ...
Since this is a multi-module Maven project, only one aggregated file will be created for
the last module, see the note on https://maven.apache.org/plugins/maven-artifact-plugin/usage.html#recording-buildinfo-file -->
<!-- The other goals of this plugin are run by the GitHub workflow to verify that
the build is reproducible (see `artifact:...` usage in the workflow) -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-artifact-plugin</artifactId>
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This includes information about the build environment to make it easier to reproduce the build. I am not sure if you would consider some of this information sensitive (e.g. OS name and version or Java vendor and version).

For example for me the generated file looks like this:

gson-2.10.2.buildinfo (click to expand)
# https://reproducible-builds.org/docs/jvm/
buildinfo.version=1.0-SNAPSHOT

name=Gson Parent
group-id=com.google.code.gson
artifact-id=gson-parent
version=2.10.2

# source information
source.scm.uri=scm:git:https://github.com/google/gson.git
source.scm.tag=gson-parent-2.10.2

# build instructions
build-tool=mvn

# effective build environment information
java.version=17.0.6
java.vendor=Eclipse Adoptium
os.name=Windows 10

# Maven rebuild instructions and effective environment
mvn.version=3.9.1
mvn.aggregate.artifact-id=gson

# aggregated output

outputs.0.coordinates=com.google.code.gson:gson-parent

outputs.0.0.groupId=com.google.code.gson
outputs.0.0.filename=gson-parent-2.10.2.pom
outputs.0.0.length=20999
outputs.0.0.checksums.sha512=36a916aea7d2e261e62a9287d0d9a2fcf9cdd94f1658bfba616dc3e9a08121b3ff080e5b574f32102311b8c6d286db977dd5b7038c5b4841c0fd63a164fecd93

outputs.1.coordinates=com.google.code.gson:gson

outputs.1.0.groupId=com.google.code.gson
outputs.1.0.filename=gson-2.10.2.pom
outputs.1.0.length=11602
outputs.1.0.checksums.sha512=e16df3d3a571fee3d6145007eb7eb20dcef0a7c9707d7ea347341cf3708ea73f197550076336dc3bb29c842ee4d00ad2ce5a9ae10302d98ac55d4437d3d7949c

outputs.1.1.groupId=com.google.code.gson
outputs.1.1.filename=gson-2.10.2.jar
outputs.1.1.length=307320
outputs.1.1.checksums.sha512=300dc667e042cb8dcc65fcf97b169b2b15bec96d4728164649f2e9dff2cc1a8b4b1f1053f12875fe59395fb42bac8abed49a5d410ecfcae8ca2287f9f8e9ada1

outputs.1.2.groupId=com.google.code.gson
outputs.1.2.filename=gson-2.10.2-sources.jar
outputs.1.2.length=202441
outputs.1.2.checksums.sha512=aa94a042ee8e3a985329eb421853668a2660ef5f1b70c5fdbcfa0b6f2cbb203ae07275ccaba02fb418773fc89ba36984e1671ee832ebe8eb31c97436e84761c9

<executions>
<execution>
<goals>
<goal>buildinfo</goal>
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This causes a warning during the build:

[WARNING] SCM source tag in buildinfo source.scm.tag=HEAD does not permit rebuilders reproducible source checkout

But I guess that can be safely ignored because during release source.scm.tag will be temporarily changed by the Maven Release Plugin.

@Marcono1234 Marcono1234 marked this pull request as ready for review November 18, 2023 16:34
Copy link
Member

@eamonnmcmanus eamonnmcmanus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks great. Thanks!

I'm not too concerned about including the build environment information. It will likely be Google's internal version of Linux in actual releases, which means nobody outside will be able to reproduce it exactly. But I don't think there's anything secret there.

We haven't had a release since January and there has been quite a lot of activity since then, so we will probably have the opportunity to put this in action soon.

Let me know if this is ready to merge or if you still have changes in mind.

@Marcono1234
Copy link
Collaborator Author

It will likely be Google's internal version of Linux in actual releases, which means nobody outside will be able to reproduce it exactly. But I don't think there's anything secret there.

Ok thanks for the confirmation; I wasn't completely sure how you performed the previous releases since you were shown as Git committer. Even if it is an internal version of Linux, I assume it might still be reproducible. The main concern the Maven guide listed was line endings (\n for Linux, \r\n for Windows) which could have an effect; there should hopefully not be any relevant differences between Linux versions.

We haven't had a release since January and there has been quite a lot of activity since then, so we will probably have the opportunity to put this in action soon.

I was thinking about the same some weeks ago, but I think it would be good to include a few more changes:

Let me know if this is ready to merge or if you still have changes in mind.

I think this should be ready; my main concern is the GPG plugin, but in the worst case it will fail on release and we can revert the changes of this PR again temporarily.

@eamonnmcmanus eamonnmcmanus merged commit b7c3e1d into google:main Nov 18, 2023
10 checks passed
@Marcono1234 Marcono1234 deleted the marcono1234/maven-reproducible-builds branch November 19, 2023 14:44
tibor-universe pushed a commit to getuniverse/gson that referenced this pull request Sep 14, 2024
However, it might still be necessary to use the same OS and JDK version
to actually be able to create identical artifacts.

This commit also formats the `pom.xml` files in the way the Maven Release
Plugin would.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants