Shell script to verify staged release candidate artifacts

site/content/community/release-guide.md

@@ -17,7 +17,7 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-linkTitle: Release Guide
+linkTitle: Release Manager Guide
 type: docs
 weight: 500
 ---

site/content/community/release-verify.md

@@ -0,0 +1,25 @@
+---
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+linkTitle: Release Verification Guide
+type: docs
+weight: 500
+---
+
+{{< readfile "/release-verify.md" >}}

site/content/release-guide.md

@@ -17,9 +17,13 @@
   under the License.
 -->
 
-# Release Guide
+# Release Manager Guide
 
-This guide walks you through the release process of the Apache Polaris podling.
+**Audience**: Release Managers
+
+This guide walks you through the process of **creating** a release of the Apache Polaris podling.
+
+Instructions how to verify a release candidate are available [here](release-verify.md).
 
 ## Setup
 

site/content/release-verify.md

@@ -0,0 +1,223 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Release Verification Guide
+
+**Audience**: Committers and interested contributors.
+
+This guide walks you through the process of **verifying** a staged Apache Polaris release candidate.
+
+Please keep the [ASF Release Policy](https://www.apache.org/legal/release-policy.html#owned-controlled-hardware),
+especially the section on "Owned and Controlled Hardware," in mind:
+**_Strictly speaking, releases must be verified on hardware owned and controlled by the committer._**
+
+Verifying a (staged) release of an Apache project has to follow a bunch of tasks, which can be
+grouped into tasks that can be automated and those that need human intervention.
+Polaris provides a tool to automate the tasks that can be automated.
+
+Tasks that are automated:
+* Checksums and PGP signatures are valid.
+* All expected artifacts are present.
+* Source code artifacts have correct names matching the current release.
+* Built artifacts are [reproducible](#reproducible-builds).
+* Build passes.
+* `DISCLAIMER`, `LICENSE` and `NOTICE` files are included.
+* main and sources jar artifacts contain `META-INF/LICENSE` and `META-INF/NOTICE` files.
+* main distribution artifacts contain `DISCLAIMER`, `LICENSE` and `NOTICE` files in the top-level directory.
+
+Tasks that need human attention:
+* Download links are valid. Check all links in the `[VOTE]` email for the release:
+    * Tag on the GitHub website
+    * Commit on the GitHub website
+    * SVN repository with the source tarball and binary release artifacts
+    * SVN repository with the Helm chart
+    * Link to the KEYS file (_MUST_ be equal to `https://downloads.apache.org/incubator/polaris/KEYS`)
+    * Maven staging repository
+    * A link to this page is included.
+* The contents of the `DISCLAIMER`, `LICENSE` and `NOTICE` files are correct for the repository.
+* Contents of jar artifacts `META-INF/LICENSE` and `META-INF/NOTICE` files are correct.
+* All files have license headers, if necessary.
+  This is (mostly) verified using the "rat" tool during builds/CI.
+* No disallowed binary artifacts are bundled in the source archive.
+  This is a (soft) requirement to be held true by committers.
+* The commits on the Git tag, created on the release branch, do only contain changes that are relevant to the
+  release process and approved cherry-picks.
+
+Other tests:
+* Running Polaris built from the Git tag and testing it manually is recommended. You can use the build
+  generated by [release verification script](#running-the-script) for this.
+* You can also manually test the Docker images and the helm chars for the release candidate.
+
+**Imply good intent!**
+Although the release manager is responsible for producing a "proper" release, mistakes can and will happen.
+The Polaris project is committed to providing reproducible builds as an essential building block of
+_Apache trusted releases_.
+The project depends on frameworks which also strive to provide reproducible builds, but not all
+these frameworks can provide fully reproducible builds yet.
+The Polaris project's release verification tool will therefore report some issues that are currently expected.
+See [below](#reproducible-builds) for details.
+
+After you have completed the verification of the release candidate, respond to the release vote email,
+detailing the checks and tests you performed, and providing a summary of their outcomes.
+Remember to include your
+[vote (+1/0/-1)](https://www.apache.org/foundation/glossary.html#MajorityApproval)
+and whether it's "binding."
+A -1 vote requires a technical justification.
+As a (P)PMC member, your vote is "binding," all other votes are "non-binding."
+
+# Verifying a release candidate
+
+Instead of performing all mentioned steps manually, you can leverage the script
+`tools/verify-release/verify-release.sh`, available in the main repository, to perform the
+automatable tasks. The script is intended to be run on Linux or macOS.
+
+## Verifying the script
+
+Verifying a release using any kind of automation, like this script, requires "trust" into the automation.
+It is therefore recommended to verify the script before running it on a release candidate.
+You can inspect it by
+[looking at the source code](https://github.com/snazy/polaris/blob/main/tools/verify-release/verify-release.sh)
+before running it.
+The link to the `Raw` file must be the same as the one used in the command below to run the script.
+
+## Running the script
+
+Run the most recent version of the script using the following command:
+```bash
+bash <(curl \
+  -s https://raw.githubusercontent.com/apache/polaris/refs/heads/main/tools/verify-release/verify-release.sh) \
+  --help
+```
+
+The tool is intended for Polaris versions 1.3 and newer.
+The tool may report issues, see [below](#reproducible-builds) for details.
+
+That script requires a couple of tools installed and will check that those are available
+and report those that need to be installed.
+
+To run the script, you need the following pieces of information from the release-vote email:
+* The *full* Git SHA of the corresponding source commit.
+* The version number of the release, something like `1.3.0`
+* The RC number of the release, for example `1` or `2`
+* The Maven staging repository ID, for example `1033` (the full number at the end of the Maven repository URL `https://repository.apache.org/content/repositories/orgapachepolaris-1033/`).
+
+Example (values taken from the 1.2.0-rc2 release)
+```bash
+bash <(curl \
+  --silent https://raw.githubusercontent.com/apache/polaris/refs/heads/main/tools/verify-release/verify-release.sh) \
+  --git-sha 354a5ef6b337bf690b7a12fefe2c984e2139b029 \
+  --version 1.3.0-incubating-rc0 \
+  --maven-repo-id 1033
+```
+
+Same example, but using the short option names:
+```bash
+bash <(curl \
+  -s https://raw.githubusercontent.com/apache/polaris/refs/heads/main/tools/verify-release/verify-release.sh) \
+  -s 354a5ef6b337bf690b7a12fefe2c984e2139b029 \
+  -v 1.3.0-incubating-rc0 \
+  -m 1033
+```
+
+The verification script creates a temporary directory that will eventually contain a fresh Git clone,
+all downloaded and all built artifacts.
+This temporary directory is deleted after the script has finished. To keep the temporary directory
+around, you can use the `--keep-temp-dir` (`-k`) option.
+
+A log file, the name matches the pattern `polaris-release-verify-*.log`,
+will be created and contain detailed information about the identified issues reported on the console.
+
+Note: The script is maintained in the Polaris source tree in the `/tools/verify-release` directory.
+
+## Verifications performed by the tool
+
+After some startup checks, the tool emits some information about the release candidate. For example:
+```
+Verifying staged release
+========================
+
+Git tag:           apache-polaris-1.3.0-incubating-rc0
+Git sha:           354a5ef6b337bf690b7a12fefe2c984e2139b029
+Version:           1.3.0-incubating
+RC:                0
+Maven repo URL:    https://repository.apache.org/content/repositories/orgapachepolaris-1033/
+Main dist URL:     https://dist.apache.org/repos/dist/dev/incubator/polaris/1.3.0-incubating
+Helm chart URL:    https://dist.apache.org/repos/dist/dev/incubator/polaris/helm-chart/1.3.0-incubating
+Verify directory:  /tmp/polaris-release-verify-2025-10-23-14-22-31-HPmmiybzk
+
+A verbose log containing the identified issues will be available here:
+    .../polaris-release-verify-2025-10-23-14-22-31.log
+```
+
+After that, release candidate verification starts immediately, performing the following operations:
+
+1. Create `gpg` keyring for signature verification from the project's `KEYS` file.
+2. Clone the Git repository directly from GitHub.
+3. Verifies that the Git tag corresponds to the provided Git SHA
+4. Verifies that the mandatory files are present in the source tree
+5. Helm chart GPG signature and checksum checks
+6. Source tarball and binary artifacts GPG signature and checksum checks
+7. Build Polaris from the Git commit/tag
+8. Compares that the list of Maven artifacts produced by the build matches those in the Nexus staging repository.
+9. Compares the individual Maven artifacts of the local build with the ones in the Nexus staging repository.
+10. Compares the main binary distribution artifacts for Polaris server and Polaris admin tool.
+11. Compares the locally built Helm chart with the one in the staging repository.
+
+Found issues are reported on the console in _red_.
+
+Details for each reported issue will be reported in the log file, depending on the issue and file type.
+The intent is to provide as much information as possible to eventually fix a reproducible build issue.
+* Text files: Output of `diff` of the local and staged files.
+* Zip/Jar files: output of `zipcmp`.
+  If `zipcmp` reports no difference, the output of `zipinfo` for both files is logged.
+* Tarballs: output of `diff --recursive` the extracted local and staged tarballs.
+  If `diff` reports no difference, the output of `tar tvf` for both files is logged.
+* Other files: Output of `diff` of the local and staged files.
+
+Note: GPG signatures are verified **only** against the project's `KEYS` file.
+
+If the script finds no issues, the temporary verification directory is deleted by default.
+You can keep it around by using the `--keep-temp-dir` (`-k`) option.
+Remember to delete it manually when you're done.
+
+# Reproducible builds
+
+A build is reproducible if the built artifacts are identical for every build on every machine from the same source.
+
+The Apache Polaris build is currently _mostly_ reproducible, with some release-version specific exceptions.
+
+References:
+* [reproducible-builds.org](https://reproducible-builds.org/)
+* [Reproducible builds at the ASF](https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds)
+* [Polaris tracking issue](https://github.com/apache/polaris/issues/2204)
+
+## Exceptions for all Apache Polaris versions
+
+Pending on full support for reproducible builds in Quarkus:
+* Jars containing generated code are not guaranteed to be reproducible. Affects the following jars:
+    * */quarkus/generated-bytecode.jar
+    * */quarkus/transformed-bytecode.jar
+    * */quarkus/quarkus-application.jar
+* Re-assembled jars are not guaranteed to be reproducible: Affects the following jars:
+    * admin/app/polaris-admin-*.jar
+    * server/app/polaris-server-*.jar
+* Zips and tarballs containing any of the above are not guaranteed to be reproducible.
+
+Helm packages are not binary reproducible yet.
+See helm-package notes on [this page](https://cwiki.apache.org/confluence/display/SECURITY/Reproducible+Builds). 

site/hugo.yaml

@@ -162,10 +162,14 @@ menu:
       parent: "community"
       url: "/community/security-report"
       weight: 70
-    - name: "Release Guide"
+    - name: "Release Manager Guide"
       parent: "community"
       url: "/community/release-guide"
       weight: 80
+    - name: "Release Verification Guide"
+      parent: "community"
+      url: "/community/release-verify"
+      weight: 81
     - name: "Blogs"
       parent: "community"
       url: "/blog"

tools/verify-release/README.md

@@ -0,0 +1,22 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+Helpers and information for verifying a release candidate can be found
+on the Polaris website: https://polaris.apache.org/ or
+in the Polaris source tree in the `site/content/release-verify.md` file.