feat: Add reproducible debian packaging and a corresponding CI workflow job

[MoeMahhouk]

This pull request introduces support for building, packaging, and testing Debian .deb packages for the project. It includes updates to the GitHub Actions workflow, the Makefile, and a new script for verifying reproducibility of Debian packages. Below are the most important changes grouped by theme:

GitHub Actions Workflow Enhancements

• Added new Debian build targets (x86_64-unknown-linux-gnu-deb and aarch64-unknown-linux-gnu-deb) to the matrix in .github/workflows/release.yml. These targets include settings for architecture-specific GCC packages and Rust targets.
• Introduced steps for installing Debian build dependencies, adding Rust targets, and installing cargo-deb for .deb package creation.
• Added separate steps for building and signing Debian packages, generating checksums, and uploading .deb artifacts as part of the release process. [1] [2]

Makefile Updates for Debian Packaging

• Added new Makefile targets for building .deb packages (deb-cargo, deb-cargo-x86_64, deb-cargo-aarch64, and deb-cargo-all), testing reproducibility, and installing or removing packages locally.
• Introduced a clean-deb target to clean up Debian packaging artifacts and a help-deb target to display Debian-specific build instructions.

Reproducibility Testing

• Added a new script scripts/test-deb-reproducible.sh to test the reproducibility of .deb packages by comparing two builds and running diffoscope for detailed analysis if differences are found.

Documentation and Release Notes

• Updated the release notes template in .github/workflows/release.yml to include information about Debian packages, their installation instructions, and links to PGP signatures and checksums.

These changes streamline the process of building and distributing Debian packages, ensuring reproducibility and providing clear documentation for end-users.

[mattsse]

cant really review this properly, this should also not prevent the existing release setup otherwise we're stuck with this

[MoeMahhouk]

cant really review this properly, this should also not prevent the existing release setup otherwise we're stuck with this

Yes, currently it is in draft stage to gather feedback and see how the reth team would like to have it best fit in their product.
To give a context on why debian packaging is desirable:
As we are moving towards building the Buildernet image with mkosi instead of yocto, it would be beneficial to have reproducible debian packages available to pull and use directly to fasten the build process and the development/deployment cycle of Buildernet.
Also, it is in general a nice to have debian packaging that encapsulate the binary with its services for ease of usage/installation.

That's being said, I would appreciate your feedback/review on it and I am open to suggestions to make it better fit inside the reth repo

[Rjected]

How critical is this to be in the main reth repo rather than a fork, or on some other infrastructure with automation? i.e., if this were not in the main repo, what would be lost?

We cannot have the following situation happen:

• We try to release
• Something in the reproducible build / deb part of the workflow fails
• We can't debug, or takes too long
• No release that day

So this should not be in any workflow that is a prereq for release, or would block release if failing

[MoeMahhouk]

How critical is this to be in the main reth repo rather than a fork, or on some other infrastructure with automation? i.e., if this were not in the main repo, what would be lost?

We cannot have the following situation happen:

• We try to release
• Something in the reproducible build / deb part of the workflow fails
• We can't debug, or takes too long
• No release that day

So this should not be in any workflow that is a prereq for release, or would block release if failing

Thanks @Rjected for your thorough response!
I do understand your concerns and they make total sense to not block release cycles for too long on this.
Why in main repo:

• User trust: Official .deb packages from paradigmxyz/reth carry much more weight than third-party packages
• Security: Same build infrastructure = clear provenance chain
• Reach: Opens reth to users who prefer debian packages over manual install

Intermediate solution:
Let's make debian builds allow_fail: true so they never block releases. We can help on fixing issues on the reproducible debian packaging without impacting your release velocity. Binary releases continue as normal, debian packages become "nice to have" additions.
Once we've proven stability over a few releases, we can revisit making them required. The implementation reuses your existing build-reproducible target, so the risk is minimal.
Would this non-blocking approach work?

[Rjected]

I have some initial comments / questions, I think building deb packages as part of releases makes sense. I want to make sure that any failures in build-reproducible are caught before we do a release. I am currently leaning towards allow_fail: true on the debian package builds, because we previously had a delayed release due to a failure in make build-reproducible.

[Rjected]

if we use target here to distinguish between deb and non-deb builds, the rust-toolchain step will fail because the -deb names are not valid targets.

[MoeMahhouk]

This is handled later here for example

[Rjected]

right, but we run this first

      - uses: dtolnay/rust-toolchain@stable
        with:
          target: ${{ matrix.configs.target }}

which would mean the -deb builds would always fail I think?

[Rjected]

I think we should add this to https://github.com/paradigmxyz/reth/blob/main/.github/workflows/reproducible-build.yml

[MoeMahhouk]

we could have it there as well, but in general, you wouldnt want to publish debian packages that are not reproducible, or do you want to separate that as well and let it publish them despite of that?

[Rjected]

let it publish them despite of that?

I think this, just add make test-deb-reproducible to the reproducible-build workflow. The reproducible-build workflow should test that the build is actually reproducible, and give a warning if something is causing us to not be reproducible. It's currently run via cron

[MoeMahhouk]

I added it there now and found out that the reproducible-build workflow is failing because the reproducible-build was not generating reproducible binaries.
So I fixed that now and now testing the workflow if it will pass with both the binaries and debian packaging reproducibility.

[Rjected]

if we do this / are relying on journald for logs, can set the RUST_LOG to debug? If people run this and something happens to the node, not having debug logs would be rough

[MoeMahhouk]

What is the minimalistic set of flags to set for this that you suggest to have here and I will change it.
The main idea is to provide it in the packaging and potentially leave it to the user to modify it with any potential extra flags.
The ones I included here are from our buildernet instance but probably we should do a simple set of flags that you (reth team) recommend.

[Rjected]

this should be run right?

[MoeMahhouk]

I left this commented for feedback if you want to include it as postrm or not. It is more of a product question.
I can include it if you find it useful

[Rjected]

not sure this is the right email, checking

[MoeMahhouk]

Sure, I will change it to the correct mail once you share that with me

[Rjected]

Need to review this again, would like to have this

[MoeMahhouk]

closing this PR as it was later on done within this PR