texlive: create reproducible .fmt files by raboof · Pull Request #196435 · NixOS/nixpkgs

raboof · 2022-10-17T15:48:24Z

Description of changes

Without these changes, building this package twice does not produce the bit-by-bit identical result each time, making it harder to detect CI breaches. You can read more about this at https://reproducible-builds.org/ .

Fixing bit-by-bit reproducibility also has additional advantages, such as avoiding hard-to-reproduce bugs, making content-addressed storage more effective and reducing rebuilds in such systems.

Based on the work done in Debian and documented at https://salsa.debian.org/live-team/live-build/-/blob/master/examples/hooks/reproducible/0139-reproducible-texlive-binaries-fmt-files.hook.chroot

Fixes #192736

Things done

veprbl · 2022-10-17T16:09:10Z

@GrahamcOfBorg build tests.texlive

raboof · 2022-10-18T06:57:07Z

@GrahamcOfBorg build tests.texlive

veprbl · 2022-10-18T16:43:05Z

@GrahamcOfBorg build tests.texlive

raboof · 2022-10-18T16:46:35Z

(I assume you retargeted this PR to master to see if the tests.texlive do succeed there, and if they do we'll retarget back to staging because of the rebuild count? Makes sense to me! 🤞 )

veprbl · 2022-10-18T16:48:36Z

texlive is only a few hundred rebuilds, we usually don't target it to master.

raboof · 2022-10-18T16:54:17Z

Hmm, ofborg is reporting 2501-5000 for linux though 🤔

Interesting to see the tests fail when targeting master as well, though - they seem to pass for me locally. Looking into it, but if you can provide any hints I'm all ears :)

veprbl · 2022-10-18T18:20:44Z

The /nix/store/h374kiml5r1gviflbdmgn75001p4dj5j-reproducible_exception_strings.patch.drv doesn't build for me

error: Normalized patch '/private/tmp/nix-build-reproducible_exception_strings.patch.drv-0/patch' is empty (while the fetched file was not)!
Did you maybe fetch a HTML representation of a patch instead of a raw patch?
Fetched file was:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>
<link rel="icon" href="/favicon.png">
<title>#1009196 - texlive-binaries: Reproducible content of .fmt files - Debian Bug report logs</title>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="/css/bugs.css" type="text/css">

<link rel="canonical" href="&lt;a href=&quot;bugreport.cgi?bug=1009196&quot;&gt;1009196&lt;/a&gt;">
<script type="text/javascript">
<!--
function toggle_infmessages()
{
        allDivs=document.getElementsByTagName("div");
        for (var i = 0 ; i < allDivs.length ; i++ )
        {
                if (allDivs[i].className == "infmessage")
                {
                        allDivs[i].style.display=(allDivs[i].style.display == 'none' | allDivs[i].style.display == '') ? 'block' : 'none';
                }
        }
}
-->
</script>
</head>
<body>
<h1>Debian Bug report logs - 
<a href="mailto:1009196@bugs.debian.org">#1009196</a><br>
texlive-binaries: Reproducible content of .fmt files</h1>
<div class="versiongraph"><a href="version.cgi?collapse=1;absolute=0;package=texlive-binaries;info=1;found=texlive-bin%2F2021.20210626.59705-1">
<div class="pkginfo">
  <p>Package:
     <a class="submitter" href="pkgreport.cgi?package=texlive-binaries">texlive-binaries</a>;
Maintainer for <a href="pkgreport.cgi?package=texlive-binaries">texlive-binaries</a> is <a href="pkgreport.cgi?maint=debian-tex-maint%40lists.de

</div>

Without these changes, building this package twice does not produce the bit-by-bit identical result each time, making it harder to detect CI breaches. You can read more about this at https://reproducible-builds.org/ . Fixing bit-by-bit reproducibility also has additional advantages, such as avoiding hard-to-reproduce bugs, making content-addressed storage more effective and reducing rebuilds in such systems. Based on the work done in Debian and documented at https://salsa.debian.org/live-team/live-build/-/blob/master/examples/hooks/reproducible/0139-reproducible-texlive-binaries-fmt-files.hook.chroot Fixes NixOS#192736

raboof · 2022-10-18T19:02:08Z

The /nix/store/h374kiml5r1gviflbdmgn75001p4dj5j-reproducible_exception_strings.patch.drv doesn't build for me

Argh, hashes not getting invalidated when the input URL changes strikes again... fixing

@GrahamcOfBorg build tests.texlive

veprbl · 2022-10-20T02:45:52Z

pkgs/tools/typesetting/tex/texlive/combine.nix

+    if [[ -d share/texmf-var/web2c/luahbtex ]]
+    then
+      cd share/texmf-var/web2c/luahbtex
+      faketime $(date --utc -d@$SOURCE_DATE_EPOCH --iso-8601=seconds) luahbtex -ini -jobname=lualatex -progname=lualatex lualatex.ini


Your link does not mention luahbtex, and the engines are supposed to be completely different. I assume you found an additional issue with it?

correct, though I didn't manage to investigate it sufficiently to write up a useful upstream report yet.

vcunat · 2022-10-30T10:12:38Z

This PR broke scheme-minimal: https://hydra.nixos.org/build/196758219
/cc PR #198236

raboof · 2022-10-30T17:23:14Z

This PR broke scheme-minimal: https://hydra.nixos.org/build/196758219 /cc PR #198236

Ouch, sorry about that. I can reproduce, though it's not obvious to me yet how this change would cause that problem. I'll dig in further, but possibly I won't get to it properly before Tuesday. If that holds up other things then feel free to revert I guess :(

veprbl · 2022-10-31T04:20:03Z

@raboof Should we revert for now? It might be worth exploring where the date is used in the source code. I've tried looking into that previously without success, but if we can patch it, it should be a simpler change, possibly even upstreamable.

raboof · 2022-10-31T08:03:53Z

@raboof Should we revert for now?

I'm not familiar enough with the staging process to judge how urgent it is to resolve this problem. If it's urgent then, as said, revert - but if it can wait until tomorrow I would prefer to "roll forward".

It might be worth exploring where the date is used in the source code. I've tried looking into that previously without success, but if we can patch it, it should be a simpler change, possibly even upstreamable.

For the order of the hyphenation exception strings, this patch has already been proposed upstream, but they seem to be discussing other ways to fix it.

For the date in the ini, I agree it might be neater to adapt this with a patch instead of using faketime. The Debian notes at https://salsa.debian.org/live-team/live-build/-/blob/master/examples/hooks/reproducible/0139-reproducible-texlive-binaries-fmt-files.hook.chroot#L12 suggest that this was also already brought up upstream and they didn't want to have it fixed, though I didn't find the upstream discussion to find out why yet.

vcunat · 2022-10-31T08:36:03Z

The current batch is only waiting for the openssl critical security update, to be released tomorrow (2022-11-01, 13:00-17:00 UTC). I'll revert around that point if not resolved by then.

Follow-up on NixOS#196435, fixing NixOS#196435 (comment) Needed to make NixOS#198236 pass

raboof · 2022-11-01T07:44:21Z

Fixed in #198892 - though I'm not sure if that is supposed to target staging or staging-next?

Follow-up on #196435, fixing #196435 (comment) Needed to make #198236 pass

raboof added the 6.topic: reproducible builds Run nix-build twice and get the same result. label Oct 17, 2022

github-actions bot added the 6.topic: TeX Issues regarding texlive and TeX in general label Oct 17, 2022

ofborg bot requested a review from veprbl October 17, 2022 15:59

raboof changed the base branch from master to staging October 17, 2022 15:59

raboof force-pushed the texlive branch from d85b0de to 6f17edd Compare October 17, 2022 21:46

veprbl changed the base branch from staging to master October 18, 2022 16:42

raboof force-pushed the texlive branch from 6f17edd to a41ae44 Compare October 18, 2022 19:00

veprbl reviewed Oct 20, 2022

View reviewed changes

veprbl changed the base branch from master to staging October 20, 2022 14:18

veprbl approved these changes Oct 20, 2022

View reviewed changes

veprbl merged commit 508429a into NixOS:staging Oct 20, 2022

ofborg bot requested a review from veprbl October 20, 2022 14:29

raboof added a commit to raboof/nixpkgs that referenced this pull request Nov 1, 2022

texlive.combined.scheme-minimal: fix build

0e61b9e

Follow-up on NixOS#196435, fixing NixOS#196435 (comment) Needed to make NixOS#198236 pass

raboof mentioned this pull request Nov 1, 2022

texlive.combined.scheme-minimal: fix build #198892

Closed

13 tasks

vcunat pushed a commit that referenced this pull request Nov 1, 2022

texlive.combined.scheme-minimal: fix build

87598a6

Follow-up on #196435, fixing #196435 (comment) Needed to make #198236 pass

Uh oh!

Conversation

raboof commented Oct 17, 2022

Description of changes

Things done

Uh oh!

veprbl commented Oct 17, 2022

Uh oh!

raboof commented Oct 18, 2022

Uh oh!

veprbl commented Oct 18, 2022

Uh oh!

raboof commented Oct 18, 2022

Uh oh!

veprbl commented Oct 18, 2022

Uh oh!

raboof commented Oct 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

veprbl commented Oct 18, 2022

Uh oh!

raboof commented Oct 18, 2022

Uh oh!

veprbl Oct 20, 2022

Choose a reason for hiding this comment

Uh oh!

raboof Oct 20, 2022

Choose a reason for hiding this comment

Uh oh!

vcunat commented Oct 30, 2022

Uh oh!

raboof commented Oct 30, 2022

Uh oh!

veprbl commented Oct 31, 2022

Uh oh!

raboof commented Oct 31, 2022

Uh oh!

vcunat commented Oct 31, 2022

Uh oh!

raboof commented Nov 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

raboof commented Oct 18, 2022 •

edited

Loading