antora-lunr-extension: init at 1.0.0-alpha.8

[trueNAHO]

Description of changes

This patchset introduces the Antora Lunr extension, complementing the pkgs.antora package:

Lunr provides a great search experience for readers without having to rely on external search services. With this extension, you can add offline, full-text search powered by Lunr to your Antora documentation site.

-- https://gitlab.com/antora/antora-lunr-extension

This extension is officially supported by Antora and heavily maintained by the AsciiDoc team.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[trueNAHO]

Changelog

v2: 465f7f2

• antora

  • verify core functionality with a minimal working example
  • move package into pkgs/by-name directory
  • declare passthru.updateScript
  • add naho as maintainer
  • minor code style improvements

• antora-ui-default

  • extract inline antora-lunr-extension package into its own package
  • fetch UI bundle from a fork to make the package reproducible

• antora-lunr-extension

  • verify core functionality with a minimal working example
  • disable NPM tests that are fetching data at runtime
  • add longDescription
  • declare passthru.updateScript

• add Link: https://github.com/NixOS/nixpkgs/pull/332341 tag to every commit

v1: 827c50d

• Remove unnecessary callPackage in pkgs/top-level/all-packages.nix

v0: 0fabb09

[trueNAHO]

The "Vet nixpkgs" action fails with the following error message:

• pkgs/by-name/an/antora-lunr-extension: File package.nix at line 26 contains the path expression "../antora/test" which may point outside the directory of that package.
  This is undesirable because it creates dependencies between internal paths, making it harder to reorganise Nixpkgs in the future.
  Alternatives include:
  [...]
  • If the path being referenced is internal and has multiple uses, consider passing the file as an explicit callPackage argument in pkgs/top-level/all-packages.nix.
    [...]

[...]
If you're having trouble, ping @NixOS/nixpkgs-vet

-- https://github.com/NixOS/nixpkgs/actions/runs/11072379203/job/30766461579?pr=332341

@philiptaron, how to pass files as an explicit argument when packages are not explicitly defined in pkgs/top-level/all-packages.nix, but in pkgs/by-name/*/*?

The ../antora/test path, introduced in commit 42a162d8fd736fae6e9d17d8c65cdf5c3dcc7f94 ("antora-lunr-extension: init at 1.0.0-alpha.8"), accesses the Antora test framework introduced in commit b32d7824bad3130e8f44d74fd1886ac92dbe0577 ("antora: verify core functionality with a minimal working example").

For reference, commit b807da48ae86f49ece76ec9d77cb288dc8f50e47 ("antora: move package into pkgs/by-name directory") explicitly removes the package declaration from pkgs/top-level/all-packages.nix.

[trueNAHO]

Cc: @ehllie

This PR (v2) modifies the antora package non-negligibly.

[philiptaron]

how to pass files as an explicit argument when packages are not explicitly defined in pkgs/top-level/all-packages.nix, but in pkgs/by-name/*/*?

Thanks for the question, Noah (@trueNAHO).

In this case, you want to have pkgs/by-name/an/antora-lunr-extension/package.nix accept antora, then do:

passthru.tests.run = antora.tests.run.override { antora-lunr-extension-test = true; };

Instead of taking a dependency in your derivation on a path, take a dependency on the package.

I think it's also fine to do some copying -- in other words, have one test for antora, and another for the lunr extension. They'll happen to share a lot of stuff, but that's OK.

[trueNAHO]

Changelog

v3: b63ef6d

• antora-lunr-extension
  • do not access files outside package

v2: 465f7f2

• antora

  • verify core functionality with a minimal working example
  • move package into pkgs/by-name directory
  • declare passthru.updateScript
  • add naho as maintainer
  • minor code style improvements

• antora-ui-default

  • extract inline antora-lunr-extension package into its own package
  • fetch UI bundle from a fork to make the package reproducible

• antora-lunr-extension

  • verify core functionality with a minimal working example
  • disable NPM tests that are fetching data at runtime
  • add longDescription
  • declare passthru.updateScript

• add Link: https://github.com/NixOS/nixpkgs/pull/332341 tag to every commit

v1: 827c50d

• Remove unnecessary callPackage in pkgs/top-level/all-packages.nix

v0: 0fabb09

[trueNAHO]

how to pass files as an explicit argument when packages are not explicitly defined in pkgs/top-level/all-packages.nix, but in pkgs/by-name/*/*?

[...]

In this case, you want to have pkgs/by-name/an/antora-lunr-extension/package.nix accept antora, then do:

passthru.tests.run = antora.tests.run.override { antora-lunr-extension-test = true; };

Instead of taking a dependency in your derivation on a path, take a dependency on the package.

I rebased your elegant proposal into commit 42a162d ("antora-lunr-extension: init at 1.0.0-alpha.8"):

From 17c1ec3fff056b4d50c0d5ef2fd9121a5a94e976 Mon Sep 17 00:00:00 2001
From: NAHO <90870942+trueNAHO@users.noreply.github.com>
Date: Sun, 29 Sep 2024 12:05:23 +0200
Subject: [PATCH] antora-lunr-extension: do not access files outside package

---
 pkgs/by-name/an/antora-lunr-extension/package.nix | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pkgs/by-name/an/antora-lunr-extension/package.nix b/pkgs/by-name/an/antora-lunr-extension/package.nix
index 222580dec7a7..4a3c78554db6 100644
--- a/pkgs/by-name/an/antora-lunr-extension/package.nix
+++ b/pkgs/by-name/an/antora-lunr-extension/package.nix
@@ -1,6 +1,6 @@
 {
+  antora,
   buildNpmPackage,
-  callPackage,
   fetchFromGitLab,
   lib,
   nix-update-script,
@@ -23,7 +23,7 @@ buildNpmPackage rec {
   npmDepsHash = "sha256-EtjZL6U/uSGSYSqtuatCkdWP0NHxRuht13D9OaM4x00=";

   passthru = {
-    tests.run = callPackage ../antora/test {
+    tests.run = antora.tests.run.override {
       antora-lunr-extension-test = true;
     };

--

I might have figured this out myself if the error message did not get me stuck on pkgs/top-level/all-packages.nix. @philiptaron Should I submit a PR to update it to mention checking other package attributes too?

[philiptaron]

@philiptaron Should I submit a PR to update it to mention checking other package attributes too?

I'm planning on doing a pass through all the errors that can come from nixpkgs-vet (NixOS/nixpkgs-vet#99). I need reviews on NixOS/nixpkgs-vet#109 to continue to make progress. Reviewing my slew of commits would enable making better errors here!

[philiptaron]

What does @ehllie think of this?

[philiptaron]

No need to pull this into a local. Just use it down below.

[ehllie]

Yes, this use of a let binding feels a little superfluous. Setting it directly inside the fetchFromGitLab call attributes would be a bit clearer.

[trueNAHO]

According to #332341 (comment), I will inline the pname value. However, the srcFetchFromGitLab values are also used to generate the following documentation URL:

https://github.com/NixOS/nixpkgs/blob/b63ef6df743c3164c259524875c896273d0008d5/pkgs/by-name/an/antora-ui-default/package.nix#L34

Should I inline all srcFetchFromGitLab instances or only pname?

[ehllie]

From my point of view as long as the antora package builds it's as good as it was. It was an auto-generated with node2nix before, and I repackaged it as an example of how other packages in node-packages.json could be moved for the purpose of this issue #229475.

[philiptaron]

@philiptaron, how should we resolve this?

Let's take a look at what the script is doing for us.

1. Removing files that contain timestamps or otherwise have non-deterministic properties.

nixpkgs/pkgs/build-support/fetchgit/deterministic-git

Lines 16 to 19 in 9e70960

	# Remove files that contain timestamps or otherwise have non-deterministic
	# properties.
	rm -rf .git/logs/ .git/hooks/ .git/index .git/FETCH_HEAD .git/ORIG_HEAD \
	.git/refs/remotes/origin/HEAD .git/config

2. Removing all remote branches.

nixpkgs/pkgs/build-support/fetchgit/deterministic-git

Lines 21 to 24 in 9e70960

	# Remove all remote branches.
	git branch -r | while read -r branch; do
	clean_git branch -rD "$branch"
	done

3. Removing tags not reachable from HEAD. If we're exactly on a tag, don't delete it.

nixpkgs/pkgs/build-support/fetchgit/deterministic-git

Lines 28 to 33 in 9e70960

	maybe_tag=$(git tag --points-at HEAD)
	git tag --contains HEAD | while read -r tag; do
	if [ "$tag" != "$maybe_tag" ]; then
	clean_git tag -d "$tag"
	fi
	done

4. Do a full repack single-threaded

nixpkgs/pkgs/build-support/fetchgit/deterministic-git

Lines 36 to 38 in 9e70960

	clean_git config pack.threads 1
	clean_git repack -A -d -f
	rm -f .git/config

5. Garbage collect unreferenced objects.

nixpkgs/pkgs/build-support/fetchgit/deterministic-git

Line 43 in 9e70960

clean_git gc --prune=all --keep-largest-pack

We don't need any of that; the repository is empty.

So we can resolve it by deleting the line and blaming me for picking an example that wasn't what was needed in this case.

[trueNAHO]

Changelog

v6: 7a31aa5

• remove redundant pkgs/build-support/fetchgit/deterministic-git call

v5: 57d984c

• format code with nixfmt

v4: a8b38f5

• antora

  • improve Git reproducibility of test framework
  • add meta attribute to test framework
  • sort meta entries

• antora-ui-default

  • inline pname value

• specify meta.platforms for all packages

• sort attribute declarations according to Nixpkgs conventions

• rebase on top of commit 8885a1e ("lib/systems/architectures: add sapphirerapids/emeraldrapids")

v3: b63ef6d

• antora-lunr-extension
  • do not access files outside package

v2: 465f7f2

• antora

  • verify core functionality with a minimal working example
  • move package into pkgs/by-name directory
  • declare passthru.updateScript
  • add naho as maintainer
  • minor code style improvements

• antora-ui-default

  • extract inline antora-lunr-extension package into its own package
  • fetch UI bundle from a fork to make the package reproducible

• antora-lunr-extension

  • verify core functionality with a minimal working example
  • disable NPM tests that are fetching data at runtime
  • add longDescription
  • declare passthru.updateScript

• add Link: https://github.com/NixOS/nixpkgs/pull/332341 tag to every commit

v1: 827c50d

• Remove unnecessary callPackage in pkgs/top-level/all-packages.nix

v0: 0fabb09

[trueNAHO]

We don't need any of that; the repository is empty.

In that case, should the deterministic-git call not also be removed in the flutter module, or at least in pkgs/development/compilers/flutter/flutter.nix:

$ git checkout --quiet 8885a1e21ad43f8031c738a08029cd1d4dcbc2f7 && rg deterministic-git
pkgs/development/compilers/flutter/flutter.nix:        (. '${../../../build-support/fetchgit/deterministic-git}'; make_deterministic_repo .)
pkgs/development/compilers/flutter/engine/source.nix:    source ${../../../../build-support/fetchgit/deterministic-git}

If yes, I can submit a PR for this.

Here is the PR introducing deterministic-git: #147107.

[trueNAHO]

Friendly ping: AFAIK, this PR is merge-ready.

[philiptaron]

nixpkgs-review pr 332341 returns an error for me (something about /nix/store/0g4slq4f5l4ps4k7gkcvx0zs3vkvpdri-antora-ui-default-0: cannot execute binary file). Could you investigate please?

[trueNAHO]

Changelog

v7: a883a99

• install $src/ui-bundle.zip to $out/ui-bundle.zip instead of $out to prevent the ZIP from being misidentified as a binary
• properly format the "antora: add naho as maintainer" commit

v6: 7a31aa5

• remove redundant pkgs/build-support/fetchgit/deterministic-git call

v5: 57d984c

• format code with nixfmt

v4: a8b38f5

• antora

  • improve Git reproducibility of test framework
  • add meta attribute to test framework
  • sort meta entries

• antora-ui-default

  • inline pname value

• specify meta.platforms for all packages

• sort attribute declarations according to Nixpkgs conventions

• rebase on top of commit 8885a1e ("lib/systems/architectures: add sapphirerapids/emeraldrapids")

v3: b63ef6d

• antora-lunr-extension
  • do not access files outside package

v2: 465f7f2

• antora

  • verify core functionality with a minimal working example
  • move package into pkgs/by-name directory
  • declare passthru.updateScript
  • add naho as maintainer
  • minor code style improvements

• antora-ui-default

  • extract inline antora-lunr-extension package into its own package
  • fetch UI bundle from a fork to make the package reproducible

• antora-lunr-extension

  • verify core functionality with a minimal working example
  • disable NPM tests that are fetching data at runtime
  • add longDescription
  • declare passthru.updateScript

• add Link: https://github.com/NixOS/nixpkgs/pull/332341 tag to every commit

v1: 827c50d

• Remove unnecessary callPackage in pkgs/top-level/all-packages.nix

v0: 0fabb09

[trueNAHO]

nixpkgs-review pr 332341 returns an error for me (something about /nix/store/0g4slq4f5l4ps4k7gkcvx0zs3vkvpdri-antora-ui-default-0: cannot execute binary file). Could you investigate please?

v7 works around the issue by installing the UI bundle in the $out directory:

diff --git a/pkgs/by-name/an/antora-ui-default/package.nix b/pkgs/by-name/an/antora-ui-default/package.nix
index b0ae50e094dc..2e1278026ae8 100644
--- a/pkgs/by-name/an/antora-ui-default/package.nix
+++ b/pkgs/by-name/an/antora-ui-default/package.nix
@@ -41,7 +41,14 @@ stdenvNoCC.mkDerivation {
   src = fetchFromGitLab srcFetchFromGitLab;

   phases = [ "installPhase" ];
-  installPhase = ''install --mode 755 -D "$src/ui-bundle.zip" "$out"'';
+
+  # Install '$src/ui-bundle.zip' to '$out/ui-bundle.zip' instead of '$out' to
+  # prevent the ZIP from being misidentified as a binary [1].
+  #
+  # [1]: https://github.com/NixOS/nixpkgs/blob/8885a1e21ad43f8031c738a08029cd1d4dcbc2f7/pkgs/stdenv/generic/setup.sh#L792-L795
+  installPhase = ''
+    install --mode 755 -D "$src/ui-bundle.zip" "$out/ui-bundle.zip"
+  '';

   meta = {
     description = "Antora default UI bundle";
@@ -56,8 +63,9 @@ stdenvNoCC.mkDerivation {
       > -- Antora
       >    https://docs.antora.org/antora/3.1/playbook/ui-bundle-url

-      This UI bundle is intended to be passed to `antora`'s `--ui-bundle-url`
-      flag or injected into the [`ui.bundle.url`
+      This UI bundle is available under `$out/ui-bundle.zip` and intended to be
+      passed to `antora`'s `--ui-bundle-url` flag or injected into the
+      [`ui.bundle.url`
       key](https://docs.antora.org/antora/3.1/playbook/ui-bundle-url/#url-key)
       to avoid irreproducible
       [`https://gitlab.com/antora/antora-ui-default/-/jobs/artifacts/HEAD/raw/build/ui-bundle.zip?job=bundle-stable`](https://gitlab.com/${srcFetchFromGitLab.owner}/${srcFetchFromGitLab.repo}/-/blob/${srcFetchFromGitLab.rev}/README.adoc#user-content-use-the-default-ui)
diff --git a/pkgs/by-name/an/antora/test/default.nix b/pkgs/by-name/an/antora/test/default.nix
index 08cba1660aa8..ce5e0d5f50bb 100644
--- a/pkgs/by-name/an/antora/test/default.nix
+++ b/pkgs/by-name/an/antora/test/default.nix
@@ -47,7 +47,7 @@ stdenvNoCC.mkDerivation {
           cache-dir = "$(mktemp --directory)";
           extension = if antora-lunr-extension-test then antora-lunr-extension else false;
           to-dir = placeholder "out";
-          ui-bundle-url = antora-ui-default;
+          ui-bundle-url = "${antora-ui-default}/ui-bundle.zip";
         }
       } "${playbook}"
     '';

It seem more reasonable to comply with the /pkgs/stdenv/generic/setup.sh requirements than to modify the script, even if this workaround makes the package slightly more cumbersome.

[trueNAHO]

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 332341

---

x86_64-linux

✅ 2 packages built:

• antora-lunr-extension
• antora-ui-default

[trueNAHO]

Changelog

v8: 4ce5400

• restrict permissions of the installed UI bundle file by replacing install --mode 755 with cp

v7: a883a99

• install $src/ui-bundle.zip to $out/ui-bundle.zip instead of $out to prevent the ZIP from being misidentified as a binary
• properly format the "antora: add naho as maintainer" commit

v6: 7a31aa5

• remove redundant pkgs/build-support/fetchgit/deterministic-git call

v5: 57d984c

• format code with nixfmt

v4: a8b38f5

• antora

  • improve Git reproducibility of test framework
  • add meta attribute to test framework
  • sort meta entries

• antora-ui-default

  • inline pname value

• specify meta.platforms for all packages

• sort attribute declarations according to Nixpkgs conventions

• rebase on top of commit 8885a1e ("lib/systems/architectures: add sapphirerapids/emeraldrapids")

v3: b63ef6d

• antora-lunr-extension
  • do not access files outside package

v2: 465f7f2

• antora

  • verify core functionality with a minimal working example
  • move package into pkgs/by-name directory
  • declare passthru.updateScript
  • add naho as maintainer
  • minor code style improvements

• antora-ui-default

  • extract inline antora-lunr-extension package into its own package
  • fetch UI bundle from a fork to make the package reproducible

• antora-lunr-extension

  • verify core functionality with a minimal working example
  • disable NPM tests that are fetching data at runtime
  • add longDescription
  • declare passthru.updateScript

• add Link: https://github.com/NixOS/nixpkgs/pull/332341 tag to every commit

v1: 827c50d

• Remove unnecessary callPackage in pkgs/top-level/all-packages.nix

v0: 0fabb09

[philiptaron]

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 332341

---

x86_64-linux

✅ 2 packages built:

• antora-lunr-extension
• antora-ui-default