test: add shebangs to shell.nix test scripts

[agucova]

Summary

Adds shebangs to the foo, bar, and ruby test scripts in tests/functional/shell.nix to fix intermittent SIGSEGV crashes on macOS.

Motivation

Some of the test scripts used for the functional test are created without shebangs, which causes a ~50% build failure rate on macOS (particularly Apple Silicon) caused by SIGSEGVs when the scripts are executed via command substitution inside the nix sandbox.

See #12121 for my root cause analysis and a minimal reproduction script.

Solution

Add #!${shell} shebangs to each test script. With this fix, builds pass consistently (10/10 vs 5/10 without the patch).

Testing

• Verified fix with 10 consecutive builds on macOS Sequoia 15.1 (M4)
• Minimal reproduction in Functional tests fail when executing home-manager switch #13106 confirms the issue and fix

Likely fixes some of the problems people had in #13106, though not all test failures reported are caused by this.

[xokdvium]

Can't even fathom how this leads to segfaults and at this point I'm too afraid to ask.

[agucova]

Can't even fathom how this leads to segfaults and at this point I'm too afraid to ask.

I did some digging, and as far as I can tell, it's because under macOS fork-without-exec is an unsafe operation. When bash doesn't see a shebang, it does a fork and then execve, but execve fails with ENOEXEC, which, depending on a race condition, the sandbox detects as an unsafe operation and returns a SIGSEGV. macOS logs this as crashed on child side of fork pre-exec.

I believe it's the same issue as the one reported at benoitc/gunicorn#2761

Having said all this, I couldn't find a minimally reproducible example that doesn't rely on an environment created by nix-build, so it's possible that it's nix's fault somehow.

[xokdvium]

nix-build, so it's possible that it's nix's fault somehow.

Probably the specific macOS sandbox profile has to do with it?

[agucova]

Probably the specific macOS sandbox profile has to do with it?

Nope, apparently it happens without the sandbox profile enabled, too. I'm now trying to figure out if it's stdenv related, in which case there might be a more serious bug elsewhere.

[agucova]

@xokdvium want to hear something even more crazy? The bug just generally affects the use of runCommand with a shebang-less script. But if one swaps in the underlying bash version to either bash-interactive or, more conservatively, just the same bootstrap bash but linked with the ncurses library, it solves the SIGSEGV.

I'm cooking the most cursed reproduction scripts right now...

Running 30 builds for each configuration...

=== Testing BOOTSTRAP BASH (no ncurses) ===
.......X..............X....X..
Bootstrap bash: PASSED=27 FAILED=3 (SIGSEGV=3)

=== Testing BASH + NCURSES ===
..............................
Bash + ncurses: PASSED=30 FAILED=0 (SIGSEGV=0)

==============================================
SUMMARY
==============================================
Bootstrap bash (no ncurses): 27/30 passed, 3/30 failed (3 SIGSEGV)
Bash + ncurses:              30/30 passed, 0/30 failed (0 SIGSEGV)

My plain version looks like:

{ pkgs ? import <nixpkgs> {} }:
let
  # Script WITHOUT a shebang - this triggers the bug
  script = pkgs.runCommand "shebang-less-script" {} ''
    mkdir -p $out/bin
    echo 'echo hello' > $out/bin/test
    chmod +x $out/bin/test
  '';
in
pkgs.runCommand "test-bootstrap-bash" {} ''
  for i in $(seq 1 100); do
    # Capture both output and exit code
    set +e
    result=$(${script}/bin/test 2>&1)
    code=$?
    set -e

    if [ $code -eq 139 ]; then
      echo "SIGSEGV (exit 139) at iteration $i"
      exit 139
    elif [ $code -ne 0 ]; then
      echo "FAILED at iteration $i with exit code $code"
      exit $code
    elif [ "$result" != "hello" ]; then
      echo "FAILED at iteration $i: unexpected output '$result'"
      exit 1
    fi
  done

  mkdir -p $out
  echo "PASSED all 100 iterations" > $out/result
''

My patched version with ncurses is:

{ pkgs ? import <nixpkgs> {} }:
let
  # Build bash with ncurses linked 
  bashWithNcurses = pkgs.bash.overrideAttrs (old: {
    pname = "bash-with-ncurses";
    buildInputs = (old.buildInputs or []) ++ [ pkgs.ncurses ];
    # Force linking ncurses even though bash doesn't use it
    NIX_LDFLAGS = "-lncursesw";
  });

  # Script WITHOUT a shebang - same trigger as the failing case
  script = pkgs.runCommand "shebang-less-script-nc" {} ''
    mkdir -p $out/bin
    echo 'echo hello' > $out/bin/test
    chmod +x $out/bin/test
  '';
in
derivation {
  name = "test-bash-ncurses";
  system = builtins.currentSystem;
  # Bash with ncurses linked
  builder = "${bashWithNcurses}/bin/bash";
  args = ["-c" ''
    export PATH="${pkgs.coreutils}/bin"

    for i in $(seq 1 100); do
      # Capture both output and exit code
      set +e
      result=$(${script}/bin/test 2>&1)
      code=$?
      set -e

      if [ $code -eq 139 ]; then
        echo "SIGSEGV (exit 139) at iteration $i"
        exit 139
      elif [ $code -ne 0 ]; then
        echo "FAILED at iteration $i with exit code $code"
        exit $code
      elif [ "$result" != "hello" ]; then
        echo "FAILED at iteration $i: unexpected output '$result'"
        exit 1
      fi
    done

    mkdir -p $out
    echo "PASSED all 100 iterations" > $out/result
  ''];
  outputs = ["out"];
  __impureHostDeps = "/bin/sh /usr/lib/libSystem.B.dylib /dev/zero /dev/random /dev/urandom";
}

I'll continue my journey into the depths of hell tomorrow, trying to figure out what's ncurses doing and if this is bash's fault.

[internal-nix-ci]

Successfully created backport PR for 2.28-maintenance:

• [Backport 2.28-maintenance] test: add shebangs to shell.nix test scripts #14937

[internal-nix-ci]

Successfully created backport PR for 2.28-maintenance:

• [Backport 2.28-maintenance] test: add shebangs to shell.nix test scripts #14938

[internal-nix-ci]

Successfully created backport PR for 2.29-maintenance:

• [Backport 2.29-maintenance] test: add shebangs to shell.nix test scripts #14939

[internal-nix-ci]

Successfully created backport PR for 2.30-maintenance:

• [Backport 2.30-maintenance] test: add shebangs to shell.nix test scripts #14941

[internal-nix-ci]

Successfully created backport PR for 2.30-maintenance:

• [Backport 2.30-maintenance] test: add shebangs to shell.nix test scripts #14940

[internal-nix-ci]

Successfully created backport PR for 2.31-maintenance:

• [Backport 2.31-maintenance] test: add shebangs to shell.nix test scripts #14942

[internal-nix-ci]

Successfully created backport PR for 2.32-maintenance:

• [Backport 2.32-maintenance] test: add shebangs to shell.nix test scripts #14943