Statically linked executable

[alandefreitas]

Because of how time-consuming it is to build mrdocs, the way mrdocs is supposed to be used is with scripts that download portable executables and run the tool. These scripts will usually be run in CI on some ubuntu-X container. So we need to provide portable executables people can use.

The problem is we're using recent versions of ubuntu (ubuntu 23.04 now) because of the C++ features we need but users (including the boost release scripts) are usually using earlier versions of ubuntu (usually from 18.04 to 22.04) for a variety of reasons. And binaries built on ubuntu X (say 23.04) don't work under ubuntu <X (say 20.04).

I read about the solutions people proposed in #boost and wrote this summary of alternatives to the problem. I'm initially tempted to experiment with musl and a system libs wrapper.

Most solutions above would require us to rebuild all dependencies with -static/musl/cosmopolitan/-ffreestanding and might end up depending on other tools. Each solution is also quite complex and with lots of caveats. There's no way to know if it's going to work before testing.

1. Some people reported -static/-static-libgcc worked sometimes
   • It's also the first thing I tried but I couldn't get it to work (more details in the PR)
     • Compiling with -static gives us an error when trying to link nlibtinfo.so
     • -static-libgcc still asks for GLIBC_2.36 in ubuntu 20.04 when looking for libc.so.6
   • Another example is https://www.github.com/muttleyxd/clang-tools-static-binaries
     • It uses -static-libgcc and builds LLVM with -DDBUILD_SHARED_LIBS=OFF -DLLVM_BUILD_STATIC=ON
     • I'm not sure -static-libgcc really works there or if it just works because they're compiling the tools with ubuntu-20.04 which should have no conflicts with later versions of ubuntu anyway
2. Use musl
   • It's a more static-friendly C standard library implementation
   • Smaller than glibc
   • It's what the hdoc guys use
   • Documentation is not great
   • More references:
     • https://github.com/georgy7/build-with-musl
     • https://blog.gibson.sh/2017/11/26/creating-portable-linux-binaries/#some-general-suggestions
3. Use cosmopolitan
   • Also a libc worth trying
   • I haven't found examples using C++ yet
4. A wrapper overriding system libs
   • https://blog.gibson.sh/2017/11/26/creating-portable-linux-binaries/#a-wrapper-that-selectively-overrides-system-libs
5. GCC -ffreestanding compile mode
   • The standard libraries are limited. Only these headers can be used: https://stackoverflow.com/a/23259112
   • We probably won't be even able to compile the dependencies.
6. A combination of Mrdocs downgrading ubuntu and release-tools upgrading ubuntu
   • Would it be even possible to use ubuntu-22.04 in the release-tools docker container?
     • There's a PR (CircleCI: update image boostorg/boost#846) in release-tools already
     • The biggest limitation I remember is a weird version of sphinx we needed
7. ApBuild: a gcc wrapper that enforces GLIB_C version and does few other binary compatibility tricks
   • CMake seems to use it
   • References:
     • https://github.com/DeaDBeeF-Player/apbuild?tab=readme-ov-file
     • https://stackoverflow.com/a/1523096
     • https://web.mit.edu/andersk/arch/share/bin/apgcc

[alandefreitas]

Update:

1. ❌ -static-libgcc won't work for us because libc is part of the kernel.
2. ❌musl won't work with the C++ version we need. The latest C++/musl toolchain is GCC 11.
3. ❌Cosmopolitan is very experimental and attempts to be much more general than what we need
4. A wrapper is still an option
5. ❌ffreestanding won't work for us because we need access to all C++ features
6. ❌Mrdocs downgrading ubuntu is still an option but it's very hard to achieve (I tried to achieve this for weeks in ci: create release for Ubuntu 20.04 #562)
7. ❌ApBuild is not being maintained

So we're left with 4) the wrapper and 6) adjusting Ubuntu versions.

I'll first experiment with the possibility of installing a recent version of Clang and libc++ on Ubuntu 20.04 and building MrDocs this way. MrDocs would use libc++. If that works, we can use that as the official Linux release, which will work on Ubuntu >20.04. This should solve the problem with Boost release-tools.

[alandefreitas]

As described in #562 (comment), solution 6 (Mrdocs downgrading ubuntu) didn't work very well: it's overly complex, hard to maintain, slow to test, not general enough, and it just failed to work properly.

Considering these limitations, the next idea I'll explore to solve #547 is solution (4) a wrapper overriding system libs. This strategy is somewhat similar to what APBuild does but it's manual and more stable. It's similar to how CMake provides its binaries. It basically consists of patching the release so that:

• The libraries required by MrDocs are identified and bundled with the MrDocs release in subdirectories of lib
• bin/mrdocs is renamed mrdocs.real (or something to this end)
• A C wrapper is compiled to replace the original bin/mrdocs
  • This wrapper adjusts the library paths and calls mrdocs.real is compiled with a very old version of GCC and Ubuntu
  • The wrapper can be compiled once or in CI

This has the following benefits:

• This wrapper also seemed like a complex solution at first but it turns out to be much simpler and more effective than the other solutions we tried.
• It's easy to maintain. We can use the same logic and the same wrapper no matter what version of Ubuntu we originally used to compile MrDocs. This also means we have more freedom to upgrade the MrDocs toolchain and use more recent C++ standards.
• Testing is much easier because there's no specific workflow for these binaries. This is a patched version of the same binaries we've been producing. Also, it involves no custom-built compilers.
• It's a much more general solution. We can build MrDocs on Ubuntu 23.04 and use the release binaries on Ubuntu 16.04 if we have to.