cc-wrapper: generating compile_commands.json using cc-wrapper

[danielbarter]

Description of changes

Modifying the cc-wrapper so that it can be used to generate compile_commands.json files. Here is the corresponding package:
https://github.com/danielbarter/mini_compile_commands

Inserts a post-wrapper-hook which the user can provide to insert a hook into the wrapper.

mini-compile-commands provides the function mini-compile-commands.wrap which can be used to add compile command generating functionality to a standard environment:

with (import <nixpkgs> {});
let llvm = llvmPackages_latest;
in (mkShell.override {stdenv = ( mini-compile-commands.wrap llvm.stdenv );}) {
   buildInputs = [ cmake gtest ];
}

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-nix-infrastructure-to-reliably-generate-compile-commands-json/21961/1

[danielbarter]

still tinkering a little with the mini compile commands code. Should have it settled by later today

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/1238

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-nix-infrastructure-to-reliably-generate-compile-commands-json/21961/14

[fricklerhandwerk]

After at first being confused what this is supposed to be about and then reading the linked Discourse thread, this seems like a valuable addition.

Once someone approves technical fitness, please ping me for a review of the user-facing documentation. I can't promise to take a lot of time or be very quick, but I really would like this to be at least reasonably good and fear that otherwise it will be quite useless to the uninitiated.

Sorry if this sounds harsh - I really appreciate your contribution! But I can't even quickly make constructive comments without a few basic preconditions:

• use one sentence per line in markdown, then one can make suggestions directly or ask questions about individual statements.
• link every term you use that is defined in this manual to the respective definition, and link specialist technical terms to authoritative definitions. otherwise users (and reviewers like me, who don't know all the context but can judge if they understand what you try to convey, as a proxy for future users) will be lost immediately and won't be able to make use of your otherwise useful feature
• I seriously don't understand the description of what the thing does, and the example doesn't help. Please run this against someone who knows software development but not the thing you're trying to describe, and answer all questions specific to the problem that the person raises in the manual section. This may be a lot of work. But just as you test your code, you have to test your documentation - before handing it in for review.

Usually I'd jump in as a test subject, but there's only so much I can do in a day, and this topic is too far away from my current priorities. And yet, I would veto merging this PR before documentation is in a state that is worthwhile to discuss in more detail.

[danielbarter]

Thanks for the feedback @fricklerhandwerk! very much appreciated. Will get to work on addressing the issues.

[danielbarter]

OK, I have put each sentence in the docs on a new line, which hopefully will make discussing easier.

I totally agree that currently, I am not doing a fantastic job explaining. I think the main issue is that there are two orthogonal things happening here, and I am conflating them.

1. Adding post-wrapper-hook.sh into the compiler wrapper. This allows users to inject their own code into the wrapper.
2. mini-compile-commands, which is an example application of the post-wrapper-hook.sh. Generating correct compile_commands.json files is currently a little difficult on nixos because of the compiler wrappers. The standard tool people use, bear, struggles to distinguish between the wrapped and unwrapped compiler invocations. Fortunately, given a programmable wrapper, it is very easy to generate a correct compile_commands.json, hence this PR!

I 100% agree that nix documentation is one of its weak spots, and am committed to getting the documentation right in this case! I'll start accumulating commits until we have this resolved so that we don't lose comments. Will squish down at the end.

[fzakaria]

This will make CLion integration with Nix first class right?

[danielbarter]

@fzakaria: I haven't used CLion before, but based on my reading of https://www.jetbrains.com/help/clion/compilation-database.html, I would say yes

[danielbarter]

I was just digging around in the ld-wrapper and noticed this:

nixpkgs/pkgs/build-support/bintools-wrapper/ld-wrapper.sh

Line 259 in 21b14e9

source @out@/nix-support/post-link-hook

which is exactly what I am adding to the cc-wrapper. For the cc-wrapper it does need to go above the compiler invocation though, because of the execs. Also, i should probably change the name post-wrapper-hook to something like cc-wrapper-hook.

Does anyone know why the compilers are execd from cc-wrapper instead of just being called like for the ld-wrapper? I could imagine it is for performance reasons, but I am not sure. If we got rid of the execs, then we could move the hook after the compiler invocation and call it post-compile-hook which would be nice and consistent with what happens in ld-wrapper

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-nix-infrastructure-to-reliably-generate-compile-commands-json/21961/20