bazel: add execlog parser

[teto]

Motivation for this change

https://docs.bazel.build/versions/master/remote-execution-caching-debug.html advises to build the execlog parser to troubleshoot bazel cache issues. I would like to add it to our bazel packaging. So far fails with

Analyzing: 2 targets (28 packages loaded, 323 targets configured)
    currently loading: src/main/protobuf
    Fetching @com_google_protobuf; fetching
WARNING: Download from https://mirror.bazel.build/github.com/protocolbuffers/protobuf/archive/v3.13.0.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Unknown host: mirror.bazel.build
Analyzing: 2 targets (28 packages loaded, 323 targets configured)
    currently loading: src/main/protobuf
WARNING: Download from https://github.com/protocolbuffers/protobuf/archive/v3.13.0.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Unknown host: github.com
Analyzing: 2 targets (28 packages loaded, 323 targets configured)
    currently loading: src/main/protobuf
ERROR: An error occurred during the fetch of repository 'com_google_protobuf':
   Traceback (most recent call last):
        File "/build/bazel_src/tools/build_defs/repo/http.bzl", line 111, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://mirror.bazel.build/github.com/protocolbuffers/protobuf/archive/v3.13.0.tar.gz, https://github.com/protocolbuffers/protobuf/archive/v3.13.0.tar.gz] to /build/tmp.Yby7S1DIZl/.cache/bazel/_bazel_nixbld/96c321f6d4067432c019ff2725c18acb/external/com_google_protobuf/temp12648161410897702521/v3.13.0.tar.gz: Unknown host: github.com

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[SuperSandro2000]

You do not have network access in the build phase. Bazel has probably some phase where it does all the fetching.

[avdv]

The only thing needed is that you add the com_google_protobuf to the srcDepsSet, just like rules_java for example.

Then you only need to fix this error:

cp: cannot stat './bazel-src/bazel-bin/src/tools/execlog/parser': No such file or directory

[teto]

thanks that fixed the build process. On run:

/nix/store/54imygzblxfvirqdwy6pagdykaxjgy8b-bazel-3.7.2/bin/execlog -h                                                                                                                                         
/nix/store/54imygzblxfvirqdwy6pagdykaxjgy8b-bazel-3.7.2/bin/execlog: Cannot locate runfiles directory. (Set $JAVA_RUNFILES to inhibit searching.)

Seems like bin/bazel does a bit of magic upon start so not sure how to accomplish something similar or what's the best strategy there.

[avdv]

The execlog parser is a Java tool, packaged as a jar. The file you copied is just the startup script which tries to run java -jar parser.jar.

You need to copy the jar too and perhaps the script needs to be patched so that it can locate it properly. (Note: the script cannot be renamed since it will look for similarly named .runfiles and its .jar file in the same directory it resides in)

Looking into this a bit more, you probably want to build target src/tools/execlog:parser_deploy.jar which is going to be a fat jar. Then package this jar and the wrapper script together, make a new wrapper script which calls the parser script with the --singlejar flag which you can put into out/bin.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/wild-idea-how-about-splitting-nixpkgs-and-nixos/11487/5

[teto]

I've added a wrapper but reached the limit of my java skill : @avdv Do you know how to solve this error when running execlog ?

Error: Could not find or load main class com.google.devtools.build.execlog.ExecLogParser
Caused by: java.lang.ClassNotFoundException: com.google.devtools.build.execlog.ExecLogParser

[avdv]

I would just add the --singlejar argument to the wrapper:

makeWrapper $out/bin/parser $out/bin/execlog --set JAVABIN '${openjdk}/bin/java' --set JAVA_RUNFILES $out/share/parser.runfiles --add-flags '--singlejar'

and I would not put those files into out/bin, but into out/share and just put the wrapper script into bin.

A further complication seems to be that the parser script wants the runfiles and these are local to the system. Does it run if you simply use java -jar parser_deploy.jar with a log file?

[avdv]

Yes, seems to work just fine with the fat jar:

$ java -jar bazel-bin/src/tools/execlog/parser_deploy.jar --log_path ~/sandbox/haskell-bazel-nix-template/exec-1.log | head
command_args: "/nix/store/ijfv8z733l3kwzigvvilc21hpyqqvvd5-bash/bin/bash"
command_args: "-c"
command_args: "source external/bazel_tools/tools/genrule/genrule-setup.sh; cp xmobar/paths.in bazel-out/k8-fastbuild/bin/xmobar/Paths_xmobar.hs"
environment_variables {
  name: "PATH"
  value: "/home/claudio/.rbenv/shims:/home/claudio/.rbenv/bin:/home/claudio/.jenv/shims:/home/claudio/.rbenv/shims:/home/claudio/.rbenv/bin:/home/claudio/.local/bin:/home/claudio/bin:/home/claudio/.local/bin:/run/wrappers/bin:/home/claudio/.nix-profile/bin:/etc/profiles/per-user/claudio/bin:/nix/var/nix/profiles/default/bin:/run/current-system/sw/bin:/home/claudio/bin:/home/claudio/.local/bin:/home/claudio/go/bin:/home/claudio/.gem/ruby/2.6.0/bin:/home/claudio/.npm-packages/bin:/home/claudio/.jenv/bin:/home/claudio/.githubcli/bin:/nix/store/2jysm3dfsgby5sw5jgj43qjrb5v79ms9-bash-4.4-p23/bin:/nix/store/w9wc0d31p4z93cbgxijws03j5s2c4gyf-coreutils-8.31/bin:/nix/store/aja0dimyn0sg5b9zf1cav4k43p8h5xqc-findutils-4.7.0/bin:/nix/store/4z79ipgxqn80ns7mpax25zmb77i3ndfw-gawk-5.1.0/bin:/nix/store/xhvk95cjr2dk339airqxqfk04c0zras6-gnugrep-3.4/bin:/nix/store/1yyn6aar4kw3vjn2fs5xv5jxywdly8wn-gnutar-1.32/bin:/nix/store/mgkwf0c30x2xdw98nh7kj40n83wwv9sf-gnused-4.8/bin:/nix/store/ljjm1r3hn8wmlh5gp38vms13hipjmygy-gzip-1.10/bin:/nix/store/zwms4mg5cf5vmzg9wlskl54fck3643z4-which-2.21/bin:/nix/store/bs0d5iz7isfa8spbp2d4w31sbkkbd6ff-unzip-6.0/bin:/nix/store/y6kf7b0bf91if7i11pk84rxszvbggnqk-file-5.39/bin:/nix/store/jcbs5xa84wk053c7mnhhl2998j97vslw-zip-3.0/bin"
...

[teto]

thanks. should I ditch the wrapper and call the fatjar directly then ?

[avdv]

thanks. should I ditch the wrapper and call the fatjar directly then ?

Yes, only copy the jar and create a simple script in bin calling it with all arguments passed through.

Maybe we should call the script something other than parser though. How about bazel-execlog-parser?!

[teto]

it should work now ! should I put it in bazel_4 as well ?

[SuperSandro2000]

We could use with srcs; here.

[teto]

with srcs; is not a net improvement IMO so I'll let it be.

[SuperSandro2000]

If we you add it in line 45 we can remove all those srcs.

[teto]

I've tried to address reviews.

[SuperSandro2000]

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 111975 run on x86_64-linux 1

10 packages marked as broken and skipped:

• gvisor
• python38Packages.arviz
• python38Packages.dm-sonnet
• python38Packages.graph_nets
• python38Packages.optuna
• python38Packages.pymc3
• python38Packages.rl-coach
• python39Packages.dm-sonnet
• python39Packages.graph_nets
• python39Packages.rl-coach

4 packages failed to build and are new build failures:

• bazel: plain log | pretty log
• (bazel_3): log was empty
• bazel-remote: log was empty
• bazel-watcher: log was empty

The following issues got detected with the above build packages.
Please fix at least the ones listed with your changed packages:

bazel:

warning: attribute-typo
A likely typo in preBuildPhase argument was found, did you mean preBuildPhases or buildPhase?

Near pkgs/development/tools/build-managers/bazel/bazel_3/default.nix:516:3:

    |
516 |   preBuildPhase = ''
    |   ^

See: https://github.com/jtojnar/nixpkgs-hammering/blob/master/explanations/attribute-typo.md

[avdv]

LGTM

[avdv]

it should work now ! should I put it in bazel_4 as well ?

Yes, this would be nice; once we got this sorted out.

[teto]

you mean in a separate commit ? actually I might prefer that to speed up the arrival of this in the cache.

[avdv]

you mean in a separate commit ? actually I might prefer that to speed up the arrival of this in the cache.

Separate commit, yes. But you could do it in this PR I guess. But I have no idea how / when something hits the cache...

[teto]

I've added similar instructions for bazl4. Tested locally and it looked fine.

[SuperSandro2000]

But I have no idea how / when something hits the cache...

When the commit is in a hydra eval set for master it should hit the cache of it builds and had not set hydra platforms to an empty list.

[teto]

@avdv I deem it ready. Could you merge if it's ok for you ?

[avdv]

@avdv I deem it ready. Could you merge if it's ok for you ?

I cannot, I have no merging permissions.

[teto]

haha I could've waited for long then. Thanks for the reviews. Going to merge then :)