linux: don't add HOSTCC to PATH during build by vcunat · Pull Request #34882 · NixOS/nixpkgs

vcunat · 2018-02-12T09:38:31Z

It was taking precedence before stdenv's normal CC, at least in case of non-cross build.

Motivation for this change

Finishing the retpoline workaround, hopefully without breaking cross-compilation.

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

vcunat · 2018-02-12T09:40:43Z

Verified that now it's really effective (non-cross):

$ cat /sys/devices/system/cpu/vulnerabilities/spectre_v2
Mitigation: Full AMD retpoline

And spectre-meltdown-checker.sh includes:

* Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)

GrahamcOfBorg · 2018-02-12T09:46:48Z

Failure on x86_64-darwin (full log)

Partial log (click to expand)

error: while evaluating the attribute 'kernel' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:13072:12:
while evaluating the attribute 'linux_4_9' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:12970:3:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:113:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:12970:15:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:72:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:117:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.9.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:74:12:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:72:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.9.nix:3:1:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:10:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:74:12:
assertion failed at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:48:1

GrahamcOfBorg · 2018-02-12T10:04:14Z

Success on aarch64-linux (full log)

Partial log (click to expand)

wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
cannot find section .dynamic
/nix/store/kf5r9hcs7vrzbhgg0idj7cv8ic3w023p-linux-4.9.80

vcunat · 2018-02-12T10:11:15Z

It also fixes the regressed build of linuxPackages_hardened.dpdk and probably most other failures in that set.

GrahamcOfBorg · 2018-02-12T10:46:43Z

Failure on x86_64-linux (full log)

Partial log (click to expand)

  LD [M]  drivers/net/wireless/realtek/rtlwifi/rtl8723ae/rtl8723ae.o
  LD      drivers/net/wireless/realtek/rtlwifi/rtl8723be/built-in.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8723be/dm.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8723be/fw.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8723be/hw.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8723be/led.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8723be/phy.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8723be/pwrseq.o
building of ‘/nix/store/ms8yzzasgxx56b8149xl216bjb5fr8v3-linux-4.9.80.drv’ timed out after 3600 seconds
error: build of ‘/nix/store/ms8yzzasgxx56b8149xl216bjb5fr8v3-linux-4.9.80.drv’ failed

Ericson2314 · 2018-02-12T16:04:16Z

@vcunat for a non-cross build, buildPackages.stdenv and stdenv should be exactly the same, so what's the problem?

This makes HOSTCC something that doesn't exist for a native build`, which could help, but I'd think is distinct from a path-order problem.

vcunat · 2018-02-12T17:34:24Z

@Ericson2314: the problem is with overridability. We have stdenv parameter to buildLinux that allows you to override it and it's specifically documented for that purpose. I used that to override gcc version (for final executables), but the compiler for "helper" executables shadowed it (silently).

Why doesn't it exist? For me it resolves to working gcc for non-cross build (i.e. "native", right?), and I thought it's the correct version for cross builds also.

Ericson2314 · 2018-02-12T18:22:20Z

the problem is with overridability

Ok makes sense. And I'm warming to your solution, actually :). At first I thought you needed to exploit a native-only special case, and that gave me pause. But actually, you were hampered by a native-only special case, namely that two different compilers were clobbering each other.

Why doesn't it exist?

Nevermind, it does exists but it wouldn't for cross. You'd have to do

"HOSTCC=${buildPackages.stdenv.cc}/bin/${buildPackages.stdenv.cc.targetPrefix}cc"

Ericson2314 · 2018-02-12T18:26:43Z

@vcunat So the thing is, though, it's sketch to not have the depsBuildBuild because then that C compiler's wrapper doesn't have its setup hook run. If the build tools built on the fly need external libraries, problems will ensure.

How about we define HOSTCC= and CC= using absolute path so there's no clobbering ever? FWIW, I already want to do this system-wide, too. nix-shell development is still a bit awkward with the PATH clobbering, but maybe we can fix that separately.

vcunat · 2018-02-12T18:47:33Z

You mean this way ^^ ?

Ericson2314 · 2018-02-12T18:54:19Z

pkgs/os-specific/linux/kernel/manual-config.nix

still need the stdenv.cc.targetPrefix like below, but otherwise good. Thanks!

OK, I assumed stdenv.cc is always prefix-less.

GrahamcOfBorg · 2018-02-12T18:54:49Z

Failure on x86_64-darwin (full log)

Partial log (click to expand)

error: while evaluating the attribute 'kernel' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:13079:12:
while evaluating the attribute 'linux_4_9' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:12977:3:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:113:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:12977:15:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:72:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:117:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.9.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:74:12:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:72:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.9.nix:3:1:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:10:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:74:12:
assertion failed at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:48:1

HOSTCC was taking precedence before stdenv's normal CC, at least in case of non-cross build.

GrahamcOfBorg · 2018-02-12T19:09:22Z

Failure on x86_64-darwin (full log)

Partial log (click to expand)

error: while evaluating the attribute 'kernel' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:13079:12:
while evaluating the attribute 'linux_4_9' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:12977:3:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:113:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:12977:15:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:72:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:117:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.9.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:74:12:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:72:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.9.nix:3:1:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:10:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:74:12:
assertion failed at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:48:1

Ericson2314

I went ahead and just made the one change, so merge whenever @vcunat!

GrahamcOfBorg · 2018-02-12T19:14:48Z

Success on aarch64-linux (full log)

Partial log (click to expand)

wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
wrong ELF type
cannot find section .dynamic
/nix/store/0vnakscc62571jippccpmppfx9jy8y5f-linux-4.9.80

GrahamcOfBorg · 2018-02-12T19:14:48Z

Success on aarch64-linux (full log)

Partial log (click to expand)

these derivations will be built:
  /nix/store/krn974zslhspmdmw1xdxjywlg6547913-linux-4.9.80.drv
waiting for locks or build slots...
/nix/store/0vnakscc62571jippccpmppfx9jy8y5f-linux-4.9.80

GrahamcOfBorg · 2018-02-12T19:54:38Z

Failure on x86_64-linux (full log)

Partial log (click to expand)

  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ce/hw.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ce/led.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ce/phy.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ce/rf.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ce/sw.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ce/table.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ce/trx.o
  LD [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ce/rtl8192ce.o
building of ‘/nix/store/asnxg3a5v33733lf7884xpxi0nlplz5g-linux-4.9.80.drv’ timed out after 3600 seconds
error: build of ‘/nix/store/asnxg3a5v33733lf7884xpxi0nlplz5g-linux-4.9.80.drv’ failed

vcunat · 2018-02-12T19:55:00Z

Uh, I accidentally re-pushed again, deleting your comment in the file, but otherwise the contents should be the same. Feel free to overwrite it again :-) (GitHub seems to have lost the commit immediately.) I have a setup that force-mirrors all branches that match a pattern, and sometimes this bites me.

Ericson2314 · 2018-02-12T19:59:38Z

No worries, easy to force push.

GrahamcOfBorg · 2018-02-12T20:16:17Z

Success on aarch64-linux (full log)

Partial log (click to expand)

/nix/store/0vnakscc62571jippccpmppfx9jy8y5f-linux-4.9.80

GrahamcOfBorg · 2018-02-12T20:16:37Z

Failure on x86_64-darwin (full log)

Partial log (click to expand)

error: while evaluating the attribute 'kernel' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:13079:12:
while evaluating the attribute 'linux_4_9' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:12977:3:
while evaluating 'callPackageWith' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:113:35, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/top-level/all-packages.nix:12977:15:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:72:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:117:8:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.9.nix:1:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:74:12:
while evaluating 'makeOverridable' at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:72:24, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/linux-4.9.nix:3:1:
while evaluating anonymous function at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:10:1, called from /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/lib/customisation.nix:74:12:
assertion failed at /private/var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/builder/lnl7-mac/pkgs/os-specific/linux/kernel/generic.nix:48:1

GrahamcOfBorg · 2018-02-12T20:54:54Z

Failure on x86_64-linux (full log)

Partial log (click to expand)

  CC [M]  drivers/net/wireless/intersil/p54/fwio.o
  CC [M]  drivers/net/wireless/intersil/p54/txrx.o
  CC [M]  drivers/net/wireless/intersil/p54/main.o
  CC [M]  drivers/net/wireless/intersil/p54/led.o
  LD [M]  drivers/net/wireless/intersil/p54/p54common.o
  CC [M]  drivers/net/wireless/intersil/p54/p54usb.o
  CC [M]  drivers/net/wireless/intersil/p54/p54pci.o
  CC [M]  drivers/net/wireless/intersil/p54/p54spi.o
building of ‘/nix/store/asnxg3a5v33733lf7884xpxi0nlplz5g-linux-4.9.80.drv’ timed out after 3600 seconds
error: build of ‘/nix/store/asnxg3a5v33733lf7884xpxi0nlplz5g-linux-4.9.80.drv’ failed

This is needed to really build linux with gcc7 after ae04052.

GrahamcOfBorg · 2018-02-12T22:04:38Z

Failure on x86_64-linux (full log)

Partial log (click to expand)

  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192de/table.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192de/trx.o
  LD [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192de/rtl8192de.o
  LD      drivers/net/wireless/realtek/rtlwifi/rtl8192ee/built-in.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ee/dm.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ee/fw.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ee/hw.o
  CC [M]  drivers/net/wireless/realtek/rtlwifi/rtl8192ee/led.o
building of ‘/nix/store/asnxg3a5v33733lf7884xpxi0nlplz5g-linux-4.9.80.drv’ timed out after 3600 seconds
error: build of ‘/nix/store/asnxg3a5v33733lf7884xpxi0nlplz5g-linux-4.9.80.drv’ failed

vcunat added the 1.severity: security Issues which raise a security issue, or PRs that fix one label Feb 12, 2018

vcunat mentioned this pull request Feb 12, 2018

kernel: fix cross compilation #33813

Merged

8 tasks

GrahamcOfBorg added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. labels Feb 12, 2018

vcunat mentioned this pull request Feb 12, 2018

Compile kernel with retpoline GCC support #34383

Closed

vcunat force-pushed the p/retpoline branch from 05ad621 to 36d525c Compare February 12, 2018 18:46

Ericson2314 approved these changes Feb 12, 2018

View reviewed changes

vcunat force-pushed the p/retpoline branch from 36d525c to 8070efb Compare February 12, 2018 18:58

linux: use absolute paths for compilers

3d24c5e

HOSTCC was taking precedence before stdenv's normal CC, at least in case of non-cross build.

Ericson2314 force-pushed the p/retpoline branch from 8070efb to 3d24c5e Compare February 12, 2018 19:01

Ericson2314 approved these changes Feb 12, 2018

View reviewed changes

vcunat force-pushed the p/retpoline branch from 3d24c5e to 8070efb Compare February 12, 2018 19:50

Ericson2314 force-pushed the p/retpoline branch from 8070efb to 3d24c5e Compare February 12, 2018 19:59

vcunat merged commit 3d24c5e into NixOS:master Feb 12, 2018

vcunat added a commit that referenced this pull request Feb 12, 2018

Merge #34882: linux: use absolute paths for compilers

5e62091

This is needed to really build linux with gcc7 after ae04052.

vcunat deleted the p/retpoline branch February 12, 2018 21:27

vcunat mentioned this pull request Apr 8, 2018

How to override the default compiler #38552

Closed

Uh oh!

Conversation

vcunat commented Feb 12, 2018

Motivation for this change

Things done

Uh oh!

vcunat commented Feb 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

vcunat commented Feb 12, 2018

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

Ericson2314 commented Feb 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vcunat commented Feb 12, 2018

Uh oh!

Ericson2314 commented Feb 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ericson2314 commented Feb 12, 2018

Uh oh!

vcunat commented Feb 12, 2018

Uh oh!

Ericson2314 Feb 12, 2018

Choose a reason for hiding this comment

Uh oh!

vcunat Feb 12, 2018

Choose a reason for hiding this comment

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

Ericson2314 left a comment

Choose a reason for hiding this comment

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

vcunat commented Feb 12, 2018

Uh oh!

Ericson2314 commented Feb 12, 2018

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

GrahamcOfBorg commented Feb 12, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vcunat commented Feb 12, 2018 •

edited

Loading

Ericson2314 commented Feb 12, 2018 •

edited

Loading

Ericson2314 commented Feb 12, 2018 •

edited

Loading