Skip to content

Comments

nix/sources.json: upgrade nixpkgs to the latest master#1189

Merged
mergify[bot] merged 1 commit intomasterfrom
basvandijk/nixpkgs-upgrade-2020-02-10
Feb 13, 2020
Merged

nix/sources.json: upgrade nixpkgs to the latest master#1189
mergify[bot] merged 1 commit intomasterfrom
basvandijk/nixpkgs-upgrade-2020-02-10

Conversation

@basvandijk
Copy link
Contributor

@basvandijk basvandijk commented Feb 10, 2020

This upgrades nixpkgs to DFINITY's fork of nixpkgs which tracks the
master branch. There're only two patches on top of master:

commit a4e23f32f8d7006e217125a9123b53597b9c0c24 (HEAD -> dfinity-motoko, dfinity-lab/dfinity-motoko)
Author: Bas van Dijk <v.dijk.bas@gmail.com>
Date:   Wed Feb 12 16:08:05 2020 +0100

    expect: remove autoreconfHook since that causes a wrong path to stty

    See: https://github.com/NixOS/nixpkgs/issues/79863

commit f359aea057db81c51e55d7250283a0b2201bd6b6
Author: Bas van Dijk <v.dijk.bas@gmail.com>
Date:	Thu Feb 6 19:14:09 2020 +0100

    libuv: 1.34.1 -> 1.34.2

    (cherry picked from commit 21ad5c123b8e76ab241b89acbc8db24925244d63)

We need these patches in order to build motoko without using the
upstream cache.nixos.org. It would be good not to use an external
cache to decrease the chance an attacker can infect the cache with a
trojan and to ensure DFINITY can always build their own code without
depending on an external cache.

@nomeata
Copy link
Contributor

nomeata commented Feb 10, 2020

Would we track upstream master again once they merge the cherry-pickes patch?

@basvandijk
Copy link
Contributor Author

Would we track upstream master again once they merge the cherry-pickes patch?

Yes we can. The patch is currently in staging because it causes a mass-rebuild.

I won't merge this yet because I'm still testing this out on https://zh-hydra-int.dfinity.systems which has the upstream cache.nixos.org disabled.

@basvandijk basvandijk force-pushed the basvandijk/nixpkgs-upgrade-2020-02-10 branch 5 times, most recently from 1d95e0c to d334dee Compare February 11, 2020 19:45
@basvandijk
Copy link
Contributor Author

basvandijk commented Feb 12, 2020

This is currently blocked on the pkgsMusl.libffi failure: NixOS/nixpkgs#79863.

EDIT: this has been worked around.

@basvandijk basvandijk force-pushed the basvandijk/nixpkgs-upgrade-2020-02-10 branch from d334dee to 7cf546c Compare February 12, 2020 15:10
@basvandijk
Copy link
Contributor Author

@nomeata @ggreif I probably need a bit of help fixing the last remaining 2 bugs in the test suite:

  • tests.ld failing with:
--- representative.linked.wat (expected)
+++ representative.linked.wat (actual)
@@ -28,4 +28,4 @@
   (table (;0;) 0 0 funcref)
   (memory (;0;) 2)
   (global (;0;) i32 (i32.const 65536))
-  (start 8))
+  (start $link_start))
  • tests.run-drun failing with:
--- caller.drun-run (expected)
+++ caller.drun-run (actual)
@@ -4,7 +4,7 @@
 ingress Completed: Canister: Payload: 0x4449444c0000
 ingress Completed: Canister: Payload: 0x4449444c0000
 Ok: Payload: 0x4449444c0000
-ingress Completed: Canister: Payload: 0x4449444c016d7b010021000000000000000000000000000000000000000000000000000000000000000000
-Ok: Payload: 0x4449444c0001798d3a7c7a
-Ok: Payload: 0x4449444c00017d21
-Ok: Payload: 0x4449444c016e7b01000100
+ingress Completed: Canister: Payload: 0x4449444c016d7b0100020101
+Ok: Payload: 0x4449444c0001792813c52f
+Ok: Payload: 0x4449444c00017d02
+Ok: Payload: 0x4449444c016e7b01000101

I guess these are due to the dfinity v0.5.0 upgrade.

@ggreif
Copy link
Contributor

ggreif commented Feb 12, 2020

  • tests.ld failing with:

@basvandijk This is a benign change due to a new wasm2wat version? I have seen (and accepted) the same thing on my DWARF branch.

I'll leave the second for @nomeata.

@ggreif
Copy link
Contributor

ggreif commented Feb 12, 2020

@basvandijk branch gabor/dwarf also has a Nixpkgs bump (though I am referencing my fork). I am happy with this change. Just asking, will 20.03 contain all of this or is that branch curated?

@nomeata
Copy link
Contributor

nomeata commented Feb 13, 2020

Indeed, changes look benign. Just run make accept in tests within a nix-shell to update them (after running make in src and rts – we can do it if you don’t want to get into the pecularities of this build and test system).

@basvandijk basvandijk force-pushed the basvandijk/nixpkgs-upgrade-2020-02-10 branch from 7cf546c to a2a0f2f Compare February 13, 2020 12:47
@basvandijk
Copy link
Contributor Author

@ggreif

will 20.03 contain all of this or is that branch curated?

Yes 20.03 will contain these fixes on our fork of nixpkgs.

@nomeata

Just run make accept

Done.

I'll open it up for review after CI is green.

This upgrades `nixpkgs` to DFINITY's fork of nixpkgs which tracks the
`master` branch. There're only two patches on top of `master`:

```
commit a4e23f32f8d7006e217125a9123b53597b9c0c24 (HEAD -> dfinity-motoko, dfinity-lab/dfinity-motoko)
Author: Bas van Dijk <v.dijk.bas@gmail.com>
Date:   Wed Feb 12 16:08:05 2020 +0100

    expect: remove autoreconfHook since that causes a wrong path to stty

    See: NixOS/nixpkgs#79863

commit f359aea057db81c51e55d7250283a0b2201bd6b6
Author: Bas van Dijk <v.dijk.bas@gmail.com>
Date:	Thu Feb 6 19:14:09 2020 +0100

    libuv: 1.34.1 -> 1.34.2

    (cherry picked from commit 21ad5c123b8e76ab241b89acbc8db24925244d63)
```

We need this patch in order to build `motoko` without using the
upstream `cache.nixos.org`. It would be good not to use an external
cache to decrease the chance an attacker can infect the cache with a
trojan and to ensure DFINITY can always build their own code without
depending on an external cache.
@basvandijk basvandijk force-pushed the basvandijk/nixpkgs-upgrade-2020-02-10 branch from a2a0f2f to 56b8565 Compare February 13, 2020 14:36
@basvandijk basvandijk marked this pull request as ready for review February 13, 2020 14:36
@basvandijk basvandijk requested a review from nomeata February 13, 2020 14:44
@basvandijk basvandijk added the automerge-squash When ready, merge (using squash) label Feb 13, 2020
@mergify mergify bot merged commit 7883956 into master Feb 13, 2020
@mergify mergify bot deleted the basvandijk/nixpkgs-upgrade-2020-02-10 branch February 13, 2020 15:25
@mergify mergify bot removed the automerge-squash When ready, merge (using squash) label Feb 13, 2020
Copy link
Contributor

@ggreif ggreif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haskell-related stuff looks good.

@nomeata
Copy link
Contributor

nomeata commented Feb 13, 2020

Hmm, it seems it now wants to build

building '/nix/store/b0sd8xvaxvbpivf9v574pjs70wlmx1rr-ghc-8.8.1.drv'...

here, which is not found in any cache. I assume this might be because we use cabal2nix, which requires haskell at evaluation phase, but the Hydra setup doesn’t ensure that evaluation-time dependencies are in our nix cache (at the moment, rumors are that the “new” hydra setup will fix this), but we also now pull in a version of nixpkgs that isn’t necessarily on the “official” nix cache? @basvandijk could that be?

@nomeata
Copy link
Contributor

nomeata commented Feb 13, 2020

Maybe we should simply commit the output of cabal2nix to the repo, it’s only four files (see nix/haskell-packages.nix).

nomeata added a commit that referenced this pull request Feb 13, 2020
see #1189 (comment)

My hope is that by adding `cabal2nix` to the build product, it will
reliably be available on our nix cache.
nomeata added a commit that referenced this pull request Feb 20, 2020
in #1189 we switched to a fork of nixpkgs to get two patches not in
nixpkgs `master` at that time. Looks like they have made it to
`nixpkgs/master`, so we can track that again (and make the Motoko repo
more self-contained).
mergify bot pushed a commit that referenced this pull request Feb 21, 2020
in #1189 we switched to a fork of nixpkgs to get two patches not in
nixpkgs `master` at that time. Looks like they have made it to
`nixpkgs/master`, so we can track that again (and make the Motoko repo
more self-contained).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants