Skip to content

Update mkCacheFile and mkCacheModule to support private repos with builtins.fetchGit#348

Merged
angerman merged 1 commit intoinput-output-hk:masterfrom
cdepillabout:mkCacheFile-fetchGit
Dec 10, 2019
Merged

Update mkCacheFile and mkCacheModule to support private repos with builtins.fetchGit#348
angerman merged 1 commit intoinput-output-hk:masterfrom
cdepillabout:mkCacheFile-fetchGit

Conversation

@cdepillabout
Copy link
Contributor

@cdepillabout cdepillabout commented Dec 4, 2019

This updates the mkCacheFile and mkCacheModule functions to support caches that are pulling from private repos with builtins.fetchGit.

At work (@xc-jp), we have a big stack-based project that has a few private libraries in the stack.yaml file.

In order to use private libraries with nix, we need to use builtins.fetchGit (so that the library is fetched at eval time with the user's SSH_AUTH stuff).

This PR modifies the mkCacheFile and mkCacheModule functions to take an is-private argument for the individual cache definitions. If is-private is true, then builtins.fetchGit is used instead of the normal pkgs.fetchgit.

buitins.fetchGit has a few differences from pkgs.fetchgit (namely that you don't have to specify a sha256), so there are a few differences in the arguments for the caches depending on whether you are using a private repo vs. a non-private repo.

It doesn't look like there are any tests for the cache stuff, so I didn't add any tests for my changes, but I could always go back and add tests if that would be necessary.


This is related to #309, although I don't think it fully fixes it.

Comment on lines +243 to +247
# Non-private repos must have sha256 set.
assert repo ? sha256;
# pkgs.fetchgit doesn't have any way of fetching from a given
# ref.
assert !(repo ? ref);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be nicer to have these assertions return an understandable error message, but for now they should be fine.

Comment on lines +259 to +260
sha256String = if isNull sha256 then self.buildPackages.lib.fakeSha256 else sha256;
in "${url} ${rev} ${subdir} ${sha256String} ${name} ${nix-expr}";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be better to add some sort of actual format (maybe json based?) for this cache line that stack-to-nix knows how to parse.

The format could be a sum-type, with one constructor for public repos (which use pkgs.fetchgit), and one constructor for private repos (which use builtins.fetchGit).

But I think that would be best for a future PR.

@cdepillabout
Copy link
Contributor Author

cdepillabout commented Dec 4, 2019

Here's an example of a cache that has both private and public repos:

let
  caches = [
    # an example of a private repo
    {
      name = "net-mqtt";
      url = "https://github.com/xc-jp/mqtt-hs.git";
      rev = "2c5ffe80cd1d765fd8ff3db2eaa54a98ca0485b2";
      ref = "fix-message-id-0"; # this is a branch
      is-private = true;
    }
    # an example of a normal public repo
    {
      name = "datasets";
      url = "https://github.com/DataHaskell/dh-core.git";
      rev = "78dee2304790fae0e301af17c7512d9c916a9b3f";
      sha256 = "1rwwfc78cgdw3b2mv9ibiphl86m32c8r7zvb2a5saq0zg9vi3f9h";
      subdir = "datasets";
     }
  ];
in ...

In the future, I think it would be possible to write a function to generate this cache file from the stack.yaml file as long as you treat all the git repos as private (since you don't need to specify the sha256 for private repos).

@cdepillabout
Copy link
Contributor Author

This PR had a conflict with #334, but I have fixed it and rebased on top of master.

Copy link
Collaborator

@hamishmack hamishmack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@angerman angerman merged commit 7e996d5 into input-output-hk:master Dec 10, 2019
@cdepillabout cdepillabout deleted the mkCacheFile-fetchGit branch December 10, 2019 03:04
@cdepillabout
Copy link
Contributor Author

Thanks for merging this in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants