cargo install <library_name> --library

[sjmackenzie]

Platform: Nixos

It would be really nice to be able to issue the command cargo install <library name>
Or at least allow for installing of libraries by explicitly appending --library i.e. cargo install <library_name> --library

We currently have an issue, in that a crate and all its dependencies are compiled to a unique folder name, in the form of /nix/store/<hash>-shared_object_name.
Now if we change a single line of the code, the code and all the dependencies will be recompiled and put in a new folder i.e. /nix/store/<new-hash>-shared_object_name.

Thus it makes using crates with large dependencies such as piston, a pain, as each time we change code, the resulting compile time can be over a minute.

Hence I'd like to split out the dependencies of each crate and put them into their own folder i.e. /nix/store/<hash>-dependency1_name and make use of the .cargo/config to override the .toml dependencies to /nix/store/<hash>-dependency1_name, so that they are not recompiled each time I change a line of code in my shared object.

[alexcrichton]

Thanks for the report! I'm a little confused as to what's going on here as I'm not very familiar with nix, however. Are you basically thinking that we could benefit from a crate cache of some form, and that specifically comes up here where everything is in a new directory? Are you using Cargo and somehow just running cargo with a different target directory each time right now?

[sjmackenzie]

The problem I'm trying to solve is improving compile times for Fractalide components.

Ah okay, let me go into more detail:

• nix expressions will call cargo.
• every time nix calls cargo the output is put into a /tmp/xyz folder, then in one of the nix phases (namely the install phase) we copy the output from the /tmp/xyz folder to a folder name stored in a bash variable called $out. $out is created by nix and resolves to a folder inside the nix store. i.e. /nix/store/<hash>-some-name

Basically I'm using nix to replace make. So far it's been a great decision except for this issue of recompiling every dependency every time we make a change to the code.

Now we have created a modular dataflow framework where every component is a Rust shared object with defined inputs and outputs. Each component is a cargo new <name> (not a binary). The framework pulls the shared objects in and orchestrates execution. So in a sense each of these shared objects can be "treated" as a binary. In some senses this could be seen as an alternative to writing bash code... but only after consuming vodka.

As I mentioned earlier, should a component have crates.io deps which are large ie piston then every time we compile a component all the deps are recompiled, thus it takes super long to compile, though if you do not make a change to a previously compiled component, nix registers no change and skips compiling the component. (nix is lazily evaluated)

This is a super niche feature, that "we" as in the greater rust community probably won't benefit much from atm. I created this issue as I'd like to sound this idea to see if it can be incorporated upstream thus I do not have to maintain a specialized version of cargo.

[sjmackenzie]

Here is an example of where we copy the output to the $out folder: https://github.com/fractalide/fractalide/blob/master/build-support/buildFractalideComponent.nix#L109-L114

(I just noticed the indentation on those sections is buggered, I'll fix it shortly.)

[alexcrichton]

Ok, thanks the explanation! Unfortunately right now binary distribution of Rust libraries is pretty hard due to the super-unstable ABI and the ways that Cargo manages dependencies. We haven't currently provided cargo install for libraries because there's actually no way to use them (Cargo doesn't like linking to system libraries).

Perhaps the output directories could be cached between builds and overlaid with each other to produce the final artifact in your build tool? That may have the effect of caching dependencies, but I admit to still not fully understand how the underlying build is working :(

[sjmackenzie]

Alex, appreciate it, I'll find a way to hack this one. Bit of a nasty problem. I also wanted to say, you're doing a really fine job working on Rust and with the community.

[alexcrichton]

Ok, feel free to keep this issue updated with your progress!

[clacke]

Just happened to drop by (I was looking around for a rust package importer for Nix) and will offer my two cents:

In the case of Nix, an unstable ABI is not an issue, as each package instance is hard-coded to depend on a specific binary, with specific build tools and any other input factors. Whether versions differ in major, minor or patchlevel doesn't matter -- bash 4.3.46 compiled with gcc 5.0.1 and glibc 2.23.1 is a different package instance from bash 4.3.46 compiled with gcc 5.0.2 or glibc 2.23.2. The semantic contents of each package is deterministic and reproducible, and the byte contents of a majority of packages is deterministic and reproducible.

The same goes for Guix.

So in the Nix/Guix case, installation of libraries would be worthwhile, and the package manager would handle the distribution.

[sjmackenzie]

Yeah, i'd make use of that rust package importer in a heartbeat for fractalide. We've worked a way around this issue on nix actually, but an importer would mean we don't have to hit crate.io so many times for each component in fractalide. Do let me know if you're going to build one please.

[davidblewett]

My use case for this is to wrap a built library from Python. I would like to issue a cargo install --library rure to get a shared object file that I can then include in a binary Python wheel. The alternative is to clone the regex repo, check out the tag I'm interested in and issue a cargo build inside the regex-capi subdir.

The Python wheel format encodes the platform information that it is building for, so this should allow the end user to pip install and pip will choose the one that matches their platform. My WIP is here: https://github.com/davidblewett/rure-python . This Python package expects the static shared object built for rure, and will embed it in the binary wheel. I've verified that given that, you can install the wheel and call into the shared object using cffi.

[shlevy]

Is there any progress on this? Would love to have this for Nix builds as well.

[shlevy]

FWIW I think GHC, the haskell compiler, has the same concerns around ABI stability, they solve this by essentially requiring the installed libraries to be using the same compiler and settings etc. As mentioned above, for Nix specifically this isn't a problem.