Remap target dir

[alepez]

Problem

I'm using ZFS filesystem, which has an instant snapshot feature very useful for disaster recovery. I use this feature to automatically make snapshots of my projects. The snapshot is applied to the whole ZFS dataset and there is no way to exclude certain directories.

The problem with Rust projects is that they may have many files in the target directory (which is usually inside the project directory). File under target directory change often (e.g. when re-building) and can be effortlessly recreated. So they don't fit well with ZFS automatic snapshotting system.

Similar problem exists if someone wants to always use a different disk or partition for target directories (which are frequently changed, effortlessly recoverable, unlike the source code).

I could override CARGO_TARGET_DIR environment variable, but I would need to remember to do it for every every project.

I could create an alias of cargo which automatically overrides CARGO_TARGET_DIR, but it would also override whatever is set as target directory in the config.toml, which can cause unpredictable issues.

Proposed Solution

I propose to add the environmental variable CARGO_TARGET_DIR_REMAP which let the user remap where target dir should be created.

Example: crate is in /home/user/projects/foo and default target directory is in /home/user/projects/foo/target. Setting CARGO_TARGET_DIR_REMAP=/home/user/projects=/tmp/cargo-build should change the real target directory so it becomes /tmp/cargo-build/foo/target

The syntax is inspired by rustc command line argument --remap-path-prefix

Notes

I've already experimented with a possible implementation. This is a simple working patch.

diff --git a/src/cargo/util/config/mod.rs b/src/cargo/util/config/mod.rs
index fddd47acb..5bc0c94ba 100644
--- a/src/cargo/util/config/mod.rs
+++ b/src/cargo/util/config/mod.rs
@@ -518,6 +518,13 @@ impl Config {
             }
 
             Ok(Some(Filesystem::new(path)))
+        } else if let Some(remap) = self.env.get("CARGO_TARGET_DIR_REMAP") {
+            remap_target_dir(remap, self.cwd.clone())
+                .map(|dir| Some(Filesystem::new(dir)))
+                .map_err(|err| anyhow!(
+                    "the target directory remap is not valid in the \
+                    `CARGO_TARGET_DIR_REMAP` environment variable. reason: {}", err)
+                )
         } else {
             Ok(None)
         }
@@ -2459,3 +2466,45 @@ macro_rules! drop_eprint {
         $crate::__shell_print!($config, err, false, $($arg)*)
     );
 }
+
+fn remap_target_dir(remap: &str, mut path: PathBuf) -> CargoResult<PathBuf> {
+    let (from, to) = remap.split_once('=')
+        .ok_or_else(|| anyhow!("must be a value of the form FROM=TO"))?;
+
+    // Change the path only if it starts with the `from` prefix
+    if path.starts_with(from) {
+        let relative = path.strip_prefix(from)?;
+        path = PathBuf::from(to).join(relative);
+    }
+
+    Ok(path)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_remap_target_dir_start() {
+        let original_path = PathBuf::from("/foo/bar/baz");
+        let remap = String::from("/foo/bar=/new/target/prefix");
+        let remapped = remap_target_dir(&remap, original_path);
+        assert_eq!(PathBuf::from("/new/target/prefix/baz"), remapped.unwrap());
+    }
+
+    #[test]
+    fn test_remap_target_dir_different_start_unchanged() {
+        let original_path = PathBuf::from("/baz/foo/bar/baz");
+        let remap = String::from("/foo/bar=/new/target/prefix");
+        let remapped = remap_target_dir(&remap, original_path);
+        assert_eq!(PathBuf::from("/baz/foo/bar/baz"), remapped.unwrap());
+    }
+
+    #[test]
+    fn test_remap_target_dir_invalid_syntax() {
+        let original_path = PathBuf::from("/baz/foo/bar/baz");
+        let remap = String::from("/new/target/prefix");
+        let remapped = remap_target_dir(&remap, original_path);
+        assert!(remapped.is_err());
+    }
+}
\ No newline at end of file

[epage]

More use cases:

• I work at a place that does continual, whole-system, unconfigurable backup and making it easy to move the target dir out of the backup would help to avoid wasted processing from the backup
• I used to work at a place that checked all run programs against a central server and I had to authorize new programs through slack and then run a command to sync that down. This is incompatible with build scripts and cargo run. We had some exception directories and I had to regularly change my target dir to one of these to even be able to do anything.

I appreciate that this proposal is looking to precedence for a design.

Since --remap-path-prefix is a repeatable option, we would need to specify what a separator character is for the env variable (most likely the same used in PATH). I'm assuming we'd also want a config field for this.

[alepez]

Being a repeatable option can be useful. I don't know which is the best delimiter in this case. Using : as in PATH is a good idea, but maybe we already have a feature in the Rust ecosystem which is using a different delimiter?

The config field makes sense. In that case I would use the same logic already implemented for the target option.

target_dir_remap = "a=b"
target_dir_remap = ["a=b"]
target_dir_remap = ["a=b", "c=d"]

[alepez]

I'm not sure what should be the correct interaction with CARGO_TARGET_DIR env var and target_dir config.

CARGO_TARGET_DIR overrides CARGO_TARGET_DIR_REMAP

Example:

cwd=/a/b
CARGO_TARGET_DIR=c
CARGO_TARGET_DIR_REMAP=/a=/x

Result target dir is /a/b/c

CARGO_TARGET_DIR works together with CARGO_TARGET_DIR_REMAP

Example:

cwd=/a/b
CARGO_TARGET_DIR=c
CARGO_TARGET_DIR_REMAP=/a=/x

Result target dir is /x/b/c

[alepez]

And how the env var and the config field interact? One could override the other, or they could be merged.

[lorenz]

I'd personally like to see Cargo adopt an approach similar to what Bazel does, i.e. keep its cache under XDG_CACHE_DIR/cargo-whatever/... and make target a symlink to a cache directory under there. This has multiple advantages:

• It doesn't break most workflows as accesses through target are still possible. Only tools explicitly resolving the symlink might behave differently.
• It can work without configuration and still have significant benefits, i.e. most backup utilities avoid XDG_CACHE_DIR out of the box.
• It allows users to centrally customize where to put all their cache data (by modifying XDG_CACHE_DIR).

WDYT?

[epage]

I'd personally like to see Cargo adopt an approach similar to what Bazel does, i.e. keep its cache under XDG_CACHE_DIR/cargo-whatever/... and make target a symlink to a cache directory under there. This has multiple advantages:

How do we maintain a unique location within XDG_CACHE_DIR for each package directory?

It allows users to centrally customize where to put all their cache data (by modifying XDG_CACHE_DIR).

• It seems like this would help the ZFS case.
• Its "meh" for the use case I mentioned as i wouldn't be wanting to put all caches into the trusted directory
• This is disruptive to the ecosystem as people and tools that are used to operate directly on targetwill need to keep a symlink in mind. Whether we would want to consider this a major or a minor breaking change, I'm unsure.

[lorenz]

How do we maintain a unique location within XDG_CACHE_DIR for each package directory?

I see two approaches:

1. Generate a random string, create a symlink target -> $XDG_CACHE_DIR/cargo/$RANDOM_STRING, then just read the symlink back to figure out what was chosen. This is resilient to moving the source directory around and having multiple views (mount namespaces, symlinks, ...) into it. But cargo would need to check if the used symlink target is actually pointing to $XDG_CACHE_DIR/cargo/... and recreate it if it is not.
2. Hash the path to the Cargo.toml file and use that, with a similar path construction. This means the cache path can just be computed from the path of the Cargo.toml file.

I personally prefer the first option as it allows for the most flexibility at moderate complexity.

Its "meh" for the use case I mentioned as i wouldn't be wanting to put all caches into the trusted directory

True, in that specific case it also depends on how the executable checking is done and if a symlink would count. I don't know enough about a workflow would look like with such a system in place to make any recommendations here.

This is disruptive to the ecosystem as people and tools that are used to operate directly on targetwill need to keep a symlink in mind. Whether we would want to consider this a major or a minor breaking change, I'm unsure.

Most tools should not notice the difference. They would need to explicitly call readlink or similar to be affected. Cargo is responsible for creating target and thus the symlink and everything under it would just work normally.

I've done tests by creating the symlink by hand and everything I tested (cargo build, run, rust-analyzer) at it worked OOTB.

[epage]

True, in that specific case it also depends on how the executable checking is done and if a symlink would count. I don't know enough about a workflow would look like with such a system in place to make any recommendations here.

In my case, the symlink does not count

Most tools should not notice the difference. They would need to explicitly call readlink or similar to be affected. Cargo is responsible for creating target and thus the symlink and everything under it would just work normally.

For myself, I am used to running rm -rf */target. That would no longer do what I expected. We'd need more due-diligence than "probably fine".

[lorenz]

For myself, I am used to running rm -rf */target. That would no longer do what I expected. We'd need more due-diligence than "probably fine".

This would still (mostly) do what you intend (if we go with option 1 for the directory mapping), it would just leave a dangling cache directory.

[epage]

This would still (mostly) do what you intend (if we go with option 1 for the directory mapping), it would just leave a dangling cache directory.

No, I generally do it to recover space from all of my projects when I upgrade cargo. If someone knows about the move, this makes it even easier. The problem is how do we help adjust workflows.

[alepez]

@lorenz proposal would help in my case. But what about users which do not have freedesktop XDG? Or CI pipelines? The behavior would be inconsistent between different operating systems.

And it would be the default behavior, possibly breaking some build systems.

With my proposal, you just need to set

CARGO_TARGET_DIR_REMAP=/=$XDG_CACHE_DIR/cargo

and everything is put inside the xdg cache dir, with the full path of Cargo.toml appended to it (though i like the idea of having an hash of it). And you need to explicitly opt-in for this feature, so it won't break any existing system.

[sunshowers]

I ran into the same issue with btrfs and wrote a wrapper as a quick hack: https://github.com/sunshowers/targo

[epage]

@sunshowers looks like targo takes an approach similar to what @lorenz was proposing, always use a central target root.

Mind explaining your design and trade offs? I'm assuming its too early to get a feel for benefits and pitfalls.

[sunshowers]

Still super early, but, yeah it's very similar to what @lorenz was proposing.

The only real (minor) difference is that it uses $CARGO_HOME/targo rather than the XDG cache dir, which is more likely to be defined across platforms. I'm definitely not wedded to this path or anything, it was just the most convenient one for me.

For directories it uses a base58-encoded blake3 hash of the workspace directory. This is just for initial setup -- there's also a metadata JSON file stored in the directory with a backlinks list.

Execution model

The execution model is that you're supposed to alias cargo to targo wrap-cargo in your environment (remaining arguments are passed through to cargo). Targo locks the targo dir for metadata mangement, does the setup (nuking the old target directory if present which is definitely hacky), and finally unlocks execs the real cargo. I haven't gotten around to Windows yet but there it could just run cargo as a subprocess.

Targo does a bit of interception of cargo's arguments. For now it just processes --manifest-path. Ideally it would also capture --target-dir, CARGO_TARGET_DIR and build.target-dir, but I don't use them and this is purely for my own purposes :) (Honestly I don't even know what targo should do if those are set -- probably needs a bunch of design work.)

Symlinks

Targo uses the same symlink strategy as proposed by @lorenz. Some interesting effects:

• cd target; cd .. doesn't do what it used to. Turns out I do this a lot! Not surprising if you're familiar with symlinks, but definitely not transparent.
• cargo clean doesn't work -- it just removes the symlink. This one I might end up fixing at some point since I do this a lot. It'll just require further interception of Cargo's arguments.

A bind mount would be more transparent but doesn't work on Windows or Mac, and requires root permissions. I'm not sure how Windows junctions manage this.

[sunshowers]

Oh, and I'd love for someone else to be co-maintainer of the project and run with it! What I have is a very basic MVP that works for me, but there's a lot of work that can be done here. I don't expect to have more than a few hours a month to work on it. (Of course, being upstreamed in cargo also works!)

[epage]

Did a quick look and at least cargo sweep respects the target_directory reported by cargo metadata.

[sunshowers]

Yeah, agreed, I should just be able to look at cargo metadata for the information. My concern is more about the design of what circumstances we should switch the target dir to being managed by targo. For example, a target dir within the workspace vs outside it, being able to manually override it, etc.

[epage]

There are several ways that can go

• A config to change the default target dir. CARGO_TARGET_DIR overrides it
• A config to set the root of a shared target dir with CARGO_TARGET_DIR overriding it

Additionally, there is the question of whether we should change the default or not. We could even have a transition where we check if target/ exists and just respect that, whether a symlink or not, until the next cargo clean.

We just need someone to enumerate and weigh the options and create a proposal from among them.

[alepez]

I ran into the same issue with btrfs and wrote a wrapper as a quick hack: https://github.com/sunshowers/targo

Great! I'm giving it a try. It seems promising!

[poliorcetics]

Sorry if that's not the best place for it, I just found the issue and tried to write something in the form of a proposal for it:

Cargo: remapping the target directory

Motivation

1. Not having to find and clean all target/ dirs everywhere
2. Being able to easily exclude a directory from saves (Apple's Time Machine, ZFS snapshots, BRTS, ...)
3. Allows easily having separate directories for Rust-Analyzer and cargo itself, allowing concurrent builds
   (technically already doable with arguments/env vars but CARGO_TARGET_DIR collides all projects)
4. Allows using a different disk, partition or mount point for cargo artifacts
5. Avoids having to set CARGO_TARGET_DIR for every project to get the same effect as proposed here

Proposed solution

Introduce CARGO_TARGET_DIR_REMAP environment variable (and an option in .cargo/config.toml and
cargo build --target-dir-remap CLI flag).

The syntax is inspired by rustc command line argument --remap-path-prefix

Syntax

The syntax would be:

For .cargo/config.toml:

[build] # Or another section, since it also affects `cargo run`
target-dir-remap = [
  "/home/poliorcetics=/caches/cargo_target_dirs/mine",
  "/home/poliorcetics/work=/caches/cargo_target_dirs/work"
]

For the CLI:

cargo build \
  --target-dir-remap "/home/poliorcetics=/caches/cargo_target_dirs/mine" \
  --target-dir-remap "/home/poliorcetics/work=/caches/cargo_target_dirs/work"

For environment variables:

• On Unix-like platforms: CARGO_TARGET_DIR_REMAP=/home/poliorcetics=/caches/cargo_target_dirs/mine:/home/poliorcetics/work=/caches/cargo_target_dirs/work
• On Windows: CARGO_TARGET_DIR_REMAP=C:\Users\poliorcetics=C:\caches\cargo_target_dirs\mine;C:\Users\poliorcetics\work=C:\caches\cargo_target_dirs\work

Note the : vs ; separators. Those are the one used for the PATH variables on those systems, which means user of
said OSes are already familiar with the concept and usage of them in a similar situation.

Behaviour

The most specific (longest matching prefix) path is the one applied, so with the following config:

target-dir-remap = [
  "/home/poliorcetics=/caches/cargo_target_dirs/mine",
  "/home/poliorcetics/work=/caches/cargo_target_dirs/work"
]

/home/poliorcetics/work/work-crate-1/target is remapped to /caches/cargo_target_dir/work/... (... scheme explained below).

Just like CARGO_TARGET_DIR does not create a symlink from target to the pointed-to directory, remapping would not
create a symlink either.

Naming scheme inside the remapped directory

Objectives:

• Is somewhat easily navigable by a human
• Guarantees unicity for different projects
• Gathers crates in a single workspace under the same directory (as is done for a regular workspace already)

Proposed solution:

<remapped-dir> / <dir name> - <blake3 hash of absolute path to main manifest>`

where the "main manifest" is the one for the whole workspace and the "dir name" is the name of the
parent directory to the main manifest.

In the case where the "parent directory" is the root, "dir name" should be left empty and the following dash (-) omitted.
(Alternative: use root-<hash> to keep the form consistent).

Example:

$ cd /home/poliorcetics/work/work-crate-1
$ cargo build --target-dir-remap "/home/poliorcetics/work=/caches/cargo_target_dirs/work"
# ...
$ echo -n "/home/poliorcetics/work/work-crate-1/Cargo.toml" | b3sum # https://crates.io/crates/blake3 for the b3sum utility
e738a37f44b76767aa51b9d5c12ad9077b9c4ad44cc13833ade787867e6f5c3e  -
$ ls -1 /caches/cargo_target_dirs/work/
# ...
work-crate-1-e738a37f44b76767aa51b9d5c12ad9077b9c4ad44cc13833ade787867e6f5c3e
# ...
$ cargo run --target-dir-remap "/home/poliorcetics/work=/caches/cargo_target_dirs/work"
# ...
     Running `/caches/cargo_target_dirs/work/work-crate-1-e738a37f44b76767aa51b9d5c12ad9077b9c4ad44cc13833ade787867e6f5c3e/debug/work-crate-1`

Of course, most user (probably) won't pass --target-dir-remap by CLI, they'll set it either in the environment or in
the config.toml, but I wrote it in full here for clarity.

Interactions with existing cargo features

• CARGO_TARGET_DIR, in all of its form (cli, option, env), would completely override CARGO_TARGET_DIR_REMAP.
• cargo metadata would report the path after remapping (when remapping is applied)
• cargo clean would clean the remapped target directory, not <project>/target, which means if target is already
  here because of some other tool or command, it would not be removed. This is already what happens with CARGO_TARGET_DIR
  today, that would not be new behaviour.

Interactions with external tools

Lots of build tools would probably be very happy with the option to entirely remap target directories to somewhere else,
for example bazel which uses $XDG_CACHE_DIR/bazel/... could put the output of cargo calls in $XDG_CACHE_DIR/bazel/cargo_target_dirs/...
while still retaining different directories for different projects and having a consistent naming scheme to fall back on.

Alternatives considered

Doing nothing

It is already possible today to use CARGO_TARGET_DIR to remap workspaces and projects but this has two problems:

1. If done globally, the CARGO_TARGET_DIR becomes a hodge-podge of every project, which is not often the goal.
2. If done per-project, it is very cumbersome to maintain.

For those reason, this option has not been retained.

Using XDG_CACHE_HOME instead of a cargo-specific env-var

Not all OSes use the XDG convention, notably Windows and macOS (though the latter can be somewhat made to) and it is
very easy to define CARGO_TARGET_DIR_REMAP=${XDG_CACHE_HOME:-~/.cache}/cargo_target_dirs if needed.

Just put the remap inside of .cargo/cache/...

There are already lots of discussion about .cargo and .rustup being home to both cache and config files and why this
is annoying for lots of users. What's more, it would not be as helpful to external build tools, they don't care about
bringing the registry cache in their build directory for example.

Use first or last matching path instead of most specific when remapping

I find those options less intuitive but I have no hard data on what others think about that. As long as the behaviour
is well-defined and well-documented, I don't especially care.

Prior works and discussions

• targo by @sunshowers
• rust-lang/cargo#11156

[epage]

The most specific (longest matching prefix) path is the one applied, so with the following config:

How does rustc command line argument --remap-path-prefix deal with this?

CARGO_TARGET_DIR, in all of its form (cli, option, env), would completely override CARGO_TARGET_DIR_REMAP.

cargo is documented as reading but not writing CARGO_TARGET_DIR. I wonder if we should guarantee that CARGO_TARGET_DIR is set if not set by the user. Granted, I mostly want this to workaround CARGO_TARGET_TMPDIR not being available for test libs. Unsure what other use cases there might be that don't need the OUTDIR or TMPDIR.

<remapped-dir> / <dir name> - <blake3 hash of absolute path to main manifest>

This is mixing proposals and, while I can understand it being here, also feels weird.

What is being mixed

• Hashing is intended for when there is a central cache dir.
• Remapping is when the user explicitly says what directory to use.

Why its helpful: it allows using this proposal to act as a way to get a central cache dir

Why it feels weird: the user explicitly gave a remap directive and we aren't following it.

[poliorcetics]

What is being mixed

• Hashing is intended for when there is a central cache dir.
• Remapping is when the user explicitly says what directory to use.

Ah you're right, what I personally want is a form of cache that does not gathers all the crates and workspaces together like CARGO_TARGET_DIR, I kept the "remap" nomemclature but that's not what it really is.

I wonder if we should guarantee that CARGO_TARGET_DIR is set if not set by the user.

I would very much like that too, and I can't think of a case where that wouldn't be useful: either it set by the user and nothing is done or it is set by cargo itself, meaning runners can rely on it for example.

[poliorcetics]

The most specific (longest matching prefix) path is the one applied, so with the following config:

How does rustc command line argument --remap-path-prefix deal with this?

It selects the last match, which really should be documented, I'll make a PR

[lorenz]

Using XDG_CACHE_HOME instead of a cargo-specific env-var
Not all OSes use the XDG convention, notably Windows and macOS (though the latter can be somewhat made to) and it is very easy to define CARGO_TARGET_DIR_REMAP=${XDG_CACHE_HOME:-~/.cache}/cargo_target_dirs if needed.

True, but other OSs have equivalent mechanisms, sometimes very well integrated with the OS. For example macOS has Library/Caches (see https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html) which the OS even cleans up automatically when running out of space.
Windows has AppData\Local\Temp which is similarly integrated with their disk cleanup utility.
Not saying we should necessarly use these, but they do in fact exist for all (major) operating systems. I'm personally also OK with just putting my cache directory into .cargo/config or similar. But I would prefer it if more software did the right thing OOTB and didn't require me to manually remap directories to conform to XDG or equivalent standards on other operating systems.

[poliorcetics]

Since that variable/config option would not be set by default, how could we make it point to the OS's default ?

With the idea I'm going for, users have to set it themselves, machine by machine, I don't see a need to depend on OS defaults here

[lorenz]

Fair, this primarily benefits users when it is already the default. I guess there is no way to actually change the default in cargo to make it use OS defaults without manual configuration.

[alepez]

Thanks @poliorcetics for your contribution! I especially like the Motivations and Alternatives part. I only feels uncomfortable about the hash part. Like @epage pointed out, there's no need to hash anything, with this option we are just asking to remap, so it feels weird to not be just remapped.

I would also keep the rustc command line argument --remap-path-prefix convention, instead of being too smart and use the most specific path. Precedence to the rightmost item is a convention used by many rustc options.

About the OSs XDG_CACHE_HOME equivalents, I would just ignore them all (xdg included). It is so simple to add an environmental variable when needed, that I don't feel it as a big improvement. And I would absolutely not want the current behavior to be changed. So: keep target inside the source directory, as cargo do now, if CARGO_TARGET_DIR_REMAP is not set.

[epage]

If we want to keep this focused and bike-shed free, we could error on any CARGO_TARGET_DIR_REMAP source or destination that isn't an absolute path. Then things conversations about XDG_CACHE_HOME vs CARGO_HOME go away.

As I mentioned, the hashing and central directory would be part of an alternative proposal for just moving the caches in an opinionated way, like targo. The proposal should have a comparison for this and why one vs the other.

One potential extension of this one is that an empty source path could mean "cache the path" and the empty destination path could mean "central directory", allowing this proposal to be extended into the other proposal over time.

[poliorcetics]

Finally found the time and motivation to write a proper RFC, here it is: rust-lang/rfcs#3371

[epage]

See also #5544

[totikom]

I've implemented a quick hack for this case:

export RUST_BUILD_BASE="$HOME/.cache/rust-builds"
WORKSPACE_ROOT=$(cargo metadata --no-deps --offline 2>/dev/null | jq -r ".workspace_root")
PACKAGE_BASENAME=$(basename $WORKSPACE_ROOT)
TARGET_DIRECTORY=$(cargo metadata --no-deps --offline 2>/dev/null | jq -r ".target_directory")

# Check if target has default location
if [ $TARGET_DIRECTORY == "$WORKSPACE_ROOT/target" ]; then
        # Run cargo with target set to $RUST_BUILD_BASE/$PACKAGE_BASENAME
        CARGO_TARGET_DIR="$RUST_BUILD_BASE/$PACKAGE_BASENAME" cargo $@
else
        # Do nothing, target dir is configured to smth non-default
        cargo $@
fi