Commits on Mar 17, 2016

1. Fix running Cargo concurrently …

   Cargo has historically had no protections against running it concurrently. This
   is pretty unfortunate, however, as it essentially just means that you can only
   run one instance of Cargo at a time **globally on a system**.
   
   An "easy solution" to this would be the use of file locks, except they need to
   be applied judiciously. It'd be a pretty bad experience to just lock the entire
   system globally for Cargo (although it would work), but otherwise Cargo must be
   principled how it accesses the filesystem to ensure that locks are properly
   held. This commit intends to solve all of these problems.
   
   A new utility module is added to cargo, `util::flock`, which contains two types:
   
   * `FileLock` - a locked version of a `File`. This RAII guard will unlock the
     lock on `Drop` and I/O can be performed through this object. The actual
     underlying `Path` can be read from this object as well.
   * `Filesystem` - an unlocked representation of a `Path`. There is no "safe"
     method to access the underlying path without locking a file on the filesystem
     first.
   
   Built on the [fs2] library, these locks use the `flock` system call on Unix and
   `LockFileEx` on Windows. Although file locking on Unix is [documented as not so
   great][unix-bad], but largely only because of NFS, these are just advisory, and
   there's no byte-range locking. These issues don't necessarily plague Cargo,
   however, so we should try to leverage them. On both Windows and Unix the file
   locks are released when the underlying OS handle is closed, which means that
   if the process dies the locks are released.
   
   Cargo has a number of global resources which it now needs to lock, and the
   strategy is done in a fairly straightforward way:
   
   * Each registry's index contains one lock (a dotfile in the index). Updating the
     index requires a read/write lock while reading the index requires a shared
     lock. This should allow each process to ensure a registry update happens while
     not blocking out others for an unnecessarily long time. Additionally any
     number of processes can read the index.
   * When downloading crates, each downloaded crate is individually locked. A lock
     for the downloaded crate implies a lock on the output directory as well.
     Because downloaded crates are immutable, once the downloaded directory exists
     the lock is no longer needed as it won't be modified, so it can be released.
     This granularity of locking allows multiple Cargo instances to download
     dependencies in parallel.
   * Git repositories have separate locks for the database and for the project
     checkout. The datbase and checkout are locked for read/write access when an
     update is performed, and the lock of the checkout is held for the entire
     lifetime of the git source. This is done to ensure that any other Cargo
     processes must wait while we use the git repository. Unfortunately there's
     just not that much parallelism here.
   * Binaries managed by `cargo install` are locked by the local metadata file that
     Cargo manages. This is relatively straightforward.
   * The actual artifact output directory is just globally locked for the entire
     build. It's hypothesized that running Cargo concurrently in *one directory* is
     less of a feature needed rather than running multiple instances of Cargo
     globally (for now at least). It would be possible to have finer grained
     locking here, but that can likely be deferred to a future PR.
   
   So with all of this infrastructure in place, Cargo is now ready to grab some
   locks and ensure that you can call it concurrently anywhere at any time and
   everything always works out as one might expect.
   
   One interesting question, however, is what does Cargo do on contention? On one
   hand Cargo could immediately abort, but this would lead to a pretty poor UI as
   any Cargo process on the system could kick out any other. Instead this PR takes
   a more nuanced approach.
   
   * First, all locks are attempted to be acquired (a "try lock"). If this
     succeeds, we're done.
   * Next, Cargo prints a message to the console that it's going to block waiting
     for a lock. This is done because it's indeterminate how long Cargo will wait
     for the lock to become available, and most long-lasting operations in Cargo
     have a message printed for them.
   * Finally, a blocking acquisition of the lock is issued and we wait for it to
     become available.
   
   So all in all this should help Cargo fix any future concurrency bugs with file
   locking in a principled fashion while also allowing concurrent Cargo processes
   to proceed reasonably across the system.
   
   [fs2]: https://github.com/danburkert/fs2-rs
   [unix-bad]: http://0pointer.de/blog/projects/locking.html
   
   Closes rust-lang#354

   

   alexcrichton committed Mar 17, 2016
   Configuration menu

   • View commit details
   • Copy full SHA for 8eac1d6
   • Browse repository at this point

   Copy the full SHA

   8eac1d6 View commit details

   Browse the repository at this point in the history