Recursive WORKSPACE file loading

[kchodorow]

Design doc: https://bazel-review.googlesource.com/c/6131

[jart]

Drive by review. I love watching the amazing work you do @kchodorow. This seems like a potential solution for #1550.

You're tackling a very difficult problem. Many people have tried to solve this problem and only introduced new ones. One of Bazel's virtues is that it's held off on trying to solve this so far until it can come up with a solution it knows is going to be awesome. I'd like to help you find that solution.

[Diamonds]

The PNG files seems to be broken in https://github.com/bazelbuild/bazel/blob/master/site/designs/_posts/2016-09-19-recursive-ws-parsing.md so it's not clear to me yet how this is going to work.

We always download/link every external dependency before a build can happen.

I want to only pay for the dependencies I use. If I depend on @foo//bar/some/tiny:library, then I don't want to have to download the externals required to build @foo//.... That could include testing dependencies like web browsers, documentation generators, and all this other stuff that could be gigabytes in size, when all I actually wanted was maybe a tiny jar file.

A lot of people have opinions on this topic, as you know. I'm sure you hear user complaints all the time about why Bazel can't solve all their problems without them having to do additional work. Allow me to counteract some of those opinions by saying this. I don't care if Bazel is convenient. I don't care if it's easy. I just care that it's performant, minimal, deterministic, reproducible, and correct (PMDRC). Even if that means I'm configuring my build on the hardest difficulty setting. I like difficult things, as long as I'm reaping significant benefits from that difficulty. This level of high precision professionalism is what sets Bazel apart from many other build systems, which play fast and loose with dependencies. I also believe that once we figure out how to make our builds PMDRC, then we can add abstractions later that make PMDRC easy.

[kchodorow]

Sorry, missed this comment! Thanks for the props, commit in progress to fix the images.

Many people have tried to solve this problem and only introduced new ones.

I've read the internal doc on it :)

I'm not really trying to solve the problem, fwiw. The tl;dr of that design doc is:

1. Use unambiguous transitive external dependencies.
2. Error out fast if there's an ambiguity.
3. Allow overriding dependency conflicts in the top-level WS file.

There's also some cool strictness stuff we'll be doing, like only allowing external dependencies that have been declared in that WS (so @x can't use @y unless it's declared in x's WS file).

We always download/link every external dependency before a build can happen.

We're not doing that!

They'll be downloaded as-needed, as it is today.