resolving dependency confusion: views, scopes, canonical gem URIs, and gem author verification

[duckinator]

Alright, time to try to turn a monster of a Slack thread into something actionable. Let's do this.

Based on that discussion, this is the list of tools I'm interested in for fighting dependency confusion.

1. scopes: a common name grouping a collection of gems. In gem name references, they're prefixed with an @.
   • Example: I could have the @duckinator scope, and maybe have @duckinator/boop and @duckinator/jim.
   • PURPOSE: gem names no longer have to be unique across a given server.
2. canonical gem URIs: treating the gem server you upload to as the "canonical" host, the canonical URI would be ${CANONICAL_HOST}/${SCOPE}/${GEMNAME}.
   • Example: If I upload jim in the @duckinator scope to gem.coop, the canonical gem URI would be https://gem.coop/@duckinator/jim
   • PURPOSE: scopes no longer have to be unique across all gem servers.
3. views: using alternative URLs to change what collections of gems RubyGems and Bundler can "see".
   • Idea: If gem.coop does not populate the unscoped namespace by default, we can use views to populate it.
   • Example: https://some-company.gem.coop could only allow access to gems some-company has authorized.
   • Example: https://rgo.gem.coop could show only things mirrored from RubyGems.org.
   • PURPOSE: Provides a more formal mechanism for private mirrors that override certain gem names.
4. gem author verification: I don't know anything about this, I defer to more knowledgeable folks. ❤️

I have a proof of concept adding Bundler support for scopes and canonical gem URIs. GitHub Packages has "namespaces" that are just a GitHub username or organization name, and it works very similar to my idea for scopes. However, since their "namespaces" don't have a leading @, my proof-of-concept has a kludge for the GitHub Packages hosts specifically. This means it can be tested in the real world, against a production-quality server, right now.

---

I think my rationale is best explained with (this slightly-cleaned-up version of) something I said earlier today:

I think we should be taking this as an opportunity to decentralize, not create a new Pillar Of Truth™️.
I'd rather see a larger number of smaller gem servers, and make our tooling support that idea.
Especially since to my knowledge we have never, at any point, had just one gem server.

Gemspecs are inherently lying to you by claiming there's only One Source Of Truth.

Or, to put it another way: We always had multiple gem servers in the wild. Starting gem.coop didn't create new problems, it highlighted ones people have known about for over 20 years. We should use this as an opportunity to address those problems.

[duckinator]

Here's a screenshot showing how my Bundler proof-of-concept works in practice:

[andrew]

One of the remaining pieces that needs to be solved to make this fully work is that dependencies in *.gemspec files don't support sources or namespaces. They only have simple gem names ( spec.add_dependency "rails"), so it's hard to know which source a dependency should come from when there are multiple options.

This problem shows up a lot in the Maven world, where maven clients will run into a name being available from two configured repositories and it doesn't know which one to pick.

It's not such a big deal for direct dependencies as they can be driven from Gemfile, but from transitive dependencies that may be in conflict when required from two different sources at once.

Gemfile
├── gem "foo" (from rubygems.org)
│   └── depends on "shared"  <-- which source?
│
└── gem "@duckinator/bar" (from gem.coop)
    └── depends on "shared"  <-- which source?

Problem: gemspec just says add_dependency "shared"
No way to know if it should fetch from rubygems.org or gem.coop

This isn't so much of a problem in npm because you can have multiple versions of the same package in a dependency tree (first class functions in the node.js runtime allow for it, they don't have one single module namespace), but in ruby you'd end up with module already loaded.

One way around this would be for @duckinator/bar to ensure all of its transitive dependencies are mirrored into the @duckinator namespace, but that's a lot of duplication and management for users.

[pboling]

Perhaps this would work?

Specify a default gem source used to install gems, e.g.,

➜  cat ~/.gemrc 
---
:sources:
- "https://gem.coop"

Then

Gemfile
├── source "rubygems.org"
│   └── gem "foo" (from rubygems.org)
│       └── depends on "shared"  <-- from locally canonical transitive dependency source "https://gem.coop" in ~/.gemrc
│
├── source "gem.coop"
│   └── gem "@duckinator/bar" (from gem.coop)
│       └── depends on "shared"  <-- from locally canonical transitive dependency source "https://gem.coop" in ~/.gemrc

It would just need to treat the sources as an ordered array, and use the first one in the list as the default for transitive dependencies.

That doesn't do us any favors for the goal of federated distributed servers though. 😢

And to that end, perhaps we need to have a "satisfies" macro, such that a gem published into a scope can indicate that it "satisfies" a root gem name.

Gem::Specification.new do |spec|
  spec.name "shared" # published to gem.coop as "@duckinator/shared"
  spec.satisfies "shared"
  # ...
end

This places a premium on the root namespace, but I think that may be unavoidable for backwards compatibility.

If we did have a satisfies feature as above, then this is possible:

Gem::Specification.new do |spec|
  spec.name "bread" # published to gem.coop as "@duckinator/bread"
  spec.satisfies "shared"
  # ...
end

and

Gemfile
├── source "rubygems.org"
│   └── gem "foo" (from rubygems.org)
│       └── depends on "shared"  <-- installs / uses bread gem from gem.coop
│
├── source "gem.coop"
│   └── gem "@duckinator/bar" (from gem.coop)
│       └── depends on "bread"  <-- installs / uses bread gem from gem.coop

[SleeplessByte]

Debian allows arbitrary number of sources (and in fact we almost always use Debian + APT to source our gems instead of RubyGems; but do note that because Bundler does not want to support this wholly, this only works if all your gems are locally vendored or system gems), and they use a very simple method for resolution if multiple sources provide the same package:

1. First look in preferences,
2. then do it "on file order" (aka alphabetically).

The Ruby equivalent would be "in order of declaring the source".
Perhaps that could be used for the issue described by Andrew.