Add --offline option and REBAR_OFFLINE environment variable

[zuiderkwast]

If --offline or REBAR_OFFLINE=1 is given, inets is not started and
all calls to httpc:request/N are prevented. If any online resource
is needed, an error occurs rather than trying to fetch the online
resource.

The purpose is described in #1281 and #2640.

Combined with placing the dependencies in _checkouts, I think it makes it possible to build using dependencies provided manually (which is what's required in the scenario that all dependencies need to be separately reviewed and verified legally and security wise). When combined with #2642, even rebar3 itself can be built in this way.

[ferd]

I don't know that this is actually a good idea or a proper way to make this work.

The original post and issue mentions that ideally the offline mode would need to be done via mirroring of hex packages. This, for example, comes with no tests (how does this work and error out on global plugins that have dependencies?), but also does not cover any of the other resources like git dependencies or custom resources that can be plugin-defined, which are fetched entirely differently.

I understand this is the simplest way to get the thing you need going, but this sounds like a real tricky and difficult thing to maintain in the future and ensure is working fine in the long run. _checkout dependencies were specifically intended to work on deps locally, not as a feature that should just become a de-facto offline mode and vendoring mechanism. For one, it's really bad at that because it moves the dependencies to be top-level apps and re-analyzes them on every build step, but it also removes entries from lockfiles which can lead to weird unexpected behaviours if an app is published using this feature and a non-offline app requires it.

I don't think we should accept this PR moving forwards. Given Rebar3 has to maintain backwards and forwards compatibility for years, this needs to be properly designed, and co-opting unrelated features for a partial (untested) implementation like that is unlikely to reach a mergeable state.

[zuiderkwast]

I wanted to push something forward and see where the discussion leads. I understand that you need to push the breaks. I'd be happy for any clues on a way forward though.

Regarding tests, I'd be happy to add some once we know where we're going.

I'm trying to keep the two things separate: (1) Use local deps over remote ones and (2) prevent fetching online resources. I hope you agree on the distinction.

1. This PR doesn't touch _checkouts. It's just mentioned it as one way to build offline that I can imagine already works. It's a hack for sure. I suppose it's also possible to place the deps directly under _build/*/lib/.

   If _checkouts removes entries in lock files, it seems already quite dangerous if used when publishing. Perhaps it deserves some warning in the docs?

   Do you have an idea of a different offline or vendoring mechanism, such as one that doesn't touch lock files and prevents publishing?

2. Regarding ...

   other resources like git dependencies or custom resources that can be plugin-defined, which are fetched entirely differently

   ... this is a very good point. We'd need to block them as well. Perhaps there is a place to hook in an offline mode check that catches all kinds of resources that I'm unaware of? I'm not very familiar with the rebar3 internals (you may have guessed).

   Running a local hex web server might work but it'd be suboptimal. Besides, we'd sometimes need to override a hex dependency with a slightly different dependency, if for example the version approved by legal/security bureaucracy staff is a tar.gz release from github while the hex package with the same versions is slightly different, the hash in the lockfile woudn't match.

[ferd]

About the local hex index, it's more about re-packaging the local cache (which is by default in ~/.cache/rebar3, containing tarballs and a copy of the hex index) in another directory (say vendored/) with only the deps required so that we wouldn't hit the network even if we could.

This could possibly be done by having a {vendored_deps, false | "local/path"} config value and calling rebar3 vendor to create that "snapshot."

This would also not take care of git dependencies, but I'd prefer a plan going in that direction. It's possible that adding such a feature would require no new resource callbacks (https://rebar3.org/docs/extending/custom_dep_resources/) if that mode changes where we call download() and have a copy operation circumvent the download() callback in normal cases.

I don't know if @tsloughter had different ideas, but if you're serious about putting the implementation time into it, we should have a talk.

[zuiderkwast]

The original issue seems to contain a few different use cases. One of them seems to be for calling rebar3 from mix. Our use case is different.

This could possibly be done by having a {vendored_deps, false | "local/path"} config value and calling rebar3 vendor to create that "snapshot."

I can try to implement this if you want. Is the idea to create a symlink to "local/path" under "_build/*/" then or would it be tar files from hex to be fetched from the vendored directory instead of being downloaded?

What about vendored deps as a dep resource? Since deps are resolved by level-order traversal (breath first), specifying them all as vendored deps in the toplevel rebar.config would override any hex or git sub-dependecies, right? IIUC, the https://github.com/benoitc/rebar3_path_deps plugin already provides this, doesn't it? Example:

{deps, [{cowboy, {path, "vendored/cowboy"}},
        {cowlib, {path, "vendored/cowlib"}}]}.

If this works, it seems much better for our use case. We can't use packages strait from hex or github anyway so I can't see how rebar3 vendor would help us.

The remaining problem is to provide vendored deps to the bootstrapping script.

[ferd]

Yeah the risk of supporting path-based vendoring is figuring out how we handle/error out when you call to a dependency that requires them, and how that bubbles out into the ecosystem.

[zuiderkwast]

I don't see what you mean.

If a dependency is missing, I'll get a build error, look at the logs and see some failed attemt to download something (or if we have an offline mode, some others error message). Then I can file a request to get the missing 3rd party dependency approved (or whatever manual verification) and then make another build.

Or do you mean version mismatch? It can happen already with nested deps.

[ferd]

I don't see what you mean.

If a dependency is missing, I'll get a build error, look at the logs and see some failed attemt to download something (or if we have an offline mode, some others error message). Then I can file a request to get the missing 3rd party dependency approved (or whatever manual verification) and then make another build.

Or do you mean version mismatch? It can happen already with nested deps.

the specific edge case here is to prevent users from doing things like publishing hex packages or even getting to build an app that can't be used as a dep when they publish it on github.

It's something we want to capture to help people properly publish apps aimed to be used by other people and to avoid the whole release cycle of "someone publishes an app that doesn't build for anybody else".
Generally the way Rebar3 is structured, this doesn't really happen unless you mess with checkouts (and we force an unlock to act as a signal there). For another example, the hex plugin prevents publishing packages that rely on private hex repos, and umbrella applications can't be used as dependencies unless they specify a single subdirectory to fetch that represents a standalone app.

The sort of question I'm asking is what do we show the user who tries to use a dep that uses vendoring, or what do we tweak as a behaviour so that someone knows their app making use of vendoring isn't going to work for others.

The scenario I want to avoid is a case where someone goes "to install this dep, install this other dep somewhere and add this one config" and then everyone down the entire call chain ends up having a separate set of garbage localpath deps that prevent the stack from working otherwise. I'd rather instead give rebar3 the ability to go "this dependency you're declaring depends on a localpath/vendored dependency and is therefore not intended to be used as a dependency", and constrain vendoring to be a top-level, final release concern.

[zuiderkwast]

OK, I understand why path-deps can cause trouble to published apps. We only need vendoring on the top-level anyway, which I think it a good restriction. I think we can drop the path-deps idea. It's not better than checkouts anyway (which works in the meantime for everything except bootstrapping).

We also use Go and apparently the dependency and vendoring handling for Go modules works very well for us. I think it can be used as an inspiration. Details: https://golang.org/ref/mod#vendoring. In short:

• go mod vendor downloads all the deps (known as modules) to a vendor directory.
• Once the vendor directory exists, build commands load packages from the vendor directory instead of accessing the network or the module cache (which is used when vendoring is not used).
• Hashes are checked when deps are downloaded, not when they're built.
• Only top-level vendoring, i.e. if there's a vendor directory inside a dependency, it is ignored.

If we design a vendoring system which is similar, it would mean that rebar3 build wouldn't try to download anything if vendoring is enabled, which is the wish from Ericsson that this PR tried to solve.

[KennethL]

If rebar3 differentiate between which dependencies (name and version) and where to fetch the dependencies the rebar.config in published packages could have pointers to hex or github for convenient use but it should only be treated as a hint which could easily be overridden with by the user. The user should not need to edit rebar.config from published packages.
Note I have not read about golang vendoring yet maybe there are more ideas to fetch there.

[tsloughter]

Note that anything can already be overriden with the overrides features.

But also, only git (and the other scm based resources) are tied to a specific place to fetch the dependency (unless overridden or found in _checkouts). Which is a legacy thing.

We specifically designed packages to not have the repository or org (a feature of hex.pm which is really just a repo, but also sort of note a repo...) declared with the dependency in rebar.config so that there is a separation between what are the dependencies (name/version) and where to fetch them. The user is then able to configure where to look for these dependencies with the default being to look at hex.pm if no user defined package repositories are configured.

[zuiderkwast]

@tsloughter Overrides seems to be "meant to allow quick fixes and workarounds". Also _checkouts is meant for temporary hacking AFAIK. It works for us as a temporary solution, but it messes with the lock file and is not intended for vendored deps so it doesn't seem like a real solution.

Besides, _checkouts doesn't work for bootstrapping and I assume overrides doesn't either.

[KennethL]

dependency in rebar.config so that there is a separation between what are the dependencies (name/version) and where to fetch them. The user is then able to configure where to look for these dependencies with the default being to look at hex.pm if no user defined package repositories are configured.

But it is not possible to tell it to look in the local file system. It requires a running server adhering to the hex.pm protocol I assume?

[tsloughter]

@zuiderkwast right. Just giving a full landscape and design goals we had.

[tsloughter]

dependency in rebar.config so that there is a separation between what are the dependencies (name/version) and where to fetch them. The user is then able to configure where to look for these dependencies with the default being to look at hex.pm if no user defined package repositories are configured.

But it is not possible to tell it to look in the local file system. It requires a running server adhering to the hex.pm protocol I assume?

Actually, hm. So first, the idea we had, @ferd correct me if I'm wrong, isn't to have a local server running the protocol, but to have a local index which is where the location of the packages is stored. That index does have to adhere to the hex format, but its not a server, it is just saying that the place to store the location of the local file system copy is in an index file.

But now that I think about it again I do wonder if the resource type couldn't be "overridden". Right now the deps list sees a dep like {atom(), string()} and matches that to the pkg resource type, which has a corresponding module implementing the resource behaviour.

You can already supply your own resource providers (there are a number out there, some for vendoring and pointing to the local filesystem) but that requires manually setting the dep to use them because the match for {atom(), string()} always goes to pkg. But if you could define the precedence and give another resource the control over those deps then you could basically do anything.

That said... I think the local index is likely the best and possibly even less work since so much is kept from the pkg resource functionality. And maybe with the clarification that a server isn't needed makes that more acceptable to you?

[ferd]

My current ideas for vendoring were:

• have a ./_vendor directory that rebar3 expects to control.
  • this directory has one subdirectory per resource type (pkg, git, hg, some plugin, etc.)
• call rebar3 vendor and each resource provider can vendor apps in it (somehow?!)
  • I imagine the hex package resource making a local index (a subset of the ETS table it maintains locally that matches deps) and having the local tarballs, essentially copying the local cache
  • I imagine the git handler copying the directories directly, and possibly making a tarball of them
• each provider is responsible for detecting if deps are outdated in the vendor directory
  • hex does it natively by having a list of hashes of tarballed deps that it knows it can use when fetching and putting them in _build
  • git and hg probably would need to do something similar and maintaining a tarball + hash in a "vendor lock" of some kind
  • a resource type that does not support vendoring warns or fails the whole operation.
• the _vendor directory is expected to be checked into source (hence using tarballs is not a bad idea)
• calling rebar3 vendor --apply or rebar3 vendor --unpack or whatever would be a preliminary step that unpacks all vendored dependencies into _build and then the rest of the application should work without needing to make network calls.
  • If the call to unbundle data isn't made, the app will just hit the network
  • I have no idea how well this would work for profiles and their various combinations.
• none of this works with global plugins, and making it work with local plugins may also be a bit of a challenge since these may try to download themselves before the vendor application can even take place, but they generally don't stall a build.

This is still all sorts of hand-wavy but feels like it's the most simple sort of approach that composes with the rest of features.

[tsloughter]

If it is based on setting the hex index to use it should work with global and local plugins. There isn't an app to load/start it is just a configuration setting to point to a specific hex index.

[max-au]

@potatosalad this feels horribly familiar, how good of an idea it would be to open source your work? (given that we rename "third-party" to ".vendor", or even make it configurable).

@ferd one caveat we ran into is how to do the actual vendoring. So far, IIRC, we create third-party/vendor folder, and within that folder we have sub-folders like argparse-1.1.4, meck-0.9.0, proper-1.4.0 and so on. Every time we update vendored dependency, we fully wipe the folder, creating a new one, e.g. argparse-1.1.5.

We also have hex package index saved in that folder, @potatosalad is the best to explain what's happening there.

[zuiderkwast]

In an EEF meeting ~1 month ago with @ferd, @jhogberg and @KennethL, we concluded that Ericsson's vendoring needs can be fulfilled by putting all deps under lib/, i.e. we don't need another vendoring feature. Apart from that, we wished for this flag/option to fail rather than try to download anything. I will update this PR to hook in to the resource plugin interface and prevent the download callback from being called. That's what was mentioned in the meeting IIRC. Sounds good?

[zuiderkwast]

@ferd (in another PR)

Yeah, #2643 may be more useful, though I'll need to take another look to properly cover network stuff there, to properly prevent all sorts of dep fetches and plugin uses. It's a bit broader than what is currently defined and there may be better hook points for it. Oh and things like hooks and plugins themselves can call the network and can't be covered by any of this either. This is always going to be an imperfect solution compared to permissions work.

I'm replying here. Yeah, it's a best effort. If we cover what's shipped with rebar3 by default, that's good enough, i.e. hex, git and mercurial. I grepped for "http" and "git" and followed the code backwards. Apart from download it's the update callback IIRC.

[KennethL]

@tsloughter it is unclear to me what you mean with:

Note that anything can already be overriden with the overrides features.
Maybe we are misunderstanding each other.
Let me give an example of what I think should work.
In the rebar.config we have a list of dependencies (in this case for rebar3 itself) like this:

{deps, [{erlware_commons,  "1.5.0"},
        {ssl_verify_fun,   "1.1.6"},
        {certifi,          "2.9.0"},
        {providers,        "1.9.0"},
        {getopt,           "1.0.1"},
        {bbmustache,       "1.12.2"},
        {relx,             "4.6.0"},
        {cf,               "0.3.1"},
        {cth_readable,     "1.5.1"},
        {eunit_formatters, "0.5.0"}]}.

By default this syntax of the dependencies means that they are to be fetched from Hex (as I understand it).
Suppose I then want to "override" where to fetch some or all of these dependencies without changing the original rebar.config.
I could for example provide a file called rebar.overrides with the following content:

{location_overrides, [
        {erlware_commons,  {tarfilepath,"./vendored_deps/lib"}},
        {ssl_verify_fun,  {localgitrepo, "/myrepos"}
...

For each dep it is then possible to override the location where it is fetched from. The location can be a tar file fetched from github or a local git repo or many other alternatives. Note that these overrides does not should not be part of the export to hex.
I am just trying to illustrate the principle solution where the user can override where dependencies should be fetched from and doing this without altering the original rebar.config.

I don't know if this is kind of already possible with overrides but I can't read that from the documentation.
Using the rebar.overrides can be deafult if it exist or it can be configurable.
To achieve the offline build requirement the existense of the override file could be interpreted as nothing should be fetched elsewhere than what is specified in this file.

[tsloughter]

I'm not sure if there is an issue of discussing bootstrapping and regular dependencies at the same time. What you describe with the deps example and rebar.overrides file makes me think you may be wanting this for bootstrap?

In case it isn't about bootstrap but about general rebar3 usage then what you describe is basically possibly but not by dropping in a file, it is done by an overrides section in the rebar.config of the project that is including the dependency.

So if you have a project that depends on say erlware_commons which depends on cf you would have in your rebar.config:

{deps, [{erlware_commons, "~> 1.0"}]}.

{overrides, [{override, erlware_commons, [{deps, [{cf,  {tarfilepath,"./vendored_deps/lib"}}]}]}]}.

tarfilepath of course requiring a plugin to work since we don't support a resource like that.

[tsloughter]

I don't think we've discussed having a global option to use whatever is in the code path first? Meaning the user could just set ERL_LIBS to a directory full of all the apps they depend on -- and could do this for bootstrap too.

It is "dangerous" since it won't be verifying anything besides the name and maybe the version, but I guess left up to the user to handle that if they want to tell rebar3 to just use whats on disk.

I still prefer a general vendor solution that will populate an ets table registry cache, then write it to disk, that includes only the dependencies of the project (and only the versions used). This file could then even be committed in the repo of the project.

[potatosalad]

As @max-au mentioned, we implemented something similar to this within WhatsApp by using a custom rebar3 plugin. For context: one of our main goals is to have hermetic or self-contained builds that do not rely on external repositories to reproduce identical builds.

We did this by having our plugin "wrap" or override the builtin rebar3 modules for various resource types with a single third_party_resource module (using rebar_state:set_resources/2 to override modules like rebar_pkg_resource, rebar_git_resource, etc). The rebar.lock file is also modified (during the lock/2 callback) to return a resource type that is different from the one in rebar.config (for example: a pkg resource is turned into a third_party_pkg resource as shown here).

I should note that this was a "get the job done for now" solution, and a general vendoring solution would be preferred.

Similar to what @tsloughter described with an ets cache, we dump the flat file version of the resource into a directory (like third-party/vendor) and write a metadata file (named third_party.config) in the root of each dependency that looks like this. For each file, the type (directory, regular, or symlink) plus the exec bit, file size, checksum/hash, or symlink target are all stored in the metadata which is used to attempt to restore the files when running rebar3 get-deps (for example, if a file has lost its exec bit, or if a directory was accidentally deleted). It also uses the SHA-256 hash of the contents of the file and warns the developer if third-party code has been tampered with. There is also a signature for the <<"protected">> contents of the metadata file itself to further prevent misuse of the vendoring system.

By default, our overriding third_party_resource module forces offline-first mode, so that attempts to fetch newly added dependencies will fail with a message explaining to the developer the steps required to add a new third-party dependency. Once they have completed those steps, running ALLOW_REMOTE=true rebar3 get-deps kicks off the whole flow described above which vendors everything automatically. This flow also works for rebar3 upgrade and other dependency related commands.

Having an "offline-first" option would be a nice-to-have as part of a rebar.config, where external requests could fail with an explanation that offline mode is currently enabled.

Also, as @tsloughter mentioned:

It is "dangerous" since it won't be verifying anything besides the name and maybe the version...

Having some sort of standardized way to vendor dependencies and check for tampering would definitely fit our use-case. My vote would be for something more plain-text-like (similar to rebar.lock) versus an ets cache on-disk, for the main reason that it's not as easily visible during code review.

[zuiderkwast]

I don't think we've discussed having a global option to use whatever is in the code path first? Meaning the user could just set ERL_LIBS to a directory full of all the apps they depend on -- and could do this for bootstrap too.

It is "dangerous" since it won't be verifying anything (...)

@tsloughter It seems you're responding to my post in the bootstrap PR: #2642 (comment). It's hard since they're related.

What if a project has rebar3 as a dependency? Then it may want to override one of rebar3's deps (e.g. erlware_commons) with another versions or even with a vendored one under lib/. Applications placed under apps/ and lib/ do override deps already, whether you think it's dangerous or not.

[tsloughter]

Having some sort of standardized way to vendor dependencies and check for tampering would definitely fit our use-case. My vote would be for something more plain-text-like (similar to rebar.lock) versus an ets cache on-disk, for the main reason that it's not as easily visible during code review.

Good point about plain text. Could be a file of Erlang terms.

[KennethL]

@tsloughter

I'm not sure if there is an issue of discussing bootstrapping and regular dependencies at the same time. What you describe with the deps example and rebar.overrides file makes me think you may be wanting this for bootstrap?

My intention was that the rebar.overrides could be used both for the bootstrap and general dependencies in any application using rebar for its build.

In case it isn't about bootstrap but about general rebar3 usage then what you describe is basically possibly but not by dropping in a file, it is done by an overrides section in the rebar.config of the project that is including the dependency.

So if you have a project that depends on say erlware_commons which depends on cf you would have in your rebar.config:

{deps, [{erlware_commons, "~> 1.0"}]}.

{overrides, [{override, erlware_commons, [{deps, [{cf,  {tarfilepath,"./vendored_deps/lib"}}]}]}]}.

tarfilepath of course requiring a plugin to work since we don't support a resource like that.

I then think it is hard to read the documentation and understand that overrides can be used this way.
After reading some example in the doc I got the impression that the

{overrides, [{override, erlware_commons, [{deps, [{cf,  {tarfilepath,"./vendored_deps/lib"}}]}]}]}.

is overriding what deps erlware_commons has while I just wanted to override where the dependency erlware_commons is fetched from. It would then also be important that the places to fetch from can be local and that the types of resources (tarfilepath, local_git, ...) you can point to are supported by default without need for extra plugins.

Having these type of location overrrides in a special file would assure that a project/package is not exported to hex with local dependencies since the type of resources used as location don't necessarily are supported in the ordinary rebar.config.

I think the cruical point is to see the deps as names and versions of the dependencies and the location to fetch them from is just a hint used as default if nothing else is specified by the user.

Of course I still want an offline flag which makes rebar3 refuse to fetch anything which is not local or explicitly specified in the separate location overrides.

[ferd]

@zuiderkwast we shouldn't override the files in the vendored subdirectory since those are generated. The place where most of the calls can be circumvented are actually in https://github.com/erlang/rebar3/blob/main/src/rebar_fetch.erl since that file is the central point to call all resource downloads and updates.

• The call to download_source should return a failure for being offline
• The call to needs_update should shortcircuit to false

I think that with these, it'll circumvent most possible attempts to call the outside world, including for git deps or plugin-based fetches.

[zuiderkwast]

@zuiderkwast we shouldn't override the files in the vendored subdirectory since those are generated. The place where most of the calls can be circumvented are actually in https://github.com/erlang/rebar3/blob/main/src/rebar_fetch.erl since that file is the central point to call all resource downloads and updates.

* The call to `download_source` should return a failure for being offline

* The call to `needs_update` should shortcircuit to `false`

I think that with these, it'll circumvent most possible attempts to call the outside world, including for git deps or plugin-based fetches.

@ferd Done.

Is there anything you think I should revert, such as intercepting httpc calls in rebar_httpc_adapter?

Should I store the offline flag in rebar_state:t() instead of in the environment variable?

[ferd]

Yeah storing it in the state would actually speed up a lot of lookups. The httpc adapter currently does not take a state however, though I'm not sure how costly that one would be given everything else that's resource heavy should be covered.

I think this looks closer to mergeable, though I'll probably have to take it for a spin rather than just review.

[ferd]

Heads up, I've gotten the vendoring ready for the experimental provider. Once this is ready to review and merge, I think we can cut 3.19.0. I'll then start moving rebar3 to an umbrella structure where it uses its own vendoring mechanisms and we can close the loop on this whole ordeal.

[ferd]

@zuiderkwast is this ready to review or still in draft mode for good reason?

[zuiderkwast]

Feel free to review. No tests have been added (yet). I haven't grasped how to set up suitable test cases. Perhaps you can comment what you think we need here.

[ferd]

I'll probably test it by hand tomorrow and consider that good enough for the bootstrap process at least, then will start seeing what a rebar3 with its own vendor structure looks like and aim for tests on that structure instead.

[zuiderkwast]

That's awesome. (It'd be quite complex to test that rebar3 doesn't connect to a fake Hex server. We'd need to setup a TCP listener and fail the test case if SUT connects to it.)