rebar in offline mode?

[t3spe]

Can rebar work in offline mode? (without any internet connectivity)
If yes, what is the setting that needs to be set?

I have the hex registry file and trying to make rebar work with it.

[tsloughter]

It works offline if it has all the dependency you need.

[ericmj]

@tsloughter I don't think that's true if you don't have the registry fetched or if some dependencies are not in the registry. Could we make rebar3 skip fetching the registry when the bare compiler is used.

For reference: hexpm/hex#265

[t3spe]

seems the it depend on the registry being present here: ~/.cache/rebar3/hex/default/ (packages.idx and registry.idx)

My use case is that I have a separate build system that is figuring out the dependencies and building a trimmed down version of the hex registry. See here how I'm generating the registry file: http://stackoverflow.com/questions/38407448/building-the-hex-registry-file-offline
Those files are not present there unless rebar3 pulls the hex registry and build them itself. Was curios if there is a way to do this.

[t3spe]

Copying the registry from the HEX_HOME to ~/.cache/rebar3/hex/default/ did the trick.
~/.hex/registry.ets -> ~/.cache/rebar3/hex/default/registry

I would still like to have a mode where rebar does not go out over the network to fetch anything (ie sort of like Hex has offline mode) but this works around my problem for now.

[tsloughter]

It only goes out over the network if it can't verify everything with what it already has. If it doesn't have what it needs locally it can't function in an offline mode anyway.

[ericmj]

Could the deps check be skipped for the bare compiler or is too complicated to pull out? Technically rebar shouldn't have to check the deps since mix has already done that.

[t3spe]

@tsloughter yes. it's the difference between knowing that it should not go out and explicitly failing if it does not find something (ie does not even attempt to go out) and working with what's cached and failing when it tries to go out and it does not have network connectivity (ie implicitly failing)

[t3spe]

agree w/ @ericmj it should probably not attempt to check the deps in bare mode.

[tsloughter]

It shouldn't be doing a deps check with the bare compile provider. It only depends on app_discovery.

[ghost]

I ran into this as well when on the road and have additional thoughts:

I could really use a comprehensive rebar3 binary that bundles more plugins. This isn't only useful when mobile, but also to minimize network dependencies in CI, which in many projects has been getting unreasonably dependent on Internet connectivity. Should I file a ticket for that?

Alternatively, is it possible to mirror Hex to disk (not regular caching) and restrict access to local paths? This would also help in many corporate networks, where a shared mirror saves network roundtrips and improves reliability of builds. I used to do this with a local Maven (JVM) proxy which cached everything ever accessed. That was a compromise and a full mirror that doesn't wait for request to cache a certain package-vsn is the proper solution.

[ferd]

You could probably just run your own rebar3 build that bundles the plugins you need as providers I'd guess. Some plugins could arguably be evaluated to start being bundled in rebar3 if they are stable and have healthy maintainership.

As far as Hex mirroring goes, it is possible to do it, though I believe it requires a full service and at least local network access. Usage for mirrors is documented at https://hex.pm/docs/mirrors and the specification is in a hex repo: https://github.com/hexpm/specifications/blob/master/endpoints.md#repository

We're waiting to be a bit closer to OTP-21 to start thinking of rewriting the hex handling to use the v2 API, since it requires maps to work (clashes in type names for records), and with that one comes a need for a lazier usage of the index (only fetch what we need). We've got some ideas to make this friendlier to offline modes, but right now it's a bit too early to do the work since R16 support prevents us from using that API.

[ghost]

You could probably just run your own rebar3 build that bundles the plugins you need as providers I'd guess.

I tried this, but I didn't figure out the right changes. Extending deps and apps in rebar.app.src wasn't sufficient, since the plugin was still fetched from Hex.

Some plugins could arguably be evaluated to start being bundled in rebar3 if they are stable and have healthy maintainership.

Say I wanted to bundle port_compiler, what would I need to change? Once that's included in a rebar3 release, I can comfortably migrate some of my non-trivial projects.

As far as Hex mirroring goes, it is possible to do it, though I believe it requires a full service and at least local network access.

That sounds semi-useful, but if it works reliably, I could live with running Hex locally during development.

[ghost]

@ferd, I'm going to submit a patch to add the rebar_alias and pc plugins.

[ghost]

#1669

[filmor]

I tried to write a proper Gentoo ebuild and am failing due to this. It's actually easy enough to get all of the dependencies and place them where bootstrap would put them, but there seems to be a forced call to the hex api during the Verify phase that fails.

[ferd]

Yeah, going fully offline would probably also require a local registry cache in the global (or configured) cache directory, which is used to handle some checksums. By default it is in ~/.cache/rebar3, but can be configured as an environment variable with REBAR_CACHE_DIR.

Going offline for now would probably mean grabbing the cache dir of a clean build, moving it to a tracked place (i.e. vendoring it) and specifying that value when running so that rebar3 does not necessarily attempt to go fetch all the information.

This is on top of pre-fetching placing the dependencies in the right directory.

I haven't tested this, but this is how it should work off the top of my mind.

[filmor]

I indeed made the registry available already, but the code still calls the web service, apparently. I'll try your suggestion of making the whole cache dir available.

[filmor]

Ah, actually it's enough to have a usable packages.idx file around. For some reason I thought that this was the same as Hex's registry.ets.gz. For bootstrapping, it's not required to actually place the packages in the cache directory, only the package index needs to exist and the packages have to be unpacked in _build.

For anyone interested, this is the final ebuild that works properly sandboxed: https://github.com/filmor/overlay/blob/master/dev-util/rebar3/rebar3-3.12.0.ebuild

[ferd]

The old ets registry was used in older versions of rebar3 (and hex); the idx package is a sparse sub-index we maintain based on the packages needed recently using the newer hex APIs. You could therefore make a minimal idx file by starting from scratch, building only the project you wish to package, and vendoring the new file.

[ferd]

I've been looking at this again.

I'm currently thinking that this can be made to work rather well. It would require the following parts:

• add a switch to get rebar3 get-deps --vendor as a command which will create a dump of everything into a directory (./vendor/rebar3, possibly configurable)
• add an env var of the form REBAR_OFFLINE=true or REBAR_VENDORED=true that will skip the network phase of a download (or try anyway but fallback to the cache in ./vendor/rebar3)

Hex deps currently work without a problem by just setting REBAR_CACHE_DIR=..., it will just fail to reach the network and fallback on the cache. So the switch will override that value. For git and mercurial deps, none of that error handling exists and I'll have to bolt it on.

@jhogberg does this sound like something that could work for your needs?

[filmor]

I know you didn't specifically ask for my needs, but I'll tell you anyway :):

For packaging in Gentoo, I find it immensely useful to not just get a dump of all dependencies, but to be able to just get the information about everything to download (essentially just a nice formatting for the lock-file, I can probably easily do this in a plugin). I did this manually for https://github.com/filmor/overlay/blob/master/dev-util/rebar3/rebar3-3.12.0.ebuild#L13-L26

Then I would place all those independently downloaded files (or cloned repos) in a directory and point rebar3 to that (REBAR_PACKAGE_SOURCE=some-dir), have it otherwise work as it would normally do.

The main issue I had with this was at the time that rebar3 unconditionally needed a package index file, not really sure why. Also, I couldn't at the time just drop the hex packages into the cache directory as you are suggesting and had to unpack them manually into the _build/... path: https://github.com/filmor/overlay/blob/master/dev-util/rebar3/rebar3-3.12.0.ebuild#L63-L73

[ferd]

The package index file is really what we need in order to know which hashes should exist for each package, and also to know which index (we support many) from which to fetch in what order. Just picking a random tarball would work with the lock file, but you couldn't create the lockfile without the index in the first place. Because we tend to need to create a lock file before using it (it's unlikely you have a cached package you never fetched from said index), it never really made sense for them to be entirely disjoint.

The thing I just tested is not a full solution on its own; the sub-index we need needs to also be moved/created to the vendor directory along with the package cache, but the idea will be exactly that. Rather than separating the cache from the index, it will be to bundle the sub-index along with the cached packages which will automatically make things work fine.

[jhogberg]

@jhogberg does this sound like something that could work for your needs?

Yes, we're happy as long as we can tell it to only work with vendored dependencies and never go out on the network, and that we can build rebar3 itself in this manner. Trying anyway but falling back to cache is something we'd like to avoid.

[filmor]

@ferd Not sure whether I understand you correctly, but for me it would be enough if I could once run the whole build process normally to create the lock file (if it's not commited) and then being able to use only that to fill and use a package directory with the correct tarballs.

$ rebar3 compile  # creates lock
$ rebar3 vendor get-sources
http://url/for/first-package.tar.gz
http://url/for/second-package.tar.gz
...
$ rebar3 clean && rm -r ~/.cache/rebar3
$ # download all packages to ./vendor
$ REBAR_PACKAGE_SOURCE=./vendor rebar3 compile --offline  # No network access, all info taken from lock and ./vendor

[ferd]

You couldn't make that work well @filmor because every time you run a command like rebar3 release then the compile task doesn't have its --offline switch so we'll have to do it from the environment. Regarding the index, I still believe the index is the simplest thing to re-build along with the cache because internal code just assumes it's there.

I'll probably have to figure out something fancier than what I had in mind though if we want all attempts at a network call. That demands going a lot deeper and spread out in what is being attempted but that can probably be made to work anyway.