RFC: requirements.lua

[wbthomason]

I'd like to make it easier for plugin authors to specify the rocks and other plugins that their plugins depend upon. To this end, I'm thinking about introducing the requirements.lua convention/format.

This would be a Lua file included at the top level of a plugin. It should be a module returning a table with two optional keys: rocks and plugins. The rocks key would specify a list of Luarocks packages the plugin depends on in the same format as the rocks key in a packer plugin specification; the plugins key would specify a list of plugins the plugin depends on in the same format as a plugin name in a packer plugin specification.

If a user includes rocks = true or requires = true in a plugin specification, then packer will search for a requirements.lua file in the relevant plugin's install directory and ensure the relevant dependencies (just the rocks, if any, if rocks = true, and both plugins and rocks if requires = true - this allows configuration of dependent packages, if customization is required).

What do people think of this? Would it be helpful? Is it a good approach to solving this problem?

[akinsho]

I think having a way for plugins to under the hood specify dependencies as either rocks or plugins would be great since it removes a lot of boiler plate for the user but also people can build much cooler plugins if they can depend on other libraries or plugins. I'm curious about why deviate from a rockspec though and have a custom format?

I don't know much about rockspecs in general but I imagine that by deviating from this you won't be able to resolve other dependencies for other lua sub-dependencies that use a rockspec (or is this taken care of automatically by luarocks)?

Even if that isn't the case is there good reason to deviate? I guess it won't be a solution for being able to specify other plugins as dependencies which would be quite nice, but that could be done separately or maybe as an addition to the rockspec not sure if you can add random keys to it that hopefully it'd just ignore

[wbthomason]

To clarify one thing: we'd need to get information out of a rockspec rather than just letting Luarocks handle everything because we need to detect which rocks should be in the managed set.

[akinsho]

It sounds like there's not much benefit to using a rockspec in an nvim context then since it doesn't really solve any of the problems and is maybe more unwieldy. Re manually parsing the spec I wonder how luarocks does it maybe it's exposed somehow or easy enough to duplicate but that's more curiosity than anything else.

Here is a link to the format I found online btw, not sure how it'd behave if you added random keys not specified there

[wbthomason]

Re: parsing, I may have been wrong about that - it looks like the rockspec format is valid Lua. I'm still a little leery to adopt that format, since I don't want to imply that packer will handle every feature rockspec supports (e.g. building C extensions), and I'm still not sure that there's any real benefit to using the format other than avoiding format fragmentation - I expect that most plugin authors will need to learn the rockspec format anyway, so there's no familiarity benefit.

[akinsho]

I think you could clearly indicate what is and isn't supported and potentially this might be room to grow into in the future 🤷🏿 . Also one potential benefit would be that libraries that plugin authors might create could be usable by the lua community (small though that may be) which might in turn invite contributions back to those libraries. Anyway I'm definitely playing rockspec-advocate despite being happy with either solution since I think just adding this principle in really will be hugely advantageous in whatever form.

Re. the fear of fragmentation I worry that other lua packer managers that might exist/appear might choose to implement a different format more so if packer as (as far I know the most popular example out there) decided on it's own custom one 🤷🏿 especially if that ever introduced functionality that was particularly geared towards packer's use cases but not others. This wouldn't really be a problem for packer but eventually for plugin developers and potentially users as they'd eventually end up potentially having to use multiple different dependency file formats or might end up getting locked in with one set of plugins supporting one file format vs another (which is possible in either case anyway).

Anyway as I mentioned I think the simple format you propose is fine and I'm likely overstating the downsides.

[wbthomason]

Yeah, the fear of fragmentation is, in my opinion, the big argument against requirements.lua. I'd also be curious to know what people like @tjdevries, @nanotee, @Iron-E, etc. think about this.

[MuhammedZakir]

Rather than a list, having required and optional keys in rocks and plugins is better IMO. This way, with a nicely commented requirements.lua, we can get an overview of all dependencies of a plugin.

[wbthomason]

Could you give an example? For instance, how should packer behave with optional dependencies?

[MuhammedZakir]

If packer can't find a required package, it downloads and install that package, right? If so,

1. packer should install required dependencies by default, unless user does not want to, in which case, packer should show an error msg about unsatisfied dependencies.
2. Whereas. optional dependencies are ignored, unless user wants to install it.

Before installing any dependencies, packer should show a list of packages & URL and ask for confirmation.

Does this make sense?

[Iron-E]

I like this implementation of the idea. Having been playing around with Rust recently, I've fallen in love with the feature feature of the Cargo spec where you can enable / disable parts of the crate's dependencies with a flag. We could do so similarly by allowing users to pick and choose which optional dependencies to enable, perhaps with feature names rather than verbose repo names.

[akinsho]

Might just be me missing something but this seems unnecessarily complicated for a first pass at a dependency specification since a lot of these concepts don't really exist/aren't widely used at all in neovim plugin development i.e. the idea of optional dependencies or features? Surely including these things would only make sense if plugins were designed this way. Although maybe the value is in being forward looking 🤷🏿

[Iron-E]

I think having features would be an easy way to describe to the end users that you have something you'd like your plugin to do but it requires something else, and you can let the plugin figure out what that something else was instead.

It would let me create, for example, a plugin that provides completion for some language and then the feature is what client to support (e.g. built-in vs coc). Then user wouldn't have to know to install nvim-lspconfig and completion-nvim if they wanted built-in support.

For a first pass, probably not needed though I agree! Something to add.

[tjdevries]

I'm coming at this more from the plugin maker space, so I think any time a user has to copy & paste into their init.lua, it's a very bad plan, because it makes it difficult to update any dependency or install instructions.

I like the idea of rocks = true in the configuration, although I think there is an argument for always looking for requirements.lua in a repository and packer installing those (I do not see any large difference between the two options, except that the user no longer has to write that into their configuration. And it will save me from several hundred issues over the next few years from "CANNOT FIND CJSON WHY NOT?!?!? BROKEN PLUGIN?SSDLKFJS" 😆 )

I think if we pioneer requirements.lua, that could be quite nice for the community.

One other random thought I had was a way of taking a rockspec and converting to requirements.lua as a one-time pass or something... idk, just a thought. This way you don't have to parse / know EVERYTHING about the rockspec, but we might be able to generate some useful info from them. No idea if that thought is useful or not. Still bouncing it around in my head.

[Iron-E]

I like this idea overall. Changes to dependencies could finally be synced through updates (and therefore reverted through downgrades, as well as tested in branches). It keeps things centrally sourced as opposed to requiring each plugin user to copy configs.

Perhaps we could create a repo of common definitions for plugins which don't want to target packer / are targeted towards vim? We would still be able to benefit from the centralization of configuration then.

Additionally, even other package managers besides packer could support something like this so having a unified dependency specification standard for Neovim plugins is probably not a bad idea in general.

[wbthomason]

Thanks, everyone - there's a lot of good thinking and suggestions here. Given the feedback, I'm proposing something like the following:

The requirements.lua format

requirements.lua is a simple format for specifying Luarocks and plugin dependencies for Neovim plugins. It is a single file at the top level of a plugin repository adhering to the structure:

return {
  rocks = {ROCK_SPECIFICATION...},
  plugins = {PLUGIN_SPECIFICATION...},
  features = {
    FEATURE_NAME = {
      rocks = {ROCK_SPECIFICATION...},
      plugins = {PLUGIN_SPECIFICATION...}
    }...
  }
}

where:

1. a ROCK_SPECIFICATION is either a string naming a rock available via Luarocks or a table containing such a string and a second string indicating a specific version of the rock to use,
2. a PLUGIN_SPECIFICATION is a string identifying a plugin in the same way that packer accepts (i.e. a full URL or a "Github style" user/repo URL),
3. a FEATURE_NAME is a string identifying an optional feature of the plugin, and
4. ... indicates that the item may be repeated

In all contexts, the rocks, plugins, and features keys are optional.

How packer uses requirements.lua

If a user includes rocks = true, requirements = true, and/or features = {NON-EMPTY LIST OF FEATURE NAMES} in a plugin specification given to packer, packer will look for a requirements.lua file in the top directory of the relevant plugin on update (including initial install) of the plugin

1. If rocks = true, then packer will ensure that all rocks specified in the top-level rocks key of the file are installed
2. If requirements = true, then packer will ensure that all rocks and all plugins specified in the top-level rocks and plugins keys (respectively) of the file are installed
3. For each string in the value of features in the plugin specification, packer will ensure that the rocks and plugins specified in the corresponding FEATURE_NAME entry are installed (should this allow only installing rocks/plugins separately?)

[akinsho]

This looks cool 👍🏿 . Although why bother having the rocks and requirements = true fields ? This seems like TJ suggested to be the sort of thing that will lead to a bunch of user errors/ issues about dependencies not being installed?

Can packer just check if this file exists and handle it itself without any user intervention? Also if a user can set rocks = false and/or requirements = false won't this mean the plugin is likely to break. As someone writing a plugin seems like you won't easily be able to guarantee what dependencies will be available/safe to use within your plugin?

[MuhammedZakir]

Although why bother having the rocks and requirements = true fields ? This seems like TJ suggested to be the sort of thing that will lead to a bunch of user errors/ issues about dependencies not being installed?

I agree with this. What's quite likely to happen is user forgetting to set true for rocks and requirements and getting a broken plugin. IMO, by default, packer should install all rocks and plugins except what are specified in features. This will make sure dependencies for all basic functionalities are met.

Even though that's the case, there should be an option to disable automatic installation of dependencies. Some users may want to disable it. Along with this, an option to ignore plugins with unsatisfied (required) dependencies. However, I am not sure whether this should be a top-level option, or plugin-level option.

[wbthomason]

Ah, I misread what TJ suggested. I'm fine removing the rocks/requirements = true fields - to @MuhammedZakir's point, we might still want a requirements = false option to locally disable dependency installation.

The rationale for the rocks/requirements granularity was that some people may wish to customize the configuration of plugins installed as dependencies. I suppose this is better resolved by changing packer's duplicate plugin logic to ignore duplicates that come from requirements.lua files and always prefer the configuration values in a manually specified plugin spec.

[Iron-E]

Here's a mock of how the proposed requirements.lua would look for my plugin nvim-typora:

return {
  plugins = {'Iron-E/nvim-libmodal'},
  features = {
    table_formatting = {'godlygeek/tabular', 'plasticboy/vim-markdown'}
  }
}

Seems pretty clean and also human-readable. I suppose if one plugin was referenced multiple times in a different features we could just create a var and use it over again:

local foo_req = {'user/foo', branch='bar'}

return {
  features = {
    do_something = {foo_req, 'user/something-else'},
    alt_do_something = {foo_req, 'user/something-else-alt'}
  }
}

[oberblastmeister]

I think this is a great idea. Maybe we should have a way to specify the version of the dependency? For example many plugins use plenary but it might have breaking changes, so we can specify a version of plenary to use so the whole system won't break when plenary updates. Maybe the version can be the commit hash. This way we can symlink the entry point of each dependency to the plugin's lua/ directory. Dependencies won't be duplicated because they are just symlinks. For example if a plugin requires plenary.nvim with a version abdc34a806921de436654adb7f68186dd8dcf796 then packer will download that version and name it plenary.nvim-abdc34a806921de436654adb7f68186dd8dcf796. Then it will symlink the plenary directory to the plugin's directory so that when the plugin requires it the correct version will be used.

[lewis6991]

Ideally plugins which are used as dependencies like plenary should be semantically versioned.

For example gitsigns might need plenary at hash 4d32 but telescope needs plenary at has 8694. How do we handle this? We don't want to only force a specific hash of plenary otherwise the user can't absorb any bugfixes (which could be security related).

Ideally gitsigns can just say it needs version >=4.1, <6 and plenary needs >=4.6, <7. That way packer can figure out that the required plenary to satisfy both is the latest version of 5. The suggestion of symlinking specific versions might work, but I'm sure there are cases where this would cause massive headaches if the dependency isn't just a pure lua library. If it has a plugin directory they would step over each other.

This might sound all a bit over-engineered but this is the truth of dependency management, it's a nightmare. Otherwise we should just require that plugins vendor in their dependencies to prevent this issue all together.

[wbthomason]

I agree with the need for dependency versioning, but I am extremely hesitant to add a version constraint resolver to packer....that feels like a lot of potential for bloat and complexity explosion. Semantic versioning also requires cooperation and coordination on the part of plugin authors, which - given the widely distributed and somewhat fragmented nature of the Neovim plugin ecosystem (particular prior to the current wave of Lua plugins) might be too much to expect.

Vendoring is one possible solution, though it bloats the disk space used and will hurt performance (as we won't benefit from e.g. Lua caching modules). I'm also not sure how this would work with autoloaded functions and module paths that would need to be manually modified to not cause a name collision.

I suppose a very limited semantic version resolver wouldn't be too awful - literally just accepting a lower and upper bound (both optional) on a dependency's version. That's pretty simple and concise to solve. We'd also probably want a way to override a version requirement, so that e.g. if I'm using plugin Foo that requires plenary < 5 and I want to add plugin Bar that requires plenary >= 5, I as a user have the ability to try and use Foo with a potentially incompatible version of plenary rather than being unable to install Foo and Bar simultaneously until the maintainer of Foo updates the dependency requirement.

[tjdevries]

One thing to consider is local old_require = require; require = PackerRequire :)

[wbthomason]

That's an option, but I'm hesitant to do something that interferes with require globally. Seems to open the possibility for massive breakage.

[tjdevries]

well my thoughts would be only for files that you have deps installed for and listed with requirements, you use PackerRequire, otherwise it calls old_require... :) idk that's just how I would do it if I wanted to add crazy good dependency management.

But it sounds like a lot of work and I wouldn't want to solve the problem :)