Skip to content

Package structure and Interop

Jump to bottom Edit New page
genotrance edited this page Sep 12, 2020 · 10 revisions

Discussion

Please post comments on the forum: https://forum.nim-lang.org/t/6738#41823.

Problems

Nimble enforces package structure rules that are confusing and annoying to users

Some nimble rules around package structure are to prevent pollution of the global namespace. In order to avoid import jester/jester, package directories are not included in the search path.

However, the nim compiler can get into actual module conflicts and resolution, that will not be nimble's responsibility and it will not raise errors in these cases.

Related issue:

A package installed by Nimble does not match how the repo looks originally

The package structure as deployed by nimble does not look the same as a git clone. Nimble removes the source control data, moves or excludes files, and configuring these details (install, skip, srcDir) leads to user confusion. Further it prevents the user from generating docs, running tests or creating PRs.

Related issues:

Nimble doesn't allow a package to install files during or post-install nor uninstall packages cleanly if any files not installed by nimble are added after the fact

Packages might need to generate or download additional metadata that aren't part of the repository. This isn't possible with nimble.

Related issue:

Nim is too aware of nimble implementation details

There is a lot of nimble specific code in the compiler. This needs to be reduced or removed as much as possible to keep the compiler simple and not locked to one package manager.

Related issue:

Nim and nimble use different algorithms to find dependencies

Nimble installs multiple versions of packages since ~/.nimble/pkgs is a shared space. While nimble parses .nimble files and calls nim with --path flags to the package requested dependency versions, nim simply uses the latest package version found in $nimbleDir. This discrepancy is little known to users and can cause various issues. This further extends to tools like nimsuggest and nim check which will not know to use the correct dependency paths.

Additional requirements

  1. Namespace conflicts are not typically discovered by package writers or nim/nimble core devs - they are discovered by users. Delegating to them might be practical but a simple warning to the package writer that globally exported modules should match the package name is still prudent.

  2. Cleaning up package structure and interop should not involve addition of new nimble specific code in the compiler (e.g. srcDir mapping). The solution should just rely on --path since it is simple and backwards compatible.

  3. All said and done, users test with multiple nim versions. $nimbleDir changes should not break just because users checked out an older nim/nimble combo. The $nimbleDir contents do not change just because user switched to an older nim/nimble combo.

  4. Considering nim.cfg is the interface between nim and nimble and dependency versions are package specific, each package needs its own nim.cfg.

  5. Solution should work for the shared ~/.nimble/pkgs, local deps and for a specific --nimbleDir so as not to break existing workflows.

  6. Solution will not introduce package namespaces either as actual directories or injected by nim/nimble. Any such change will break both library authors and users of libraries and is at not backwards compatible with shipping nim/nimble versions.

  7. Solution should handle repos with multiple packages in subdirectories.

Solution

Nimble changes

Installation

  • After git clone, copy entire directory structure including .git contents into $nimbleDir
  • Respect skip directives but deprecate and noop install directives
  • Leave $srcDir as is and don't move files around
  • Reduce or eliminate nimblemeta.json - no need to maintain list of installed files, vcs version, url, etc. since git already has all that info
  • Package uninstall should simply delete the entire package-x.y.z directory

nim.cfg interop

  • Generate or update nim.cfg when any nimble command that processes deps is run within a project with --path entries to all deps
    • Maintain a separate # added by nimble section to isolate from user edits
  • Include $srcDir in --path if defined in .nimble
  • Include --noNimblePath in nim.cfg - prevents older Nims from scanning
  • Verify that generated nim.cfg is valid
  • All commands invoking nim will no longer pass --path info via flags since nim.cfg already contains it
  • Interop should work with standard ~/.nimble, localdeps as well as a custom --nimbleDir
  • All --path added to nim.cfg should be relative to $cfg
  • All packages installed by older nimble will have modified package structure and latest nimble should ignore srcDir declaration when generating nim.cfg --path entries since srcDir is not present on the file system
  • If paths are invalid when executing nim (fresh clone, no deps installed, etc.), user is expected to invoke nimble to refresh nim.cfg
  • Invalid paths in the nim.cfg file getting checked into source control shouldn't matter since nimble will update the nim.cfg when cloned later
  • Examples
    • Has srcDir: --path:$cfg/../path/to/pkgs/nimble-0.1.0/src 
    src/nimble.nim
    src/nimblepkg/version.nim
  • No srcDir: --path:$cfg/../path/to/pkgs/jester-0.1.0 
    jester.nim
    jester/request.nim

    nimterop/cimport.nim
  • For the case where a repo has multiple packages, nimble will do a sparse checkout of just that folder and its contents into $nimbleDir and update --path to point to the subdir.
  • srcDir will still be honored if specified - --path:$cfg/../path/to/pkgs/multi-0.1.0/pname/src
  • Nimble will need minor code changes to handle subdir structure - findNimbleFile(), etc.

Namespace guidance

  • Nimble will warn if package exports a module to the global namespace that doesn't match the package's name.
    • OK: pname-0.1.0/pname.nim
    • Warn: pname-0.1.0/mname.nim
  • Nimble will warn if a package exports a directory containing modules to the global namespace if that name doesn't start with the package's name.
    • OK: pname-0.1.0/pname/mname.nim
    • OK: pname-0.1.0/pnamepkg/mname.nim
    • Warn: pname-0.1.0/other/mname.nim

Nim

  • Deprecate all nimble related code including awareness of ~/.nimble, flags: --clearNimblePath, --nimblePath, pathSubs, lazyPaths and $nimblePath scanning
  • Retain --noNimblePath as a noop since nim.cfg will contain --noNimblePath for older nim
  • If legacy nimble features used, display message to use nimble to setup package directory with a nim.cfg instead of relying on legacy pkg scanning
  • Future release can consider removing nimble related code

Backwards compatibility

As far as nim is concerned, once latest nimble sets up a repo with nim.cfg, any older nim will work with the project since the paths are specified correctly. So the approach is backwards compatible with nim.

When it comes to an older nimble, there are issues:

  • If user has an existing package checked out from before, nim.cfg will not exist and latest nim won't be able to build even though it works just fine with older nim
    • User needs to run latest nimble to setup nim.cfg
  • If user installs package with older nimble in a project with an existing nim.cfg setup by latest nimble, neither older nor latest nim will work since nim.cfg contains --noNimblePath but older nimble does not add --path for new package
    • User needs to run latest nimble to fix nim.cfg or manually adds --path to package into nim.cfg
  • If user uses older nimble commands, they will have issues when working with a $nimbleDir setup by latest nimble
    • Nimble does not read nim.cfg and invokes nim directly in nimble c, nimble build and others with --path flags
    • Older nimble expects srcDir to be gone so some --path flags will be wrong and nim won't work
    • User needs to use the latest nimble even with older nim or use nim directly

Given nimble is compatible and tested with upto nim v0.20.2 and nimble install nimble can setup the latest nimble even on older nim installs, it might be a better option to suggest to avoid some of these issues.

Add a custom footer
Add a custom sidebar
Clone this wiki locally