Implement midenc toolchain command

[bitwalker]

Background

Currently, the compiler supports linking against Miden libraries/packages by specifying a library name with -l NAME (or --link-library NAME), and additional library search paths with -L PATH (or --search-path PATH). When a link library is requested, it is searched for in all available search paths by name. For example, -l std tells the compiler to link against a library named std, so we will look for std.masl or a Miden project in a std directory, in all available search paths.

However, because of how essential the Miden standard library is (as well as the Miden transaction kernel), we have baked in those two libraries to the compiler binary itself. Thus, when -l std or -l base are specified/implied, we load them directly from memory, and skip the normal search process. We do this because these libraries are an essential part of the Miden toolchain. A toolchain is a named/versioned collection of toolchain components, consisting of tools and libraries that are expected to be available in a typical installation.

Problem

The current situation has some issues:

1. Baking libraries into the compiler means that it is not possible to get the latest bug fixes and features of those libraries when they are released. This is because a new release of the compiler is required in order to pick up those changes.
2. Because the compiler hardcodes the handling of those libraries, there is no way to "opt out" of the baked-in library in favor of a different one on disk. Thus, even if you wanted to manually override what libraries are linked, you cannot specify alternate versions of baked-in libraries.
3. We do not currently have any toolchain management. There is no unified installer for toolchain components, no way to switch between toolchains on a per-project basis, nor a scheme by which to group toolchain components that depend on each other.

Proposal

This issue proposes to implement a new experimental command, as part of midenc for now, called midenc toolchain. It's purpose would be to provide compiler toolchain management facilities. In particular, the ability to install a toolchain, manipulate toolchain components, update and remove toolchains, and specify project-specific toolchain configuration.

When the compiler run, it will perform the following steps as part of initializing the compiler session:

1. Check if there is a midenc-toolchain.toml file in the current working directory. If so, initialize the toolchain configuration for the session based on the configuration in that file. If no midenc-toolchain.toml file exists, then the default toolchain configuration for the stable toolchain will be used.
2. Check if the configured toolchain is installed to the sysroot, which is by default $XDG_DATA_DIR/miden/<toolchain>. If it is, add the toolchain lib path to the search path, i.e. $XDG_DATA_DIR/miden/<toolchain>/lib, and proceed with compilation normally.
3. The configured toolchain is not installed, so we must install it. See the description of miden toolchain install for details. Once installed, proceed as described in Step 2.

This ensures that the compiler can always rely on a toolchain being present.

What's in a toolchain?

Currently, the compiler toolchain is expected to consist of:

• The midenc executable (implied)
• The Miden standard library, in compiled form (the std component)
• The Miden transaction kernel library, in compiled form (the tx-kernel component)

In the future, we expect it will also consist of:

• The miden executable (for proving, verifying, and analyzing Miden programs)
• The miden-client executable (for interacting with the Miden rollup)
• Other "standard" Miden libraries

miden toolchain install <toolchain>

This command installs the specified toolchain, if not already installed. Valid toolchains are one of:

• stable, corresponding to the latest stable miden-vm release
• nightly, corresponding to the latest commit on next of all components
• MAJ.MIN.PATCH, corresponding to the specified miden-vm release.

To perform the installation, a Cargo script will be used, generated in $XDG_CACHE_DIR/miden/install-<toolchain>/install.rs, and executed with CARGO_TARGET_DIR set to $XDG_CACHE_DIR/miden/install-<toolchain>/target. For example, to install the stable toolchain, the following would be generated to install.rs:

#!/usr/bin/env -S cargo +nightly -Zscript
---cargo
[dependencies]
miden-stdlib = "> 0.0"
miden-base = "> 0.0"
---

fn main() {
    // MIDENC_SYSROOT is set by the compiler when invoking this script, and will contain
    // the resolved (and prepared) sysroot path of `$XDG_DATA_DIR/miden`
    let install_dir = std::path::Path::new(env!("MIDENC_SYSROOT")).join("stable");

    // Write stdlib to $XDG_DATA_DIR/miden/<toolchain>/std.masl
    let stdlib = miden_stdlib::StdLibrary::default();
    stdlib.write_to_file(install_dir.join("std").with_extension("masl")).unwrap();

    // Write transaction kernel to $XDG_DATA_DIR/miden/<toolchain>/tx.masl
    ...
}

This script could also be used to build and install other components we're interested in, but this is the basic structure. For the nightly toolchain, we would specify git dependencies instead of using those published to crates.io. Likewise, installing specific versioned toolchains would pin the dependency versions. In this way, we can control the versions of all installed components for a toolchain, without having to bake any of those components into the compiler itself.

midenc toolchain update <toolchain>

This would update the specified installed toolchain, using the generated Cargo script in $XDG_CACHE_DIR/miden/install-<toolchain>/install.rs, by running cargo +nightly -Zscript update install.rs, to fetch all updates, and then re-run the script to perform the install again.

NOTE: This command can only be used to update "sliding" releases, e.g. stable and nightly, which can change as new versions are released.

midenc toolchain set <toolchain>

This would generate a midenc-toolchain.toml in the current working directory, for example:

[toolchain]
name = "nightly"

This file would also support a toolchain.components key, containing a list of named components to install for the toolchain. For now, the set of components is inferred to be all of them.

Compiler Changes

Once toolchain support is implemented, we would modify the compiler in the following ways:

• Remove the hardcoded mapping of -l std and -l base to in-memory libraries
• Remove dependency on miden-stdlib and miden-base
• Add toolchain libraries to search path by default

Future Work

While the proposal above ensures that the way libraries are handled is uniform, it isn't quite enough for us to remove some of the information about toolchain libraries hardcoded in the compiler. Specifically, the compiler currently hardcodes the type signatures for each procedure in std or base that we have added support for. This is essential for binding to these procedures from Rust, in particular so that we can emit the appropriate ABI adapter (if applicable) for each procedure.

Once type signatures are expressible in Miden Assembly, we can instead make this information available via library/package metadata, and have the compiler use that as the source for its type signature database and ABI analysis. While we currently hardcode the specific ABI transformation strategy for each procedure, we should be able to derive this from the type signature automatically, but it only really makes sense to do so once the type signatures are no longer hardcoded.

Tasks

Beta Give feedback

1. Stub out midenc toolchain command (and subcommands) in driver
2. Implement support for midenc-toolchain.toml and midenc toolchain set
3. Add toolchain detection/selection to driver, including adding toolchain sysroot to search path
4. Remove hardcoded loading of libraries from miden-stdlib and miden-base crates
5. Implement and test midenc toolchain install stable (for production usage)
6. Implement and test midenc toolchain install nightly (for testing with components under development)
7. Implement midenc toolchain update