[Feature]: Add external dependencies to project

[jonaro00]

Describe the feature

Some projects need external dependencies installed, such as ffmpeg, youtube-dl, or things related to cmake.

This feature allows users to specify dependencies in Shuttle.toml (or macros?) that are then installed to the shuttle container before build time using apt or other suitable way (nix??). This allows more and more "niche" projects to deploy.

Or perhaps just allow an arbitrary shell script to execute before the build stage?

Suggestion/example of how the feature would be used

Shuttle.toml

dependencies = [ "ffmpeg" ]

or

apt = [ "cowsay" ]
nix = [ "libopus" ]

or arbitrary script, say shuttlebuild.sh

apt install -y cmake
rm -rf / # 🤪jk

Not sure which approach suits shuttle the best.

EDIT: Realized this is a duplicate of #135
EDIT 2: Added script example

[jonaro00]

Update Oct 2024

On shuttle.dev, it is now possible to run an arbitrary shell script before compilation and in the runtime image: https://docs.shuttle.dev/docs/builds#experimental-hook-scripts

---

I made a temporary workaround to enable installing apt packages (or any prep work) in the container before a deploy. It is a bit ugly and tedious, but works for me.

Example code

For top level projects that need external tools

The approach below is used for when a dependency of your project needs something at compile time. If your project only needs something during its own compile time or run time, you can skip ahead and use the build.rs approach at the end, but in your main crate (no helper crate).

When a dependency needs tools at compile time

Add a workspace-excluded crate, with the same Shuttle project name as the main crate. I called mine dep-installer-hack.

# dep-installer-hack/Cargo.toml

[package]
name = "dep-installer-hack"
version = "0.1.0"
edition = "2021"
publish = false

# For some reason, the exclude from the main workspace doesn't
# work well with shuttle's `cargo-metadata`.
# Therefore this crate is marked as workspace.
[workspace]

[dependencies]
shuttle-runtime = "0.24.0"
tokio = "1"

# dep-installer-hack/Shuttle.toml

name = "wallace-minion"  # <--- use the same project name as your main project.

The service in this crate is an empty Shuttle service that does nothing at runtime:

// dep-installer-hack/src/main.rs

#[shuttle_runtime::main]
async fn shuttle_main() -> Result<MyService, shuttle_runtime::Error> {
    Ok(MyService {})
}

struct MyService {}

#[shuttle_runtime::async_trait]
impl shuttle_runtime::Service for MyService {
    async fn bind(self, _addr: std::net::SocketAddr) -> Result<(), shuttle_runtime::Error> {
        tokio::time::sleep(tokio::time::Duration::from_secs(2)).await;
        Ok(())
    }
}

Now, add a build script that runs the apt command (or anything really) when this helper crate is being built in Shuttle.

// dep-installer-hack/build.rs

fn main() {
    // Install external dependency (in the shuttle container only)
    if std::env::var("HOSTNAME")
        .unwrap_or_default()
        .contains("shuttle")
    {
        if !std::process::Command::new("apt")
            .arg("install")
            .arg("-y")
            .arg("libopus-dev") // the apt package that a dependency of my project needs to compile
            // can add more here
            .status()
            .expect("failed to run apt")
            .success()
        {
            panic!("failed to install dependencies")
        }
    }
}

This bare-bones crate can now be deployed to a freshly restarted Shuttle container to prepare for the compilation of your real deployment. I use this sequence to deploy my Discord bot. Modify to your needs.

cargo shuttle project restart
# installs stuff in the container. do this after container restarts (any "project restart").
cargo shuttle deploy --no-test --working-directory dep-installer-hack

cargo shuttle deploy # deploy real project in prepped container

The compilation time of the first deployment isn't useless either, all the compiled shuttle-runtime and tokio artifacts are reused :)

[virtualritz]

I like the solution with [dependencies] in shuttle.toml. It would also allow the distinction between compile- & runtime deps with a [dev-dependency] section.

I wouldn't have apt or nix sections. Too much hassle for the user.

The shuttle team should maintain a mapping from accepted package names in those sections to actual package names on the distro/os where shuttle deploys in the cloud. I.e. I would think that you often may need to add a -dev postfix etc.

[jonaro00]

Now available as an experimental feature: https://docs.shuttle.dev/docs/builds#experimental-hook-scripts