fix(runtime): Remove 2s startup sleep (#1012)

* dont upload .git * Remove the 2 second sleep at runtime startup + logging and docs * some markdown formatting * move .git ignore (pun intended) to other PR
shuttle-hq · Jun 23, 2023 · e9906f5 · e9906f5
1 parent 7075917
commit e9906f5
Showing 8 changed files with 111 additions and 97 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -56,11 +56,13 @@ You can now start a local deployment of Shuttle and the required containers with
 USE_PANAMAX=disable make up
 ```
 
+<!-- markdownlint-disable no-blanks-blockquote -->
 > Note: `make up` does not start [panamax](https://github.com/panamax-rs/panamax) by default, if you do need to start panamax for local development, run this command with `make COMPOSE_PROFILES=panamax up`.
 
 > Note: `make up` can also be run with `SHUTTLE_DETACH=disable`, which means docker-compose will not be run with `--detach`. This is often desirable for local testing.
 
 > Note: Other useful commands can be found within the [Makefile](https://github.com/shuttle-hq/shuttle/blob/main/Makefile).
+<!-- markdownlint-restore -->
 
 The API is now accessible on `localhost:8000` (for app proxies) and `localhost:8001` (for the control plane). When running `cargo run --bin cargo-shuttle` (in a debug build), the CLI will point itself to `localhost` for its API calls.
 
@@ -167,11 +169,13 @@ Next up we need to insert an admin user into the `auth` state using the ID of th
 container and the auth CLI `init` command:
 
 ```bash
-AUTH_CONTAINER_ID=$(docker ps -qf "name=auth") \
-    docker exec $AUTH_CONTAINER_ID ./usr/local/bin/service \
+AUTH_CONTAINER_ID=$(docker ps -qf "name=auth")
+
+docker exec $AUTH_CONTAINER_ID ./usr/local/bin/service \
     --state=/var/lib/shuttle-auth \
     init --name admin --key dh9z58jttoes3qvt
 ```
+
 > Note: if you have done this already for this container you will get a "UNIQUE constraint failed"
 > error, you can ignore this.
 
@@ -185,7 +189,7 @@ cargo shuttle login --api-key dh9z58jttoes3qvt
 We're now ready to start a local run of the deployer:
 
 ```bash
-cargo run -p shuttle-deployer -- --provisioner-address http://localhost:3000 --auth-uri http://localhost:8008 --proxy-fqdn local.rs --admin-secret test-key --local --project <project_name>
+cargo run -p shuttle-deployer -- --provisioner-address http://localhost:3000 --auth-uri http://localhost:8008 --proxy-fqdn local.rs --admin-secret dh9z58jttoes3qvt --local --project <project_name>
 ```
 
 The `<project_name>` needs to match the name of the project that will be deployed to this deployer. This is the `Cargo.toml` or `Shuttle.toml` name for the project.
@@ -205,30 +209,41 @@ cargo run --bin cargo-shuttle --manifest-path="../../../Cargo.toml" -- deploy
 If you want to use Podman instead of Docker, you can configure the build process with environment variables.
 
 Use Podman for building container images by setting `DOCKER_BUILD`.
-```
+
+```sh
 export DOCKER_BUILD=podman build --network host
 ```
 
 The Shuttle containers expect access to a Docker-compatible API over a socket. Expose a rootless Podman socket either
+
 - [with systemd](https://github.com/containers/podman/tree/main/contrib/systemd), if your system supports it,
+
     ```sh
     systemctl start --user podman.service
     ```
+
 - or by [running the server directly](https://docs.podman.io/en/latest/markdown/podman-system-service.1.html).
+
     ```sh
     podman system service --time=0 unix://$XDG_RUNTIME_DIR/podman.sock
     ```
+
 Then set `DOCKER_SOCK` to the *absolute path* of the socket (no protocol prefix).
+
 ```sh
 export DOCKER_SOCK=$(podman system info -f "{{.Host.RemoteSocket.Path}}")
 ```
 
 Finally, configure Docker Compose. You can either
+
 - configure Docker Compose to use the Podman socket by setting `DOCKER_HOST` (including the `unix://` protocol prefix),
+
     ```sh
     export DOCKER_HOST=unix://$(podman system info -f "{{.Host.RemoteSocket.Path}}")
     ```
+
 - or install [Podman Compose](https://github.com/containers/podman-compose) and use it by setting `DOCKER_COMPOSE`.
+
     ```sh
     export DOCKER_COMPOSE=podman-compose
     ```
@@ -275,10 +290,9 @@ Before committing:
 
 ## Opening a Pull Request
 
-Before opening a pull request it's a good idea to first open an issue, even if the change is small. This way you can get
-feedback and suggestions on the issue, as well as a confirmation from the maintainers that this is something
-we want to implement. This also greatly increases the likelihood of the pull request getting merged, and it reduces the 
-chance that multiple contributors start working on the same issue in parallel.
+Before opening a pull request it's a good idea to first open an issue, even if the change is small.
+This way you can get feedback and suggestions on the issue, as well as a confirmation from the maintainers that this is something we want to implement.
+This also greatly increases the likelihood of the pull request getting merged, and it reduces the chance that multiple contributors start working on the same issue in parallel.
 
 We will squash commits before merging your PR to main. If you do want to squash commits, please do not do so
 after the review process has started, the commit history can be useful for reviewers.
@@ -291,7 +305,7 @@ friendly and respectful of the contributor as well as in line with our [code of
 We will always strive to review pull requests as soon as we can, but during certain periods we may be too busy to
 review every pull request in a timely manner. This does not mean we are not excited about the contribution or that
 there is anything wrong with it, we are grateful for every contribution and the time spent on them. If you feel that
-your PR has gone under the radar for too long feel free to ping us and we'll try to get back to you with an update. 
+your PR has gone under the radar for too long feel free to ping us and we'll try to get back to you with an update.
 
 ## Project Layout
 
@@ -349,16 +363,18 @@ and renewing SSL certificates through the acme client in the `gateway`.
 
 - `common` contains shared models and functions used by the other libraries and binaries.
 - `codegen` contains our proc-macro code which gets exposed to user services from `runtime`.
-The redirect through `runtime` is to make it available under the prettier name of `shuttle_runtime::main`.
-- `runtime` contains the `alpha` runtime, which embeds a gRPC server and a `Loader` in a service with the `shuttle_runtime::main` macro. The gRPC server receives commands from `deployer` like `start` and `stop`. The `Loader` sets up a tracing subscriber and provisions resources for the users service. The `runtime` crate also contains the `shuttle-next` binary, which is a standalone runtime binary that is started by the `deployer` or the `cargo-shuttle` CLI, responsible for loading and starting `shuttle-next` services.
-- `service` is where our special `Service` trait is defined. Anything implementing this `Service` can be loaded by the `deployer` and the local runner in `cargo-shuttle`. The `service` library also defines the `ResourceBuilder` and `Factory` trais 
-which are used in our codegen to provision resources. The `service` library also contains the utilities we use for compiling users
-crates with `cargo`.
-- `proto` contains the gRPC server and client definitions to allow `deployer` to communicate with `provisioner`, and to allow
-the `deployer` and `cargo-shuttle` cli to communicate with the `alpha` and `shuttle-next` runtimes.
+  The redirect through `runtime` is to make it available under the prettier name of `shuttle_runtime::main`.
+- `runtime` contains the `alpha` runtime, which embeds a gRPC server and a `Loader` in a service with the `shuttle_runtime::main` macro.
+  The gRPC server receives commands from `deployer` like `start` and `stop`.
+  The `Loader` sets up a tracing subscriber and provisions resources for the users service.
+  The `runtime` crate also contains the `shuttle-next` binary, which is a standalone runtime binary that is started by the `deployer` or the `cargo-shuttle` CLI, responsible for loading and starting `shuttle-next` services.
+- `service` is where our special `Service` trait is defined.
+  Anything implementing this `Service` can be loaded by the `deployer` and the local runner in `cargo-shuttle`.
+  The `service` library also defines the `ResourceBuilder` and `Factory` trais which are used in our codegen to provision resources.
+  The `service` library also contains the utilities we use for compiling users crates with `cargo`.
+- `proto` contains the gRPC server and client definitions to allow `deployer` to communicate with `provisioner`, and to allow the `deployer` and `cargo-shuttle` cli to communicate with the `alpha` and `shuttle-next` runtimes.
 - `resources` contains various implementations of `ResourceBuilder`, which are consumed in the `codegen` to provision resources.
-- `services` contains implementations of `Service` for common Rust web frameworks. Anything implementing `Service` can be deployed
-by shuttle.
+- `services` contains implementations of `Service` for common Rust web frameworks. Anything implementing `Service` can be deployed by shuttle.
 - `e2e` just contains tests which starts up the `deployer` in a container and then deploys services to it using `cargo-shuttle`.
 
 Lastly, the `user service` is not a folder in this repository, but is the user service that will be deployed by `deployer`.
@@ -383,4 +399,4 @@ git config --global core.autocrlf true
 
 After you run this command, you should be able to checkout projects that are maintained using CRLF (Windows) again.
 
-[^1]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration#_core_autocrlf
+[^1]: <https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration#_core_autocrlf>
diff --git a/README.md b/README.md
@@ -51,6 +51,7 @@ Shuttle is built for productivity, reliability and performance:
 ![star](https://i.imgur.com/kLWmThm.gif)
 
 ## Console Sneak Peek (gif)
+
 ![console-sneak-peek](https://i.imgur.com/1qdWipP.gif)
 *The gif above demonstrates the ease of adding/managing resources to your project(s) in our upcoming console. Bear in mind that the ease of adding resources is already there, albeit without the visualization coming from the console.*
 
@@ -87,9 +88,7 @@ To initialize your project, simply write:
 cargo shuttle init --axum hello-world
 ```
 
-> Note: if you use [sparse registries](https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol.html) (which
-> is enabled by default on `nightly`), you may encounter [this bug](https://github.com/shuttle-hq/shuttle/issues/821) when 
-> running the `init` command. To resolve this, see [this comment](https://github.com/shuttle-hq/shuttle/issues/821#issuecomment-1525317860).
+> Note: if you use [sparse registries](https://blog.rust-lang.org/inside-rust/2023/01/30/cargo-sparse-protocol.html) (which is enabled by default on `nightly`), you may encounter [this bug](https://github.com/shuttle-hq/shuttle/issues/821) when running the `init` command. To resolve this, see [this comment](https://github.com/shuttle-hq/shuttle/issues/821#issuecomment-1525317860).
 
 And to deploy it, write:
 

diff --git a/cargo-shuttle/src/lib.rs b/cargo-shuttle/src/lib.rs
@@ -1,40 +1,35 @@
 mod args;
 mod client;
-pub mod config;
+mod config;
 mod init;
 mod provisioner_server;
 
-use args::LogoutArgs;
-use indicatif::ProgressBar;
-use indoc::printdoc;
-use shuttle_common::claims::{ClaimService, InjectPropagation};
-use shuttle_common::models::deployment::{
-    get_deployments_table, DeploymentRequest, GIT_STRINGS_MAX_LENGTH,
-};
-use shuttle_common::models::project::IDLE_MINUTES;
-use shuttle_common::models::resource::get_resources_table;
-use shuttle_common::project::ProjectName;
-use shuttle_common::{resource, ApiKey};
-use shuttle_proto::runtime::runtime_client::RuntimeClient;
-use shuttle_proto::runtime::{self, LoadRequest, StartRequest, StopRequest, SubscribeLogsRequest};
-
-use tokio::process::Child;
-use tokio::task::JoinHandle;
-use tonic::transport::Channel;
-use tonic::Status;
-
-use std::collections::HashMap;
+use std::collections::{BTreeMap, HashMap};
 use std::ffi::OsString;
 use std::fs::{read_to_string, File};
 use std::io::stdout;
 use std::net::{Ipv4Addr, SocketAddr};
 use std::path::{Path, PathBuf};
-
 use std::process::exit;
 use std::str::FromStr;
 
+use shuttle_common::{
+    claims::{ClaimService, InjectPropagation},
+    models::{
+        deployment::{get_deployments_table, DeploymentRequest, GIT_STRINGS_MAX_LENGTH},
+        project::{self, IDLE_MINUTES},
+        resource::get_resources_table,
+        secret,
+    },
+};
+use shuttle_common::{project::ProjectName, resource, ApiKey};
+use shuttle_proto::runtime::{
+    self, runtime_client::RuntimeClient, LoadRequest, StartRequest, StopRequest,
+    StorageManagerType, SubscribeLogsRequest,
+};
+use shuttle_service::builder::{build_workspace, BuiltService};
+
 use anyhow::{anyhow, bail, Context, Result};
-pub use args::{Command, DeployArgs, InitArgs, LoginArgs, ProjectArgs, RunArgs, ShuttleArgs};
 use cargo_metadata::Message;
 use clap::CommandFactory;
 use clap_complete::{generate, Shell};
@@ -47,16 +42,22 @@ use futures::{StreamExt, TryFutureExt};
 use git2::{Repository, StatusOptions};
 use ignore::overrides::OverrideBuilder;
 use ignore::WalkBuilder;
-use shuttle_common::models::{project, secret};
-use shuttle_service::builder::{build_workspace, BuiltService};
+use indicatif::ProgressBar;
+use indoc::printdoc;
 use std::fmt::Write;
 use strum::IntoEnumIterator;
 use tar::Builder;
+use tokio::process::Child;
+use tokio::task::JoinHandle;
+use tonic::transport::Channel;
+use tonic::Status;
 use tracing::{debug, error, trace, warn};
 use uuid::Uuid;
 
+pub use crate::args::{Command, ProjectArgs, RunArgs, ShuttleArgs};
 use crate::args::{
-    DeploymentCommand, ProjectCommand, ProjectStartArgs, ResourceCommand, EXAMPLES_REPO,
+    DeployArgs, DeploymentCommand, InitArgs, LoginArgs, LogoutArgs, ProjectCommand,
+    ProjectStartArgs, ResourceCommand, EXAMPLES_REPO,
 };
 use crate::client::Client;
 use crate::provisioner_server::LocalProvisioner;
@@ -606,12 +607,13 @@ impl Shuttle {
             }
         };
 
+        // Child process and gRPC client for sending requests to it
         let (mut runtime, mut runtime_client) = runtime::start(
             is_wasm,
-            runtime::StorageManagerType::WorkingDir(working_directory.to_path_buf()),
+            StorageManagerType::WorkingDir(working_directory.to_path_buf()),
             &format!("http://localhost:{provisioner_port}"),
             None,
-            run_args.port - (1 + i),
+            run_args.port - i - 1,
             runtime_path,
         )
         .await
@@ -809,7 +811,7 @@ impl Shuttle {
 
     #[cfg(target_family = "unix")]
     async fn local_run(&self, run_args: RunArgs) -> Result<()> {
-        let services = Shuttle::pre_local_run(self, &run_args).await?;
+        let services = self.pre_local_run(&run_args).await?;
         let (provisioner_server, provisioner_port) = Shuttle::setup_local_provisioner().await?;
         let mut sigterm_notif =
             tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
@@ -921,7 +923,7 @@ impl Shuttle {
 
     #[cfg(target_family = "windows")]
     async fn local_run(&self, run_args: RunArgs) -> Result<()> {
-        let services = Shuttle::pre_local_run(&self, &run_args).await?;
+        let services = self.pre_local_run(&run_args).await?;
         let (provisioner_server, provisioner_port) = Shuttle::setup_local_provisioner().await?;
 
         // Start all the services.
@@ -975,10 +977,11 @@ impl Shuttle {
             let repo_path = dunce::canonicalize(repo_path)?;
             trace!(?repo_path, "found git repository");
 
-            if !args.allow_dirty {
-                self.is_dirty(&repo).context("dirty not allowed")?;
+            let dirty = self.is_dirty(&repo);
+            if !args.allow_dirty && dirty.is_err() {
+                bail!(dirty.unwrap_err().context("dirty not allowed"));
             }
-            deployment_req.git_dirty = Some(self.is_dirty(&repo).is_err());
+            deployment_req.git_dirty = Some(dirty.is_err());
 
             if let Ok(head) = repo.head() {
                 // This is typically the name of the current branch
@@ -1233,7 +1236,7 @@ impl Shuttle {
 
         // Add all the entries to a map to avoid duplication of the Secrets.toml file
         // if it is in the root of the workspace.
-        let mut entries = HashMap::new();
+        let mut entries = BTreeMap::new();
 
         for dir_entry in WalkBuilder::new(working_directory)
             .hidden(false)
@@ -1279,6 +1282,7 @@ impl Shuttle {
 
         let encoder = tar.into_inner().context("get encoder from tar archive")?;
         let bytes = encoder.finish().context("finish up encoder")?;
+        debug!("Archive size: {} bytes", bytes.len());
 
         Ok(bytes)
     }
@@ -1439,7 +1443,7 @@ mod tests {
         };
 
         let mut shuttle = Shuttle::new().unwrap();
-        Shuttle::load_project(&mut shuttle, &mut project_args).unwrap();
+        shuttle.load_project(&mut project_args).unwrap();
 
         assert_eq!(
             project_args.working_directory,

diff --git a/deployer/README.md b/deployer/README.md
@@ -1,22 +1,4 @@
 # Deployment System
 
 Service that manages the building, loading, and deployment of the Shuttle service(s) that make up a user's Shuttle project.
-
-## Checklist
-
-* [ ] Implement building of incoming services.
-  * `deployment/queue.rs`
-* [ ] Implement the loading/execution/deployment of built services.
-  * `deployment/run.rs`
-* [ ] Populate logs table.
-  * `persistence.rs`
-  * `deployment/queue.rs`
-  * `deployment/run.rs`
-* [ ] Send build logs over WebSockets.
-* [ ] Properly cache `crates.io`.
-* [ ] Server-side pre-deployment testing of services.
-  * `deployment/run.rs`
-* [ ] Integrate with gateway (i.e., start instances of deployer from gateway with correct project secret specified).
-  * Depends on gateway being complete/merged.
-* [ ] End-to-end/integration testing.
-  * Fairly independent of the rest of the code base.
+This binary runs inside a Docker image that is spun up at every 'project start'.
diff --git a/deployer/src/deployment/queue.rs b/deployer/src/deployment/queue.rs
@@ -327,9 +327,11 @@ async fn extract_tar_gz_data(data: impl Read, dest: impl AsRef<Path>) -> Result<
 
     for entry in archive.entries()? {
         let mut entry = entry?;
-        let path: PathBuf = entry.path()?.components().skip(1).collect();
+        let name = entry.path()?;
+        let path: PathBuf = name.components().skip(1).collect();
         let dst: PathBuf = dest.as_ref().join(path);
         std::fs::create_dir_all(dst.parent().unwrap())?;
+        trace!("Unpacking {:?} to {:?}", name, dst);
         entry.unpack(dst)?;
     }