Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use linux-micro sandbox image for local development #1360

Merged
merged 2 commits into from
Apr 29, 2021
Merged

Use linux-micro sandbox image for local development #1360

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
75fb6c8
Add support for overriding remote docker images
jyn514 Apr 17, 2021
1980ac8
Use the linux-micro image for tests
jyn514 Apr 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env.sample
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ AWS_ACCESS_KEY_ID=cratesfyi
AWS_SECRET_ACCESS_KEY=secret_key
S3_ENDPOINT=http://localhost:9000
DOCSRS_INCLUDE_DEFAULT_TARGETS=false
DOCSRS_DOCKER_IMAGE=ghcr.io/rust-lang/crates-build-env/linux-micro
6 changes: 1 addition & 5 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,10 @@ jobs:
- name: fast tests
run: cargo test --workspace --locked

- name: create small build-environment
run: |
docker build -t buildenv - < dockerfiles/Dockerfile-small-build-env

- name: slow tests
env:
DOCSRS_INCLUDE_DEFAULT_TARGETS: true
DOCS_RS_LOCAL_DOCKER_IMAGE: buildenv
DOCSRS_DOCKER_IMAGE: ghcr.io/rust-lang/crates-build-env/linux-micro
run: cargo test --workspace --locked -- --ignored

- name: Clean up the database
Expand Down
6 changes: 0 additions & 6 deletions dockerfiles/Dockerfile-small-build-env
View file
Open in desktop

This file was deleted.

5 changes: 3 additions & 2 deletions src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ pub struct Config {
pub(crate) build_attempts: u16,
pub(crate) rustwide_workspace: PathBuf,
pub(crate) inside_docker: bool,
pub(crate) local_docker_image: Option<String>,
pub(crate) docker_image: Option<String>,
pub(crate) toolchain: String,
pub(crate) build_cpu_limit: Option<u32>,
pub(crate) include_default_targets: bool,
Expand Down Expand Up @@ -108,7 +108,8 @@ impl Config {

rustwide_workspace: env("CRATESFYI_RUSTWIDE_WORKSPACE", PathBuf::from(".workspace"))?,
inside_docker: env("DOCS_RS_DOCKER", false)?,
local_docker_image: maybe_env("DOCS_RS_LOCAL_DOCKER_IMAGE")?,
docker_image: maybe_env("DOCS_RS_LOCAL_DOCKER_IMAGE")?
.or(maybe_env("DOCSRS_DOCKER_IMAGE")?),
toolchain: env("CRATESFYI_TOOLCHAIN", "nightly".to_string())?,
build_cpu_limit: maybe_env("DOCS_RS_BUILD_CPU_LIMIT")?,
include_default_targets: env("DOCSRS_INCLUDE_DEFAULT_TARGETS", true)?,
Expand Down
11 changes: 8 additions & 3 deletions src/docbuilder/rustwide_builder.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ use docsrs_metadata::{Metadata, DEFAULT_TARGETS, HOST_TARGET};
use failure::ResultExt;
use log::{debug, info, warn, LevelFilter};
use postgres::Client;
use rustwide::cmd::{Command, SandboxBuilder, SandboxImage};
use rustwide::cmd::{Command, CommandError, SandboxBuilder, SandboxImage};
use rustwide::logging::{self, LogStorage};
use rustwide::toolchain::ToolchainError;
use rustwide::{Build, Crate, Toolchain, Workspace, WorkspaceBuilder};
Expand Down Expand Up @@ -53,8 +53,13 @@ impl RustwideBuilder {

let mut builder = WorkspaceBuilder::new(&config.rustwide_workspace, USER_AGENT)
.running_inside_docker(config.inside_docker);
if let Some(custom_image) = &config.local_docker_image {
builder = builder.sandbox_image(SandboxImage::local(&custom_image)?);
if let Some(custom_image) = &config.docker_image {
let image = match SandboxImage::local(&custom_image) {
Ok(i) => i,
Err(CommandError::SandboxImageMissing(_)) => SandboxImage::remote(custom_image)?,
Err(err) => return Err(err.into()),
};
Comment on lines +57 to +61
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is what we want: this would result in the image never being updated, as once the image is downloaded it's available locally. A good heuristic could be to treat an image as remote if it contains /.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good heuristic could be to treat an image as remote if it contains /.

Hmm, there are plenty of remote images that don't have / though, like ubuntu or alpine. Personally I would be ok with it being cached indefinitely; you can always remove the image if it gets out of date. Is there a way to tell if a local image was built locally or downloaded?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full name for ubuntu is docker.io/ubuntu, docker just has a hardcoded "try docker.io if no registry specified". That said, just using a locally cached image without checking that it's up to date is pretty standard behaviour of docker tooling, you either need to explicitly docker pull ... to update it, or pass --pull to commands like docker build to have them attempt pulling updates.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The full name for ubuntu is docker.io/ubuntu, docker just has a hardcoded "try docker.io if no registry specified".

Well yes, but if someone overrides the image, they'll be using the shorthand.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly I'm not sure: I don't want us accidentally use an older image in production because we end up changing the environment variable without realizing it.

Hmm, there are plenty of remote images that don't have / though, like ubuntu or alpine.

I don't think that's a problem, you can't realistically build any Rust crate using those images.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly I'm not sure: I don't want us accidentally use an older image in production because we end up changing the environment variable without realizing it.

Oh I see - I don't think this will have any impact on prod, we don't override the image there.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we do want to change the image in prod, I think we should do that by changing the default, not through environment variables.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, ok.

builder = builder.sandbox_image(image);
}
if cfg!(test) {
builder = builder.fast_init(true);
Expand Down