Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize OCI archive imports, fix up Bass's build #281

Merged
merged 12 commits into from
Apr 16, 2023
Merged

Optimize OCI archive imports, fix up Bass's build #281

merged 12 commits into from
Apr 16, 2023

Conversation

vito
Copy link
Owner

@vito vito commented Apr 16, 2023 •
edited
Loading

A bit of a grab-bag from dogfooding:

  • Fix the pesky unhelpful end of source error in :lines and :unix-table protocols.
  • Use a 1MB buffer when exporting files through the gateway. Major perf improvement for loading a 1.2GB image (~52s => ~16s).
  • Cache OCI archive imports in-memory, which also obviously helps a lot considering the above duration would be redone every time the image was used.
  • Fix an issue with Bass's own build that led to redundant shim and image building.
  • Use a single gateway client for the full run, to avoid repeated host path operations.
  • Cache host paths, which also seems to help avoid repeated host path operations.

@vito
ignore .tar files
78d428d
@vito
fix up weird image behavior caused by (subgit)
6503d44 
previously we'd end up using both the last nix-affecting commit and also
the nix-affecting commit prior to the last shim-affecting commit,
because we were passing the result of one (subgit) to another. really
these both need to be from HEAD.

+ fix double shim building along the way
  * cmd/bass/bass had a dependency on the 'shims' phony target, so they
    would always build
  * added make dist
@vito
propagate scanner error
2b4ddff
@vito
cache OCI archive imports, show in progress UI
9d0366a
@vito
use a large buffer for reading files over gateway
f4608cf 
this speeds up imports by a LOT; ~52s => 16s
@vito
pin dependencies in buildkit.bass
23cc216
@vito vito added the enhancement New feature or request label Apr 16, 2023
@vito
fix with-go-cache
79cc9b4
@vito
use single gateway whenever possible, cache host paths
71d0616 
Previously repeated references to a host path would upload it each and
every time. Same goes for running thunks in general. Now that everything
uses one gateway client there's a lot less churn.

The exception is when we need to export, since exporting isn't possible
with a gateway client. In those cases the host path will likely be
synced all over again, and builds might run again.
@vito
remove theoretically no longer used deps
48b25e7
@vito
lazily initialize runtimes for remote runner
928d882 
...so that progress can be captured on a per-run basis
@vito
nevermind, back to long-lived runtime
71ee57b
@vito
make long-lived gateway configurable
a7d2e48 
This fixes progress in the REPL, too! Funny that both "loops" need it.
Makes sense.
@vito vito merged commit 7285013 into main Apr 16, 2023
@vito vito deleted the optimize-build branch April 16, 2023 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@vito