-
Notifications
You must be signed in to change notification settings - Fork 14
Fix docker cache #369
Fix docker cache #369
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,6 +11,9 @@ on: | |
options: | ||
type: string | ||
default: -DENABLE_CUDA=off | ||
secrets: | ||
DOCKERHUB_PASSWORD: | ||
required: true | ||
|
||
jobs: | ||
build: | ||
|
@@ -25,47 +28,44 @@ jobs: | |
|
||
- name: Set env context | ||
run: | | ||
set -vx | ||
echo RUN_STAMP=docker-${{ runner.os }}-${{ inputs.name }} >>$GITHUB_ENV | ||
cat omniscidb/docker/dev/Dockerfile omniscidb/docker/dev/Dockerfile.${{ inputs.name }} >Dockerfile | ||
cat docker/Dockerfile docker/Dockerfile.${{ inputs.name }} >Dockerfile | ||
echo DOCKER_NAME=docker.io/dataved/build.${{ inputs.name }}:${{ hashFiles('docker/Dockerfile', 'docker/Dockerfile.${{ inputs.name }}') }} >>$GITHUB_ENV | ||
|
||
- name: Set up Docker Buildx | ||
id: buildx | ||
uses: docker/setup-buildx-action@master | ||
|
||
- name: Cache Docker layers | ||
uses: actions/cache@v3 | ||
- name: Login to image repository | ||
uses: docker/login-action@v2 | ||
with: | ||
path: /tmp/.buildx-cache | ||
key: ${{ runner.os }}-buildx-${{ hashFiles('**/Dockerfile') }} | ||
restore-keys: | | ||
${{ runner.os }}-buildx- | ||
|
||
username: dataved | ||
password: ${{ secrets.DOCKERHUB_PASSWORD }} | ||
|
||
- name: Test the image | ||
id: test-docker | ||
run: | | ||
docker run -u ghrunner ${{ env.DOCKER_NAME }} dpkg -l | ||
continue-on-error: true | ||
|
||
- name: Build image | ||
if: steps.test-docker.outcome != 'success' | ||
uses: docker/build-push-action@v3 | ||
with: | ||
context: . | ||
builder: ${{ steps.buildx.outputs.name }} | ||
file: Dockerfile | ||
push: false | ||
load: true | ||
tags: build.${{ inputs.name }} | ||
cache-from: type=local,src=/tmp/.buildx-cache | ||
cache-to: type=local,dest=/tmp/.buildx-cache-new | ||
|
||
# https://github.com/docker/build-push-action/issues/252 | ||
- name: Move cache | ||
run: | | ||
rm -rf /tmp/.buildx-cache | ||
mv /tmp/.buildx-cache-new /tmp/.buildx-cache | ||
push: true | ||
tags: ${{ env.DOCKER_NAME }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Minor note. It's possible to reduce amount of pulls from docker.io with caching docker local storage with github cache. As it reduces build time ~ 2 mins we decided to keep current (more simple and direct) approach. Also github cache is limited with 10 GB and current image size is ~5.6 Gb, conda env/ maven should also be cached so caching for docker storage will be checked later. |
||
|
||
- name: Start the container | ||
run: | | ||
mkdir -p build | ||
docker run -id --name build.${{ inputs.name }} --network host -v $(pwd):/_work build.${{ inputs.name }}:latest | ||
docker run -id --name build.${{ inputs.name }} --network host -v $(pwd):/_work ${{ env.DOCKER_NAME }} | ||
|
||
- name: Configure and build the project | ||
run: | | ||
docker exec -u ghrunner build.${{ inputs.name }} dpkg -l | ||
docker exec -u ghrunner build.${{ inputs.name }} sh /_work/omniscidb/scripts/conda/build.sh ${{ inputs.options }} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think maven cache also can used from cache. |
||
docker exec -u ghrunner build.${{ inputs.name }} tar -zcf /tmp/build.tgz -C /_work . | ||
docker cp build.${{ inputs.name }}:/tmp/build.tgz . | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought that secrets are global vars for all github jobs, but after discussion with Alexei I understood that it's special github behavior https://github.blog/changelog/2022-05-03-github-actions-simplify-using-secrets-with-reusable-workflows/ . Maybe there are some WA, because secrets added as https://docs.github.com/en/actions/learn-github-actions/variables#creating-configuration-variables-for-an-environment .
This is only to mark and summarize our discussion.