docker/install: Support `version: master` #438

vvoland · 2024-09-06T10:27:34Z

related to: Provide binaries using container images crazy-max/ghaction-setup-docker#81

Add support for installing Docker master packages from moby/moby-bin and dockereng/cli-bin images.

This could also allow to install arbitrary version from these images but for now it's only used for master.

Test run: https://github.com/docker/moby-private/actions/runs/10773693004/job/29874033854?pr=9#step:2:336

src/hubRepository.ts

crazy-max · 2024-09-06T10:42:25Z

src/docker/install.ts

+    core.info(`Downloading Docker ${this.version} from ${this.channel}`);
+
+    this._version = this.version;
+    if (this.version == 'master') {


I think it would be better to rely on csv values so we can either choose archive or image format:

v24.0.9 similar to type=archive,version=v24.0.9

type=image,tag=master

type=image,tag=24.0.9

WDYT?

But that would be smth to handle in setup-docker action as well.

Do we want to have another input variable for cli version alone?
Like:

version: type=image,tag=master # and optional repository=moby/moby-bin-someotherepo cli-version: type=archive,version=v20.10.25 # if not specified, will default to tag from `version` an

or just sth like

version: | component=engine,type=image,tag=master component=cli,type=archive,version=v20.10.25

Or, if we want to go in the direction of: crazy-max/ghaction-setup-docker#81

Then maybe we should have a separate image input:

images: | moby/moby-bin:master dockereng/cli-bin:master

that would be mutually exclusive with version?

Do we want to have another input variable for cli version alone?

I think we should keep both under the same terminology instead of being separated. Don't think we want to allow custom registry or image repository imo.

Or, if we want to go in the direction of: crazy-max/ghaction-setup-docker#81

I think csv values like #438 (comment) are easier to maintain so we don't introduce a new input and keep backward compat with current version input. I will update this proposal if we are ok with this. WDYT?

My main concern here was the ability to install a different engine and cli version. I think it's fine to leave the image repository hardcoded, but it could be useful to be able to install different versions of CLI and Engine.

but it could be useful to be able to install different versions of CLI and Engine.

Ok if there is this use case then what do you think of:

v24.0.9 > type=archive,version=v24.0.9 or type=archive,engine_version=v24.0.9,cli_version=v24.0.9 if you want specific engine/cli version

Same for images:

type=image,tag=24.0.9 > type=image,engine_tag=24.0.9,cli_tag=24.0.9

SGTM, although it's not an urgent use-case so we can handle that in a follow up.

Updated to support the CSV syntax.
Example run: https://github.com/docker/moby-private/actions/runs/10773693004/job/29874033854?pr=9#step:2:336

src/docker/install.ts

vvoland · 2024-09-09T16:55:09Z

Also opened a PR on the ghaction to make use of the new options: crazy-max/ghaction-setup-docker#106

crazy-max

Along the change for source can you add new cases in https://github.com/docker/actions-toolkit/blob/main/__tests__/docker/install.test.ts to download from image?

src/docker/install.ts

crazy-max · 2024-10-16T07:45:17Z

https://github.com/docker/actions-toolkit/actions/runs/11353388604/job/31578295350?pr=438#step:11:583

Downloading from moby/moby-bin and dockereng/cli-bin tag: 27.3.1
FAIL __tests__/docker/install.test.itg.ts (355.416 s)
  ● install › install docker { type: 'image', tag: '27.3.1' }

    expect(received).resolves.not.toThrow()

    Received promise rejected instead of resolved
    Rejected to value: [Error: Cannot find manifest for darwin/amd64/]

Hum right, on macos runners it will use darwin looking at https://github.com/docker/actions-toolkit/pull/438/files#diff-0f2ffb1716f62956ee604fae748f3369d867fa5c39a60388479728f013479b46R120. Might need to enforce linux in this case but not for cli.

vvoland · 2024-10-16T07:46:47Z

Or just skip the daemon on darwin and only download the CLI?

OTOH, this could be confusing behavior as it will not error out and won't be able to connect to the daemon..

crazy-max · 2024-10-16T08:02:16Z

Or just skip the daemon on darwin and only download the CLI?

OTOH, this could be confusing behavior as it will not error out and won't be able to connect to the daemon..

Yeah provisioning for macos is a bit different. It downloads static bins from download.docker.com: https://github.com/docker/actions-toolkit/actions/runs/11289888751/job/31400662984#step:11:17 which only contains cli as expected.

But then during lima provisioning it installs the engine using get.docker.com script:

actions-toolkit/src/docker/assets.ts

Line 224 in a59a5f8

    
               curl -fsSL https://get.docker.com | sh -s -- --channel {{dockerBinChannel}} --version {{dockerBinVersion}}

so platform normalization would match the VM.

vvoland · 2024-10-16T08:10:10Z

Right, so we'd need to also handle the image download inside the lima provisioning script. I think it's easiest to just use undock there? The lima provisioning is already doing a lot of stuff, so the extra binary won't hurt there 😅

EDIT: Actually it might be easier to just mount the binaries to lima vm, let me try that.

crazy-max · 2024-10-16T10:43:48Z

Can you rebase to fix the QEMU issue related to #459

crazy-max · 2024-10-17T09:14:31Z

src/docker/assets.ts

+      wget https://raw.githubusercontent.com/moby/moby/{{srcImageTag}}/contrib/init/systemd/docker.service \
+        https://raw.githubusercontent.com/moby/moby/v{{srcImageTag}}/contrib/init/systemd/docker.service \
+        -O /etc/systemd/system/docker.service || true
+      wget https://raw.githubusercontent.com/moby/moby/{{srcImageTag}}/contrib/init/systemd/docker.socket \
+        https://raw.githubusercontent.com/moby/moby/v{{srcImageTag}}/contrib/init/systemd/docker.socket \
+        -O /etc/systemd/system/docker.socket || true
+
+      mkdir -p /usr/local/bin
+      cp /tool/* /usr/local/bin/
+      sed -i 's|^ExecStart=.*|ExecStart=/usr/local/bin/dockerd -H fd://|' /etc/systemd/system/docker.service
+      sed -i 's|containerd.service||' /etc/systemd/system/docker.service
+      if ! getent group docker; then
+        groupadd --system docker
+      fi
+      systemctl daemon-reload
+      if ! systemctl enable --now docker; then
+        systemctl status docker.socket
+        systemctl status docker.service
+      fi


Maybe instead of doing this we could just run curl -fsSL https://get.docker.com | sh and override binaries with ones from tooldir and restart daemon?

I guess this could be the same behavior for both archive and image mode

I was considering that, but that could be problematic in cases where there are packaging changes between the version installed by curl get.docker.com | sh and the actual target version.

Add support for installing Docker `master` packages from `moby/moby-bin` and `dockereng/cli-bin` images. This could also allow to install arbitrary version from these images but for now it's only used for `master`. Signed-off-by: Paweł Gronowski <[email protected]>

Signed-off-by: Paweł Gronowski <[email protected]>

Use InstallSource instead Signed-off-by: Paweł Gronowski <[email protected]>

Signed-off-by: Paweł Gronowski <[email protected]>

vvoland · 2024-10-17T10:41:04Z

Weird qemu failure:

[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[    0.000000] APIC: Switch to symmetric I/O mode setup
[    0.000000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.000000] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[    0.000000] ...trying to set up timer (IRQ0) through the 8259A ...
[    0.000000] ..... (found apic 0 pin 2) ...
[    0.016000] ....... failed.
[    0.016000] ...trying to set up timer as Virtual Wire IRQ...
[    0.032000] ..... failed.
[    0.032000] ...trying to set up timer as ExtINT IRQ...
[    0.044000] ..... failed :(.
[    0.044000] Kernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option.

Is this known to happen sometimes? 🤔

vvoland · 2024-10-17T10:41:44Z

The previous failure on macos-12 was related to the long /var/tmp path - it didn't mount correctly into lima vm.
Fixed by copying to a shorter path in .lima directory.

EDIT: Looks like it's still the case 🫠

Signed-off-by: Paweł Gronowski <[email protected]>

vvoland force-pushed the install-from-binimage branch from ea3d0b7 to 91a2ddd Compare September 6, 2024 10:30

crazy-max reviewed Sep 6, 2024

View reviewed changes

src/hubRepository.ts Outdated Show resolved Hide resolved

vvoland force-pushed the install-from-binimage branch 2 times, most recently from 5d3b32c to 2941b11 Compare September 6, 2024 10:40

crazy-max reviewed Sep 6, 2024

View reviewed changes

vvoland commented Sep 9, 2024

View reviewed changes

src/docker/install.ts Outdated Show resolved Hide resolved

vvoland force-pushed the install-from-binimage branch from 7f4cb8d to 9a7c8c8 Compare September 9, 2024 13:02

vvoland mentioned this pull request Sep 9, 2024

Support downloading binaries from docker images crazy-max/ghaction-setup-docker#106

Draft

crazy-max requested changes Sep 18, 2024

View reviewed changes

src/docker/install.ts Outdated Show resolved Hide resolved

crazy-max reviewed Sep 18, 2024

View reviewed changes

src/docker/install.ts Outdated Show resolved Hide resolved

vvoland force-pushed the install-from-binimage branch 4 times, most recently from 2c74c55 to 5a0fe00 Compare October 15, 2024 19:53

vvoland force-pushed the install-from-binimage branch from 5a0fe00 to 97bef8d Compare October 16, 2024 07:45

vvoland force-pushed the install-from-binimage branch from 97bef8d to 20a30f2 Compare October 16, 2024 07:49

vvoland force-pushed the install-from-binimage branch 8 times, most recently from 9662b9d to 23c945d Compare October 16, 2024 10:32

vvoland force-pushed the install-from-binimage branch 8 times, most recently from 5234b17 to bcb34bf Compare October 17, 2024 08:50

crazy-max reviewed Oct 17, 2024

View reviewed changes

vvoland force-pushed the install-from-binimage branch from 2ed7185 to 81d3249 Compare October 17, 2024 09:27

vvoland added 5 commits October 17, 2024 12:18

docker/install: Install source

10424fa

Signed-off-by: Paweł Gronowski <[email protected]>

docker/install: Handle missing v prefix when searching GH release

b8a9607

Signed-off-by: Paweł Gronowski <[email protected]>

docker/install: Remove deprecated version and channel

de390e0

Use InstallSource instead Signed-off-by: Paweł Gronowski <[email protected]>

docker/install: Add tests for installing from image

b143889

Signed-off-by: Paweł Gronowski <[email protected]>

vvoland force-pushed the install-from-binimage branch from 81d3249 to 668161b Compare October 17, 2024 10:18

vvoland force-pushed the install-from-binimage branch 5 times, most recently from 41c8df2 to 0ab806d Compare October 17, 2024 13:09

Support image source on darwin

a74c891

Signed-off-by: Paweł Gronowski <[email protected]>

vvoland force-pushed the install-from-binimage branch 4 times, most recently from 7ec9dc8 to 6bc0f4b Compare October 17, 2024 15:20

increase test timeout

8d62dc6

Signed-off-by: Paweł Gronowski <[email protected]>

vvoland force-pushed the install-from-binimage branch from 6bc0f4b to 8d62dc6 Compare October 17, 2024 15:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker/install: Support `version: master` #438

docker/install: Support `version: master` #438

vvoland commented Sep 6, 2024 •

edited

Loading

crazy-max Sep 6, 2024

crazy-max Sep 6, 2024 •

edited

Loading

vvoland Sep 6, 2024

vvoland Sep 6, 2024

crazy-max Sep 9, 2024 •

edited

Loading

vvoland Sep 9, 2024

crazy-max Sep 9, 2024

vvoland Sep 9, 2024

vvoland Sep 9, 2024

vvoland commented Sep 9, 2024

crazy-max left a comment •

edited

Loading

crazy-max commented Oct 16, 2024 •

edited

Loading

vvoland commented Oct 16, 2024 •

edited

Loading

crazy-max commented Oct 16, 2024

vvoland commented Oct 16, 2024 •

edited

Loading

crazy-max commented Oct 16, 2024 •

edited

Loading

crazy-max Oct 17, 2024

crazy-max Oct 17, 2024

vvoland Oct 17, 2024

vvoland commented Oct 17, 2024

vvoland commented Oct 17, 2024 •

edited

Loading

docker/install: Support version: master #438

Are you sure you want to change the base?

docker/install: Support version: master #438

Conversation

vvoland commented Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

crazy-max Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazy-max Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvoland commented Sep 9, 2024

crazy-max left a comment • edited Loading

Choose a reason for hiding this comment

crazy-max commented Oct 16, 2024 • edited Loading

vvoland commented Oct 16, 2024 • edited Loading

crazy-max commented Oct 16, 2024

vvoland commented Oct 16, 2024 • edited Loading

crazy-max commented Oct 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvoland commented Oct 17, 2024

vvoland commented Oct 17, 2024 • edited Loading

docker/install: Support `version: master` #438

docker/install: Support `version: master` #438

vvoland commented Sep 6, 2024 •

edited

Loading

crazy-max Sep 6, 2024 •

edited

Loading

crazy-max Sep 9, 2024 •

edited

Loading

crazy-max left a comment •

edited

Loading

crazy-max commented Oct 16, 2024 •

edited

Loading

vvoland commented Oct 16, 2024 •

edited

Loading

vvoland commented Oct 16, 2024 •

edited

Loading

crazy-max commented Oct 16, 2024 •

edited

Loading

vvoland commented Oct 17, 2024 •

edited

Loading