Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker build cache broken after v24.0.7 when building "ltsc2019/1809" Windows container images on Windows 11 #47307

Closed
pbering opened this issue Feb 2, 2024 · 4 comments · Fixed by #47330
Assignees
Labels
area/builder/classic-builder Issues affecting the classic builder area/builder kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. platform/windows status/0-triage version/25.0
Milestone

Comments

@pbering
Copy link

pbering commented Feb 2, 2024

Description

Docker Desktop 4.27.x was release which now comes with Docker Engine v25.0.2, previous version 4.26.1 came with Docker Engine v24.0.7.

Using Docker Engine v25.0.2 to build images based on ltsc2019/1809 Windows container images breaking the build cache and nothing is cache or it is never used.

Reproduce

Using this Dockerfile:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2022

RUN echo "hello"

run docker image build . and you get expected first build output:

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
 ---> e2547c7dd84d
Step 2/2 : RUN echo "hello"
 ---> Running in 7e39ef7bbc05
"hello"
 ---> Removed intermediate container 7e39ef7bbc05
 ---> b8baf9fb556c

run docker image build . again and you get expected build output where the RUN statement is cached:

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
 ---> e2547c7dd84d
Step 2/2 : RUN echo "hello"
 ---> Using cache
 ---> b8baf9fb556c
Successfully built b8baf9fb556c

Now change the image to use ltsc2019:

FROM mcr.microsoft.com/windows/nanoserver:ltsc2019

RUN echo "hello"

run docker image build . and you get expected first build output:

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2019
 ---> 82ef3885248c
Step 2/2 : RUN echo "hello"
 ---> Running in 3c6de6846b97
"hello"
 ---> Removed intermediate container 3c6de6846b97
 ---> f55e40c44f69
Successfully built f55e40c44f69

run docker image build . yet again and now you can see the RUN statement is NOT CACHED:

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM mcr.microsoft.com/windows/nanoserver:ltsc2019
 ---> 82ef3885248c
Step 2/2 : RUN echo "hello"
 ---> Running in 3c6de6846b97
"hello"
 ---> Removed intermediate container 3c6de6846b97
 ---> f55e40c44f69
Successfully built f55e40c44f69

Expected behavior

Building ltsc2019/1809 images should use the build cache as it did before on v24.0.7.

docker version

failed to get console mode for stdout: The handle is invalid.
Client:
 Cloud integration: v1.0.35+desktop.10
 Version:           25.0.2
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Thu Feb  1 00:24:09 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.27.1 (136059)
 Engine:
  Version:          25.0.2
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       fce6e0c
  Built:            Thu Feb  1 00:23:01 2024
  OS/Arch:          windows/amd64
  Experimental:     false

docker info

failed to get console mode for stdout: The handle is invalid.
Client:
 Version:    25.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.3-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.22
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.3.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 970
 Server Version: 25.0.2
 Storage Driver: windowsfilter
  Windows: 
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 22631 (22621.1.amd64fre.ni_release.220506-1250)
 Operating System: Microsoft Windows Version 23H2 (OS Build 22631.3085)
 OSType: windows
 Architecture: x86_64
 CPUs: 24
 Total Memory: 63.91GiB
 Name: amd
 ID: d27c76ac-01ee-4ca5-b776-e0a90411f084
 Docker Root Dir: D:\Data\Docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional Info

No response

@pbering pbering added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. status/0-triage labels Feb 2, 2024
@vvoland vvoland self-assigned this Feb 2, 2024
@thaJeztah thaJeztah changed the title Docke build cache broken after v24.0.7 when building "ltsc2019/1809" Windows container images on Windows 11 Docker build cache broken after v24.0.7 when building "ltsc2019/1809" Windows container images on Windows 11 Feb 2, 2024
@vvoland
Copy link
Contributor

vvoland commented Feb 2, 2024

Thanks for reporting!

I'm taking a look into it.

@pbering
Copy link
Author

pbering commented Feb 2, 2024

Thanks @vvoland, let me know if you need anything and have a nice weekend!

@vvoland
Copy link
Contributor

vvoland commented Feb 7, 2024

@pbering The fix is included in the 25.0.3. Let me know if you have any further issues!

@pbering
Copy link
Author

pbering commented Feb 9, 2024

Confirmed working, thank you @vvoland !

idodod added a commit to earthly/dind that referenced this issue Apr 22, 2024
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [docker/docker](https://github.com/docker/docker) | patch | `25.0.1`
-> `25.0.5` |

---

### Release Notes

<details>
<summary>docker/docker (docker/docker)</summary>

### [`v25.0.5`](https://github.com/moby/moby/releases/tag/v25.0.5)

[Compare
Source](https://github.com/docker/docker/compare/v25.0.4...v25.0.5)

#### 25.0.5

For a full list of pull requests and changes in this release, refer to
the relevant GitHub milestones:

- [docker/cli, 25.0.5
milestone](https://github.com/docker/cli/issues?q=is%3Aclosed+milestone%3A25.0.5)
- [moby/moby, 25.0.5
milestone](https://github.com/moby/moby/issues?q=is%3Aclosed+milestone%3A25.0.5)
- Deprecated and removed features, see [Deprecated
Features](https://github.com/docker/cli/blob/v25.0.5/docs/deprecated.md).
- Changes to the Engine API, see [API version
history](https://github.com/moby/moby/blob/v25.0.5/docs/api/version-history.md).

##### Security

This release contains a security fix for [CVE-2024-29018], a potential
data exfiltration from 'internal' networks via authoritative DNS
servers.

##### Bug fixes and enhancements

- [CVE-2024-29018]: Do not forward requests to external DNS servers for
a container that is only connected to an 'internal' network. Previously,
requests were forwarded if the host's DNS server was running on a
loopback address, like systemd's 127.0.0.53.
[moby/moby#47589](https://github.com/moby/moby/pull/47589)
- plugin: fix mounting /etc/hosts when running in UserNS.
[moby/moby#47588](https://github.com/moby/moby/pull/47588)
- rootless: fix `open /etc/docker/plugins: permission denied`.
[moby/moby#47587](https://github.com/moby/moby/pull/47587)
- Fix multiple parallel `docker build` runs leaking disk space.
[moby/moby#47527](https://github.com/moby/moby/pull/47527)

[CVE-2024-29018]:
https://github.com/moby/moby/security/advisories/GHSA-mq39-4gv4-mvpx

### [`v25.0.4`](https://github.com/moby/moby/releases/tag/v25.0.4)

[Compare
Source](https://github.com/docker/docker/compare/v25.0.3...v25.0.4)

#### 25.0.4

For a full list of pull requests and changes in this release, refer to
the relevant GitHub milestones:

- [docker/cli, 25.0.4
milestone](https://github.com/docker/cli/issues?q=is%3Aclosed+milestone%3A25.0.4)
- [moby/moby, 25.0.4
milestone](https://github.com/moby/moby/issues?q=is%3Aclosed+milestone%3A25.0.4)
- Deprecated and removed features, see [Deprecated
Features](https://github.com/docker/cli/blob/v25.0.4/docs/deprecated.md).
- Changes to the Engine API, see [API version
history](https://github.com/moby/moby/blob/v25.0.4/docs/api/version-history.md).

##### Bug fixes and enhancements

- Restore DNS names for containers in the default "nat" network on
Windows. [moby/moby#47490](https://github.com/moby/moby/pull/47490)
- Fix `docker start` failing when used with `--checkpoint`
[moby/moby#47466](https://github.com/moby/moby/pull/47466)
- Don't enforce new validation rules for existing swarm networks
[moby/moby#47482](https://github.com/moby/moby/pull/47482)
- Restore IP connectivity between the host and containers on an internal
bridge network.
[moby/moby#47481](https://github.com/moby/moby/pull/47481)
- Fix a regression introduced in v25.0 that prevented the classic
builder from ADDing a tar archive with xattrs created on a non-Linux OS
[moby/moby#47483](https://github.com/moby/moby/pull/47483)
- containerd image store: Fix image pull not emitting `Pulling fs layer`
status [moby/moby#47484](https://github.com/moby/moby/pull/47484)

##### API

- To preserve backwards compatibility, make read-only mounts not
recursive by default when using older clients (API version < v1.44).
[moby/moby#47393](https://github.com/moby/moby/pull/47393)
- `GET /images/{id}/json` omits the `Created` field (previously it was
`0001-01-01T00:00:00Z`) if the `Created` field is missing from the image
config. [moby/moby#47451](https://github.com/moby/moby/pull/47451)
- Populate a missing `Created` field in `GET /images/{id}/json` with
`0001-01-01T00:00:00Z` for API version <= 1.43.
[moby/moby#47387](https://github.com/moby/moby/pull/47387)
- Fix a regression that caused API socket connection failures to report
an API version negotiation failure instead.
[moby/moby#47470](https://github.com/moby/moby/pull/47470)
- Preserve supplied endpoint configuration in a container-create API
request, when a container-wide MAC address is specified, but
`NetworkMode` name-or-id is not the same as the name-or-id used in
`NetworkSettings.Networks`.
[moby/moby#47510](https://github.com/moby/moby/pull/47510)

##### Packaging updates

- Upgrade Go runtime to
[1.21.8](https://go.dev/doc/devel/release#go1.21.8).
[moby/moby#47503](https://github.com/moby/moby/pull/47503)
- Upgrade RootlessKit to
[v2.0.2](https://github.com/rootless-containers/rootlesskit/releases/tag/v2.0.2).
[moby/moby#47508](https://github.com/moby/moby/pull/47508)
- Upgrade Compose to
[v2.24.7](https://github.com/docker/compose/releases/tag/v2.24.7).
[https://github.com/docker/docker-ce-packaging/pull/998](https://github.com/docker/docker-ce-packaging/pull/998)
- Upgrade Buildx to
[v0.13.0](https://github.com/docker/buildx/releases/tag/v0.13.0).
[https://github.com/docker/docker-ce-packaging/pull/997](https://github.com/docker/docker-ce-packaging/pull/997)

**Full Changelog**:
moby/moby@v25.0.3...v25.0.4

### [`v25.0.3`](https://github.com/moby/moby/releases/tag/v25.0.3)

[Compare
Source](https://github.com/docker/docker/compare/v25.0.2...v25.0.3)

#### 25.0.3

For a full list of pull requests and changes in this release, refer to
the relevant GitHub milestones:

- [docker/cli, 25.0.3
milestone](https://github.com/docker/cli/issues?q=is%3Aclosed+milestone%3A25.0.3)
- [moby/moby, 25.0.3
milestone](https://github.com/moby/moby/issues?q=is%3Aclosed+milestone%3A25.0.3)

##### Bug fixes and enhancements

- containerd image store: Fix a bug where `docker image history` would
fail if a manifest wasn't found in the content store.
[moby/moby#47348](https://github.com/moby/moby/pull/47348)
- Ensure that a generated MAC address is not restored when a container
is restarted, but a configured MAC address is preserved.
[moby/moby#47304](https://github.com/moby/moby/pull/47304)

    > **Note**
    >
> - Containers created with Docker Engine version 25.0.0 may have
duplicate MAC addresses.
    >     They must be re-created.
> - Containers with user-defined MAC addresses created with Docker
Engine versions 25.0.0 or 25.0.1
> receive new MAC addresses when started using Docker Engine version
25.0.2.
    >     They must also be re-created.

<!---->

- Fix `docker save <image>@&#8203;<digest>` producing an OCI archive
with index without manifests.
[moby/moby#47294](https://github.com/moby/moby/pull/47294)
- Fix a bug preventing bridge networks from being created with an MTU
higher than 1500 on RHEL and CentOS 7.
[moby/moby#47308](https://github.com/moby/moby/issues/47308),
[moby/moby#47311](https://github.com/moby/moby/pull/47311)
- Fix a bug where containers are unable to communicate over an
`internal` network.
[moby/moby#47303](https://github.com/moby/moby/pull/47303)
- Fix a bug where the value of the `ipv6` daemon option was ignored.
[moby/moby#47310](https://github.com/moby/moby/pull/47310)
- Fix a bug where trying to install a pulling using a digest revision
would cause a panic.
[moby/moby#47323](https://github.com/moby/moby/pull/47323)
- Fix a potential race condition in the managed containerd supervisor.
[moby/moby#47313](https://github.com/moby/moby/pull/47313)
- Fix an issue with the `journald` log driver preventing container logs
from being followed correctly with systemd version 255.
[moby/moby47243](https://github.com/moby/moby/pull/47243)
- seccomp: Update the builtin seccomp profile to include syscalls added
in kernel v5.17 - v6.7 to align the profile with the profile used by
containerd. [moby/moby#47341](https://github.com/moby/moby/pull/47341)
- Windows: Fix cache not being used when building images based on
Windows versions older than the host's version.
[moby/moby#47307](https://github.com/moby/moby/pull/47307),
[moby/moby#47337](https://github.com/moby/moby/pull/47337)

##### Packaging updates

- Removed support for Ubuntu Lunar (23.04).
[docker/ce-packaging#986](https://github.com/docker/docker-ce-packaging/pull/986)

### [`v25.0.2`](https://github.com/moby/moby/releases/tag/v25.0.2)

[Compare
Source](https://github.com/docker/docker/compare/v25.0.1...v25.0.2)

#### 25.0.2

For a full list of pull requests and changes in this release, refer to
the relevant GitHub milestones:

- [docker/cli, 25.0.2
milestone](https://github.com/docker/cli/issues?q=is%3Aclosed+milestone%3A25.0.2)
- [moby/moby, 25.0.2
milestone](https://github.com/moby/moby/issues?q=is%3Aclosed+milestone%3A25.0.2)

##### Security

This release contains security fixes for the following CVEs
affecting Docker Engine and its components.

| CVE | Component | Fix version | Severity |
| ----------------------------------------------------------- |
------------- | ----------- | ---------------- |
| [CVE-2024-21626](https://scout.docker.com/v/CVE-2024-21626) | runc |
1.1.12 | High, CVSS 8.6 |
| [CVE-2024-23651](https://scout.docker.com/v/CVE-2024-23651) | BuildKit
| 1.12.5 | High, CVSS 8.7 |
| [CVE-2024-23652](https://scout.docker.com/v/CVE-2024-23652) | BuildKit
| 1.12.5 | High, CVSS 8.7 |
| [CVE-2024-23653](https://scout.docker.com/v/CVE-2024-23653) | BuildKit
| 1.12.5 | High, CVSS 7.7 |
| [CVE-2024-23650](https://scout.docker.com/v/CVE-2024-23650) | BuildKit
| 1.12.5 | Medium, CVSS 5.5 |
| [CVE-2024-24557](https://scout.docker.com/v/CVE-2024-24557) | Docker
Engine | 25.0.2 | Medium, CVSS 6.9 |

The potential impacts of the above vulnerabilities include:

-   Unauthorized access to the host filesystem
-   Compromising the integrity of the build cache
- In the case of CVE-2024-21626, a scenario that could lead to full
container escape

For more information about the security issues addressed in this
release,
refer to the [blog
post](https://www.docker.com/blog/docker-security-advisory-multiple-vulnerabilities-in-runc-buildkit-and-moby/).
For details about each vulnerability, see the relevant security
advisory:

-
[CVE-2024-21626](https://github.com/opencontainers/runc/security/advisories/GHSA-xr7r-f8xq-vfvv)
-
[CVE-2024-23651](https://github.com/moby/buildkit/security/advisories/GHSA-m3r6-h7wv-7xxv)
-
[CVE-2024-23652](https://github.com/moby/buildkit/security/advisories/GHSA-4v98-7qmw-rqr8)
-
[CVE-2024-23653](https://github.com/moby/buildkit/security/advisories/GHSA-wr6v-9f75-vh2g)
-
[CVE-2024-23650](https://github.com/moby/buildkit/security/advisories/GHSA-9p26-698r-w4hx)
-
[CVE-2024-24557](https://github.com/moby/moby/security/advisories/GHSA-xw73-rw38-6vjc)

##### Packaging updates

- Upgrade containerd to
[v1.6.28](https://github.com/containerd/containerd/releases/tag/v1.6.28).
- Upgrade containerd to v1.7.13 (static binaries only).
[moby/moby#47280](https://github.com/moby/moby/pull/47280)
- Upgrade runc to v1.1.12.
[moby/moby#47269](https://github.com/moby/moby/pull/47269)
- Upgrade Compose to v2.24.5.
[docker/docker-ce-packaging#985](https://github.com/docker/docker-ce-packaging/pull/985)
- Upgrade BuildKit to v0.12.5.
[moby/moby#47273](https://github.com/moby/moby/pull/47273)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "after 6am on monday" (UTC), Automerge
- At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/earthly/dind).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yOTMuMCIsInVwZGF0ZWRJblZlciI6IjM3LjI5My4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZSJdfQ==-->

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: idodod <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/builder/classic-builder Issues affecting the classic builder area/builder kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. platform/windows status/0-triage version/25.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants