disabling of layers in builds (squash) is not respected

[akostadinov]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Trying to build image without layers still creates an image with layers.

Steps to reproduce the issue:

1. BUILDAH_LAYERS=false podman build ... --layers=false .
2. podman inspect 7efe57c60a46309fb7de852

Describe the results you received:

STEP 3: LABEL maintainer="My Team"
--> Using cache 22373ed96c6bbd3ad8e7d66a053d02861008051fc4aa86355efd14a54991620b
--> 22373ed96c6
STEP 4: RUN set -x && .....
--> Using cache 1d229bef2c1e84fcabf5a5cad6187786d9cb0825022f89f90d34f4062dd46e54
--> 1d229bef2c1
STEP 5: ADD . $HOME/verification-tests
--> 3433c722429
STEP 6: RUN scl enable...
....

btw caching is interrupted by the ADD command. I assume this is expected.

Then podman inspect shows under History a number of non-empty layers.

Describe the results you expected:

I expect layers are not used and image contain only one non-empty layer on top of base image.

Additional information you deem important (e.g. issue happens only occasionally):

This has worked properly in the past. if I'm not mistaken with podman-2:2.0.3-1.fc32.x86_64
With podman-2:2.1.1-7.fc32.x86_64 and 2.2 same issue.

Output of podman version:

Version:      2.2.0
API Version:  2.1.0
Go Version:   go1.14.10
Built:        Tue Dec  1 19:52:38 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 3
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: fedoraw
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 2000
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 90000
  kernel: 5.9.11-100.fc32.x86_64
  linkmode: dynamic
  memFree: 2313236480
  memTotal: 6176157696
  ociRuntime:
    name: crun
    package: crun-0.16-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 551727104
  swapTotal: 858779648
  uptime: 39h 57m 8.5s (Approximately 1.62 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/avalon/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.3.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/avalon/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 32
  runRoot: /run/user/1000/containers
  volumePath: /home/avalon/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1606845158
  BuiltTime: Tue Dec  1 19:52:38 2020
  GitCommit: ""
  GoVersion: go1.14.10
  OsArch: linux/amd64
  Version: 2.2.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.2.0-2.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora 32 virtual machine on a Fedora host.

[rhatdan]

Does this work properly with buildah bud?

[akostadinov]

I don't know about buildah bud. It is just a regression from previous behavior and what is documented in man podman run.

I noticed that --squash is now supported and it seems to be working as I expect. podman inspect now shows most layers as empty. So it is not clear what is the difference between --layers=false and --squash. I think documentation update is due and possibly deprecate outdated flags.

[rhatdan]

--layers=true is just a speed enhancement for building. It means that podman build will look to see if previous builds had successfully built this layer, so it will not be necessary to build it a gain.
With --layers=true or False, you will still see all of the layers.

--squash says to remove all layers down to the base layer.
With --squash=false, you should see only one squashed layer.

[akostadinov]

@rhatdan , In the past I've been building with BUILDAH_LAYERS=false and --layers=false to achieve squashing. Now --squash seems to be doing that . so It is kind of a behavior change.

Now the two options are also incompatible to be applied to the same build. Which makes some sense according to your explanations. And what I understand from your explanation is that --layers now only controls whether cached layers should be used in the build. But then when I run with --layers=false it still reuses existing layers.

In man page I see

  Cache intermediate images during the build process (Default is true).

  Note:  You can also override the default value of layers by setting the
  BUILDAH_LAYERS environment variable. export BUILDAH_LAYERS=true

In case we do NOT squash (and we can't specify --layers when squashing anyway), then how can we NOT cache layers? They
will be part of the resulting image anyway.

$ podman build -f Dockerfile -t docker-registry.upshift.redhat.com/aosqe/cucushift:oc46 --squash --layers=true .
Error: cannot specify --squash, --squash-all and --layers options together
$ podman build -f Dockerfile -t docker-registry.upshift.redhat.com/aosqe/cucushift:oc46 --squash --layers=false .
Error: cannot specify --squash, --squash-all and --layers options together

I hope I managed to explain where the confusion comes from. Let me try to summarize the points:

• behavior changed compared to the past
• when --squash is used, then --layers can't be used anyway
• when NOT using --squash then layers will be cached anyway; it is not a usage modifier either because with --layers=false I still observed cached layers used by the build (as can be seen in my log snippets earlier)

[rhatdan]

Ok What I see is
podman build --layers=false .

Causes layers to not be used.

If I add
BUILDAH_LAYERS=false podman build --layers=false .
or
BUILDAH_LAYERS=false podman build .
or
BUILDAH_LAYERS=false podman build --layers=true .

I see layers being used.