podman version 1.4.1 errors when starting containers - "run failed after [conmon:e]: No such log driver json-file"

[rlandy]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Using podman version 1.4.1 on RHEL 8 (updated) we see the following error when starting containers:

ERROR: 27947 -- ['/usr/bin/podman', 'start', '-a', 'container-puppet-glance_api'] run failed after [conmon:e]: No such log driver json-file",
"Error: unable to start container : write child: broken pipe",
...
ERROR: 27949 -- ['/usr/bin/podman', 'start', '-a', 'container-puppet-iscsid'] run failed after [conmon:e]: No such log driver json-file"
Repeated for all containers.

Steps to reproduce the issue:

1. Install podman version 1.4.1 on RHEL 8 with updated rpms

2. Run containers

3. See driver error

Describe the results you received:

The following errors on every attempt to run/start containers:

WARNING: 27947 -- Retrying running container: glance_api",
ERROR: 27946 -- ['/usr/bin/podman', 'start', '-a', 'container-puppet-crond'] run failed after [conmon:e]: No such log driver json-file",
"Error: unable to start container : write child: broken pipe",
2019-06-18 14:59:02 | " attempt(s): 2",

Describe the results you expected:

No errors when starting containers.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.4.1
RemoteAPI Version:  1
Go Version:         go1.12.6
OS/Arch:            linux/amd64

+ podman info
host:
  BuildahVersion: 1.8.4
  Conmon:
    package: podman-1.4.1-2.module+el8.1.0+3401+5fb5c576.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 0.3.0, commit: unknown'
  Distribution:
    distribution: '"rhel"'
    version: "8.0"
  MemFree: 326901760
  MemTotal: 8191172608
  OCIRuntime:
    package: runc-1.0.0-57.rc5.dev.git2abd837.module+el8.1.0+3394+8ce796a4.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 8579956736
  SwapTotal: 8589930496
  arch: amd64
  cpus: 8
  hostname: standalone.localdomain
  kernel: 4.18.0-80.el8.x86_64
  os: linux
  rootless: false
  uptime: 1h 19m 7.14s (Approximately 0.04 days)
registries:
  blocked: null
  insecure:
  - 192.168.24.1
  search:
  - registry.redhat.io
  - quay.io
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 48
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 60
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

[mheon]

Seems like it could be a Conmon version mismatch - @haircommander can you confirm?

[mheon]

@haircommander Assuming it is... maybe we need something similar to what c/storage does for overlayfs - code to probe Conmon to determine what features it supports

[haircommander]

Actually somehow log driver is being set as json-file, which used to be a noop but now is an error (we used to pretend we supported json file). @rlandy can you post your libpod.conf? And what image are you trying to run?

[haircommander]

@rlandy you should be able to do your create/run command above with --log-driver k8s-file to fix your problem. It doesn't appear the libpod.conf has a member for log_driver, so I'm not sure where json-file is coming from...

[cjeanner]

Hello,
Basically, dropping the "json-file" broke things. That kind of change shouldn't happen like that imho, since it was the only driver available for previous versions..... Not really cool :(.

[mheon]

I think this one's my fault. I didn't realize we already had a --log-driver flag when we added support for journald logging (and implemented the backend for the --log-driver flag), so I requested that we remove an alias of json-file to k8s-file in code review to leave it open for future implementation (the default log driver which we use isn't actually logging JSON, or to any sort of standardized format for that matter).

It seems unlikely that we're going to see proper JSON logging implemented any time soon, so we could discuss aliasing json-file to k8s-file (I'm still a bit leery because it's effectively lying to the user. We're definitely not logging JSON).

[cjeanner]

@mheon thanks for your explanation.

I don't think aliasing now the "non-json" «json-file» to k8s-file is the right thing to do - we're already changing things in our project (TripleO) - it should be good soon. Hopefully ;).

[amoralej]

Even if using "json-file" for a non-json log is uggly, i'd say that keeping some sort of backwards compatibility wins unless there have been proper deprecation period, notification, etc... I'm not sure if it's the case here, but IMHO backwards compatibility wins.

[haircommander]

Added a fix in #3375 . Now, we will make json-file an alias, but give an error message describing such :)

[sshnaidm]

from docs:

--log-driver="k8s-file"
Logging driver for the container. Currently not supported. This flag is a NOOP provided solely for scripting compatibility.

https://github.com/containers/libpod/blob/master/docs/podman-run.1.md
Is it still correct?

[haircommander]

aah nope that is no longer correct. Thanks @sshnaidm for pointing that out

[haircommander]

Added a doc fix in #3375 as well