"pebble logs" does not show logs of short-running / failing processes

[thp-canonical]

The command pebble logs doesn't capture stdout/stderr of short-running (kind of "oneshot") processes. The output of the service is only visible by inspecting the task of the "start" change.

How to reproduce

With a fresh Pebble checkout, build the binary:

go build ./cmd/pebble

In one terminal session, run Pebble with an empty pebble dir:

PEBBLE=$(pwd)/tmp/ ./pebble run --verbose

In another terminal session, create a new file test.yaml:

services:
  list-files:
    command: ls
    override: replace

Then, add this layer to the Pebble plan (of the running instance) and start it:

PEBBLE=$(pwd)/tmp/ ./pebble add test test.yaml
PEBBLE=$(pwd)/tmp/ ./pebble start list-files

The pebble start command prints:

error: cannot perform the following tasks:
- Start service "list-files" (cannot start service: exited quickly with code 0)

And trying to retrieve the log files produces empty output:

PEBBLE=$(pwd)/tmp/ ./pebble logs

However, changing the command to something that doesn't immediately exit makes Pebble properly take care of the process and capture its output (restart the Pebble daemon and run the above steps with the updated test.yaml from here):

services:
  list-files:
    command: bash -c "ls; sleep 10"
    override: replace

As especially the cases where the service fails to start are the "interesting" ones, having pebble logs output something in those cases would be nice. If not, maybe the command-line help of "pebble logs" should be updated to notice that logs are only captured for successfully-running services, not for failing services(?).

Doing a pebble changes and noticing the failed change ID and then running pebble tasks <change-id> (e.g. pebble tasks 1) DOES show the service output.

In any case, a service exiting with exit status 0 may in some cases be considered "successful" (maybe with a kind of "oneshot" service type?), and still cause pebble logs to contain its output?

[benhoyt]

Huh, yeah, this is an interesting one. At first I thought this was caused by the same issue as #56, but it's not really.

When the service doesn't start correctly (in 1s), we remove it from the list of running services (m.services) so that GET /v1/logs can't see it (it calls ServiceManager.Services and ServiceManager.ServiceLogs).

We should probably keep the service object (with its logs) around for a while, or until the service is next started.

This is somewhat related to #240: "Pebble does not try to restart service if exited too quickly for the first time".