Make debugging of our containers easier

[mnagy]

Debugging our database images has been a bit of a pain for me, especially when I do not have access to the machine where the container in question is being run (think QE and cooperation through issues/bugzilla bugs). I often find myself in a need to craft a custom image with debugging statements which is often tedious and can waste a lot of time (especially if OpenShift does not use the image I want).

It's hard to make sense from the container logs. I don't know if something is being printed out by the entrypoint script, database initialization script, or the final exec'd binary. In case of either MySQL or PostgreSQL (don't remember which), the last line runs along the lines of "server is shutting down". This is from the pre-exec'd process and it doesn't indicate failure. For a user who does not know how the image works, this must be extremely confusing.

Proposed solutions:

1. Let's have a unified way of printing out information about what we are doing and use that. Let's tell the user that we're starting a mysql/postgres daemon just to check/modify the database. Let's tell him we're calling the final exec. Let's tell him when and what failed. Let's make sure the output from entrypoint script stands out and that it's recognized who prints out what.
2. Let's have a debugging mode. It could at least do a set -x. This will be helpful for us. We're just going to tell our users to set a variable and send us output from that. We should also check out the volumes. We often have to deal with permission-related bugs or user mistakes (nfs). Putting couple of ls -la statements on volumes might provide useful information. As well as calling id.

@bparees @thesteve0 @rhcarvalho @hhorak @praiskup @eliskasl

[praiskup]

Something like this?
$ docker run --rm -ti -e CONT_DEBUG=3 -e POSTGRESQL_CONTAINER_OPTS='assert_external_data = false' fedora/postgresql

[rhcarvalho]

I like the item 1, a bit more verbose but easier to understand the transitions between each part. PostgreSQL's log "server is shutting down" looks indeed confusing.

From item 2 I'd pick only ls -la and id when looking at the data dir, but set -x I'd leave out, as it won't gain us much after exec, and for the other parts we have context to tell what was happening (as it would get better by doing item 1).

[mnagy]

@praiskup I'm not 100% sure we want to use any kinds of levels, but maybe.

@rhcarvalho now I'm thinking about doing id in all cases when the container starts.

[praiskup]

something like mount command could be helpful too

[bparees]

@mnagy +1, this sounds like an excellent idea.

[hhorak]

@mnagy I like the idea, we need something for debugging. Instead of ls -la I'd suggest ls -laZ so we also cover SELinux issues.

[mnagy]

@hhorak, @praiskup agree. I'm also thinking of using -n instead of -l. It's sometimes useful to see uids (especially when using NFS).

Any takers for first implementation? :)

[mnagy]

Also, a question. I don't like very much the name of the variable proposed, "CONT_DEBUG". I don't think I have any reasonable objective objections, I just don't like the sound of it.

Anyone have any better ideas? Any reason why we should not use an application specific name, such as MYSQL_DEBUG? It would be inline with environment variables that we have now.

[rhcarvalho]

MYSQL_DEBUG

The problem is that we're not debugging MySQL itself, but our image.
We should leave MYSQL_* to options that will change MySQL's config.

I thought that "CONT_DEBUG" was short for "CONTAINER_DEBUG".

Why don't we simply have those improvements as part of our images without introducing a new env var? IOW, combine @mnagy's idea 1+2 and have clear messages and debug mode always on.
It would make life easier, we don't need to ask "please enable XYZ and send the logs".

[hhorak]

Some general name will allow us to share code of some bash functions if we have any. CONT_DEBUG seems fine to me, since it's obvious what it does.

Also, output of set -x uses + by default as prefix when telling what command is being executed. I don't find this way very clear, so I'd propose to change it to PS4='+CONT_DEBUG '

Then secret data like passwords, that's something that shouldn't be seen in logs at all, but with set -x it would be hard to hide it.

[bparees]

Debug always on is ugly if we're going to be running ls - alZ, plus the
password issue.

You could make the same argument about defaulting build logging to verbose
and we don't do it for the same reason.

Why not just call the variable CONTAINER_DEBUG? Abbreviating it doesn't
seem worth it, and having a common variable name across our containers
seems more user friendly. Plus it really is debugging the container, not
just mysql.

Ben Parees | OpenShift
On Nov 26, 2015 9:17 AM, "Rodolfo Carvalho" [email protected]
wrote:

MYSQL_DEBUG

The problem is that we're not debugging MySQL itself, but our image.
We should leave MYSQL_* to options that will change MySQL's config.

I thought that "CONT_DEBUG" was short for "CONTAINER_DEBUG".

Why don't we simply have those improvements as part of our images without
introducing a new env var? IOW, combine @mnagy https://github.com/mnagy's
idea 1+2 and have clear messages and debug mode always on.
It would make life easier, we don't need to ask "please enable XYZ and
send the logs".

—
Reply to this email directly or view it on GitHub
#112 (comment).

[mfojtik]

• CONTAINER_DEBUG, please remove the CONT_ from your minds :-))

If we are going to print any debugging informations, they have to be properly annotated and printed in a way that one can understand what he see. Each section should have extra echo message saying "[DEBUG] This is XYZ"

Otherwise you will just get tons of messages and finding a bug there will be pain.

But I like the idea

[mnagy]

Ping? I'm finding myself debugging MySQL right now and the debugging functionality would come in handy. Any taker for this issue?

[bparees]

@mnagy i think you're in the best position to implement the feature.

[mfojtik]

i'm nominating @mnagy as well :-)