Run retrace in a podman container instead of mock

[michalfabik]

This PR uses the existing mechanisms currently used for mock config to create a Dockerfile in the crash directory. This is then used to create a podman container into which the coredump and metadata and a gdb.sh script are copied and where the retrace itself is run. The use of podman is optional and enabled by setting the newly introduced RetraceEnvironment config variable to "podman". Changes to the retrace user are required during deployment in order to run retrace in a podman container, as described in DEPLOYING.md.

[ernestask]

Why remove the exploitable check?

[michalfabik]

Why remove the exploitable check?

I don't remember quite precisely to be honest but the gist is that the exploitable plugin (/usr/libexec/abrt-gdb-exploitable) is now part of the abrt-addon-ccpp package which gets installed in the Dockerfile (or the equivalent mock configuration) just before the check happens, so checking for its presence makes no sense. @mkutlak suggested doing away with the checks completely.

Now I see I shoud remove the EXPLOITABLE_PLUGIN_PATH variable as well, since it isn't used anywhere else.

[ernestask]

I don't remember quite precisely to be honest but the gist is that the exploitable plugin (/usr/libexec/abrt-gdb-exploitable) is now part of the abrt-addon-ccpp package which gets installed in the Dockerfile (or the equivalent mock configuration) just before the check happens, so checking for its presence makes no sense. @mkutlak suggested doing away with the checks completely.

Now I see I shoud remove the EXPLOITABLE_PLUGIN_PATH variable as well, since it isn't used anywhere else.

Right, that makes sense. Which checks did Martin want to get rid of, exactly?

[mkutlak]

I would add a weak dependecy on podman to the spec file.

[mkutlak]

I encourted a problem with this. I already have a retrace user on my system (from older installation) with a home dir /usr/share/retrace.

Upgrading retrace-server does not change the home dir to /var/lib/retrace.

https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/

[michalfabik]

Seems to be working now. It's quite ugly because simple usermod -m -d will fail if there is a process run by the user retrace. So I just userdel -f the user and then add it again with a different home dir.

Side note for @ernestask: userdel removes the appropriate entries from /etc/sub[ug]id but it leaves the empty lines in place, meaning that the files did end up slightly malformed. After many testing installs and uninstalls they contained one entry for my user followed by many \n's. Fortunately, sort -n puts the blank lines at the top but I added a sed call to get rid of those anyway.

[ernestask]

In cut(1):

       -s, --only-delimited
              do not print lines not containing delimiters

That should do the trick as well.

[ernestask]

By “malformed” I meant anything that isn’t empty or in the right format. Probably should have clarified.

[michalfabik]

I dropped this whole part from the .spec file altogether and just documented it instead.

[abrt-bot]

Can one of the admins verify this patch?

[mkutlak]

@trapas add to whitelist

[mkutlak]

1. It would be good to have podman honor RequireGPGCheck from config.
2. Fix indentation commits squashed into 1.
3. Maybe use run() instead of Popen/call.

[mkutlak]

Thank you @michalfabik

[DaveWysochanskiRH]

This is very good work!

Unfortunately there must be something missed here because I'm seeing regressions in my testing which does not use either mock or podman but just vmcores and crash directly.

If I build on 45ed3a6 it gives "not enough arguments for format string"
[2020-01-29 15:14:06] [I] Calling prepare_debuginfo
[2020-01-29 15:14:06] [I] Version: '2.6.32'; Release: '504.el6'; Arch: 'x86_64'; _arch: 'x86_64'; Flavour: 'None'; Realtime: False
[2020-01-29 15:14:06] [I] Found cached vmlinux at path: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-504.el6.x86_64/vmlinux
[2020-01-29 15:14:06] [I] Searching for kernel-debuginfo package for 2.6.32-504.el6.x86_64
[2020-01-29 15:14:06] [E] prepare_debuginfo failed: not enough arguments for format string

Earlier commits will fail with "local variable 'container_id' referenced before assignment:
[2020-01-29 14:56:52] [I] Found cached vmlinux at path: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-504.el6.x86_64/vmlinux
[2020-01-29 14:56:52] [I] Searching for kernel-debuginfo package for 2.6.32-504.el6.x86_64
[2020-01-29 14:56:57] [I] Generating backtrace
[2020-01-29 14:56:57] [I] Stripping to 1 would have no effect
[2020-01-29 14:57:02] [E] local variable 'container_id' referenced before assignment

Here is the bisect log for the first failing commit about 'container_id':

$ git bisect log
git bisect start
# bad: [c7a97005054a9de84de32dcc33b2384d232b8dcd] Run hooks with podman as well
git bisect bad c7a97005054a9de84de32dcc33b2384d232b8dcd
# good: [437e002ac01c04d67f5a91fdb67ab7cfea2cab5e] Add RetraceEnvironment config item
git bisect good 437e002ac01c04d67f5a91fdb67ab7cfea2cab5e
# bad: [964ebe03a1f3733b3b1f7c710fbdee39175e8439] Fix indentation
git bisect bad 964ebe03a1f3733b3b1f7c710fbdee39175e8439
# good: [efd0a6f0162c339de3eeffb5506a8c074d86e000] Make mock-specific parts conditional
git bisect good efd0a6f0162c339de3eeffb5506a8c074d86e000
# bad: [a8f0cdd16432f2d19968de8a79e9198774dceec5] Run retrace in podman container
git bisect bad a8f0cdd16432f2d19968de8a79e9198774dceec5
# good: [5689ef8afdea417c87e165c38d26e700eb086711] Create Dockerfiles
git bisect good 5689ef8afdea417c87e165c38d26e700eb086711
# first bad commit: [a8f0cdd16432f2d19968de8a79e9198774dceec5] Run retrace in podman container

[DaveWysochanskiRH]

FWIW, I think we can remove the 'set_mock' and 'get_mock' and related logic as that is legacy code that was a wart anyway and it's not needed anymore. That might be related to this failure but I'm not sure yet.

[DaveWysochanskiRH]

Well a bug in run_crash_cmd exception handling code does not help (facepalm)
#293

[DaveWysochanskiRH]

Something in fd40267 needs fixed up I think.

[DaveWysochanskiRH]

Simple fixup b319f0a