enable journal logging to kernel ring buffer #445
Conversation
This will enable early log messages from systemd service to go to the kernel buffer which means they should be seen on various terminals. Even if a user is not looking at the correct primary console of the machine they should still get the error messages because now they would be going to all console= device and not just the primary one.
|
Can one of the admins verify this patch? |
| # Also disable rate-limiting at kernel ring buffer level | ||
| { | ||
| echo "kernel.printk_devkmsg = on" | ||
| } >> "$initdir/etc/sysctl.d/10-dont-ratelimit-kmsg.conf" |
There was a problem hiding this comment.
the biggest problem here is that the default for this setting (ratelimit) will not get restored after this has run. I don't know how to resolve that. Really I only want to change it to on during initramfs and after that drop it back to whatever it was set to before (either the default or the value from the kernel command line).
There was a problem hiding this comment.
If printk.devkmsg= is set from the kernel command line, this won't even work, at least according to the docs:
printk_devkmsg:
Control the logging to /dev/kmsg from userspace:
ratelimit: default, ratelimited
on: unlimited logging to /dev/kmsg from userspace
off: logging to /dev/kmsg disabled
The kernel command line parameter printk.devkmsg= overrides this and is
a one-time setting until next reboot: once set, it cannot be changed by
this sysctl interface anymore.
If you set it via sysctl, though, it shouldn't be any problem setting it back to ratelimit again once journald is online. The lockout only seems to apply when it's set from the kernel command line. (At least, I just flipped it between on and ratelimit three times on my F28 box using sysctl, with no apparent problems.)
There was a problem hiding this comment.
ahh, i think that's actually good news. IOW if a user sets it from the CLI then we don't have to worry about stepping on them. so if we set it and then set it back to the default after switch root then we'd be fine
|
bump. would like to at least have a discussion about this. |
|
ping @poettering |
|
I am not sure I follow. So far we shied away from logging to kmsg except when we have no other place to log to (because the kernel folks are pretty nasty about it, see LWN). Changing the default in Fedora to log to kmsg automatically certainly has the power to piss off those people again, and quite frankly I am a bit tired of dealing with those folks. I am not entirely sure why this is needed though. AFAICS the main change from this would be that people could type "dmesg" rather than "journalctl" to get all system logs during the initrd, but I am not sure that is such a great benefit, no? What am I missing? |
understood. does it help that this would only have effect for the journal during the initrd. After switch root the settings from the jourald.conf from the root filesystem would take over.
that depends. a lot of times people might not have actual access to the correct console device that is configured in the image so they aren't able to "type" Part of the reason for this request is we are starting to do more provisioning of systems in the initramfs using github.com/coreos/ignition and making it so that regardless of how people are using the images we produce they will get relevant log output would be super useful. I honestly think this information would be valuable for all of fedora to have, which is why I opened this PR. The biggest open question I have is related to the ratelimiting, which I mentioned above. |
|
friendly ping @poettering - I tried to answer your questions above ^^ |
well, but that's still a lot of logging in the usual case. i mean the main part of its logging systemd does during boot…
Wouldn't a better approach be to fix /dev/console in the kernel to multiplex to all console= exactly like kmsg does it? Quite frankly, if this is just about wanting multiplexing to multiple consoles then kmsg sounds like an awfully big hammer to me. I am pretty sure that in the generic versions of Fedora this really shouldn't be turned on. Maybe turning this on as proposed in the cloud edition of Fedora might be OK, but it still appears to be a hack to me, and a proper fix would be to add multiplexing to /dev/console inside the kernel as suggested... |
|
also: serial ports are actually really really slow, even emulated VM ones. I am a bit concerned about doing multiplexing too agressively (i.e. to possibly unconnected serial potrs and stuff) by default, as that will seriously slow down the boot process. |
|
(and sorry for the late reply, been travelling) |
yes. That is exactly what I want, but I don't know how to make that happen or even if it would be accepted. |
|
bump. I hit this again today. One example is where you are trying to build a disk using the same kickstart for more than one architecture so you add a bunch of any idea how we could make the fix /dev/console in the kernel to multiplex to all console= exactly like kmsg does it approach happen? |
not really, it's the (early) boot-time message those kernel folks had issues with, it's anyway where 95% of our log messages are generated, hence saying "only during early boot" is not that great... |
prep a kernel patch for that? of course this would need to be opt-in (probably compile time), for compat reasons, but we'd turn this on in fedora kernels by default. |
FYI opened discussion on kernel mailing list to see if this is a patch that would be accepted. |
|
@dustymabe any progress? |
johannbg
added
modules
|
This issue is being marked as stale because it has not had any recent activity. It will be closed if no further activity occurs. If this is still an issue in the latest release of Dracut and you would like to keep it open please comment on this issue within the next 7 days. Thank you for your contributions. |
6 participants
This will enable early log messages from systemd service to go
to the kernel buffer which means they should be seen on various
terminals. Even if a user is not looking at the correct primary
console of the machine they should still get the error messages
because now they would be going to all console= device and not
just the primary one.