ship system logs to cloudwatch

[errordeveloper]

We need a way to make troubleshooting node bootstrap without having to enable SSH, and this would be useful generally also. Mostly we just need cloud-init logs and kubelet logs, but if we could ship the entire journal, it could be handy. We consider using journald's built-in exporter capability (which will not require us to use any storage).

[mumoshu]

@errordeveloper Sounds great 👍

FYI, I've been using journald-cloudwatch-logs for 2 years or so without any issue.

https://github.com/kubernetes-incubator/kube-aws/blob/master/docs/add-ons/journald-logging-to-cloudwatch.md

[errordeveloper]

@mumoshu it'd be nice if we could actually run something outside of kubelet, so that we can use it to debug nodes that fail to join. It'd be great if we didn't have to introduce new software, so I was thinking of systemd-journal-gatewayd, but that will require an additional connector for shipping the logs to cloudwatch (unless we just let it serve them over HTTP, accessible inside the VPC...).
Alternatively, we could consider a few options, like use systemd-journal-gatewayd with Lambda function to import the logs into CloudWatch, or even use docker run --log-driver=awslogs and run a process that reads out the journal inside of that container (yet that's not too different from docker run jollinshead/journald-cloudwatch-logs). However depending on Docker for this is not particularly exciting either, as switching to containerd must soon become possible. So I'm quite puzzled.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.