Add the ability to dynamically capture server logs from a running server.

[ojarjur]

Problem

When you have a Jupyter server up and running, and encounter an unexpected issue, you might need debug logs from the server to triage the issue, but changing the existing log level on a server is very disruptive and might even be infeasible in the case of a Jupyter server being remotely hosted.

Additionally, the debug logs for Jupyter server are very verbose, so changing the log level and restarting the Jupyter server might make the root issue very difficult to find due to the amount of additional debug logs that have to be sifted through to find the root cause.

Proposed Solution

I would like to be able to capture the debug logs for a Jupyter server on the fly without having to restart it, and would like to be able to do so for just the period of time during which I am reproducing the issue I want to debug.

Additionally, I want to be able to do this from directly within the JupyterLab UI, but first the Jupyter server has to support it.

My proposal is to add two additional endpoints to the Jupyter server for capturing logs.

1. A startLogsCapture endpoint that takes a POST request and adds an in-memory handler to the ServerApp log.
2. A stopLogsCapture endpoint that takes a POST request and removes a previously added in-memory handler and returns the contents of all the logs it had received.

The request to start logs capture could include configuring things like the level of logs to capture, and a max amount of log entries to capture.

Additionally, these two endpoints should be optional, where a server administrator can disable them entirely if they do not want users to be able to access the server logs.

Additional context

This would be doable completely outside of the Jupyter server via a JupyterLab extension, but I believe it might be valuable enough that it would be used/desired by any client of the Jupyter server.

There is a longstanding proposal that is very similar to this about allowing the users to download logs. The differences between that proposal and this one are:

1. That proposal is concerned with all the logs written to stderr/stdout, whereas this one is only about the Python logs.
2. That proposal does not include any ability to change the log level, whereas the key goal of this one is that different levels of logging be able to be captured on the fly as needed by the user.

I believe the two proposals unblock non-overlapping scenarios, and ideally both would be implemented.