Improved monitoring/telemetry for Jupyter Notebooks

[ca-scribner]

To help detecting/fixing bugs, let's improve the information we collect from Jupyter Notebooks. Some ideas:

• Jupyter has logs that will record detailed failure info, etc. Can we always capture these and store then centrally for later use if users find issues?
• Are there ways to collect telemetry like how many python/r/... kernels have been run? when/what packages do people install? ...

More brainstorming would be helpful here.

[blairdrummond]

The answer is "Very Yes". We already have prometheus metrics available and we just need to pull them.

[blairdrummond]

We might be able to work with this:

kubeflow/kubeflow#5216

I think we'd be able to scrape for pods with the label notebook (like in our minio credential injector), and then scrape metrics using the blackbox exporter that @Ito-Matsuda has been playing around with.

It looks like it's possible to right a cross-namespace ServiceMonitor using that notebook label as a selector.

https://stackoverflow.com/a/64451100