Make logging verbosity configurable per package for verbose packages.

[indam23]

What problem are you trying to solve?

Logs from pika (rabbitmq) and kafka libraries can be very verbose, cluttering debug logs.
Usually turning on --debug mode is done to debug model prediction.

What's your suggested solution?

Make logging configurable for the more verbose libraries separately from overall log level.
These could be env vars e.g. PIKA_DEBUG_LEVEL, KAFKA_DEBUG_LEVEL.

Some investigation required to see which other libraries are most verbose; these are the most obvious ones.

Examples (if relevant)

No response

Is anything blocking this from being implemented? (if relevant)

None known

Definition of Done

• logging made configurable for specific overly verbose packages [Kafka, Pika, matplotlib for starters]

[b-quachtran]

Would be great to get an option for this in Rasa X as well

[tayfun]

I've been working on this ticket. There are a few problems that I can see:

1. First is that we already have LOG_LEVEL_LIBRARIES which is not documented, and not used for matplotlib correctly (eg. loaded modules are printed by matplotlib because the parent logger level is not set with LOG_LEVEL_LIBRARIES). I can fix that easily and also add a specific setting just for matplotlib/kafka/pika etc.

2. Current default log level is INFO. So when we run rasa shell we see INFO logs (and other higher level logs ie. WARNING, ERROR, CRITICAL). Docs say default is None which is actually correct but None for verbosity actually means INFO in practice :) This also means that -v doesn’t do anything as it should be setting log level to INFO and this is default as I’ve said. What do we want default log level to be? I feel INFO level is what everyone really uses so that we can see what’s going on, so this should stay? I can update docs to say INFO is default but then -v doesn’t do anything and we’d be missing ERROR (though we do have --quiet for WARNING which is good enough I think - it’s also normally the default for Python programs as in root logger comes with WARNING level).

Note we had already worked on a similar problem with #6900

I'm planning on adding specific log levels for 3 of the libraries (rabbitmq, matplotlib and kafka). I'm renaming pika to rabbitmq since it is really about connecting to RabbitMQ and also because the loggers to disable are aio_pika and aiormq.

[tayfun]

Just noticed in addition to undocumented LOG_LEVEL_LIBRARIES we also have the ability to set loglevel in endpoint configuration, but only for kafka at the moment :D (pika does not have this capability).

I wonder what is the best policy here? Options are:

1. Env variables for general (LOG_LEVEL_LIBRARIES) and specific libraries can override (LOG_LEVEL_RABBITMQ etc.)
2. For matplotlib etc. use 1 but for endpoints, use its own config so support loglevel in endpoints.yml file for pika etc.
3. Support a single logging config file from where you can easily set log levels for everything (this is most advanced but requires a bit more knowledge).