Make log level configurable

[timebertt]

How to categorize this PR?

/area logging usability
/kind enhancement
/priority normal

What this PR does / why we need it:

This PR adds the following flags:

• --zap-log-level (defaults to info): configures the log level
• --zap-devel (defaults to false meaning json encoder, info level): if set to true, will turn on development/console encoding and debug log level

Both options can also be set via make variables during make start.

Additionally, this PR cleans up logging across the whole codebase in order to make full use of structured logging and the newly configurable log level.

Which issue(s) this PR fixes:
Fixes #66

Special notes for your reviewer:

Release note:

The log level of gardener-resource-manager is now configurable via `--zap-log-level`.

The log messages have been improved across the whole codebase to make log output more helpful and readable.

[timebertt]

#96 is merged, rebased.
/ready

[timuthy]

Looks good to me, just one suggestion.

[timuthy]

I know that this isn't your implementation but I somehow don't like the inconsistency between named log levels here and int levels here.

1. The user doesn't know which log levels exist (could be part of the cmd help)
2. As a developer using log.V(1).Info or log.Info it's unclear at which log level which entry is printed.

Do you think it makes sense to either (1) mask the --zap-log-level and use --v with an Integer which maps back to a zap-log-level log level or (2) use a constant instead of Integers in the code, e.g. log.V(1).Info --> log.V(Debug).Info.

[timebertt]

Discussed this with @timuthy on the phone:
We both find the current implementation to be a bit confusing, especially the stuff with the level inversion that is going on in the zap options in c-r and in zapr.

We think it makes sense to check, how other logging implementations are handling the log levels, e.g. the new klog implementation.
It probably makes sense to add own flags (like --v) to a) be flexible to change the underlying logging library one day without breaking compatibility and b) make it more aligned with kubernetes components.

In general, it might also make sense to do similar refactorings in the extensions and also gardener itself and have everything consistent.
So let's look into this a bit, so we can have a bit of a guideline for this later on, before we end up with a zoo of different ways to handle logging in our components :D

/author-action
/needs changes

[ghost]

@timebertt The pull request was assigned to you under author-action. Please unassign yourself when you are done. Thank you.

[timebertt]

I evaluated our options wrt choosing a logging library together with @timuthy again.

Although, logging implementations in zapr, klog, klogr, controller-runtime and all the plumbing in between are kind of confusing, we reached to the following conclusion:
It probably makes sense for our components (grm, extensions and later on also g/g), to switch from zap/zapr and the flags defined by controller-runtime to klog/klogr and the flags defined by k8s.io/component-base/logs. Then, we can use to logr-adapter of klog (klogr) in our controllers and the controller-runtime managers.

This brings several advantages and solves some challenges:

• we are consistent with kubernetes components in terms of usage (mainly command line flags)
  • our components will also have --v and --logging-format as all other k8s components, as opposed to --zap-log-level and so on, which have different meanings (inverted logic for int values and also unexpected correlations with the named levels)
  • if we want, we can mark all the other klog flags as hidden to not influence user/developer experience negatively
• we are consistent with kubernetes code in terms of meaning of log levels
  • we can therefore reference and apply the official k8s logging guidelines (V(1) is meant to be debug log level which doesn't fit the k8s logging guidelines)
  • this will minimize confusing for developers/contributors
• standard logs of kubernetes libraries (e.g. client-go) will have the same log format as the component's logs (everything goes through klog at the end)

The problem with this currently is, that the new structured logging options in component-base and the other k8s libraries are only available from v1.19 on, which means c-r would also need to be upgraded to v0.7.0, which is not done in g/g yet (ref gardener/gardener#3109).
There might still be some undiscovered drawbacks and uncertainties, which I couldn't check in detail because of this (I already expect some pitfalls with different log levels in the c-r manager's logs).

I would propose to close this PR for now and revisit it, once gardener/gardener#3109 is done.
Then we will be able to get a clearer picture of how all of this can be implemented in a clean way and decide on a target picture for grm and all our other components.

/close