Introduce metrics

[zqzten]

This PR introduce metrics to kine.

It'll expose below metrics:

• SQL Total & Errors: Counter metrics which hold the total number of SQL operations and failed SQL operations. This is helpful for quickly tracking and alerting local SQL errors which have a variant of causes (network, db, etc.) thus not easy to monitor.
• SQL Time: Histogram metric which keeps track of the duration of SQL operations. This duration is measured locally and can be used as a supplement to the SQL exec time measurement of db.
• Compact Errors: Counter metrics which hold the total number of compact errors. Helpful for alerting when continuous compact errors occur which can lead to uncontrollable growth of db size.
• DBStats metrics of sql.DB used by kine.
• Process and Go runtime metrics when running stand-alone.

If kine is running stand-alone, we can use the metrics-bind-address flag to configure the metrics http server address or disable the server.
If kine is used as a library, we can set MetricsRegisterer of endpoint.Config to our own prometheus registerer to let kine register its metrics to it.

fixes #119

[brandond]

Looks good! Added a few comments about TLS and metrics selection.

I also notice that you didn't add any metrics support to the jetstream driver. Is this an intentional omission?

[zqzten]

@brandond Thanks for the comments! I have made changes accrodingly, PTAL.

For the JetStream driver, I omit it because it's not a SQL backend and cannot be included in the metrics introduced in this PR. Also, I'm not quite familiar with it so maybe we shall leave its own metric implementations to some other one who really need them.

[brandond]

Rather than just labeling these with success/failure, I think we can extract the actual error code/number? It'll need to be done on a per-driver basis, but if you add type ErrCode func(error) string to generic.go, and ErrCode ErrCode to the Generic struct, you should be able to do something like this on each driver:

+       dialect.ErrCode = func(err error) string {
+               if err == nil {
+                       return ""
+               }
+               if err, ok := err.(*mysql.MySQLError); ok {
+                       return fmt.Sprint(err.Number)
+               }
+               return err.Error()
+       }

That's an example of mysql. postgres has err.Code, sqlite has err.ExtendedCode, and so on.

[zqzten]

@brandond Brilliant solution 👍 I have modifed the code accroding to your solution, PTAL.

[zqzten]

@brandond Hi, are there any blocking issue on this PR? Seems that the test can't proceed with arm64 build for some unknown reason.

[brandond]

Restarted CI

[zqzten]

@brandond Mind merging this? We'd like to use this feature in community vendor, thanks!