improve kafka-mdm stats/priority

[Dieterbe]

cleaning up the code a bit in prep for #1022

• refactor stats a bit
• separate ingestion and stats tracking into two separate loops so they can't block each other. fix kafka offset tracking should run in separate goroutine to consumer #1113
• do stats updates more frequently

TODO: same for metricpersist (maybe in a separate PR)

[Dieterbe]

i ran benchmarks to try to find a performance impact, but there is none. only noise.
this is
fakemetrics backfill --kafka-mdm-addr localhost:9092 --offset 22h --period 10s --speedup 50 --mpo 20000 + docker-dev-custom-cfg-kafka + for every commit, compiling MT and restarting the container to measure ingest speed, using a modified dashboard that also shows movinaverage of 9minutes of ingest data, to boil down the bench run to a single number.
the main reason i wanted to check performance is because of i thought there could be false sharing between multiple goroutines accessing the same memory pages because of the stats.
(note for some of the runs, there's no kafka lag data because of a bug)

https://snapshot.raintank.io/dashboard/snapshot/Q523wHqQ21IwSp9BygfxmSr34DwTxW9x?orgId=2
bench-log.txt (see bottom for avg ingest rate)

https://snapshot.raintank.io/dashboard/snapshot/FrXPzkYC6zGRTJJE7Eq4D8ECdkjArnvb?orgId=2
bench-log-2.txt (see bottom for avg ingest rate)

for completeness, i'll do another run with padded stats structs (to rule out false sharing), once i have time to leave my computer unattended.

[Dieterbe]

for completeness, i'll do another run with padded stats structs (to rule out false sharing), once i have time to leave my computer unattended.

diff --git a/input/kafkamdm/kafkamdm.go b/input/kafkamdm/kafkamdm.go
index b0398b18..585a569d 100644
--- a/input/kafkamdm/kafkamdm.go
+++ b/input/kafkamdm/kafkamdm.go
@@ -177,6 +177,7 @@ type KafkaStats struct {
        offset  stats.Gauge64
        logSize stats.Gauge64
        lag     stats.Gauge64
+       _p1     [5]uint64
 }
 

-> no gains

[replay]

detail: this could be initialized further down on :274 so we can save the allocation if there's an error in getting the offset

[replay]

LGTM with one last comment