feat!: add new kafka backend and refactor metrics sender engine

The SNMPCollector is currently focused and oriented to send metrics to
InfluxDB, using internally specific functions of InfluxDB client to
manage and send those metrics.

This PR tries to break with this dependency on metric internal management
adopting the current telegraf metric definitions.
With this abstracion, a new engine is created in order to send those
generic metrics to specific backends in a centralized way.

The new metric sender engine also adopts the current telegraf buffer. The metrics
are now stored on the buffer instead of being written each time that
a measurement produces a small set of them.

The proposed engine is changed and it is based on the current telegraf
way: a ticker is defined and the metrics are flushed on several batches,
reducing drastically the number of request that are being done and with
the change to control the current buffer size and make operations on top
the output.

Since all the metrics are stored on a top-level buffer, the send process
is being done by the backend implementation. This PR redefines the
current influxdb to be a backend that only performs the connection, write
close.

The internal selfmon metrics -measurement, runtime, outdb stats- are
refactored to follow the same generic metric definition, using, as always,
the 'default' backend defined, now, as an Output.

With this behaviour, a new Kafka Backend has been created to be able to
write metrics to several brokers using JSON output format. The code is
copied/modified from the current telegraf.output Kafka

The SNMPCollector OutDB now is related with an Output instead of
InfluxDB Server. All the management of Outputs and new Kafka Server
can be done on the UI and via API. The output relation with the backend
is, right now 1:1, and a Device can have only 1 output attached

BREAKING CHANGE: the metrics are now stored on internal buffer and
written to the final backend based on time. The write process is being
done each FlushInterval and splitted into several request based on
the buffer length and the MetricsBatchSize.

BREAKING CHANGE: the InfluxDB UI component is being moved to a new
section and the relation with the SNMPDevice is break and changed to an
Output. The field BufferSize is being removed from API request and new
DB initializtion, but mantained on old configurations to perform a
migration process

fix #343

CHANGELOG.md

@@ -1,3 +1,21 @@
+# v0.14.0 ( unreleased )
+
+### New Features
+
+* add Kafka as available Backend - on backend and UI
+* add internal buffer to store metrics and improved management/control/writes to the backend, based on time (see breaking change)
+* add new dropped series metric to selfmon stats
+
+### Fixes
+
+* fix #343
+
+### Breaking Changes
+
+* refactor metrics sender engine with new output/backend refactor. The metrics are now sent in batches based on an interval and it
+* UI: new Output component and InfluxDB is moved as an Output backend
+* SNMPCollector OutDB refers now to an Output instead of an InfluxDB Server, a migration script is done to migrate current InfluxDBServers to Outputs
+
 # v0.13.0 ( 2022-02-16 )
 
 ### New Features

README.md

@@ -1,6 +1,6 @@
 # SnmpCollector [![Go Report Card](https://goreportcard.com/badge/github.com/toni-moreno/snmpcollector)](https://goreportcard.com/report/github.com/toni-moreno/snmpcollector)
 
-SnmpCollector is a full featured Generic SNMP data collector with Web Administration Interface Open Source tool which has as main goal simplify  the configuration for getting data from any  device which snmp protocol support and send resulting data to an influxdb backend.
+SnmpCollector is a full featured Generic SNMP data collector with Web Administration Interface Open Source tool which has as main goal simplify  the configuration for getting data from any  device which snmp protocol support and send resulting data to an InfluxDB and Kafka backends.
 
 For complete information on installation from binary package and configuration you could read the [snmpcollector wiki](https://github.com/toni-moreno/snmpcollector/wiki).
 
@@ -13,6 +13,7 @@ If you want to build a package yourself, or contribute. Here is a guide for how
 
 - Go 1.5 for snmpcollector < 0.8
 - Go 1.11 for snmpcollector >= 0.8
+- Go 1.17 for snmpcollector >= 0.13
 - NodeJS >=6.2.1
 
 ### Get Code and  setup example config

go.mod

@@ -4,47 +4,84 @@ go 1.17
 
 require (
 	github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible
+	github.com/Shopify/sarama v1.36.0
 	github.com/go-macaron/binding v1.1.1
 	github.com/go-macaron/session v1.0.2
 	github.com/go-macaron/toolbox v0.0.0-20200329073429-4401f4ce0f55
 	github.com/go-sql-driver/mysql v1.6.0
-	github.com/google/go-cmp v0.5.3
-	github.com/gosnmp/gosnmp v1.32.0
+	github.com/gofrs/uuid v4.2.0+incompatible
+	github.com/google/go-cmp v0.5.9
+	github.com/gosnmp/gosnmp v1.34.0
 	github.com/influxdata/influxdb1-client v0.0.0-20200827194710-b269163b24ab
+	github.com/influxdata/telegraf v1.24.2
 	github.com/kelseyhightower/envconfig v1.4.0
 	github.com/lib/pq v1.10.2
 	github.com/mattn/go-sqlite3 v1.14.8
 	github.com/pkg/errors v0.9.1
-	github.com/sirupsen/logrus v1.4.2
+	github.com/sirupsen/logrus v1.9.0
 	github.com/spf13/viper v1.2.1
 	gopkg.in/macaron.v1 v1.4.0
 	xorm.io/xorm v1.2.5
 )
 
 require (
+	github.com/blues/jsonata-go v1.5.4 // indirect
+	github.com/caio/go-tdigest v3.1.0+incompatible // indirect
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/eapache/go-resiliency v1.3.0 // indirect
+	github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21 // indirect
+	github.com/eapache/queue v1.1.0 // indirect
+	github.com/fatih/color v1.13.0 // indirect
 	github.com/fsnotify/fsnotify v1.4.9 // indirect
 	github.com/go-macaron/inject v0.0.0-20200308113650-138e5925c53b // indirect
+	github.com/gobwas/glob v0.2.3 // indirect
 	github.com/goccy/go-json v0.7.4 // indirect
-	github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
+	github.com/golang/snappy v0.0.4 // indirect
+	github.com/hashicorp/errwrap v1.1.0 // indirect
+	github.com/hashicorp/go-multierror v1.1.1 // indirect
+	github.com/hashicorp/go-uuid v1.0.3 // indirect
 	github.com/hashicorp/hcl v1.0.0 // indirect
-	github.com/json-iterator/go v1.1.11 // indirect
-	github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect
+	github.com/jcmturner/aescts/v2 v2.0.0 // indirect
+	github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
+	github.com/jcmturner/gofork v1.7.6 // indirect
+	github.com/jcmturner/gokrb5/v8 v8.4.3 // indirect
+	github.com/jcmturner/rpc/v2 v2.0.3 // indirect
+	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/klauspost/compress v1.15.10 // indirect
+	github.com/konsorten/go-windows-terminal-sequences v1.0.3 // indirect
 	github.com/kr/pretty v0.3.0 // indirect
-	github.com/magiconair/properties v1.8.0 // indirect
-	github.com/mitchellh/mapstructure v1.1.2 // indirect
+	github.com/magiconair/properties v1.8.6 // indirect
+	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/mattn/go-isatty v0.0.16 // indirect
+	github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
+	github.com/mitchellh/mapstructure v1.5.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
-	github.com/modern-go/reflect2 v1.0.1 // indirect
+	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/pelletier/go-toml v1.8.1 // indirect
-	github.com/spf13/afero v1.1.2 // indirect
+	github.com/philhofer/fwd v1.1.1 // indirect
+	github.com/pierrec/lz4/v4 v4.1.15 // indirect
+	github.com/prometheus/client_model v0.2.0 // indirect
+	github.com/prometheus/common v0.37.0 // indirect
+	github.com/prometheus/prometheus v1.8.2-0.20210430082741-2a4b8e12bbf2 // indirect
+	github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
+	github.com/spf13/afero v1.2.2 // indirect
 	github.com/spf13/cast v1.3.0 // indirect
 	github.com/spf13/jwalterweatherman v1.0.0 // indirect
-	github.com/spf13/pflag v1.0.3 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
 	github.com/syndtr/goleveldb v1.0.0 // indirect
+	github.com/tinylib/msgp v1.1.6 // indirect
 	github.com/unknwon/com v1.0.1 // indirect
-	golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e // indirect
-	golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1 // indirect
-	golang.org/x/text v0.3.6 // indirect
-	gopkg.in/ini.v1 v1.62.0 // indirect
-	gopkg.in/yaml.v2 v2.3.0 // indirect
+	github.com/wavefronthq/wavefront-sdk-go v0.10.4 // indirect
+	github.com/xdg/scram v1.0.5 // indirect
+	github.com/xdg/stringprep v1.0.3 // indirect
+	golang.org/x/crypto v0.0.0-20220829220503-c86fa9a7ed90 // indirect
+	golang.org/x/net v0.0.0-20220909164309-bea034e7d591 // indirect
+	golang.org/x/sys v0.0.0-20220919091848-fb04ddd9f9c8 // indirect
+	golang.org/x/text v0.3.7 // indirect
+	google.golang.org/protobuf v1.28.1 // indirect
+	gopkg.in/ini.v1 v1.67.0 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
 	xorm.io/builder v0.3.9 // indirect
 )