feat!: add new kafka backend and refactor metrics sender engine #522
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sbengo
force-pushed
the
feature/kafka_output
branch
3 times, most recently
from
f022eeb
to
491d76b
Compare
sbengo
force-pushed
the
feature/kafka_output
branch
from
491d76b
to
35ad282
Compare
The SNMPCollector is currently focused and oriented to send metrics to InfluxDB, using internally specific functions of InfluxDB client to manage and send those metrics. This PR tries to break with this dependency on metric internal management adopting the current telegraf metric definitions. With this abstracion, a new engine is created in order to send those generic metrics to specific backends in a centralized way. The new metric sender engine also adopts the current telegraf buffer. The metrics are now stored on the buffer instead of being written each time that a measurement produces a small set of them. The proposed engine is changed and it is based on the current telegraf way: a ticker is defined and the metrics are flushed on several batches, reducing drastically the number of request that are being done and with the change to control the current buffer size and make operations on top the output. Since all the metrics are stored on a top-level buffer, the send process is being done by the backend implementation. This PR redefines the current influxdb to be a backend that only performs the connection, write close. The internal selfmon metrics -measurement, runtime, outdb stats- are refactored to follow the same generic metric definition, using, as always, the 'default' backend defined, now, as an Output. With this behaviour, a new Kafka Backend has been created to be able to write metrics to several brokers using JSON output format. The code is copied/modified from the current telegraf.output Kafka The SNMPCollector OutDB now is related with an Output instead of InfluxDB Server. All the management of Outputs and new Kafka Server can be done on the UI and via API. The output relation with the backend is, right now 1:1, and a Device can have only 1 output attached BREAKING CHANGE: the metrics are now stored on internal buffer and written to the final backend based on time. The write process is being done each FlushInterval and splitted into several request based on the buffer length and the MetricsBatchSize. BREAKING CHANGE: the InfluxDB UI component is being moved to a new section and the relation with the SNMPDevice is break and changed to an Output. The field BufferSize is being removed from API request and new DB initializtion, but mantained on old configurations to perform a migration process fix #343
sbengo
force-pushed
the
feature/kafka_output
branch
from
35ad282
to
c71c091
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The SNMPCollector is currently focused and oriented to send metrics to InfluxDB, using internally specific functions of InfluxDB client to manage and send those metrics.
This PR tries to break with this dependency on metric internal management adopting the current telegraf metric definitions.
With this abstracion, a new engine is created in order to send those generic metrics to specific backends in a centralized way.
The new metric sender engine also adopts the current telegraf buffer. The metrics are now stored on the buffer instead of being written each time that a measurement produces a small set of them.
The proposed engine is changed and it is based on the current telegraf way: a ticker is defined and the metrics are flushed on several batches, reducing drastically the number of request that are being done and with the change to control the current buffer size and make operations on top the output.
Since all the metrics are stored on a top-level buffer, the send process is being done by the backend implementation. This PR redefines the current influxdb to be a backend that only performs the connection, write close.
The internal selfmon metrics -measurement, runtime, outdb stats- are refactored to follow the same generic metric definition, using, as always, the 'default' backend defined, now, as an Output.
With this behaviour, a new Kafka Backend has been created to be able to write metrics to several brokers using JSON output format. The code is copied/modified from the current telegraf.output Kafka
The SNMPCollector OutDB now is related with an Output instead of InfluxDB Server. All the management of Outputs and new Kafka Server can be done on the UI and via API. The output relation with the backend is, right now 1:1, and a Device can have only 1 output attached
BREAKING CHANGE: the metrics are now stored on internal buffer and written to the final backend based on time. The write process is being done each FlushInterval and splitted into several request based on the buffer length and the MetricsBatchSize.
BREAKING CHANGE: the InfluxDB UI component is being moved to a new section and the relation with the SNMPDevice is break and changed to an Output. The field BufferSize is being removed from API request and new DB initializtion, but mantained on old configurations to perform a migration process
fix #343