Skip to content

in_premetheus_remote_write: Implement handler of payloads of prometheus remote write protocol#8725

Merged
edsiper merged 6 commits intomasterfrom
cosmo0920-in_prometheus-remote-write
May 16, 2024
Merged

in_premetheus_remote_write: Implement handler of payloads of prometheus remote write protocol#8725
edsiper merged 6 commits intomasterfrom
cosmo0920-in_prometheus-remote-write

Conversation

@cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Apr 17, 2024 •
edited
Loading

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
$  bin/fluent-bit -i prometheus_remote_write -pport=8080 -phttp2=off -o stdout -v
  • Debug log output from testing the change
Fluent Bit v3.0.3
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/04/19 20:24:33] [ info] Configuration:
[2024/04/19 20:24:33] [ info]  flush time     | 1.000000 seconds
[2024/04/19 20:24:33] [ info]  grace          | 5 seconds
[2024/04/19 20:24:33] [ info]  daemon         | 0
[2024/04/19 20:24:33] [ info] ___________
[2024/04/19 20:24:33] [ info]  inputs:
[2024/04/19 20:24:33] [ info]      prometheus_remote_write
[2024/04/19 20:24:33] [ info] ___________
[2024/04/19 20:24:33] [ info]  filters:
[2024/04/19 20:24:33] [ info] ___________
[2024/04/19 20:24:33] [ info]  outputs:
[2024/04/19 20:24:33] [ info]      stdout.0
[2024/04/19 20:24:33] [ info] ___________
[2024/04/19 20:24:33] [ info]  collectors:
[2024/04/19 20:24:33] [ info] [fluent bit] version=3.0.3, commit=ec1c65f73a, pid=186265
[2024/04/19 20:24:33] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/04/19 20:24:33] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/04/19 20:24:33] [ info] [cmetrics] version=0.7.3
[2024/04/19 20:24:33] [ info] [ctraces ] version=0.4.0
[2024/04/19 20:24:33] [ info] [input:prometheus_remote_write:prometheus_remote_write.0] initializing
[2024/04/19 20:24:33] [ info] [input:prometheus_remote_write:prometheus_remote_write.0] storage_strategy='memory' (memory only)
[2024/04/19 20:24:33] [ info] [output:stdout:stdout.0] worker #0 started
[2024/04/19 20:24:33] [debug] [prometheus_remote_write:prometheus_remote_write.0] created event channels: read=21 write=22
[2024/04/19 20:24:33] [debug] [downstream] listening on 0.0.0.0:8080
[2024/04/19 20:24:33] [ info] [input:prometheus_remote_write:prometheus_remote_write.0] listening on 0.0.0.0:8080
[2024/04/19 20:24:33] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2024/04/19 20:24:33] [ info] [sp] stream processor started
[2024/04/19 20:24:38] [debug] [task] created task=0x5faf030 id=0 OK
2024-04-19T11:24:36.216000000Z fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 1
2024-04-19T11:24:35.170000000Z fluentbit_input_bytes_total{__name__="fluentbit_input_bytes_total",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_records_total{__name__="fluentbit_input_records_total",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_input_metrics_scrapes_total{__name__="fluentbit_input_metrics_scrapes_total",name="fluentbit_metrics.0"} = 1
2024-04-19T11:24:35.170000000Z fluentbit_output_proc_records_total{__name__="fluentbit_output_proc_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_proc_bytes_total{__name__="fluentbit_output_proc_bytes_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_errors_total{__name__="fluentbit_output_errors_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retries_total{__name__="fluentbit_output_retries_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retries_failed_total{__name__="fluentbit_output_retries_failed_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_dropped_records_total{__name__="fluentbit_output_dropped_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retried_records_total{__name__="fluentbit_output_retried_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_process_start_time_seconds{__name__="fluentbit_process_start_time_seconds",hostname="cosmo-desktop2"} = 1713525875
2024-04-19T11:24:36.216000000Z fluentbit_build_info{__name__="fluentbit_build_info",hostname="cosmo-desktop2",version="3.0.3",os="linux"} = 1713525875
2024-04-19T11:24:36.216000000Z fluentbit_hot_reloaded_times{__name__="fluentbit_hot_reloaded_times",hostname="cosmo-desktop2"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_chunks{__name__="fluentbit_storage_chunks"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_mem_chunks{__name__="fluentbit_storage_mem_chunks"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_fs_chunks{__name__="fluentbit_storage_fs_chunks"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_fs_chunks_up{__name__="fluentbit_storage_fs_chunks_up"} = 0
2024-04-19T11:24:36.216000000Z fluentbit_storage_fs_chunks_down{__name__="fluentbit_storage_fs_chunks_down"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_ingestion_paused{__name__="fluentbit_input_ingestion_paused",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_overlimit{__name__="fluentbit_input_storage_overlimit",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_memory_bytes{__name__="fluentbit_input_storage_memory_bytes",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks{__name__="fluentbit_input_storage_chunks",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_up{__name__="fluentbit_input_storage_chunks_up",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_down{__name__="fluentbit_input_storage_chunks_down",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_busy{__name__="fluentbit_input_storage_chunks_busy",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_busy_bytes{__name__="fluentbit_input_storage_chunks_busy_bytes",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_upstream_total_connections{__name__="fluentbit_output_upstream_total_connections",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_upstream_busy_connections{__name__="fluentbit_output_upstream_busy_connections",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_chunk_available_capacity_percent{__name__="fluentbit_output_chunk_available_capacity_percent",name="prometheus_remote_write.0"} = 100
[2024/04/19 20:24:38] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/04/19 20:24:38] [debug] [out flush] cb_destroy coro_id=0
[2024/04/19 20:24:38] [debug] [task] destroy task=0x5faf030 (task_id=0)
2024-04-19T11:24:38.216000000Z fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 3
2024-04-19T11:24:35.170000000Z fluentbit_input_bytes_total{__name__="fluentbit_input_bytes_total",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_records_total{__name__="fluentbit_input_records_total",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_input_metrics_scrapes_total{__name__="fluentbit_input_metrics_scrapes_total",name="fluentbit_metrics.0"} = 2
2024-04-19T11:24:37.278000000Z fluentbit_output_proc_records_total{__name__="fluentbit_output_proc_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:37.278000000Z fluentbit_output_proc_bytes_total{__name__="fluentbit_output_proc_bytes_total",name="prometheus_remote_write.0"} = 6175
2024-04-19T11:24:35.170000000Z fluentbit_output_errors_total{__name__="fluentbit_output_errors_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retries_tota[2024/04/19 20:24:39] [debug] [task] created task=0x6180f00 id=0 OK
l{__name__="fluentbit_output_retries_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retries_failed_total{__name__="fluentbit_output_retries_failed_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_dropped_records_total{__name__="fluentbit_output_dropped_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_output_retried_records_total{__name__="fluentbit_output_retried_records_total",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_process_start_time_seconds{__name__="fluentbit_process_start_time_seconds",hostname="cosmo-desktop2"} = 1713525875
2024-04-19T11:24:38.216000000Z fluentbit_build_info{__name__="fluentbit_build_info",hostname="cosmo-desktop2",version="3.0.3",os="linux"} = 1713525875
2024-04-19T11:24:38.216000000Z fluentbit_hot_reloaded_times{__name__="fluentbit_hot_reloaded_times",hostname="cosmo-desktop2"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_chunks{__name__="fluentbit_storage_chunks"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_mem_chunks{__name__="fluentbit_storage_mem_chunks"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_fs_chunks{__name__="fluentbit_storage_fs_chunks"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_fs_chunks_up{__name__="fluentbit_storage_fs_chunks_up"} = 0
2024-04-19T11:24:38.216000000Z fluentbit_storage_fs_chunks_down{__name__="fluentbit_storage_fs_chunks_down"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_ingestion_paused{__name__="fluentbit_input_ingestion_paused",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_overlimit{__name__="fluentbit_input_storage_overlimit",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_memory_bytes{__name__="fluentbit_input_storage_memory_bytes",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks{__name__="fluentbit_input_storage_chunks",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_up{__name__="fluentbit_input_storage_chunks_up",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_down{__name__="fluentbit_input_storage_chunks_down",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_busy{__name__="fluentbit_input_storage_chunks_busy",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:35.170000000Z fluentbit_input_storage_chunks_busy_bytes{__name__="fluentbit_input_storage_chunks_busy_bytes",name="fluentbit_metrics.0"} = 0
2024-04-19T11:24:37.216000000Z fluentbit_output_upstream_total_connections{__name__="fluentbit_output_upstream_total_connections",name="prometheus_remote_write.0"} = 1
2024-04-19T11:24:37.278000000Z fluentbit_output_upstream_busy_connections{__name__="fluentbit_output_upstream_busy_connections",name="prometheus_remote_write.0"} = 0
2024-04-19T11:24:37.278000000Z fluentbit_output_chunk_available_capacity_percent{__name__="fluentbit_output_chunk_available_capacity_percent",name="prometheus_remote_write.0"} = 100
[2024/04/19 20:24:39] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/04/19 20:24:39] [debug] [task] destroy task=0x6180f00 (task_id=0)
[2024/04/19 20:24:39] [debug] [out flush] cb_destroy coro_id=1
^C[2024/04/19 20:24:40] [engine] caught signal (SIGINT)
[2024/04/19 20:24:40] [ warn] [engine] service will shutdown in max 5 seconds
[2024/04/19 20:24:40] [ info] [engine] service has stopped (0 pending tasks)
[2024/04/19 20:24:40] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/04/19 20:24:40] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found
==186265== 
==186265== HEAP SUMMARY:
==186265==     in use at exit: 0 bytes in 0 blocks
==186265==   total heap usage: 7,158 allocs, 7,158 frees, 3,974,699 bytes allocated
==186265== 
==186265== All heap blocks were freed -- no leaks are possible
==186265== 
==186265== For lists of detected and suppressed errors, rerun with: -s
==186265== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1363

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cosmo0920 cosmo0920 changed the title in_premetheus_remote_write: Implement ingestor of payloads of prometheus remote write protocol in_premetheus_remote_write: Implement handler of payloads of prometheus remote write protocol Apr 17, 2024
@edsiper
Copy link
Member

edsiper commented Apr 18, 2024

@cosmo0920 I see the metrics generated have an _ as a prefix, e.g:

2024-04-17T05:13:34.937000000Z _fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 1

maybe is something around the definition of the metrics name,subsystem,etc ?

@cosmo0920
Copy link
Contributor Author

cosmo0920 commented Apr 19, 2024

@cosmo0920 I see the metrics generated have an _ as a prefix, e.g:

2024-04-17T05:13:34.937000000Z _fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 1

maybe is something around the definition of the metrics name,subsystem,etc ?

This could be generated from cmetrics but I'm not sure why the reason. I just following how to decode on the open telemetry payloads.

@cosmo0920 cosmo0920 force-pushed the cosmo0920-in_prometheus-remote-write branch from bae0fa5 to 0b714aa Compare April 19, 2024 11:19
@cosmo0920
Copy link
Contributor Author

cosmo0920 commented Apr 19, 2024

I got it.

@cosmo0920 I see the metrics generated have an _ as a prefix, e.g:

2024-04-17T05:13:34.937000000Z _fluentbit_uptime{__name__="fluentbit_uptime",hostname="cosmo-desktop2"} = 1

maybe is something around the definition of the metrics name,subsystem,etc ?

This could be generated from cmetrics but I'm not sure why the reason. I just following how to decode on the open telemetry payloads.

This was generated from cmetrics' msgpack decoder:
fluent/cmetrics@8cb2739

@cosmo0920 cosmo0920 force-pushed the cosmo0920-in_prometheus-remote-write branch from 0b714aa to a2d1869 Compare April 22, 2024 07:54
@cosmo0920 cosmo0920 force-pushed the cosmo0920-in_prometheus-remote-write branch from a2d1869 to 3435eb7 Compare April 25, 2024 06:52
@cosmo0920
in_prometheus_remote_write: Use abbreviated names for promethues remo…
4781bdd 
…te write

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920 cosmo0920 force-pushed the cosmo0920-in_prometheus-remote-write branch from 336cc0b to 4781bdd Compare May 1, 2024 06:05
@cosmo0920
in_prometheus_remote_write: Remove a needless FIXME
04da871 
Because prometheus remote write just needs to decode the protobuf
payloads.

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920
in_prometheus_remote_write: Use one line declarations
1b985cc 
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920
in_prometheus_remote_write: Check return value from flb_io_net_write
95bfcf0 
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@cosmo0920
in_prometheus_remote_write: Add allocation check for mk_server
4dc32b4 
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@edsiper edsiper merged commit 7de2c45 into master May 16, 2024
@edsiper edsiper deleted the cosmo0920-in_prometheus-remote-write branch May 16, 2024 02:57
@stevehipwell stevehipwell mentioned this pull request May 16, 2024
Closed
@BrewTestBot BrewTestBot mentioned this pull request May 21, 2024
Merged
markuman pushed a commit to markuman/fluent-bit that referenced this pull request May 29, 2024
@cosmo0920 @markuman
in_premetheus_remote_write: Implement handler of payloads of promethe…
65fc34d 
…us remote write protocol (fluent#8725)

 in_prometheus_remote_write: Implement prometheus remote write input plugin. This plugin is able to handle the following types currently:

- Counter
- Gauge
- Untyped
- Histogram

Summary type of metrics shouldn't be handled and decoded correctly for now.

---------

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Markus Bergholz <git@osuv.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@edsiper edsiper edsiper requested changes

@fujimotos fujimotos Awaiting requested review from fujimotos

@niedbalski niedbalski Awaiting requested review from niedbalski niedbalski is a code owner

@patrick-stephens patrick-stephens Awaiting requested review from patrick-stephens patrick-stephens is a code owner

@celalettin1286 celalettin1286 Awaiting requested review from celalettin1286 celalettin1286 is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich

@koleini koleini Awaiting requested review from koleini

Assignees

No one assigned

Labels

docs-required

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cosmo0920 @edsiper