You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to test the file storage configuration for the ClickHouse exporter.
No matter what queue size I put and how much PVC I allocate I get the error around ~1500 batches: error exporterhelper/queue_sender.go:213 Dropping data because sending_queue is full. Try increasing queue_size. {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "dropped_items": 10000}
Steps to Reproduce
Configure Clickhouse Exporter with file_storage storage.
Set batch and make queue size large, 100k+
Get the pipeline flowing with 10000 metrics a second.
Shutdown ClickHouse
Watch the send queue size grow: max(otelcol_exporter_queue_size{exporter=~".*",job="otel"}) by (exporter )
Expected Result
Queue size continues to grow util queue_size is reached.
Actual Result
Around 1.5k this error is raised: batch_processor.go:258 Sender failed {"kind": "processor", "name": "batch", "pipeline": "metrics", "error": "write /etc/otel-collector/buffer/exporter_clickhouse__metrics: no space left on device"}
I have tried increasing Doubling) the PVC and different batch sizes at it always hits this threshold and stops persisting.
Never mind. My PVC wasn't getting increased even though I had updated the Collector's claim. 🤷
Seems to be working, it could be useful to expose free storage bytes in addition to the send queue size.
Component(s)
ExporterHelper
FileStorage
ClickHouseExporter
What happened?
Description
I am trying to test the file storage configuration for the ClickHouse exporter.
No matter what queue size I put and how much PVC I allocate I get the error around ~1500 batches:
error exporterhelper/queue_sender.go:213 Dropping data because sending_queue is full. Try increasing queue_size. {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "dropped_items": 10000}
Steps to Reproduce
Configure Clickhouse Exporter with file_storage storage.
Set batch and make queue size large, 100k+
Get the pipeline flowing with 10000 metrics a second.
Shutdown ClickHouse
Watch the send queue size grow:
max(otelcol_exporter_queue_size{exporter=~".*",job="otel"}) by (exporter )
Expected Result
Queue size continues to grow util queue_size is reached.
Actual Result
Around 1.5k this error is raised:
batch_processor.go:258 Sender failed {"kind": "processor", "name": "batch", "pipeline": "metrics", "error": "write /etc/otel-collector/buffer/exporter_clickhouse__metrics: no space left on device"}
I have tried increasing Doubling) the PVC and different batch sizes at it always hits this threshold and stops persisting.
Collector version
v0.92.0
Environment information
Environment
GKE OpenTelemetry Operator 0.46.0
OpenTelemetry Collector configuration
Log output
Additional context
I have tried increasing the PVC and various queue_size and send_batch_size values.
This error doesn't make sense, why would it be limited?
StatefulSet
The text was updated successfully, but these errors were encountered: