Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EXPORTER] Prometheus: Error on ingesting samples with different value but same timestamp #2125

Closed
Mifodix opened this issue Apr 28, 2023 · 2 comments · Fixed by #2200
Closed
Assignees
Labels
bug Something isn't working

Comments

@Mifodix
Copy link

Mifodix commented Apr 28, 2023

Describe your environment
opentelemetry-cpp 1.9.0, prometheus 2.43.0.

Steps to reproduce

	auto provider = std::make_shared<metrics_sdk::MeterProvider>();
	provider->AddMetricReader(std::move(prometheus_exporter));
        // ...
       	auto pluginCounter = metrics_api::Provider::GetMeterProvider()->GetMeter("test")->CreateUInt64Counter("plugin_counter3");
	while (1)
	{
		pluginCounter->Add(1);
		std::this_thread::sleep_for(std::chrono::milliseconds(500));
	}

PrometheusExporter generates such output:

# TYPE exposer_transferred_bytes_total counter
exposer_transferred_bytes_total 21388
# HELP exposer_scrapes_total Number of times metrics were scraped
# TYPE exposer_scrapes_total counter
exposer_scrapes_total 31
# HELP exposer_request_latencies Latencies of serving scrape requests, in microseconds
# TYPE exposer_request_latencies summary
exposer_request_latencies_count 31
exposer_request_latencies_sum 9972
exposer_request_latencies{quantile="0.5"} 276
exposer_request_latencies{quantile="0.9"} 524
exposer_request_latencies{quantile="0.99"} 553
# TYPE plugin_counter3 counter
plugin_counter3 155 1682692038761

Timestamp (1682692038761) for plugin_counter3 is not changing.

What is the expected behavior?
timestamp of plugin_counter3 is changed after each call Add() in cpp-code.

What is the actual behavior?
Timestamp is not changing. Errors in Prometheus logs:
msg="Error on ingesting samples with different value but same timestamp" num_dropped=1
Prometheus is ignoring changes of metric value.

Additional context
In my opinion fix must be in PrometheusExporterUtils::TranslateToPrometheus:

 * Helper function to convert OpenTelemetry metrics data collection
 * to Prometheus metrics data collection
 *
 * @param records a collection of metrics in OpenTelemetry
 * @return a collection of translated metrics that is acceptable by Prometheus
 */
std::vector<prometheus_client::MetricFamily> PrometheusExporterUtils::TranslateToPrometheus(
    const sdk::metrics::ResourceMetrics &data)
{

  // initialize output vector
  std::vector<prometheus_client::MetricFamily> output;

  for (const auto &instrumentation_info : data.scope_metric_data_)
  {
    for (const auto &metric_data : instrumentation_info.metric_data_)
    {
      auto origin_name = metric_data.instrument_descriptor.name_;
      auto sanitized   = SanitizeNames(origin_name);
      prometheus_client::MetricFamily metric_family;
      metric_family.name = sanitized;
      metric_family.help = metric_data.instrument_descriptor.description_;
      // ============= !!! FIX !!! ==============================================
      auto time = metric_data.end_ts.time_since_epoch();  // metric_data.start_ts.time_since_epoch();
      //======================================================================== 
@Mifodix Mifodix added the bug Something isn't working label Apr 28, 2023
@lalitb
Copy link
Member

lalitb commented May 2, 2023

@Mifodix thanks for raising the issue. would you like to contribute the fix as you suggested ?

@renan028
Copy link

@lalitb
I had the same issue and the suggestion fixed the problem.
Thanks @Mifodix

@lalitb lalitb added the help wanted Good for taking. Extra help will be provided by maintainers label May 17, 2023
@esigo esigo self-assigned this May 17, 2023
@lalitb lalitb added this to the Metrics post GA release - 2 milestone Jun 19, 2023
marcalff added a commit to marcalff/opentelemetry-cpp that referenced this issue Jun 20, 2023
@marcalff marcalff assigned marcalff and unassigned esigo Jun 20, 2023
@marcalff marcalff removed the help wanted Good for taking. Extra help will be provided by maintainers label Jun 20, 2023
@marcalff marcalff changed the title "Error on ingesting samples with different value but same timestamp" in Prometheus [EXPORTER] Prometheus: Error on ingesting samples with different value but same timestamp Jun 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
5 participants