[receiver/prometheusremotewrite] Send data multiple times, keeping only one instance for the same dimension and discarding the rest.

[strive-run]

Component(s)

receiver/prometheusremotewrite

Describe the issue you're reporting

When the cache exists, the value is not returned. I'm not sure if it's a code issue or a misunderstanding on my part.

Environment: v1.131.0

/label receiver/prometheusremotewrite help-wanted

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• receiver/prometheusremotewrite: @dashpole @ArthurSens @perebaj

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ArthurSens]

Hey, could you elaborate a bit about the behavior you're seeing? I'm not sure I understand the problem just by looking at the code

[strive-run]

The data was sent 9 times repeatedly, but only the first set of data was successfully retained, while the rest of the indicators were empty.

{
  "symbols": [
    "__name__",
    "Gao",
    "app",
    "cmee-exporter",
    "engine",
    "MongoDB",
    "engineVersion",
    "8.0",
    "id",
    "dds-2ze3ab70793579e4",
    "instanceId",
    "instanceType",
    "sharding",
    "job",
    "cmee-exporter-pods",
    "lockMode",
    "Unlock",
    "measure",
    "ShardingQPS",
    "measure_desc",
    "QPS Unit:count/s",
    "namespace",
    "env-cloud-cn-beijing-1864785455605260",
    "period",
    "60",
    "product",
    "mongodb_sharding",
    "promEnvId",
    "promInstanceId",
    "rw-5d9fe4024e93846fa53bf8b07dcc",
    "regionId",
    "cn-beijing",
    "resourceGroupId",
    "rg-acfm3qyhxsi6mda",
    "resourceGroupName",
    "\u9ed8\u8ba4\u8d44\u6e90\u7ec4",
    "role",
    "Primary",
    "service",
    "status",
    "Running",
    "subinstanceId",
    "d-2zeabc5831fa3a84",
    "userId",
    "1864785455605260",
    "zoneId",
    "cn-beijing-j"
  ],
  "timeseries": [
    {
      "labelsRefs": [
        0,
        1,
        2,
        3,
        4,
        5,
        6,
        7,
        8,
        9,
        10,
        9,
        11,
        12,
        13,
        14,
        15,
        16,
        17,
        18,
        19,
        20,
        21,
        22,
        23,
        24,
        25,
        26,
        27,
        22,
        28,
        29,
        30,
        31,
        32,
        33,
        34,
        35,
        36,
        37,
        38,
        26,
        39,
        40,
        41,
        42,
        43,
        44,
        45,
        46
      ],
      "samples": [
        {
          "value": 19.0,
          "timestamp": "1756107396718"
        }
      ],
      "metadata": {
        "type": "METRIC_TYPE_GAUGE"
      }
    }
  ]
}

result:

[danelson]

I think I am seeing the same thing. I have a small reproduction at https://github.com/danelson/otel-prometheus-remote-write-receiver

[ArthurSens]

Thank you both, this is super helpful!

[perebaj]

@strive-run One doubt, if you are sending the same data multiple times, why would you expect that all records will be cached?

The data was sent 9 times repeatedly

What is different in all these samples?

[ArthurSens]

Prometheus scrapes endpoint every so often, and then sends whatever is scraped via remote write to the collector.

Labels will remain the same, including job and instance which are translated to OTel resource attributes and become the key to our cache.

If the resource attributes are the same for all 9 samples, I expect a cache hit for all of them 🤔

[strive-run]

@strive-run One doubt, if you are sending the same data multiple times, why would you expect that all records will be cached?

The data was sent 9 times repeatedly

What is different in all these samples?

This data is for demonstration purposes only. In reality, apart from job and instance, all other labels (including the timestamp) are inconsistent. However, all data entries will result in cache hits, causing only the first entry to be sent downstream.

[leegin]

@danelson I used to your repo to reproduce the issue. When testing with identical requests (same job/instance), I'm observing:

First request: resource metrics: 1, metrics: 6, data points: 6
Subsequent requests: resource metrics: 0, metrics: 0, data points: 0 (the reported bug)

Each test request contains 1 timeseries with 1 sample, so I expected each request to produce
resource metrics: 1, metrics: 1, data points: 1.

Is the first request expected behavior, or does this indicate an additional issue beyond the cache bug? I want to make sure I understand the correct expected behavior before proposing a fix.

I added somed debug statements to see the behaviour with the scope. Here are my observations

Request 1: Creates 1 metric → scope now has 1 metrics
Request 2: Reuses cached ResourceMetrics → scope now has 2 metrics
Request 3: Reuses cached ResourceMetrics → scope now has 3 metrics
Request 4: Reuses cached ResourceMetrics → scope now has 4 metrics

The same ResourceMetrics object is being shared across all requests, and metrics are accumulating in the same scope. Only the first request's consumer sees the accumulated result (6 metrics), while subsequent requests get empty results because their metrics went into the first request's object. Please do correct me if i am wrong.

@ArthurSens @perebaj Any guidance would be helpful. Thanks for your time.