[receiver/iis] iis.request.queue.age.maxconsistently fails to scrape

[BinaryFissionGames]

What happened?

Description

iis.request.queue.age.maxconsistently fails to scrape. This MAY be because the counter that it scrapes, \HTTP Service Request Queues(*)\MaxQueueItemAge doesn't seem to be populated consistently (seems to be cleared if nothing is in the queue), and gives this error:

2022-09-28T17:47:57.094Z        warn    [email protected]/scraper.go:122      some performance counters could not be scraped;         {"kind": "receiver", "name": "iis", "pipeline": "metrics", "error": "A counter with a negative denominator value was detected.\r\n"}

This happens on every scrape, which ends up filling the logs with the same warning.

Even when generating load, I can't figure out how to get this metric to consistently report.

Steps to Reproduce

Set up a default site for IIS, with no load.
Attempt to scrape using the iis receiver.
See error message in logs.

Expected Result

No error message if this is expected behavior; Not sure if omitting the metric in this case is the correct behavior or not.

Actual Result

Error message is printed on every scrape

Collector version

v0.60.0

Environment information

Encountered on Windows server 2019, with the default IIS website.

OpenTelemetry Collector configuration

receivers:
  iis:
    collection_interval: 30s

processors:
  batch:

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [iis]
      processors: [batch]
      exporters: [logging]

Log output

2022-09-28T17:47:57.094Z        warn    [email protected]/scraper.go:122      some performance counters could not be scraped;         {"kind": "receiver", "name": "iis", "pipeline": "metrics", "error": "A counter with a negative denominator value was detected.\r\n"}

Additional context

No response

[github-actions]

Pinging code owners: @Mrod1598 @djaglowski. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

I recently looked into this same error and concluded that it was due to the counter rolling over. (See https://support.microfocus.com/kb/doc.php?id=7010545)

I opened #14343 with what I think may fix a flaky integration test that occasionally showed this error. That said, if it is happening every time, I'm not sure what to make of that. We should probably remove the metric unless someone can get to the bottom of it.

[BinaryFissionGames]

Yeah, it's happening on every scrape, so that's not the problem here. This counter is of type ELAPSED_TIME (see: https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2003/cc756820(v=ws.10))

The denominator being negative, in this case, seems to be just the normal behavior when nothing is in the web server http queue. I think we just need to ignore this specific error for this specific metric.

I'd like to know what counter is failing for that other case; I think knowing the type of the counter might help to figure out what's going on. I don't know if counter rollover makes sense there; They both are measuring time since startup, it'd seem odd that the counters would roll over in the short time that the tests run.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/iis: @Mrod1598 @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Doron-Bargo]

Hi, we are still see this error . Any plan to fix it?

[BinaryFissionGames]

@djaglowski I can take a look into this if you like;

We can either

• Emit 0 for this metric when we get the "A counter with a negative denominator value was detected.\r\n" error, and skip logging the error.
• Remove this metric from the receiver.

I think the first option is the correct behavior, since nothing is in the queue when we get this error.

[djaglowski]

@BinaryFissionGames, that sounds good to me. Thanks.

[BinaryFissionGames]

Just want to update that this is still on my radar; I was looking into this last week and found it was a little more complicated than I thought it would be due to how the data is retrieved from the Windows API.

[djaglowski]

@BinaryFissionGames, any update on this?

[BinaryFissionGames]

@djaglowski I'll put a PR out for this tonight and we can iterate on it.