[Subgraph Insights] Add Apollo Subgraph Fetch Histogram to Telemetry Plugin

[rregitsky]

---

Description
This change adds a new, experimental histogram to capture subgraph fetch duration for Apollo Studio. The instrument,
apollo.router.operations.fetch.duration has the following attributes:

• apollo.client.name
• apollo.client.version
• has_errors
• apollo.operation.id
• graphql.operation.kind
• graphql.operation.name
• subgraph.name

This can be toggled on using a new boolean config flag:

telemetry:
  apollo:
    experimental_subgraph_metrics: true

The instrument is currently only sent to GraphOS and is not available in 3rd-party OTel export targets. It is not
user customizable. For this purpose, users can take advantage of the existing customizable instrument
http.client.request.duration measuring the same value.

Implementation
This is implemented using the custom instrument framework. These are Apollo-controlled metrics, so rather than allowing for user customization like existing use cases, we hardcode the CustomHistogram with specific attributes. This allows us to add more easily add apollo metrics in the future and means we can use the shared standard ways of pulling data off of the request/response/context.

Cardinality is high, so the new metric follows the realtime_metrics path which allows for the scheduled_delay config to change how often the metric is sent.

Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

• PR description explains the motivation for the change and relevant context for reviewing
• PR description links appropriate GitHub/Jira tickets (creating when necessary)
• Changeset is included for user-facing changes
• Changes are compatible¹
• Documentation² completed
• Performance impact assessed and acceptable
• Metrics and logs are added³ and documented
• Tests added and passing⁴
  • Unit tests
  • Integration tests
  • Manual tests, as necessary

Exceptions

Note any exceptions here

Notes

Footnotes

1. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩

2. Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩

3. A lot of (if not most) features benefit from built-in observability and debug-level logs. Please read this guidance on metrics best-practices. ↩

4. Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩

[apollo-librarian]

✅ Docs preview has no changes

The preview was not built because there were no changes.

Build ID: 7643b6341b64a907fc66240e
Build Logs: View logs

[bonnici]

The parts that I understand are looking good to me.

[bonnici]

I guess seconds is the standard unit to use? Milliseconds or nanoseconds makes more sense to me but again I'm new to this code.

[rregitsky]

We actually don't have control over this for CustomHistogram.

Some(instant.elapsed().as_secs_f64())

In fact, all histogram refs I could find in the router were all using f64 to measure seconds.

[timbotnik]

I suppose if it was possible for this to be milliseconds maybe we would be able to use a 32 bit value and save some space over the wire. I'd need to also handle this on the ingestion side since we don't respect the unit yet and just assume that everything is in seconds. https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto#L198 - not sure that this would work since the protobuf specifies a double type for the bounds.

[goto-bus-stop]

As far as I understand it is indeed the standard unit to use for all time measurements, but anyone displaying it would convert the unit to a most-appropriate scale at that point

[rregitsky]

@timbotnik as I mentioned, CustomHistogram controls the unit recorded to the underlying histogram. We could likely add a way to specify a unit in its constructor, but if that's the route we want I'd prefer to add it as a follow-up to this PR.

[timbotnik]

IMO there's not a great reason to change this now (an argument about precision would be the only reason, but the numbers we're dealing with are precise enough). Let's make a tech debt ticket which would include updating the ingestion side to fail gracefully on different units (or support them!)

[rregitsky]

https://apollographql.atlassian.net/browse/PULSR-1750

[timbotnik]

We're looking for graphql.operation.type on the ingestion side so this probably should come from the supergraph operation type and shouldn't require an alias. Then again, I wonder if there are any use-cases where the supergraph operation type and the subgraph operation type can be different? Actually in the case of federation or subscriptions perhaps it can be different... in which case we might actually want to track both the supergraph type and subgraph type. Let's ask someone from the Router team to confirm that it's possible.

[rregitsky]

The implementation here actually notes that it should always be the same:

router/apollo-router/src/plugins/telemetry/config_new/subgraph/attributes.rs

Line 99 in 542c416

// Subgraph operation type wil always match the supergraph operation type

That said, I may as well pull it from the context to make it "future proof" in case that changes in the future.

[rregitsky]

Asked about whether they can differ here: https://apollograph.slack.com/archives/C02UX05LF4K/p1753980692829289?thread_ts=1753971835.882889&cid=C02UX05LF4K