[iguehring-nvidia]

Problem

FastAPI 0.142.0 to 0.142.4 (release notes, fastapi/fastapi#16403) configure OpenTelemetry export by themselves at ASGI startup whenever OTEL_EXPORTER_OTLP_ENDPOINT is set. Gym sets that variable for every server process so that nemo-lens finds the collector, and nemo-lens has already registered providers with their own exporters. FastAPI attaches a second BatchSpanProcessor and a second PeriodicExportingMetricReader to those providers. Gym does not pin FastAPI, so every job environment resolved since 2026-09-29 picked this up.

Observed in every gym-native run since then:

• every span is exported twice: each trace appears twice in Tempo and the collector's span-metrics counts are doubled;
• every metric is exported by two readers with separate aggregation state, so the cumulative _sum of the Gym histograms carries last-digit dips that Prometheus treats as counter resets; increase()/rate() on a sum then adds the whole running value (mean rollout duration showed hours instead of minutes, 60 of 60 runs affected).

Change

FastAPI 0.143.0 (fastapi/fastapi#16476, from discussion #16473) makes the automatic exporter setup opt-in, so only the 0.142 series double-exports. This PR excludes that series in pyproject.toml (fastapi!=0.142.*) and refreshes uv.lock; the locked version stays 0.135.2. An earlier revision wrapped the six FastAPI() sites in a factory that switched the native telemetry off; the exclusion replaces it because it also covers the bare FastAPI() apps outside the server bases and needs nothing from future servers.

Environments already built with FastAPI 0.142 keep exporting twice until they are reinstalled.

Verification

A gym-native run with the factory revision as the job-side Gym (terminal-bench miniswe, 8 rollouts, 1 node) showed one gym.rollout/gym.verify span per rollout, one metric datapoint per 10 s with consistent sums, and resets(gym_rollout_duration_ms_milliseconds_sum[1d]) = 0, against 2 spans per rollout and up to 684 resets per run on main. The exclusion produces the same single export by resolving FastAPI to 0.143.0 instead of switching the feature off.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[ananthsub]

FastAPI has since changed the default upstream. In FastAPI 0.143.0 (fastapi/fastapi#16476, from fastapi/fastapi#16473), automatic exporter setup is opt-in: auto_configure defaults to False and is enabled only with FASTAPI_OTEL_AUTO_CONFIGURE=true or FastAPI(telemetry={"auto_configure": True}). With 0.143.0 or later, FastAPI uses nemo-lens's providers without adding a second exporter, so only 0.142.0 through 0.142.4 export twice.

That means this PR can shrink to excluding those releases in pyproject.toml:

    # FastAPI: Used for server infrastructure
    # Updated Thu Mar 26, 2026 with fastapi==0.135.2
    # 0.142.* adds its own OTLP exporters to nemo-lens's providers whenever
    # OTEL_EXPORTER_OTLP_ENDPOINT is set, so every span and metric is exported twice.
    # 0.143.0 made that opt-in (fastapi/fastapi#16476).
    # License: MIT https://github.com/fastapi/fastapi/blob/25a3697cedc6e7dfb84e93c8ff965801486f00f4/LICENSE
    "fastapi!=0.142.*",

I prefer this exclusion over keeping the factory:

• The factory only covers apps built under nemo_gym/. These still call bare FastAPI() and would keep exporting twice on 0.142: responses_api_agents/mini_swe_agent/app.py, responses_api_agents/mini_swe_agent_2/app.py, responses_api_agents/harbor_agent/app.py, responses_api_agents/harbor_agent/custom_envs/singularity/server.py, environment_servers/legacy_agent/app.py, environment_servers/single_agent_turn_legacy/app.py and scripts/harness_conformance/provider.py.
• Every future server would have to remember to use the factory. That requirement no longer serves a purpose.

On the native fastapi.* and POST spans in your table: FastAPI's own request spans are still on by default in 0.143.0. FastAPI stops creating them when the OpenTelemetry instrumentation middleware is on the app, and Gym only adds that middleware when the server span group is enabled (instrument_app_for_telemetry in nemo_gym/server_utils.py). So they likely showed up because that group was off in your run. They go through nemo-lens's providers once, without duplication, so they don't need to be part of this PR.

Environments already built with FastAPI 0.142 will keep exporting twice until they're reinstalled.

[iguehring-nvidia]

@ananthsub Thanks a lot for the great bias for action! I updated the PR following your recommendation.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37803428457