Skip to content

fix(tests): Mock HF in integration tests - #259

Merged
matthewgrossman merged 8 commits into
mainfrom
mgrossman/aircore-744-python-integration-tests-hit-hugging-face-rate-limits
Jun 15, 2026
Merged

fix(tests): Mock HF in integration tests#259
matthewgrossman merged 8 commits into
mainfrom
mgrossman/aircore-744-python-integration-tests-hit-hugging-face-rate-limits

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Problem

TestTrustRemoteCodePermission integration tests create HuggingFace-backed filesets (storage.type=huggingface), which triggers real HF Hub API calls (repo_info, get_hf_file_metadata) during validate_storage() and resolve_config(). These calls are rate-limited by HuggingFace, causing all 6 tests to fail with:

nemo_platform.InternalServerError: Error code: 502 - {'detail': 'Storage backend unavailable:
Rate limited for https://huggingface.co/api/models/Qwen/Qwen3-0.6B/revision/main'}

Setting HF_TOKEN in CI (PR #202) reduced the frequency but didn't eliminate the issue — rate limits still apply to authenticated users, and concurrent CI runs sharing the same token compound the problem.

Strategy

Mock at the Files service layer, not in individual tests. These tests verify authorization behavior (who can set trust_remote_code), not HuggingFace connectivity. The real HF API calls are an unnecessary side effect of fileset creation.

A no_hf_network fixture (defined in conftest.py) patches HuggingfaceStorageImpl.validate_storage and resolve_config to no-ops. This is the same pattern already used by the Files service auth tests in tests_filesets_with_auth_secrets.py. It's better than mocking HfApi directly because it only disables the two methods that egress to HF, leaving everything else (like list_files) intact.

TestTrustRemoteCodePermission uses this fixture via an autouse wrapper so every test in the class gets the mock automatically.

What about other tests?

  • Files service HF endpoint tests (test_huggingface_endpoints.py) — already safe, use ASGI transport (no real network)
  • Files external storage tests (test_huggingface_storage.py) — gated behind RUN_EXTERNAL_STORAGE_TESTS=1, skipped by default
  • Files auth+secrets tests (tests_filesets_with_auth_secrets.py) — already use no_hf_network
  • Models parallelism tests — guarded by pytest.importorskip("torch"), skipped in CI (torch not installed)

The only unprotected tests hitting real HF in CI were the 6 TestTrustRemoteCodePermission tests fixed here.

Test plan

  • All 6 TestTrustRemoteCodePermission tests pass locally (verified with HF_HUB_OFFLINE=1)
  • CI integration tests pass
  • No other integration test regressions

Fixes AIRCORE-744

Summary by CodeRabbit

Release Notes

  • Tests
    • Enhanced test isolation to prevent unintended external API calls during test execution
    • Improved integration test configuration for authorization behavior verification

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The conftest.py patches hf_hub_download and AutoConfig.from_pretrained
to serve from local fixtures, preventing HuggingFace API calls.
The download_fixtures.py script regenerates fixtures when new models
are added to tests.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@github-actions

github-actions Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 19462/25829 75.3% 60.9%
Integration Tests 11372/24601 46.2% 20.2%

The 5K lines of HF config fixtures were unnecessary — the parallelism
tests are always skipped in CI (torch not installed). Stripped down to
only the change that fixes the actual CI failures: mocking HfApi in
TestTrustRemoteCodePermission.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

@mckornfield mckornfield left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serenity now!

…ts-hit-hugging-face-rate-limits

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@github-actions github-actions Bot added the fix label Jun 15, 2026
Replace the broad HfApi class mock with the no_hf_network fixture
(defined in conftest.py) which patches only validate_storage and
resolve_config on HuggingfaceStorageImpl. This is the same pattern
used by the files service tests and avoids breaking downstream
calls like list_files that also use HfApi.

All 6 TestTrustRemoteCodePermission tests pass locally.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
matthewgrossman marked this pull request as ready for review June 15, 2026 18:46
@matthewgrossman
matthewgrossman requested review from a team as code owners June 15, 2026 18:46
Add a safety net: if validate_storage or resolve_config patches ever
stop working, HfApi.repo_info will raise RuntimeError instead of
silently making real HuggingFace API calls.

Verified locally: all 6 TestTrustRemoteCodePermission tests pass with
the poison active, proving no real HF egress occurs.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ca119e33-bc5e-4ed0-876b-3a46168ad132

📥 Commits

Reviewing files that changed from the base of the PR and between 6b2af4b and df78943.

📒 Files selected for processing (2)
  • services/core/models/tests/integration/conftest.py
  • services/core/models/tests/integration/test_models_with_auth.py

📝 Walkthrough

Walkthrough

no_hf_network fixture gains a monkeypatch that replaces HfApi.repo_info with a function raising RuntimeError, actively blocking live HuggingFace egress. TestTrustRemoteCodePermission gains an autouse _no_hf fixture that activates no_hf_network for all its tests.

Changes

HuggingFace Network Isolation

Layer / File(s) Summary
Block HfApi.repo_info in no_hf_network fixture
services/core/models/tests/integration/conftest.py
Expands docstring and monkeypatches HfApi.repo_info to raise RuntimeError, ensuring any unmocked HuggingFace API call fails loudly.
Wire autouse fixture into TestTrustRemoteCodePermission
services/core/models/tests/integration/test_models_with_auth.py
Adds _no_hf autouse fixture depending on no_hf_network so all tests in the class run without HF connectivity.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately captures the main change: mocking HuggingFace API calls in integration tests to prevent rate-limiting failures.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch mgrossman/aircore-744-python-integration-tests-hit-hugging-face-rate-limits

Comment @coderabbitai help to get the list of available commands and usage tips.

Comment thread services/core/models/tests/integration/conftest.py Outdated
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
matthewgrossman added this pull request to the merge queue Jun 15, 2026
Merged via the queue into main with commit a3c9e2f Jun 15, 2026
47 of 48 checks passed
@matthewgrossman
matthewgrossman deleted the mgrossman/aircore-744-python-integration-tests-hit-hugging-face-rate-limits branch June 15, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants