Wave 3a: restore auth foundation (closes JWT verification gap)

[RobbieJ]

First step of Wave 3 — restores the auth subsystem deleted from red-hat-v2.1.0 at fork time. Closes the JWT signature-verification gap flagged in project_jwt_verification_gap.md (the standing MUST-FIX). 4 cherry-picks from upstream/develop + 1 small follow-up fix.

Upstream	Cherry-pick	Title
NVIDIA-AI-Blueprints#169 880fbec	eca963c	Add register_token_fetcher plugin hook for auth extensibility
NVIDIA-AI-Blueprints#173 7fec684	7fe0508	feat: expose AI-Q as an API with Auth Middleware (KEYSTONE)
NVIDIA-AI-Blueprints#178 1a5747c	e51e0f8	allow access to /docs
NVIDIA-AI-Blueprints#180 287f7d5	61af149	fix: set size cap on reads (CVE-shaped JWKS hardening)
follow-up	5d93b8f	fix(test): make JWTValidator kid-match test uptime-independent (caught locally on 13-day-uptime DGX Spark)

What this PR does

Restores frontends/aiq_api/src/aiq_api/auth/ (deleted on RH at migration), which adds:

• JWTValidator — PyJWT-backed RS256/ES256/ES512 signature verification, OIDC discovery (${issuer}/.well-known/openid-configuration), JWKS fetch + cache (6h TTL, 64 KB read cap from fix: set size cap on reads NVIDIA-AI-Blueprints/aiq#180)
• AuthMiddleware — raw ASGI middleware that gates non-allowlisted endpoints when REQUIRE_AUTH=true. /docs, /redoc, /openapi.json allowlisted (fix: allow access to /docs NVIDIA-AI-Blueprints/aiq#178)
• TokenValidator ABC + AuthError — extension contract for additional validators (e.g. API keys, opaque tokens)
• register_token_fetcher plugin hook (Add register_token_fetcher plugin hook for auth extensibility NVIDIA-AI-Blueprints/aiq#169) on aiq_agent.auth — extensibility for upstream-aligned token sourcing

Plus integrations:

• chat_researcher reads skip_clarifier from authenticated user context (set by future validators for headless / API callers)
• deep_researcher propagates AuthError to the user instead of generic retry
• frontends/aiq_api/plugin.py registers the middleware, scans aiq_api.validators entry-point group at startup
• New env vars: AIQ_EXTERNAL_HOSTNAMES, REQUIRE_AUTH, AIQ_JWT_ISSUER, AIQ_JWT_AUDIENCE (added to deploy/.env.example in a later wave)

Why now

Closes the JWT verification gap identified during PR #7 (Wave 2a) review. Per feedback_stay_in_sync_with_upstream.md: the deletion of upstream's auth/ subsystem on red-hat-v2.1.0 was an undocumented deviation introducing real attack surface (any code path that puts attacker-controlled tokens into the idToken cookie could spoof identity). Restoring upstream is the cleanest remediation and aligns with the "prefer NVIDIA" principle.

The middleware is permissive by default (REQUIRE_AUTH=false). Enforcement gets turned on by deployment configuration once the env-var documentation is staged in Wave 3c (NVIDIA-AI-Blueprints#203). This staging keeps the PR low-risk for existing consumers — Wave 3a delivers the machinery; the legacy unverified decode_jwt_payload retires in 3b/3c.

Skipped / deviated from upstream

Upstream change	What we did	Why
pyproject.toml adds nvidia-nat-core==1.5.0 and nvidia-nat-eval==1.5.0	Adjusted both to ==1.6.0; kept our existing 1.6.0 separate-pin layout	We're on NAT 1.6 post-PR-#8; verbatim adoption would downgrade
uv.lock upstream diff	Regenerated locally with uv lock	Upstream's 1.5-resolution lockfile diff doesn't apply on top of our 1.6 baseline. Result: PyJWT 2.12.1 already resolved transitively from PR #8, no new packages
frontends/cli/auth/local_token_store.py (122 lines, fork-side dead code)	Took the deletion	No external callers (verified by grep across the repo)
scripts/setup.sh data-source install lines	Took upstream's --no-deps removal for the lines they modify; added our serper_news_search line to the cleaned-up sequence	Maintaining transitive resolution consistency across all data-source installs
frontends/aiq_api/tests/test_auth.py::test_matches_key_by_kid	Patched to use time.monotonic() for the JWKS fetched-at marker	Upstream's _jwks_keys_fetched_at = 0.0 makes the test flaky on long-uptime hosts (>11.6 days) where time.monotonic() exceeds _jwks_cache_ttl=999999.0 and trips the cache-refresh branch. Trivial test fix, no production-code change

All other upstream lines applied verbatim per feedback_stay_in_sync_with_upstream.md.

Branding / vLLM exception scan — clean

Per the sharpened rule (feedback_stay_in_sync_with_upstream.md): "prefer NVIDIA unless it touches branding or vLLM."

• 0 files under frontends/ui/
• 0 changes to docs/source/customization/
• 0 changes to README.md, MIGRATION.md
• 0 changes to deploy/.env.example, configs/config_web_vllm.yml, docs/source/customization/vllm-*.md
• frontends/aiq_api/README.md — NEW upstream content only, NVIDIA-authored, no fork-framing claims
• scripts/setup.sh — RH serper_news_search install line preserved; upstream's --no-deps removal applied

Verification

Static

• uv lock --check — 366 packages, lockfile self-consistent
• ruff check . — clean
• pytest tests/ frontends/aiq_api/tests/ — 975 passed, 19 skipped, 4.90s (up 73 from PR chore: bump NeMo Agent Toolkit pin to 1.6.0 #8 baseline of 902 — #169 adds ~5, #173 adds ~68 in test_auth.py)
• uv run python -c "from aiq_api.auth.{base,errors,jwt_validator,middleware} import *" — all 4 modules import cleanly

Broader smoke (per feedback_upstream_sync_planning.md)

Sync /chat/stream against NIM (Nemotron3-Nano-30B):

6 LLM events (nvidia/nemotron-3-nano-30b-a3b)
4 Tool events (web_search_tool start/end + tavily_search start/end)
8 Function start/end pairs (workflow → intent_classifier → shallow_research_agent → web_search_tool)

Async /v1/jobs/async/submit + /stream against NIM:

TERMINAL=success
event types: artifact.update, workflow.start/end, tool.start/end, llm.start/end (full hierarchy)

WebSocket /websocket against NIM:

18 system_intermediate_message + 2 system_response_message events

Auth allowlist (REQUIRE_AUTH=false):

/health → 200, /docs → 200, /redoc → 200, /openapi.json → 200

Auth gate (REQUIRE_AUTH=true, no validators registered):

Error: Error loading front end worker aiq_api.plugin.AIQAPIWorker:
REQUIRE_AUTH=true but no validators have been registered. Either call
aiq_api.plugin.register_validator() before starting the server, or declare
an 'aiq_api.validators' entry point in your package.

✅ Refuses to start (correct behavior — prevents footgun deploys).

What's NOT in this PR

• Wave 3b — fix: auth refactor — eliminate refresh race, increase buffer, add error semantics NVIDIA-AI-Blueprints/aiq#194 (refresh race + AuthError handling), feat: provider lifecycle hooks for composable auth extensions NVIDIA-AI-Blueprints/aiq#195 (provider lifecycle hooks), test: close remaining auth bug fix test coverage gaps NVIDIA-AI-Blueprints/aiq#196 (auth test coverage). Includes UI changes and the larger aiq_agent.auth.__init__.py API surface (Principal, get_verified_current_user, etc.). Per-file branding-aware UI review required.
• Wave 3c — fix auth trust boundary and enforce async job ownership NVIDIA-AI-Blueprints/aiq#199 (trust boundary + async ownership), Add request trace classification and pseudonymous ids NVIDIA-AI-Blueprints/aiq#203 (request trace classification). Touches routes/jobs.py heavily, deploy/.env.example (vLLM exception territory — needs careful diff), docs/source/{deployment,customization}/ (fork-framing exception territory).

🤖 Generated with Claude Code