Fix 1-hour client timeout in long-running generation jobs

[VahidooX]

Problem

litellm caches openai.AsyncOpenAI / httpx clients with a hardcoded 1-hour TTL (_DEFAULT_TTL_FOR_HTTPX_CLIENTS = 3600 in litellm.constants). When the cached client expires, it is garbage-collected, closing its underlying httpx.AsyncClient. Any in-flight async requests crash with:

RuntimeError: Cannot send a request, as the client has been closed.

This is invisible for short-lived API calls but consistently kills long-running generation jobs after exactly ~1 hour of generation time.

Fix

Patch _DEFAULT_TTL_FOR_HTTPX_CLIENTS to 4 hours (14400s) in all three modules that hold a copy of the constant:

• litellm.constants — the source definition
• litellm.llms.custom_httpx.http_handler — copied via from litellm.constants import ...
• litellm.llms.openai.common_utils — copied via from litellm.constants import ...

Patching only the source module (litellm.constants) is not sufficient because the submodules bind the value locally at their own import time via from ... import, so the patched value in the source is never seen by them.

Summary by CodeRabbit

• Bug Fixes
  • Improved reliability for long-running and high-concurrency inference by extending HTTP client session timeouts to 4 hours, reducing premature connection closures and dropped requests during extended workloads.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8085ff41-5004-487a-823e-1c33e45a1578

📥 Commits

Reviewing files that changed from the base of the PR and between 9d78ea1 and 6882bbd.

📒 Files selected for processing (1)

• nemo_skills/inference/model/base.py

🚧 Files skipped from review as they are similar to previous changes (1)

• nemo_skills/inference/model/base.py

---

📝 Walkthrough

Walkthrough

Update nemo_skills/inference/model/base.py: add _EXTENDED_CLIENT_TTL = 14400, import litellm.constants, litellm.llms.custom_httpx.http_handler, and litellm.llms.openai.common_utils, and set their _DEFAULT_TTL_FOR_HTTPX_CLIENTS attributes to 14400.

Changes

Cohort / File(s)	Summary
HTTPX Client TTL patch nemo_skills/inference/model/base.py	Adds _EXTENDED_CLIENT_TTL = 14400, imports litellm.constants, litellm.llms.custom_httpx.http_handler, and litellm.llms.openai.common_utils, and assigns their _DEFAULT_TTL_FOR_HTTPX_CLIENTS to _EXTENDED_CLIENT_TTL (set to 14400).

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly addresses the main problem: a 1-hour client timeout affecting long-running generation jobs, which matches the core issue and fix in the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/litellm-httpx-client-ttl

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@nemo_skills/inference/model/base.py`:
- Around line 23-31: The patch in base.py runs too late because
litellm.caching.caching (and its Cache) is imported earlier via
nemo_skills/inference/generate.py -> litellm_hybrid_cache.py, so
litellm.constants._DEFAULT_TTL_FOR_HTTPX_CLIENTS in base.py doesn't affect
already-loaded caching code; move the TTL patch to a guaranteed-early import
(for example the package entrypoint or nemo_skills.__init__) so it executes
before any litellm submodule imports, or after startup explicitly set both
litellm.constants._DEFAULT_TTL_FOR_HTTPX_CLIENTS and the already-imported
litellm.caching.caching module's TTL/constant (e.g., adjust the value referenced
by litellm.caching.caching or reconfigure Cache) to 14400 to ensure the longer
TTL is in effect regardless of import order.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 2aed68db-aa86-4623-999a-3827a56e7371

📥 Commits

Reviewing files that changed from the base of the PR and between a5da597 and 55a5654.

📒 Files selected for processing (1)

• nemo_skills/inference/model/base.py

[coderabbitai]

🧹 Nitpick comments (1)

nemo_skills/inference/model/base.py (1)

23-37: Monkeypatching is the correct approach for this use case, but the 4-hour expiry cliff remains a LiteLLM limitation.

The extended TTL via monkeypatching is appropriate—LiteLLM does not expose a supported configuration option for disabling or safely extending eviction of global httpx handlers beyond their wall-clock lifetime. However, the cliff persists: a cached client that's already near 4 hours old will still be garbage-collected mid-request.

To eliminate the cliff entirely, switch to litellm.Router (if the architecture allows) and configure client_ttl to exceed the maximum request timeout, as the Router does support this setting. Otherwise, document this as a known limitation of LiteLLM's global handler caching and consider periodic client refresh if request lifetimes may exceed 4 hours.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@nemo_skills/inference/model/base.py` around lines 23 - 37, The current
monkeypatch extends LiteLLM's global httpx client TTL via _EXTENDED_CLIENT_TTL
and by assigning litellm.constants._DEFAULT_TTL_FOR_HTTPX_CLIENTS (and the two
submodule targets) but does not eliminate the "cliff" where a near-expiry cached
client can still be garbage-collected mid-request; to fix, either switch to
using litellm.Router and set its client_ttl to a value longer than your maximum
request timeout (preferred when architecture allows), or explicitly document
this limitation and implement a safe refresh strategy (e.g., periodic recreation
of clients before they approach the original TTL) when continuing to rely on the
global handler overrides (refer to symbols: _EXTENDED_CLIENT_TTL,
litellm.constants._DEFAULT_TTL_FOR_HTTPX_CLIENTS,
litellm.llms.custom_httpx.http_handler._DEFAULT_TTL_FOR_HTTPX_CLIENTS,
litellm.llms.openai.common_utils._DEFAULT_TTL_FOR_HTTPX_CLIENTS, and
litellm.Router.client_ttl).

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@nemo_skills/inference/model/base.py`:
- Around line 23-37: The current monkeypatch extends LiteLLM's global httpx
client TTL via _EXTENDED_CLIENT_TTL and by assigning
litellm.constants._DEFAULT_TTL_FOR_HTTPX_CLIENTS (and the two submodule targets)
but does not eliminate the "cliff" where a near-expiry cached client can still
be garbage-collected mid-request; to fix, either switch to using litellm.Router
and set its client_ttl to a value longer than your maximum request timeout
(preferred when architecture allows), or explicitly document this limitation and
implement a safe refresh strategy (e.g., periodic recreation of clients before
they approach the original TTL) when continuing to rely on the global handler
overrides (refer to symbols: _EXTENDED_CLIENT_TTL,
litellm.constants._DEFAULT_TTL_FOR_HTTPX_CLIENTS,
litellm.llms.custom_httpx.http_handler._DEFAULT_TTL_FOR_HTTPX_CLIENTS,
litellm.llms.openai.common_utils._DEFAULT_TTL_FOR_HTTPX_CLIENTS, and
litellm.Router.client_ttl).

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: afa40167-6ef2-477d-9f40-ef3d9e0038ab

📥 Commits

Reviewing files that changed from the base of the PR and between 7c9624d and 9d78ea1.

📒 Files selected for processing (1)

• nemo_skills/inference/model/base.py

[Kipok]

@VahidooX could you please fix lint and dco?