fix(frontend): raise soft RLIMIT_NOFILE at startup to avoid accept() EMFILE spiral

[mikekg]

Fixes the accept() failure spiral described in #11801.

The frontend's TCP accept loop (lib/runtime/src/pipeline/network/tcp/server.rs) retries accept() with no backoff on failure. Under high concurrency with a non-keep-alive HTTP client, established connections can reach roughly 2x the nominal benchmark concurrency, and the default soft RLIMIT_NOFILE (1024 on most distros) gets exhausted, causing accept() to fail with EMFILE in a tight zero-backoff loop that also floods stdout (observed 15.8GB/86.9GB logs within minutes).

This raises the process's soft RLIMIT_NOFILE at startup (components/src/dynamo/frontend/main.py), bounded by whatever hard limit the environment provides. No-op if the environment already has enough headroom. No changes to the accept-loop logic itself.

See #11801 for full repro details, environment, and logs.

Summary by CodeRabbit

• Bug Fixes
  • Improved frontend startup reliability by automatically increasing the process file-descriptor limit when needed.
  • Respects the system’s maximum allowed limit and records the adjustment.

[github-actions]

👋 Hi mikekg! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

Walkthrough

The frontend defines a target soft file-descriptor limit, raises the current limit within the hard-limit bound when needed, and performs this setup before launching the asynchronous runtime.

Changes

Frontend startup

Layer / File(s)	Summary
File-descriptor limit configuration and startup wiring components/src/dynamo/frontend/main.py	Adds FRONTEND_FD_LIMIT_TARGET and _raise_fd_limit(), which conditionally updates the soft RLIMIT_NOFILE limit and logs the change. main() invokes it before uvloop.run(async_main()).

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description does not follow the required template and is missing the Overview, Details, reviewer start, and Related Issues sections.	Rewrite the PR description using the template, including Overview, Details, Where should reviewer start?, and Related Issues with either Closes #XXXX or a confirmed no-issue checkbox.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: raising the frontend soft RLIMIT_NOFILE at startup.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

components/src/dynamo/frontend/main.py (2)

64-64: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Move resource to module scope.

This standard-library import is inside _raise_fd_limit(), hiding a module dependency. Add it with the existing standard-library imports and remove the function-local import.

As per coding guidelines, imports must remain at module top.

Proposed change

+# with the other standard-library imports
+import resource
+
 def _raise_fd_limit(target: int = FRONTEND_FD_LIMIT_TARGET) -> None:
-    import resource

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/frontend/main.py` at line 64, Move the resource import
from inside _raise_fd_limit() to the module-level standard-library imports, and
remove the function-local import while leaving the function’s behavior
unchanged.

Sources: Coding guidelines, Path instructions

---

70-72: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use lazy logging formatting.

Pass values as logging arguments instead of constructing an f-string eagerly.

As per coding guidelines, logger calls must use lazy formatting.

Proposed change

         logger.info(
-            f"Raised RLIMIT_NOFILE soft limit {soft} -> {new_soft} (hard={hard})"
+            "Raised RLIMIT_NOFILE soft limit %s -> %s (hard=%s)",
+            soft,
+            new_soft,
+            hard,
         )

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@components/src/dynamo/frontend/main.py` around lines 70 - 72, Update the
logger.info call reporting the RLIMIT_NOFILE change to use a format string with
soft, new_soft, and hard passed as logging arguments, eliminating the eagerly
constructed f-string while preserving the existing message.

Sources: Coding guidelines, Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/src/dynamo/frontend/main.py`:
- Around line 66-69: Update the RLIMIT_NOFILE handling around resource.getrlimit
so resource.RLIM_INFINITY, including negative sentinel representations, is
treated as an unlimited hard limit rather than passed to min. Preserve the
existing cap for finite hard limits, and add coverage for both finite and
unlimited cases.

---

Nitpick comments:
In `@components/src/dynamo/frontend/main.py`:
- Line 64: Move the resource import from inside _raise_fd_limit() to the
module-level standard-library imports, and remove the function-local import
while leaving the function’s behavior unchanged.
- Around line 70-72: Update the logger.info call reporting the RLIMIT_NOFILE
change to use a format string with soft, new_soft, and hard passed as logging
arguments, eliminating the eagerly constructed f-string while preserving the
existing message.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d2694af6-e839-4400-93f6-be2fcb3a3f5e

📥 Commits

Reviewing files that changed from the base of the PR and between 01d2cb2 and 4ff5352.

📒 Files selected for processing (1)

• components/src/dynamo/frontend/main.py

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 24.24%
• Overall Coverage: 41.98% (-3.94%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 1626577 | Docs | Give us feedback!}

[RitwijParmar]

The EMFILE failure mode is real and the startup limit is a reasonable first layer. One portability issue needs attention before merge: resource is Unix-only, so importing it at module scope can make the frontend fail during import on Windows. I left an inline comment with the suggested lazy import and fallback. A small platform test would keep this safe across the supported runners.

[rmccorm4]

/ok to test f672fd8

[biswapanda]

/ok to test 1626577

[dagil-nvidia]

/ok to test 1626577