Skip to content

feat(server): optional API-only port - #3206

Open
MuhammadUsamaMX wants to merge 6 commits into
decolua:masterfrom
MuhammadUsamaMX:feat/api-only-port
Open

feat(server): optional API-only port#3206
MuhammadUsamaMX wants to merge 6 commits into
decolua:masterfrom
MuhammadUsamaMX:feat/api-only-port

Conversation

@MuhammadUsamaMX

@MuhammadUsamaMX MuhammadUsamaMX commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Stacked on #3205 → #3203 → #2895 → #2879. The diff below is only the last commit (one file).

Problem

The dashboard and the OpenAI-compatible endpoint share a single port. Exposing the endpoint to anything — a tunnel, a mesh network, another host — exposes the dashboard along with it, including /api/*, which can read and write provider credentials. The only way to separate them today is to put a reverse proxy in front and filter by path.

Change

Set API_PORT and 9router serves /v1, /v1beta, /responses and /codex on a second listener. Everything else returns 404 there, so the dashboard and its /api/* routes are not reachable on that port at all.

  • API_HOSTNAME controls the bind and defaults to 127.0.0.1, so widening it is always a deliberate act.
  • The main port is unchanged and still serves both, so this is opt-in: leave API_PORT unset and nothing differs.
  • Both listeners share the same handler, so the existing client-IP derivation and forwarded-header stripping apply identically on the API port.

Verification

Running with API_PORT=20129:

20129  /v1/models              200
20129  /v1/chat/completions    200   (normal completion)
20129  /dashboard              404
20129  /dashboard/model-access 404
20129  /api/keys               404
20129  /api/providers          404  {"error":{"message":"Not found. This port serves the API only."}}

20128  /dashboard              200   (unchanged)
20128  /v1/models              200   (unchanged)

Startup logs ▲ API-only listener on http://0.0.0.0:20129 (/v1, /v1beta, /responses, /codex), and a bind failure is reported without taking the main server down.

@MuhammadUsamaMX
feat(fallback): lock accounts for the provider's real rate-limit reset
8575d68 
429 fallback currently locks an account for a fixed exponential backoff
(getQuotaCooldown: base 2s … max 5m), ignoring the reset window the
provider actually reports. So an account whose quota resets in 30m is
retried every 5m (wasted 429s and fallback churn), while one that resets
in seconds is held out far too long.

Parse the provider's own reset signal from the 429 body/headers and use
it as the cooldown instead:

  - Google/Antigravity RetryInfo   "retryDelay": "1976.365s"  (relative s)
  - OpenAI/Codex                    "resets_at" / "reset_at"  (epoch s|ms)
  - OpenRouter / RateLimit headers  "X-RateLimit-Reset"       (epoch s|ms)
  - Retry-After: N                  (seconds)

resolveCooldownMs() prefers the parsed reset, floors it at the static
backoff (never retry sooner than before) and caps it at
MAX_RATE_LIMIT_COOLDOWN_MS so a bogus/huge value can't wedge an account.
Cap raised 30m -> 6h to accommodate real multi-hour resets (e.g. Codex).
No signal present -> unchanged static backoff.
@MuhammadUsamaMX
feat(fallback): per-provider retry-delay control
5374552 
Add a per-provider retry-delay option so operators can set the cooldown
applied when an account hits a rate-limit/error and the provider reports
no reset window of its own.

- Dashboard: a "Retry delay" selector on each provider page
  (Auto / 15s / 30s / 1m / 2m / 5m / 10m / 30m / 1h), saved to the new
  `retryDelayByProvider` setting via the existing /api/settings PATCH.
- markAccountUnavailable honors it as a fallback only: a provider-reported
  reset (retryDelay / Retry-After / resets_at / X-RateLimit-Reset) is always
  used and never shortened. When the provider reports nothing, a configured
  value sets the lock duration; with "auto", free/free-tier provider pools
  cap the static backoff at 60s so a small account set recovers within its
  per-minute window instead of all locking for ~5m at once.

Builds on the provider-reset-aware cooldown (uses parseProviderResetMs).
@MuhammadUsamaMX
feat(fallback): per-account RPM cap, default 40 for NVIDIA
5c91639 
NVIDIA NIM enforces a per-API-key requests-per-minute limit. Crossing it
returns 429, which parks that account on a cooldown, so a burst can knock the
whole pool out for minutes at a time even though the accounts are healthy.

Count our own requests per account and skip an account that has used its
minute budget, so the provider never sees the 429 in the first place. When
every account for a provider is at its cap the request reports the wait like
any other rate-limit instead of failing hard.

The cap is per provider and editable from Providers -> [id] ("RPM / account",
next to Retry delay): blank uses DEFAULT_PROVIDER_RPM (40 for nvidia, unset
elsewhere) and 0 means unlimited, stored in settings.rpmByProvider.

Counters are in-process, which is the right scope for a single gateway; a
multi-instance deployment would need shared state.
@MuhammadUsamaMX
feat(keys): per-key rate limits, budget, model allowlist and expiry
2008ea1 
An API key was only ever on or off. Anything sharing a 9router instance --
a second tool, a teammate, a script -- could spend the whole account pool and
reach every model, with no way to bound it short of revoking the key.

Give each key its own controls, enforced before any upstream call:

- RPM / TPM: sliding one-minute windows per key; over the limit returns 429
  with the seconds until capacity returns.
- Max budget + budget period ("1d", "30d", "1mo"): spend accrues from the
  cost already computed for usage history, so the number shown in the
  dashboard is the number enforced. The window rolls over on its own.
- Model allowlist: empty means every model, otherwise anything not listed is
  rejected with 403.
- Expiry: the key stops working at the configured time.
- Priority: stored and shown, for capacity reservation to build on.

All of it is optional -- existing keys migrate to NULL, which every check
reads as no limit, so behaviour is unchanged until a limit is set.

Enforcement lives in one place (services/keyPolicy.js) called from the chat
handler, so every inbound path applies the same rules. Token and spend
accounting hangs off saveRequestUsage, which already has the key and the cost.
Rate-limit counters are in-process, matching the existing per-account cap.
@MuhammadUsamaMX
fix(keys): pick allowlist models from a dropdown, and fit the editor …
f224d65 
…on mobile

The allowlist was a comma-separated text field, so a typo silently pinned a
key to a model that does not exist and the mistake only showed up as a 403 on
the next request. Offer the models the endpoint can actually serve instead,
and show the picks as removable chips.

Also drop the limits grid to a single column on small screens and let long
model ids truncate rather than widen the row.
@MuhammadUsamaMX
feat(server): optional API-only port
f3ab6f4 
The dashboard and the OpenAI-compatible endpoint share one port, so exposing
the endpoint to anything -- a tunnel, a mesh network, another host -- exposes
the dashboard with it. The only alternative was a reverse proxy in front.

Set API_PORT to serve /v1, /v1beta, /responses and /codex on a second port.
Everything else 404s there, so the dashboard and its /api/* routes are simply
not reachable on it. API_HOSTNAME controls the bind and defaults to loopback,
so widening it is always deliberate.

The main port is unchanged and still serves both, so this is opt-in: unset
API_PORT and nothing differs.
golamrabbi696 added a commit to golamrabbi696/EzRouter that referenced this pull request Aug 11, 2026
@golamrabbi696
Merge upstream PR decolua#3206
37d0132
@lexi-apps lexi-apps mentioned this pull request Sep 1, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MuhammadUsamaMX