Skip to content

fix(grpc-proxy): render ratelimiterToken for the rate-limit client - #905

Merged
Max-NV merged 1 commit into
mainfrom
fix/grpc-proxy-ratelimiter-token
Aug 17, 2026
Merged

fix(grpc-proxy): render ratelimiterToken for the rate-limit client#905
Max-NV merged 1 commit into
mainfrom
fix/grpc-proxy-ratelimiter-token

Conversation

@Max-NV

@Max-NV Max-NV commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

TL;DR

gRPC functions were never rate limited on self-hosted installs. The grpc-proxy chart never rendered the ratelimiterToken key its rate-limit client reads, so the client fell back to OAuth2 with placeholder credentials and every check failed Unauthenticated. This adds the missing vault-agent template block.

Additional Details

NewRateLimitClient in src/invocation-plane-services/grpc-proxy/proxy/ratelimit/rate_limit.go takes the fixed-bearer path only if ratelimiterToken exists in the vault-agent secrets file; otherwise it uses OAuth2 with the id/secret from that same file, which the chart hardcodes to no-id/no-secret. Both paths were therefore dead, and IsRateLimited returns false (not limited) on any check error, so enforcement was silently off for the whole gRPC path with only a WARN per call.

OpenBao already creates the grpc-proxy-proxy JWT signer role under services/ratelimiter-api/jwt with the ratelimit:check_invocation scope and attaches the sign policy to the auth role (migrations/openbao/migrations/12_setup_grpc-proxy.sh). Only the chart block was missing, so no OpenBao change is needed.

deploy/helm/http-invocation/nvcf-invocation-service/vault-agent-templates/secrets.json.tmpl already renders its ratelimiter token, which is why the HTTP path enforces correctly on the same cluster. This change mirrors it.

Not addressed here: a permanent auth failure on the rate-limit client still fails open and is indistinguishable from healthy operation except in logs. Tracked in the issue for separate discussion.

For the Reviewer

Single template block. Worth confirming the signer path and role name match 12_setup_grpc-proxy.sh, and that no hosted deployment relies on the current behavior. Hosted grpc-proxy renders real OAuth2 client credentials rather than placeholders, so it takes the OAuth2 path and is not expected to change.

For QA

Verified end to end on a local self-hosted k3d cluster (control plane plus compute plane, ratelimiter enabled, gRPC echo function with rateLimit: 5-M and syncCheck: true), by A/B/A: patched chart, stock chart, patched chart again.

Patched chart:

  • /vault/secrets/secrets.json contains ratelimiterToken, claims aud: [grpc-proxy-proxy, s:ratelimiter], scopes: [ratelimit:check_invocation], sub: grpc-proxy-proxy
  • startup log: Using fixed bearer token authentication for rate limit client
  • 8 unary calls in one minute: 5 OK, calls 6-8 RESOURCE_EXHAUSTED "exceeded rate limit"
  • zero external rate limit check failed WARN lines

Stock chart 1.6.7, same function and same limit:

  • no ratelimiterToken; id/secret are no-id/no-secret
  • startup log: Using OAuth2 authentication for rate limit client
  • 8 calls, 8 OK, no throttling, and exactly 8 WARN lines external rate limit check failed ... code = Unauthenticated desc = Request unauthenticated with bearer

Re-applying the patched chart returned to 5 OK / 3 throttled with zero WARNs.

Streaming gRPC and syncCheck: false were not exercised; both go through the same IsRateLimited call. QA on an amd64 cluster is still worth doing since local coverage was arm64.

Notes

The chart has no pod annotation tied to the vault-agent template contents, so an in-place upgrade updates the ConfigMap without restarting grpc-proxy, and the new key only appears after a rollout restart. Fresh installs are unaffected. Left as is here; worth a follow-up if operators are expected to pick this up without a restart.

Issues

Closes #902

Checklist

  • I am familiar with the Contributing Guidelines.
  • I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
  • New or existing tests cover these changes. Vault-agent templates are rendered by the vault agent at runtime, not by chart tests; covered by the local end-to-end verification above.
  • The documentation is up to date with these changes.

The rate-limit client reads a fixed bearer token under the ratelimiterToken
key of the vault-agent secrets file, but the chart never rendered that key,
so it fell back to OAuth2 with the placeholder no-id/no-secret credentials.
Every check was rejected Unauthenticated and IsRateLimited fails open, so
gRPC functions were never throttled on self-hosted installs.

OpenBao already provisions the signer role and sign policy for
services/ratelimiter-api/jwt/sign/grpc-proxy-proxy, so only the template
block was missing. Mirrors what the invocation-service chart does for the
HTTP path.

Closes #902

Signed-off-by: Max Xing <mxing@nvidia.com>
@Max-NV
Max-NV requested a review from a team as a code owner August 17, 2026 15:15
@Max-NV
Max-NV requested a review from jjayaraman-1 August 17, 2026 15:15
@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown

Caution

Review failed

An error occurred during the review process. Please try again later.


Comment @coderabbitai help to get the list of available commands.

@Max-NV Max-NV self-assigned this Aug 17, 2026
@Max-NV
Max-NV enabled auto-merge August 17, 2026 18:04
@Max-NV
Max-NV added this pull request to the merge queue Aug 17, 2026
Merged via the queue into main with commit 654d417 Aug 17, 2026
18 of 19 checks passed
@Max-NV
Max-NV deleted the fix/grpc-proxy-ratelimiter-token branch August 17, 2026 18:16
@balajinvda

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version helm-nvcf-grpc-proxy-v1.7.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

grpc-proxy never enforces rate limits on self-hosted: chart omits ratelimiterToken

4 participants