Skip to content

fix(#705): refresh OIDC token during agent runs to prevent expiry - #726

Merged
ascerra merged 1 commit into
fullsend-ai:mainfrom
ascerra:fix/705-oidc-token-refresh
May 7, 2026
Merged

fix(#705): refresh OIDC token during agent runs to prevent expiry#726
ascerra merged 1 commit into
fullsend-ai:mainfrom
ascerra:fix/705-oidc-token-refresh

Conversation

@ascerra

@ascerra ascerra commented May 7, 2026

Copy link
Copy Markdown
Contributor

The GitHub Actions OIDC JWT expires after ~10 minutes. Agent runs exceeding this window fail with invalid_grant when the Google auth library re-exchanges the stale token.
Add a background goroutine that re-fetches the OIDC token from the GitHub OIDC endpoint every 4 minutes and SCPs the fresh token into the sandbox, keeping agent sessions alive for the full timeout. Closes #705

Assisted-by: cursor

@github-actions

github-actions Bot commented May 7, 2026

Copy link
Copy Markdown

Site preview

Preview: https://19ba7353-site.fullsend-ai.workers.dev

Commit: 8d2439dd76791a3c71dcf334988e39fc9164b964

@fullsend-ai-review

fullsend-ai-review Bot commented May 7, 2026

Copy link
Copy Markdown

Review: #726

Head SHA: 8d2439d
Timestamp: 2026-05-07T00:00:00Z
Outcome: approve

Summary

Clean, well-scoped fix for the OIDC token expiry bug (#705) that was killing 69% of WIF-mode agent runs. The implementation adds a background goroutine in runAgent that re-fetches the GitHub OIDC JWT every 4 minutes and SCPs it into the sandbox, keeping GCP credentials valid for the full run duration. The goroutine lifecycle is properly managed with context cancellation and a WaitGroup, following the same pattern as the existing heartbeat goroutine. The shell script changes correctly export the OIDC endpoint URL and auth credential file path so the CLI can access them. Test coverage is thorough — unit tests cover the auth file reader, the HTTP fetch with multiple error scenarios (HTTP error, empty response, non-JSON, cancelled context), and an integration-style test verifying the ticker loop fires and stops on cancellation. No security concerns: credentials are file-based with 0600 permissions, the OIDC URL comes from trusted infrastructure config, and response bodies are size-limited. SA-key mode is correctly unaffected (no-op when env var is unset).

Findings

Info

  • [design-note] internal/cli/run.go (refreshOIDCToken) — sandbox.SCP creates its own internal context, so an in-flight SCP during shutdown won't respond to parent context cancellation. The oidcWg.Wait() defer could block briefly. This is pre-existing behavior with negligible impact; no action required.

Footer

Outcome: approve
This review applies to SHA 8d2439dd76791a3c71dcf334988e39fc9164b964. Any push to the PR head clears this review and requires a new evaluation.

Previous run

Review: #726

Head SHA: 0c26979
Timestamp: 2026-05-07T18:00:00Z
Outcome: approve

Summary

This PR adds a background OIDC token refresh goroutine to runAgent that periodically re-fetches the GitHub Actions OIDC JWT and SCPs it into the sandbox, preventing the invalid_grant errors that were killing ~69% of agent runs after ~10 minutes. The implementation is clean, well-scoped, correctly handles cancellation and shutdown, and includes comprehensive unit tests. The shell script change to export FULLSEND_GCP_OIDC_URL and FULLSEND_GCP_OIDC_AUTH into GITHUB_ENV is minimal and well-placed. No critical or high findings.

Findings

Info

  • [correctness] internal/cli/run.go (refreshOIDCToken) — The SCP to the sandbox destination path (/tmp/workspace/.gcp-oidc-token) is not atomic. If the Google auth library inside the sandbox reads the token file mid-write, it could get a truncated JWT and fail the STS exchange. In practice this is extremely unlikely (the file is <2KB, the write window is sub-millisecond) and any such failure would self-heal on the next API retry. No action needed; noting for awareness.

  • [style] internal/cli/run.go — The oidcRefreshInterval package-level var is mutated by TestRunOIDCRefresh_TicksAndStops. This is fine since Go test packages run serially by default, but if t.Parallel() is ever added to this test file, it could cause flakiness. The existing pattern is acceptable.

  • [intent-alignment] Issue error during agent runs "API Error: Error code invalid_grant: ID Token issued at 1778083730 is stale to sign-in" #705 also identifies false-positive exit codes (agent dies from token error but fullsend run exits 0) as a secondary problem. This PR does not address that, which is appropriate — the title and scope are specifically about token refresh. The exit code issue should be tracked separately.

Footer

Outcome: approve
This review applies to SHA 0c26979c5153179deb25e96c1175ceecda7a6bff. Any push to the PR head clears this review and requires a new evaluation.

Previous run (2)

Review: #726

Head SHA: 0c5d081
Timestamp: 2026-05-07T17:00:00Z
Outcome: approve

Summary

This PR adds a background goroutine to refresh the GitHub Actions OIDC token every 4 minutes during agent runs, solving the critical ~10-minute token expiry issue that was silently killing 69% of Code agent runs. The implementation is clean, well-scoped, follows existing patterns (mirrors the heartbeat goroutine), and includes solid test coverage. No blocking findings.

Findings

Low

  • [Correctness] internal/cli/run.go:~960refreshOIDCToken does not accept a context.Context, so an in-flight HTTP request cannot be cancelled when the done channel is closed. The oidcWg.Wait() in the defer could block process shutdown for up to 120 seconds (the httpClient timeout) if the OIDC endpoint or SCP call is slow. In practice this is very unlikely since both calls are fast, but adding a context derived from done would make cancellation immediate.
    Remediation: Consider passing a context.Context to refreshOIDCToken and using http.NewRequestWithContext. Create the context with a cancel func triggered by the done channel close.

Info

  • [Style/conventions] internal/cli/run_test.go:~358TestRefreshOIDCToken_Success asserts an error (from the SCP step), which is correct behavior but the test name suggests a fully successful path. Consider renaming to TestRefreshOIDCToken_FetchSucceedsSCPFails or similar for clarity.

Footer

Outcome: approve
This review applies to SHA 0c5d0819d5c19ca3fdd4400eb3b06f509ad77257. Any push to the PR head clears this review and requires a new evaluation.

ascerra added a commit to ascerra/fullsend that referenced this pull request May 7, 2026
… expiry

The GitHub Actions OIDC JWT expires after ~10 minutes. Agent runs
exceeding this window fail with invalid_grant when the Google auth
library re-exchanges the stale token.
Add a background goroutine that re-fetches the OIDC token from the
GitHub OIDC endpoint every 4 minutes and SCPs the fresh token into
the sandbox, keeping agent sessions alive for the full timeout.
Changes:
- prepare-sandbox-credentials.sh: export FULLSEND_GCP_OIDC_URL and
  FULLSEND_GCP_OIDC_AUTH to GITHUB_ENV so the runner can re-fetch
  tokens during the run.
- run.go: add runOIDCRefresh/refreshOIDCToken using context.Context
  for immediate cancellation on shutdown, with sync.WaitGroup to
  guarantee the goroutine exits before sandbox teardown.
- run_test.go: unit tests for HTTP fetch, error handling, empty
  response, context cancellation, and goroutine lifecycle.
Review feedback addressed:
- fullsend-ai#726 (fullsend-ai-review)
  - context.Context threaded through for immediate shutdown cancellation
  - Test renamed for clarity (FetchSucceedsSCPFails)
  - Added TestRefreshOIDCToken_CancelledContext
Closes fullsend-ai#705

Signed-off-by: Adam Scerra <ascerra@redhat.com>
Assisted-by: Cursor
@ascerra
ascerra force-pushed the fix/705-oidc-token-refresh branch from 0c5d081 to 0c26979 Compare May 7, 2026 18:29
@github-actions

github-actions Bot commented May 7, 2026

Copy link
Copy Markdown

fullsend review is working on this — view logs

@ralphbean ralphbean left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, well-scoped fix for #705. The goroutine lifecycle (context + WaitGroup) is solid, test coverage is good, and the approach mirrors the existing heartbeat pattern. Three minor notes inline — none blocking.

Comment thread internal/scaffold/fullsend-repo/scripts/prepare-sandbox-credentials.sh Outdated
Comment thread internal/cli/run.go
Comment thread internal/cli/run.go
… expiry

The GitHub Actions OIDC JWT expires after ~10 minutes. Agent runs
exceeding this window fail with invalid_grant when the Google auth
library re-exchanges the stale token.
Add a background goroutine that re-fetches the OIDC token from the
GitHub OIDC endpoint every 4 minutes and SCPs the fresh token into
the sandbox, keeping agent sessions alive for the full timeout.
Changes:
- prepare-sandbox-credentials.sh: export FULLSEND_GCP_OIDC_URL and
  FULLSEND_GCP_OIDC_AUTH_FILE to GITHUB_ENV. The auth header is
  written to a file (credentials-in-files pattern) rather than
  exported as a bare env var.
- run.go: add readOIDCAuthFile, runOIDCRefresh, refreshOIDCToken
  using context.Context for immediate cancellation on shutdown,
  with sync.WaitGroup to guarantee the goroutine exits before
  sandbox teardown. Validates OIDC response is JSON before writing.
- run_test.go: unit tests for auth file reading, HTTP fetch, error
  handling, empty/non-JSON response, context cancellation, and
  goroutine lifecycle.
Review feedback addressed (PR fullsend-ai#726):
- fullsend-ai-review: context.Context for shutdown cancellation,
  test renamed for clarity, added CancelledContext test
- @ralphbean: credentials-in-files pattern for OIDC auth header,
  json.Valid check on response body
Closes fullsend-ai#705

Signed-off-by: Adam Scerra <ascerra@redhat.com>
assisted-by: cursor
@ascerra
ascerra force-pushed the fix/705-oidc-token-refresh branch from 0c26979 to 8d2439d Compare May 7, 2026 19:43
@github-actions

github-actions Bot commented May 7, 2026

Copy link
Copy Markdown

fullsend review is working on this — view logs

@ascerra
ascerra added this pull request to the merge queue May 7, 2026
Merged via the queue into fullsend-ai:main with commit 5e7991e May 7, 2026
49 checks passed
@ascerra
ascerra deleted the fix/705-oidc-token-refresh branch May 7, 2026 20:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

error during agent runs "API Error: Error code invalid_grant: ID Token issued at 1778083730 is stale to sign-in"

2 participants