docs(sandbox): document sticky sessions via metadata.session_id - #452
Conversation
Covers the new sticky-session support in the code interpreter interceptor (BerriAI/litellm#31688): curl + OpenAI SDK examples, per-key hash isolation, 10-session LRU cap, TTL reset on access, and a Notes update so the "one sandbox per request" line no longer contradicts the feature.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e60a9dbb6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| ### Sticky sessions | ||
|
|
||
| By default each request spins up a fresh sandbox that gets deleted when the agentic loop ends. Pass `metadata.session_id` to reuse the same sandbox across sequential requests so variables, imports, and files defined in one turn stay live for the next. |
There was a problem hiding this comment.
This sentence promises that a stable metadata.session_id reuses the same sandbox across sequential requests, but the feature added in litellm#31688 keeps sessions in the interceptor's in-process _container_cache and prunes entries after 15 minutes idle. In multi-worker/multi-pod proxy deployments, or after the idle TTL, a follow-up request with the same session_id can land on a different process or an expired entry and get a fresh sandbox, so variables/files will not carry over. Please document those conditions or the need for proxy affinity/single-process routing.
Useful? React with 👍 / 👎.
2 participants
Summary
Documents the sticky-session support added in litellm#31688. A new "Sticky sessions" subsection under the Code Interpreter Interceptor explains that passing
metadata.session_idreuses the same e2b sandbox across sequential requests so variables, imports, and files carry over. Includes curl and OpenAI SDK examples.Also updates the Notes paragraph so the "one sandbox per request, deleted on completion" line no longer contradicts the new behavior.
Covers the security guarantees users need to trust the feature: cross-tenant isolation via
{user_api_key_hash}:{session_id}cache keying, the 10 live session cap per API key with LRU eviction, and TTL reset on access so active sessions don't expire mid-conversation.Test plan
npm startlocally and verified the new subsection renders at/docs/sandbox#sticky-sessions