feat(sandbox): add retry logic for transient container failures

[zmanian]

Summary

Adds retry logic for transient Docker container failures (#1224).

SandboxManager::execute_with_policy() previously had no retry logic -- transient Docker errors (daemon restart, container creation race, cgroup conflicts) caused immediate job failure.

Changes:

• Extract container execution into try_execute_in_container() helper
• Wrap in retry loop: up to 2 retries (3 total attempts) with exponential backoff (2s, 4s)
• Add is_transient_sandbox_error() classifier for retry-eligible errors
• Only retry: DockerNotAvailable, ContainerCreationFailed, ContainerStartFailed
• Never retry: Timeout, ExecutionFailed, NetworkBlocked, Config, etc.
• Container cleanup is safe: ContainerRunner::execute() always force-removes before returning

Test plan

• transient_errors_are_retryable -- verifies all 3 transient variants
• non_transient_errors_are_not_retryable -- verifies Timeout, ExecutionFailed, NetworkBlocked, Config
• All 10 sandbox::manager tests pass
• cargo clippy --all --all-features -- zero warnings

Closes #1224

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the robustness of sandbox execution by introducing a retry mechanism for transient Docker container failures. By identifying and retrying temporary issues like Docker daemon glitches or container creation races, it prevents jobs from failing prematurely due to intermittent infrastructure problems, thereby improving overall system reliability.

Highlights

• Retry Logic: Implemented retry logic for transient Docker container failures within SandboxManager::execute_with_policy().
• Refactoring: Extracted the core container execution logic into a new helper function, try_execute_in_container().
• Error Classification: Added is_transient_sandbox_error() to identify specific errors eligible for retry, such as DockerNotAvailable, ContainerCreationFailed, and ContainerStartFailed.
• Retry Policy: Configured up to 2 retries (3 total attempts) with an exponential backoff strategy (2s, 4s).
• Non-Retryable Errors: Ensured that non-transient errors like Timeout, ExecutionFailed, NetworkBlocked, and Config do not trigger retries.
• Container Cleanup: Confirmed that container cleanup remains safe, as ContainerRunner::execute() always force-removes containers.

Changelog

• src/sandbox/manager.rs
  • Refactored execute_with_policy to incorporate a retry loop with exponential backoff.
  • Introduced try_execute_in_container to encapsulate a single attempt at container execution.
  • Added is_transient_sandbox_error function to classify errors as retryable or non-retryable.
  • Included new unit tests (transient_errors_are_retryable and non_transient_errors_are_not_retryable) to validate the error classification and retry behavior.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces retry logic for transient Docker container failures, which is a great improvement for robustness. The implementation is solid, extracting the container execution into a separate function and wrapping it in a retry loop with exponential backoff. The classification of transient errors and the accompanying tests are well-done. I have one suggestion to simplify the error handling after all retries are exhausted to make it more direct and improve error reporting.

[gemini-code-assist]

The logic of the retry loop ensures that last_err will always be Some if the loop completes, as it would have been set in the final iteration. The unwrap_or_else is therefore defensive against an unreachable state. Using unwrap() is more idiomatic here to assert this invariant. Furthermore, the fallback error SandboxError::ExecutionFailed is generic and loses the valuable context of the last actual transient error that occurred. Returning the last error is more informative for debugging, aligning with the principle of creating specific error variants for different failure modes to provide semantically correct and clear error messages.

        Err(last_err.unwrap())

References

1. Create specific error variants for different failure modes (e.g., DownloadFailed with a URL string vs. ManifestRead with a file path) to provide semantically correct and clear error messages.

[zmanian]

Re: Gemini's suggestion to use .unwrap() — per CLAUDE.md: "No .unwrap() or .expect() in production code." The unwrap_or_else fallback is the correct defensive pattern even though last_err should always be Some when the loop exhausts. The fallback error message is clear enough for debugging.