feat(logging): add retry settings for generic API logger

[milan-berri]

Relevant issues

Addresses Generic API Logger batch send failures where transient callback endpoint timeouts, such as litellm.Timeout / httpx.ConnectTimeout, cause the batch send to fail without a configurable retry.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Targeted tests

pytest tests/logging_callback_tests/test_generic_api_callback.py -q
# 10 passed in 21.36s

pytest tests/litellm_utils_tests/test_logging_callback_manager.py::test_generic_api_callback_settings_retry_config -q
# 1 passed in 0.16s

Local timeout repro

Configured Generic API callback via YAML callback_settings:

callback_settings:
  generic_api_timeout_then_success_repro:
    callback_type: generic_api
    endpoint: http://127.0.0.1:5011/logs
    headers:
      Content-Type: application/json
    max_retries: 1
    retry_delay: 0
    timeout: 0.2

Repro sink delayed the first callback response for 1s, exceeding the callback timeout (0.2s), then responded immediately on retry.

Observed sink attempts:

{"attempt": 1, "path": "/logs", "status_code": 200, "delay_seconds": 1.0}
{"attempt": 2, "path": "/logs", "status_code": 200, "delay_seconds": 0}

Observed proxy retry log:

Generic API Logger - retrying request to http://127.0.0.1:5011/logs after error: litellm.Timeout: Connection timed out. Timeout passed=0.2, time taken=0.203 seconds (attempt 1/2)

The completion request still succeeded and /health returned 200.

Type

🆕 New Feature
✅ Test

Changes

• Adds opt-in retry settings to GenericAPILogger:
  • max_retries
  • retry_delay
  • timeout
• Wires those settings through existing YAML callback_settings for callback_type: generic_api.
• Retries LiteLLM timeout errors, HTTP transport errors, and HTTP 5xx errors.
• Does not retry HTTP 4xx errors.
• Preserves current behavior by default with max_retries=0.
• Adds mocked tests for:
  • timeout retry then success
  • HTTP 5xx retry then success
  • HTTP 4xx no retry
  • YAML callback_settings propagation into GenericAPILogger

[greptile-apps]

Greptile Summary

This PR adds opt-in retry settings (max_retries, retry_delay, timeout) to GenericAPILogger with exponential backoff, retrying on timeouts, transport errors, and HTTP 5xx responses while skipping 4xx errors. The new fields are wired through YAML callback_settings and are included in the logger cache equality check so stale loggers are correctly replaced on config change. The 5xx retry path works correctly in production because AsyncHTTPHandler.post() already calls raise_for_status() and re-raises as MaskedHTTPStatusError, which is a subclass of httpx.HTTPStatusError.

Confidence Score: 5/5

Safe to merge; the only finding is a P2 style note about an unreachable RuntimeError guard

All logic is correct: the 5xx retry integrates correctly with MaskedHTTPStatusError, the cache invalidation covers all new fields, defaults preserve current behavior (max_retries=0), and new tests are mock-only. Only a cosmetic dead-code line was identified.

No files require special attention

Important Files Changed

Filename	Overview
litellm/integrations/generic_api/generic_api_callback.py	Adds opt-in retry settings (max_retries, retry_delay, timeout) with exponential backoff via _post_with_retries; one unreachable RuntimeError guard at the end of the method is dead code but otherwise logic is correct
litellm/litellm_core_utils/logging_callback_manager.py	Correctly wires max_retries, retry_delay, and timeout from YAML callback_settings into GenericAPILogger and includes them in the cache equality check to invalidate stale loggers on config change
tests/logging_callback_tests/test_generic_api_callback.py	Adds three well-isolated mock-only tests covering timeout retry, 5xx retry, and 4xx no-retry; uses AsyncMock with no real network calls, consistent with the folder's test rules
tests/litellm_utils_tests/test_logging_callback_manager.py	Adds a test verifying that YAML callback_settings retry fields propagate correctly into GenericAPILogger attributes; cleans up litellm.callback_settings and the cache in a finally block

Sequence Diagram

sequenceDiagram
    participant CB as CustomBatchLogger
    participant GL as GenericAPILogger
    participant PR as _post_with_retries
    participant HC as AsyncHTTPHandler.post()
    participant EP as Callback Endpoint

    CB->>GL: async_send_batch()
    GL->>PR: _post_with_retries(data)
    loop attempt = 0..max_retries
        PR->>HC: post(url, headers, data, [timeout])
        HC->>EP: HTTP POST
        alt Success (2xx)
            EP-->>HC: 200 OK
            HC-->>PR: httpx.Response
            PR-->>GL: httpx.Response
        else Timeout / TransportError
            HC-->>PR: raises litellm.Timeout / httpx.TransportError
            PR->>PR: _should_retry_exception → True
            PR->>PR: _sleep_before_retry(attempt) [exponential]
        else 5xx Error
            EP-->>HC: 5xx
            HC->>HC: raise_for_status() → MaskedHTTPStatusError
            HC-->>PR: raises MaskedHTTPStatusError (subclass of HTTPStatusError)
            PR->>PR: _should_retry_exception → status >= 500 → True
            PR->>PR: _sleep_before_retry(attempt)
        else 4xx Error
            EP-->>HC: 4xx
            HC->>HC: raise_for_status() → MaskedHTTPStatusError
            HC-->>PR: raises MaskedHTTPStatusError
            PR->>PR: _should_retry_exception → status < 500 → False
            PR-->>GL: re-raises exception
        end
    end
    GL->>GL: log_queue.clear() (finally)

Loading

_{Reviews (2): Last reviewed commit: "Refine generic API retry behavior" | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 89.74359% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...m/integrations/generic_api/generic_api_callback.py	88.57%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!