retry openaicompatible requests if invalid content received

[leondz]

came back to a run and found this:

2024-06-27 20:16:29,399  DEBUG  response_closed.started
2024-06-27 20:16:29,399  DEBUG  response_closed.complete
2024-06-27 20:16:29,400  DEBUG  HTTP Response: POST https://integrate.api.nvidia.com/v1/chat/completions "202 Accepted" Headers([('date', 'Thu, 27 Jun 2024 18:16:29 GMT'), ('content-length', '0'), ('connection', 'keep-alive'), ('nvcf-reqid', '69f6d2a5-6564-4178-921f-bbe2fe5874dc'), ('nvcf-status', 'pending-evaluation'), ('referrer-policy', 'no-referrer'), ('strict-transport-security', 'max-age=31536000 ; includeSubDomains'), ('vary', 'Origin'), ('vary', 'Origin'), ('vary', 'Access-Control-Request-Method'), ('vary', 'Access-Control-Request-Headers'), ('x-content-type-options', 'nosniff'), ('x-frame-options', 'DENY'), ('x-xss-protection', '0')])
2024-06-27 20:16:29,400  DEBUG  request_id: None
2024-06-27 20:16:29,400  DEBUG  Could not read JSON from response data due to <class 'json.decoder.JSONDecodeError'> - Expecting value: line 1 column 1 (char 0)
2024-06-27 20:16:29,635  DEBUG  close.started
2024-06-27 20:16:29,636  DEBUG  close.complete

This patch assumes JSON output failures are transient, and so catches them with backoff. I would ideally like this to be configurable on/off - it's easy to imagine cases where both having it disable or enabled could be unexpected/frustrating. Conditional decorators look like a pain in Python, though.

[jmartin-tech]

One thought for disable or enabled may be to match the patterns in restGenerator used for backoff error codes and raise a local error type when a JSON parsing error occurs based on an instance variable. So the wrapping would error type to backoff based on:

try:
# ... existing model generation here ...
except JSONDecodeError as e:
    logger.exception(e)
    if self.retry_json:
        raise GarakOpenAIBackoff from e
    else:
        raise e

[jmartin-tech]

It looks like this was just moved so no action to take at this time. Just an inquiry to think on how to approach.

How can this occur? in theory _load_client would raise an error not part of the backoff set if it fails to set self.generator.

Should we look for a way to enforce validation of this earlier in the run?

[leondz]

It should be caught before here - it's def not intended to be mutable after init (ignoring the load/clear client mechanic). I guess _load_client is a good place to check, yeah.

[leondz]

_load_client is a blank method in OpenAICompatible which seems a distracting place to put this check; the check is already there in the OpenAIAPI class. Moved it to __init__, after the first _load_client.