GET /me/player + 429 Rate Limit exceeded = blocking calls

[gorenje]

Describe the bug
Currently my API limit is used up so I'm constantly getting API rate limit exceeded (error code 429) from the Spotify API. Which means that spotipy is doing a lot of request retrying. However this causes calls to block and hang because Spotify is setting
the retry_after header to 3600. The retry_after header is respected by the urllib3 (I assume) and it's waiting for an hour to retry the call.

I've set the request_timeout option to 5 (i.e. 5 seconds) and retries to 3 but the very first retry will block. So the only fix at the moment is to set retries to zero. Then the call will fail immediately.

Your code
spoitpy.Spotify().current_playback() # <--- this hangs/blocks for an hour....

Expected behavior
I would expect that request_timeout would be respected and the call would timeout after request_timeout seconds. Then a second retry would be made that would also fail after request_timeout ... etc.

Output
No ouptut, call hangs/blocks.

Environment:

• OS: Linux
• Python version 3.9.2
• spotipy version 2.19.0
• your IDE: Emacs

My point is that this was unexpected behaviour and I initially assumed that Spotify was having an issue. The only thing I was seeing was that my calls to current_playback were hanging and blocking my code. Then I came to discover that spotipy does automagical retries. And then I realised there was a retry_after header ...

So the end result was that Spotify killed my application my setting a header that spotipy blindly respects even though that an hour is far more that than the request_timeout value set in spotipy. I guess it would be nice if the default settings of spotipy would be more obvious when a API rate limit is hit - I only found out that I had a rate limit issue by doing a corresponding curl request.

[gorenje]

Besides setting retries=0, another fix is to add respect_retry_after_header=False at about this point - i.e. disable urllib3 from respecting the header.

I'm all for services provide feedback about rate limits but these should also be communicated to the end user. So I would find it a better solution not to retry on 429 error codes and instead fail. Rate limits only get worse with constant retrying ....

[Peter-Schorn]

but the very first retry will block.

The entire requests library uses blocking APIs. The thread is blocked during network requests as well; requests does not support async. The retry delay is just generally longer than the delay between making a network request and receiving a response, so it's more noticable.

I would expect that request_timeout would be respected and the call would timeout after request_timeout seconds.

Presumably, the request_timeout parameter refers to how long the library will wait for a response from the server after making each network request and does not factor in the retry delay. This is an issue you'll have to address with the developers of the requests library, not spotipy.

[gorenje]

I would expect that request_timeout would be respected and the call would timeout after request_timeout seconds.

Presumably, the request_timeout parameter refers to how long the library will wait for a response from the server after making each network request and does not factor in the retry delay. This is an issue you'll have to address with the developers of the requests library, not spotipy.

True that since the request_timeout value is passed onto the urllib3 client.

However urllib3 does offer the possibility of deactivating the retry_after header. So the spotipy library could support that option and pass the option up to the user of spotipy (as part of the initialization of the client).

Guess what I found confusing was that my call to current_playback() simply froze and it would have continued to stay frozen for an hour, since the retry_after value was 3600. And it did that simply out of the blue (ok, I'd been hitting the Spotify API too often and Spotify decided enough was enough)!

On the other hand, I didn't know I was hitting rate limits since spotipy was retrying for me. My workaround now is to have retries=0 and instead deal with exceptions when my rate limit is reached. I'd rather know when I've reached my rate limit than have my calls to Spotify simply freeze and block the rest of my code. I understand spotipy does automagical retries because the retry_after value is generally low (i.e. 1 or 2 seconds) but when it becomes extreme, this can become painful.

[Peter-Schorn]

So the spotipy library could support that option and pass the option up to the user of spotipy (as part of the initialization of the client).

The Spotify initializer does have a parameter for the number of retries, so you can set that to 0 to disable any retries. For more customization, you can create a custom requests.Session, customize the retry policy, and pass that into the Spotify initializer instead. Using this method, you can ignore the retry-after header:

import spotipy, requests, urllib3

session = requests.Session()

retry = urllib3.Retry(
    total=0,
    connect=None,
    read=0,
    allowed_methods=frozenset(['GET', 'POST', 'PUT', 'DELETE']),
    status=0,
    backoff_factor=0.3,
    status_forcelist=(429, 500, 502, 503, 504),
    respect_retry_after_header=False  # <---
)

adapter = requests.adapters.HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

spotify = spotipy.Spotify(
    auth_manager=spotipy.SpotifyPKCE(
        scope="",
        state=None,
        cache_handler=spotipy.CacheFileHandler()
    ),
    requests_session=session
)

[gorenje]

Of course! I noticed the requests_session parameter but didn't think of rolling my own! 👍

I would go back to 3 retries but simply ignore the retry_after header.

Cheers and thanks for the heads-up!

[CalColistra]

Hi, I have been running into a similar (if not the same) issue. When I try to get spotify data about an artist by using their artist() function, it gets stuck in the sleep_for_retry() function and it does not tell me how to to sleep for.

Here is my code for getting data for an artist: currentResults = spotify.artist(<artist id>)

for more details about my original problem, I previously created a issue that was closed here: issue #956

I just tried the solution from @Peter-Schorn here in this issue and I can't figure out how to get it working or if that would even solve my sleep_for_retry() problem.

I created a session like this, *note my spotipy.Spotify() function wouldn't work unless I added my client_id and redirect_uri within the auth_manager:

session = requests.Session()

retry = urllib3.Retry(
    total=0,
    connect=None,
    read=0,
    allowed_methods=frozenset(['GET', 'POST', 'PUT', 'DELETE']),
    status=0,
    backoff_factor=0.3,
    status_forcelist=(429, 500, 502, 503, 504),
    respect_retry_after_header=False  # <---
)

adapter = requests.adapters.HTTPAdapter(max_retries=retry)
session.mount('http://', adapter)
session.mount('https://', adapter)

CLIENT_ID='<my id>'
CLIENT_SECRET='<my secret>'
SPOTIPY_REDIRECT_URI = "https://localhost:8888/callback",

spotify = spotipy.Spotify(
    auth_manager=spotipy.SpotifyPKCE(
        client_id=CLIENT_ID, 
        redirect_uri = SPOTIPY_REDIRECT_URI,
        scope="",
        state=None,
        cache_handler=spotipy.CacheFileHandler()
    ),
    requests_session=session
)

After that I try making a request to get data about an artist like so:
currentResults = spotify.artist(<artist id>)

But I get this error that seems like there is a problem getting my access token:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[/usr/local/lib/python3.9/dist-packages/spotipy/client.py](https://localhost:8080/#) in _auth_headers(self)
    235         try:
--> 236             token = self.auth_manager.get_access_token(as_dict=False)
    237         except TypeError:

TypeError: get_access_token() got an unexpected keyword argument 'as_dict'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
11 frames
[<ipython-input-43-c0a87c547256>](https://localhost:8080/#) in <cell line: 10>()
     10 for i in range(0,len(section2)):
     11   #time.sleep(2)
---> 12   currentResults = spotify.artist(section2[i])
     13   if (currentResults['popularity'] > 23):
     14     name.append(currentResults['name'])

[/usr/local/lib/python3.9/dist-packages/spotipy/client.py](https://localhost:8080/#) in artist(self, artist_id)
    388 
    389         trid = self._get_id("artist", artist_id)
--> 390         return self._get("artists/" + trid)
    391 
    392     def artists(self, artists):

[/usr/local/lib/python3.9/dist-packages/spotipy/client.py](https://localhost:8080/#) in _get(self, url, args, payload, **kwargs)
    319             kwargs.update(args)
    320 
--> 321         return self._internal_call("GET", url, payload, kwargs)
    322 
    323     def _post(self, url, args=None, payload=None, **kwargs):

[/usr/local/lib/python3.9/dist-packages/spotipy/client.py](https://localhost:8080/#) in _internal_call(self, method, url, payload, params)
    243         if not url.startswith("http"):
    244             url = self.prefix + url
--> 245         headers = self._auth_headers()
    246 
    247         if "content_type" in args["params"]:

[/usr/local/lib/python3.9/dist-packages/spotipy/client.py](https://localhost:8080/#) in _auth_headers(self)
    236             token = self.auth_manager.get_access_token(as_dict=False)
    237         except TypeError:
--> 238             token = self.auth_manager.get_access_token()
    239         return {"Authorization": "Bearer {0}".format(token)}
    240 

[/usr/local/lib/python3.9/dist-packages/spotipy/oauth2.py](https://localhost:8080/#) in get_access_token(self, code, check_cache)
    900             "client_id": self.client_id,
    901             "grant_type": "authorization_code",
--> 902             "code": code or self.get_authorization_code(),
    903             "redirect_uri": self.redirect_uri,
    904             "code_verifier": self.code_verifier

[/usr/local/lib/python3.9/dist-packages/spotipy/oauth2.py](https://localhost:8080/#) in get_authorization_code(self, response)
    841         if response:
    842             return self.parse_response_code(response)
--> 843         return self._get_auth_response()
    844 
    845     def validate_token(self, token_info):

[/usr/local/lib/python3.9/dist-packages/spotipy/oauth2.py](https://localhost:8080/#) in _get_auth_response(self, open_browser)
    784                     'complete the authorization.')
    785 
--> 786         redirect_info = urlparse(self.redirect_uri)
    787         redirect_host, redirect_port = get_host_port(redirect_info.netloc)
    788 

[/usr/lib/python3.9/urllib/parse.py](https://localhost:8080/#) in urlparse(url, scheme, allow_fragments)
    390     Note that % escapes are not expanded.
    391     """
--> 392     url, scheme, _coerce_result = _coerce_args(url, scheme)
    393     splitresult = urlsplit(url, scheme, allow_fragments)
    394     scheme, netloc, url, query, fragment = splitresult

[/usr/lib/python3.9/urllib/parse.py](https://localhost:8080/#) in _coerce_args(*args)
    126     if str_input:
    127         return args + (_noop,)
--> 128     return _decode_args(args) + (_encode_result,)
    129 
    130 # Result objects are more helpful than simple tuples

[/usr/lib/python3.9/urllib/parse.py](https://localhost:8080/#) in _decode_args(args, encoding, errors)
    110 def _decode_args(args, encoding=_implicit_encoding,
    111                        errors=_implicit_errors):
--> 112     return tuple(x.decode(encoding, errors) if x else '' for x in args)
    113 
    114 def _coerce_args(*args):

[/usr/lib/python3.9/urllib/parse.py](https://localhost:8080/#) in <genexpr>(.0)
    110 def _decode_args(args, encoding=_implicit_encoding,
    111                        errors=_implicit_errors):
--> 112     return tuple(x.decode(encoding, errors) if x else '' for x in args)
    113 
    114 def _coerce_args(*args):

AttributeError: 'tuple' object has no attribute 'decode'

I tried passing my access token in the original auth_manager as different parameters such as 'client_secret' or 'code' as it shows in the error message but had no luck. Does anyone have an advice or tips for me?

[Peter-Schorn]

I tried passing my access token in the original auth_manager as different parameters such as 'client_secret' or 'code'

The Access token, client id, and authorization code are all different things.

spotify = spotipy.Spotify(
auth_manager=spotipy.SpotifyPKCE(
client_id=CLIENT_ID,
redirect_uri = SPOTIPY_REDIRECT_URI,
scope="",
state=None,
cache_handler=spotipy.CacheFileHandler()
),
requests_session=session
)

Try providing a non-empty authorize scope string.