API requests suddenly stopped working and hang forever

[julienbeisel]

Hi,

I recently started using Spotipy to get information about artists for a dataviz project.

As you can see, one of my apps makes a lot of requests, but I make sure to wait between requests so I don't get limited by the 30 second time-window limit.

Sometimes, the API just stops working for 1 to 24 hours. I don't get any error message, it just keeps running without responding.

Here is how I initialize the client

SP = spotipy.Spotify(
    auth_manager=SpotifyClientCredentials(
        client_id=os.environ.get("CLIENT_ID"),
        client_secret=os.environ.get("CLIENT_SECRET"),
    ),
    requests_timeout=10,
    retries=10,
)

Do you know why I have this error ? I try every 12h to run my code, sometimes it works and sometimes it doesn't.

I can't find any information about the rate limite of Spotify, but I think I'm not reaching any limit otherwise I would get an error as a response?

Thanks for your help!

[ulysses82]

Same issue here. The API stopped working for me too (with no error messages). Pretty sure it's rate limiting.

[justingrisanti]

I am also having this issue now when I use the .search() or .track() methods. Was this ever resolved for you?

[Peter-Schorn]

If your requests are getting rate-limited, then you will receive a very specific error message indicating that you have been rate-limited.

Whenever your code freezes, you should step through it in the debugger to see exactly which function is blocking. Have you done this?

[julienbeisel]

Hi @Peter-Schorn, I managed to debug by doing this :

• Go on the Spotify doc
• Use the API playground with my ID and secret_key to try the API
• Check the result

When I tried it, the API sent an error message because I was rate-limited and asked me to try after x minutes. The issue was the rate-limitation but the behaviour Spotipy was not what I was expected, it just hanged forever without returning an error.

I think the Spotipy package should expose this information so we could automate the process of retrying after x seconds.

After looking at the code, I think it should work, so I don't know what the issue was: https://github.com/plamere/spotipy/blob/master/spotipy/exceptions.py#L8

[dasbts]

I'm getting this issue when trying to do anything, and it works fine in the Spotify Dev API console, thus confirming the issue is somewhere else. Started getting the issue yesterday, then after some time it worked again and today it's just nothing at all. No changes to the code, but I tried multiple auth flows such as:

auth_manager = SpotifyClientCredentials(client_id=client_id, client_secret=client_secret)
sp = spotipy.Spotify(auth_manager=auth_manager)
token = util.prompt_for_user_token(scope=scope, client_id=client_id, client_secret=client_secret, redirect_uri=redirect_uri)
sp = spotipy.Spotify(auth=token)
token

Or:

sp = spotipy.Spotify(auth_manager=SpotifyOAuth(client_id=client_id,
                                               client_secret=client_secret,
                                               redirect_uri=redirect_uri,
                                               scope=scope))

I don't know if I'm using debugger incorrectly, but as soon as I run any of the sp.XX functions from the Spotipy library, it just freezes and doesn't add anything to the output.

I'm using Anaconda JupyterLab (the newest version) by the way, and for some reason, this doesn't just freeze the code, it freezes the whole Python, so I have to close it manually in Windows Task Manager before starting a new one, otherwise, it's just gonna keep spinning even tho Anaconda doesn't realize it's still open, this as the port is still available for the Jupyter.

Also to add, similarly to the OP, I run a lot of requests in a loop until I'm satisfied, but again, it works on the console so it makes no sense. Removed the cache as well to try because initially I got a refused refresh token error.

EDIT: My first calls are sp.current_user_playlists() and sp.playlist(), haven't tested any other ones yet but it fails on these. Sometimes it almost seems like restarting the computer works.

[dasbts]

Ok I noticed that a different access token is generated from Spotipy with the code examples posted above, than when generating one in the console. The Spotipy doesn't work but rather just loads in their console too.

So, it seems like after using the token generated on the console that this works fine when added to the Spotipy code, while Spotipy is no longer able to generate whatever they have changed the access token requests into.

[Peter-Schorn]

You need to either use a different IDE or figure out how to use the debugger on Anaconda JupyterLab. With a proper debugger, you can step through each line of code to figure out exactly where the code is blocking. Merely saying that the code freezes at sp.current_user_playlists() is not specific enough because this function calls through to another function, which calls through to another function, and so on.

[dasbts]

You need to either use a different IDE or figure out how to use the debugger on Anaconda JupyterLab. With a proper debugger, you can step through each line of code to figure out exactly where the code is blocking. Merely saying that the code freezes at sp.current_user_playlists() is not specific enough because this function calls through to another function, which calls through to another function, and so on.

Alright, but as I told you in my next post, I already found where the issue is without debugger, in my case it's not in any of the features, but rather in the authentication.

I run the code successfully by going to the console and requesting a new token manually for the scopes I need, then copying that into my python code and running.

However, when I run the authentication code (any of the above, even the other Python library Tekore, gonna try manual request too later) then it freezes, and even if I paste this token in the console at Spotify, it freezes in their API too. Which means something has happened in the way you request tokens, and something has happened in the how their token standard looks like.

Please look into this ASAP if you can, I'm dependent on it for multiple products.

[Peter-Schorn]

Alright, but as I told you in my next post, I already found where the issue is without debugger, in my case it's not in any of the features, but rather in the authentication.

No you didn't find where the issue is. The fact that the authentication code freezes is not nearly specific enough. There are a million places where the authentication code can freeze. You can find the specific place where the code freezes by using the debugger. That's why it exists.

[Peter-Schorn]

Ok I noticed that a different access token is generated from Spotipy with the code examples posted above, than when generating one in the console.

A different and unique access token is generated every time you go through the authorization process, no matter how you do it.

[dasbts]

Ok I noticed that a different access token is generated from Spotipy with the code examples posted above, than when generating one in the console.

A different and unique access token is generated every time you go through the authorization process, no matter how you do it.

I know. But the ones with any Python library does not work any longer. It even freezes the Spotify Web console. If you try it out you will see what I mean. They changed something in the standard format of access tokens the recent days it seems like. When generated on the console and then manually entered into the other Spotipy functions it works fine as I explained.

[dasbts]

Alright, but as I told you in my next post, I already found where the issue is without debugger, in my case it's not in any of the features, but rather in the authentication.

No you didn't find where the issue is. The fact that the authentication code freezes is not nearly specific enough. There are a million places where the authentication code can freeze. You can find the specific place where the code freezes by using the debugger. That's why it exists.

The issue is in the generation of the access token. Since I can reproduce the issue without a library in the https://developer.spotify.com/console/ website when using an access token it's clear that the issue is in generating the access token, and not in any other functions of Spotipy. Yes, generating an access token works, but anything that token is used in, including the console at https://developer.spotify.com/console/ is going to freeze the output, while a token generated in https://developer.spotify.com/console/ is working in all cases.

[dasbts]

Still getting this after running the code for about a day, then this happens for about 24h where I have to manually enter the token generated in https://developer.spotify.com/console/

The weird thing is, it works for some functions even with the token updated from Spotipy (which seems to freeze no matter where it's used in some functions). So to clarify:
When I generate a token with:

token = util.prompt_for_user_token(username='XYZ', scope=scope, client_id=client_id, client_secret=client_secret, redirect_uri=redirect_uri)
sp = spotipy.Spotify(auth=token)

The code works when running:
sp.available_markets()
But freezes (likely infinitely looping) when running for example:
sp.search('Test', limit=50, offset=0, type='playlist', market=None)
Or as previously mentioned for example:
sp.current_user_playlists()

This makes me think there's some try: except: somewhere that's depressing whatever error this is about.
Why? I don't get any error, the token from the console is working (thus, my account does not rate limited), and seeing that OAuth is required for Get Markets as described here, yet works while some other OAuth functionality does not work:
https://developer.spotify.com/console/get-available-markets/

Could it maybe be some functionality built-in for automatically avoiding rate limits? I have implemented time.sleep(0.3), but it seems like that's not helping prevent this from happening.

I have yet to get a debugger going that actually shows me valuable information. From what I've achieved so far, the debugger hasn't shown me anything of value, but I will look into this in more detail in the next weeks. Also, if I can't figure out the debugger to identify exactly where, and if it's not already found by someone else by then, I will just try all the functionalities to see what works or not and go through the Spotipy library code manually to see what causes this.

[Peter-Schorn]

When I generate a token with:

token = util.prompt_for_user_token(username='XYZ', scope=scope, client_id=client_id, client_secret=client_secret, redirect_uri=redirect_uri)
sp = spotipy.Spotify(auth=token)

The code works when running:

Never user util.prompt_for_user_token! It's officially deprecated; the token will not get automatically refreshed.

Look at the examples in the README:

import spotipy
from spotipy.oauth2 import SpotifyOAuth

sp = spotipy.Spotify(auth_manager=SpotifyOAuth(client_id="YOUR_APP_CLIENT_ID",
                                               client_secret="YOUR_APP_CLIENT_SECRET",
                                               redirect_uri="YOUR_APP_REDIRECT_URI",
                                               scope="user-library-read"))

This is how you authorize your app.

[dasbts]

While I appreciate your tips for authentication, Peter, it does not help this issue. I already have a code to automatically refresh my token whenever I get an error that it expired, and I get a token fine, it's just whenever running certain functions (even with your code above) it's not working (which I have described in detail in the previous posts, how some OAuth functions work, and some don't whenever this happens), so util works fine for me. I tried the following and still get an infinitely frozen code, thus, indicating it is in some form of an infinite loop. I can look into the Spotify source code sometime likely next week to try to reproduce it manually.

My bet is either there's a try: except: somewhere in the defined functions talking to the Spotify API, or there's something in the handling of cache that contains some bug.

[BenAM63]

I was having this issue occur when trying to call sp.search() or sp.playlist_items(). Using a debugger I found that eventually in the retry.py file in urllib3, a function called sleep_for_retry() was being called:

def sleep_for_retry(self, response=None):
        retry_after = self.get_retry_after(response)
        if retry_after:
            time.sleep(retry_after)
            return True

        return False

It seems that time.sleep(retry_after) is used to wait instead of sending more requests and being rate limited. Since no message is printed, it seems like it just freezes. I couldn't find a consistent fix, as using different credentials when initializing the client didn't fix it. Waiting out the retry_after timer appears to work, but that could take many hours.

[Peter-Schorn]

I already have a code to automatically refresh my token whenever I get an error that it expired, and I get a token fine

If you use the library correctly, as I have mentioned before, the token will get automatically refreshed before it expires. This saves a lot of time, if nothing else.

[Terima]

I was having this issue occur when trying to call sp.search() or sp.playlist_items(). Using a debugger I found that eventually in the retry.py file in urllib3, a function called sleep_for_retry() was being called:

def sleep_for_retry(self, response=None):
        retry_after = self.get_retry_after(response)
        if retry_after:
            time.sleep(retry_after)
            return True

        return False

It seems that time.sleep(retry_after) is used to wait instead of sending more requests and being rate limited. Since no message is printed, it seems like it just freezes. I couldn't find a consistent fix, as using different credentials when initializing the client didn't fix it. Waiting out the retry_after timer appears to work, but that could take many hours.

Apparently this happened to me too. It would be better if at least we know how long we need to wait before the next retry.

[TomBer0-0]

This happened to me today too. It's the third time, and it's more annoying than ever because my app isn't working after waiting for over 2 hours already. It almost seems like the cooldown time is random.

[stephanebruckert]

That's not spotipy related. Well, the only thing we could do is add a debug line to let the users know how long they can expect to wait. See this thread where some element of answers were provided in #937 and #913 (comment)

I imagine that switching to a new Spotipfy app would solve the issue, for some time at least, depending on how much the Spotify API rate limits your app.

Let's continue this discussion in #913