Not showing all games #264
Comments
|
Same, it seems that app showing games that are in priority list and no more |
|
I can confim this too. KOTC is an active campaign and when i open Twitch channel for it manually, it states I am receiving drops but this is not showing up in the Inventory or in the games list. |
|
Same problem. 2 games on the list |
|
Hello o/ After some simple debugging, it seems that Twitch no longer reports all ongoing campaigns via GQL, if one is using the SmartTV login (like this miner does). The response comes through, but the
This means the miner has lost the ability to discover new, unprogressed campaigns, and thus won't add their games to the UI combobox. The good news is that the "Game name" entry is just a text entry with a dropdown that lets you pre-select a "known" game name, but otherwise functions as any ordinary text entry. This allows you to just type in the name of a game you'd want to add, and add it to the list that way. Just make sure to not make any spelling mistakes, the comparison is case-sensitive, and requires the exact game name, as displayed on the Twitch campaigns page. Another good news is that Inventory view seems to still be working, so any campaign that's already started, will continue to be returned, and thus handled by the miner like expected. This means you can login to Twitch on your PC or phone, pick a campaign you want to start, tune into some random stream to start the campaign (get the first minute of progress), and then let the miner handle it from there. The bad news is that I have no immediate resolution to this problem. There is a chance this issue is only temporary, and campaigns view will return to normal later. Or it's permanent, and I'll need to figure out another way of getting that information later. |
|
Thanks for being so quick to look into this DevilXD you are awesome |
|
I just happened to be off work delegation and home today, so could spare a quick look at least. I'm going back tomorrow though, and it'll be another week before I'll be able to get some more free time. Still not sure what to do here, the web version returns campaigns, but the miner version doesn't. Web version is protected by the integrity system, so can't use it. For now, all campaigns need to be started manually - after that, they'll appear in the inventory view and everything should work from there as expected. |
|
No problems, not sure if this is related but KOTC aren't being picked up. I'm not linked (but i usually can get the drop anyway). even with manual addition (and the drop started), its not being picked up in the miner main section. It's recognised in the inventory. I have manually added King of the Castle,. Not sure if this is a naming issue or not being linked but I was able to pick up the drop in the past (can't remember if by the miner or on my own) Additional Info: it is in the priority list, there were streams active. |
|
The miner requires a linked account to consider a drop "worth mining". That's most likely why. |
|
would it be possible to have the app skim the webpage for looking at available campaigns or something like that? |
I deleted my current Twitch Drops Miner install and downloaded a fresh version, re-linked my Twitch, and still only seeing 1 option show in the list of games. |
|
@MuffinzCat Viewing https://www.twitch.tv/drops/campaigns requires logging in, and logging in requires an authorization token. The miner already uses one, but it's been locked out of the page contents per this issue. What you see when you open the browser, is the page via your web authorization token, which is protected by the integrity system, which isn't beatable at this time, so using it is not possible. So no, it's not possible to "skim" the website in any way. @KevinNovak That's expected, as this issue is caused by Twitch itself, not the miner installation or linking status, or anything like that. Note that you can still add games like before, albeit you'll have to type out their names yourself (or just copy-paste them from the website). |
Seconding this, even when I manually write the names of games in the miner still won't find and idle streams. |
|
I'm starting to see some games show up in my list without me manually adding them. anyone else? |
The only time I see the list update is if a new drop was started then it'll be added to the list. Assuming it's caching "known" drop campagins/streams/games (i.e. if it shows up in inventory on Twitch page, it shows up on the games list) |
|
You can watch it on the webpage for 1 minute, activate the treasure drop, and then open the software to continue hanging the treasure drop of this game normally |
|
Since there seem to be some confusion around what exactly happens, so I'll try to explain. The miner utilizes the "contents" of these two Twitch pages: These are also known in-short as just "inventory" and "campaigns". The contents are fetched via the underlaying page's GQL request, each of the pages having it's own, different one. The "campaigns" page request has currently been crippled/disabled/god-knows-what-exactly-happened, and as the result, only the "inventory" one remained. The inventory request contains all information that you can see on the inventory page - this includes primarily the drops your account is currently in the process of earning. Having at least 1 minute or percent of progress on a drop is enough for it to appear on the inventory page, and be recognized by the miner. Thus, you can "manually" force certain drops/campaigns to appear there, by watching an eligible stream for at least one minute (eg. in the browser), enough to gain some progress on the drop - and the miner should pick it up from there, as it's able to still see the inventory like before. The "campaigns" page was used to let the miner know about all campaigns that have no progress done towards them, and can be started - since that's gone now, that information is missing. This is also the main reason the game picker isn't displaying the games you'd expect - because it can only see those on the "inventory" page instead. You can open https://www.twitch.tv/drops/inventory at any time, and whatever drops, campaigns and progress you can see there, should appear in the miner as well (may require a reload first). |
|
would this mean that previously completed games would appear on the list again if it gets another campaign? |
|
Yes and no. Yes, because normally, that's what usually happens, especially for games that get new weekly campaigns. No, because it's not guaranteed to happen, as not progressing a particular game's campaigns for a while, somehow disables this process and they stop appearing. At least that's how it used to work before, it could've changed now. |
|
I feel for anyone developing for Twitch. There's just no consistency with them. Thanks for the continuous support DevilXD. Twitch HQ feels very strong to destroy any user apps they set their eyes on. |
|
would it be possible to manually export/import the https://www.twitch.tv/drops/campaigns list or something along those lines if we are not able to fully automate it anymore |
|
@Larsy93 I've already explained why it's not possible here: #264 (comment) |
"Manually" means that the user DLs the page in some fashion, or runs javascript that exports a JSON/.. of active campaigns to file/clipboard, and TwitchDropsMiner loads this info from file/clipboard. |
|
This is a Python program, I don't want to have to introduce any Javascript or side loading into the application. Definitely won't be developing any browser extensions for this either. I've remembered that there's an This pretty much calls for a way of manually adding channels to the miner, and the entire logic flow having to be reversed. Instead of the drop campaigns deciding which channels to use for progressing them, the channels themselves would report which drop campaigns they can progress. This also means having to replace the Priority game selector with a channel selector, although I think it can be left as-is and instead only adding a way to add channels would be enough. I didn't want to do this as the whole thing was supposed to be as automatic as it was possible to do at the time, but now the times have changed, and this sounds like the only option that could work. The problem is, I've recently started the 4th month of my work delegation on a project in a remote location, so I still barely have any time for any actual coding and changes. I really wish this wasn't the case, but I can't help it. I'll try getting to this in any spare time I will be able to find, but can't make any promises on when this will be added/changed. |
|
Yeah that's okay, it'd be selfish of people to demand that you'd work on an issue, let alone one you didn't create yourself |
|
Can say so much that only the games that show up in the dropdown is the only games it will mine. i have Overwatch 2 in my prioity list, but it won't gather the new torbjörn skin. so ATM it's only useable for 2 games. at least for me. |
|
A very slight workaround is to start getting watch progress by clicking on a stream with drops available to start gaining progress and then letting the bot continue gaining the progress. |
|
Yes, the quick patch resolves only the Spade URL extraction step, it does not resolve your connectivity to Twitch. "Twitch is down" message is shown on 5XX errors (Twitch server error), while "Cannot connect..." message is shown on a timeout (connecting took too long), or another error related strictly to connecting to Twitch (even before any response data is transmitted back). I cannot help you solve this one, it can be anything:
..., and the list just goes on and on. It'd take an eternity to debug every single connectivity issue case, everyone has it different, etc. I can only suggest where to look for a possible fix, but other than that, you're more or less on your own. None of this happens on my PC. I can say that after the change, the requests definitely take a longer time for me, as each loading step executes, but I can only blame Twitch being slow on it, since nothing else besides the client ID has changed. |
|
Switched back to 137f540 for now. |
|
So, I was having that issue. I was getting multiple "twitch is down" or "twitch stopped responding" messages. I'm watching two different channels on my main computer while TwitchDropsMiner is running on a RaspberryPi, so I was also getting "Earned points for watching" notifications on the Miner from these other channels (since I imagine the points management is account wide, a connection to Twitch servers seems to be established). I can also watch twitch on a browser on the RasberryPi that the Miner is running with no problems at all. I'm from a different country, so I put on a US IP Address proxy (maybe this new "TV Client" has more features on a different region? - I'm spitballing here) - didn't change anything. Got the same behavior. I deleted my environment, re-built it, re-downloaded the application, re-did the login and I was getting equally slow and unresponsive behavior - no proxy running. I stopped paying attention to my RaspberryPi desktop (where Miner is loading) and went to do something else. In that time, the miner managed to load the channel list and is currently idling for items.
As you can see, it took 22 minutes to retrieve the channel list and start farming. The amount of "twitch is not answering" and "twitch is down" changed from before, but it still there. I can't really tell you why, as I have no issues watching twitch on the RaspberryPi instance or on any other computer on my network. I don't really know how to debug a python application, so I can't really help much more than this :D Edit: I have now scored and claimed a drop. We're now back on the "Gathering channels" section of the loop, with no log messages of what is happening. I imagine that, whatever way the software uses to grab channels is either being throttled by Twitch or some internal issues are happening. I imagine we will go back to the "twitch is down" and/or "twitch stopped responding" loop until it eventually retrieves the necessary information. -- Thank you for the care and help with the quick patches. I'm sure other people will be more useful than I am in figuring this one out. |
|
Dev, my twitch is currently working with none of the changes poeple are giving a go. so if you need a test subject, I can volunteer :) |
|
so far claimed R6 siege drop with only 5 min of gathering channels. no lost connection. 09:15:34: Claimed drop: Tom Clancy's Rainbow Six Siege Reveal Y8S3 Charm (1/1) -- |
|
I've put the latest pre-release on my work PC (used the Github version, didn't build myself), and startup took roughly 2m and 30s: This whole "Mobile Web" Client ID seems to just be slow in general, for some reason. I wonder if Twitch is intentionally throttling it's speed, to push people into downloading the app instead. I guess only Twitch knows. There isn't much I can do about this. The current request timeout is set to 10 seconds, which should be plenty to get a response already, but with Twitch being this slow, it may just not be enough. Increasing the timeout may reduce overall app performance in situations where the connection is unstable, leading to slower operation, but it may also improve overall reliability. @DJ4everkill Please file a separate issue with that error, seems like something that needs to be worked on some more. |
|
Can anyone confirm if the new release with the MOBILE_WEB Client ID uses more bandwidth than the previous one? |
|
I've just been stuck on "gathering channels" for like the past 2 hours.. running commit version d5410b2 this is the only program that seems to be having an issue connecting to twitch once the switch to the MOBILE_WEB Client ID happened.. Funny thing is, it took 2 hours before it said anything in the output. Edit: Downgraded to commit 29fa220 before I went to bed and after about ~4 hours it seems to finally actually be doing something, but before it started doing anything it did throw a bunch of error message:
|
"Slight" difference :)
But at least the functionality is working again. Each rose has it's thorns. |
|
Wait, that's the difference? That shouldn't be happening, unless Twitch sends a bunch of random shit in addition to the expected data. This would also explain why it's so slow all of the sudden. I'll try looking into it in some free time. EDIT: I took a quick look, and there's nothing that could cause such an increase, the usual set of data is returned. Are you sure this chart doesn't include a bunch of other things you've downloaded along the miner running? |
|
On commit 137f540 (i.e. before the recent mobile-related changes), HTTP requests the app make looked like this:
On commit d5410b2 (i.e.
A few observations:
After adding the changes I proposed, the list of HTTP requests should look like this:
Which is still not as good as before, but a tiny bit better. There are still a bunch of requests being made to Twitch's main page, which I don't understand why since I'm not too familiar with how the app works internally. |
|
I think I got it. This change removed the ability to extract Line 520 in d5410b2 And, most importantly, here: Line 523 in d5410b2 This second check is what is triggering the extra requests to Twitch's main page. Removing those
If someone wants my changes to test for themselves: diff --git a/constants.py b/constants.py
index a40ca71..79023a8 100644
--- a/constants.py
+++ b/constants.py
@@ -80,7 +80,7 @@ MAX_TOPICS = (MAX_WEBSOCKETS * WS_TOPICS_LIMIT) - BASE_TOPICS
MAX_CHANNELS = MAX_TOPICS // TOPICS_PER_CHANNEL
# Misc
DEFAULT_LANG = "English"
-BASE_URL = URL("https://twitch.tv")
+BASE_URL = URL("https://www.twitch.tv")
# Intervals and Delays
PING_INTERVAL = timedelta(minutes=3)
PING_TIMEOUT = timedelta(seconds=10)
diff --git a/twitch.py b/twitch.py
index d940a26..7ea2c27 100644
--- a/twitch.py
+++ b/twitch.py
@@ -517,12 +517,12 @@ class _AuthState:
async def _validate(self):
if not hasattr(self, "session_id"):
self.session_id = create_nonce(CHARS_HEX_LOWER, 16)
- if not self._hasattrs("client_version", "device_id", "access_token", "user_id"):
+ if not self._hasattrs("device_id", "access_token", "user_id"):
session = await self._twitch.get_session()
jar = cast(aiohttp.CookieJar, session.cookie_jar)
- if not self._hasattrs("client_version", "device_id"):
+ if not self._hasattrs("device_id"):
async with self._twitch.request(
- "GET", BASE_URL, headers=self.headers()
+ "GET", "https://m.twitch.tv/", headers=self.headers()
) as response:
page_html = await response.text("utf8")
assert page_html is not None |
It's filtered by app that's why I can easily get a graph for either versions. I attached a file of 5/08 which was a high traffic date on the old version. Also 17/08 which is today so far on the 29fa220 version. edit: also added console output of 17/08. CnnHist-2023-08-17T20-39-57.csv |
|
@guihkx i tried your "fix" and it seems to work. Ran through the whole night and ahvent encountered any problems so far. |
|
someone could send the compiled bot pls :P |
|
@DevilXD Testing the latest dev build as of right now and looks like everything is working perfectly! Starts up quickly, doesn't have the issue of getting stuck on the |
|
After 20 hours of running e16ddea and mining drops it also seems to be back at normal bandwidth. No longer m.twitch.net connections transferring gibs of data every day. |
Would it be possible to create a "Chromium"-based cookie via headless browser and feed the miner that cookie? Which the miner parrots as if it's the actual browser? Even if it'd require a headless Chromium running in the background, it'd be great regaining the browser's access to the master campaign "index". Just a thought. |
Okay, but how would you login on Twitch using a headless browser (or any other remotely-controlled browser, for that matter)? As far as I know, Twitch's integrity system (which @DevilXD literally spoke about in the message you quoted) is currently undefeated. In any case, right now I'd consider this issue fixed. |
|
@Coldblackice This was already a part of this project at some point. Here's a short history summary:
The gist of it is that, Chrome login/access tends to work for a while, then not work for a while, then again work for a while, then not work, and so on. Every time it stops working, someone from the Selenium project sits down, makes it work again, and pushes out an update. Updating Selenium as a dependency in a pre-compiled project is impossible, without rebuilding and re-releasing the entire thing, a way I really do not like to go with. The new mobile login option works, so lets stick to this - I can try going back to Chrome flow later, once there will be no other option. |
|
As @guihkx said, this issue is closed now. If a new issue arises, please open a new one. |
and others
It shows only few games, when there are a lot available
The text was updated successfully, but these errors were encountered: