-
-
Notifications
You must be signed in to change notification settings - Fork 951
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[twitter][inquiry][possible feature request] Skip download for already downloaded text-tweets? #3786
Comments
Metadata postprocessors support |
Ah! Didn't think of that. Thanks! :D |
Hmm. It does no longer overwrite the files, good, that was the important part. On the other hand, I measured it for a small test-sample, and that still takes just as long as if it did overwrite them compared to it being faster when removing the "postprocessor" section from the *.config altogether. Not sure what's happening here to slow things down, hard to say. I know it used to take a couple hours to fully reparse twitter before I started also downloading text-tweets, even without an abort parameter. Now it takes about a day just to do it with abort set to 1000. Could of course be other factors at play here, maybe twitter just slowed down too. Ah well, slow and steady wins the race, and not overwriting the files was the more important of the two inquiries.
EDIT: mikf added a commit that referenced this issue 4 minutes ago Ooops, didn't see that! :) |
Last time I did a rough measurement, GraphQL endpoints were around 4x slower than the previous REST API from before cb43f77. Also, I did try to improve You can also disable
I think you need to also set a custom You might also want to look into |
Is |
@a84r7a3rga76fg |
Ops, meant |
@a84r7a3rga76fg I'm not planning on removing support for |
My gallery-dl config is set up to download text-tweets too. However, it re-downloads them every time to overwrite the old file.
This significantly slows down the entire download process.
(Yes, I compared it and it really is significantly slower. Setting "text-tweets: false" in my config didn't change much, but removing the corresponding post-processor setting from my config sped things up significantly. So it's the act of needlessly re-saving the text-files that's the problem.)
It also creates a tremendous amount of overhead for when I run a backup of my gallery-dl directory, with literally hundreds of thousands of "updated" text-files being backed up needlessly which takes a frankly absurd amount of time and probably isn't good for the hard drive.
Is this something that can be changed in the config already?
If not, is this something that would be easy to implement? "If text-file with name X exists, skip and don't even attempt download"?
My config:
The text was updated successfully, but these errors were encountered: