[ci-en] 403 when attempting to download a followed user's gallery

[nomakewan]

When attempting to download an entire gallery at once for a followed user using the pattern gallery-dl https://ci-en.dlsite.com/creator/####/article, the process runs for a little bit (specifically, about 21 files' worth) then throws multiple [cien][error] HttpError: '403 Forbidden' for https://ci-en.dlsite.com/creator/####/article/... until merely trying to get the next page also throws a 403, which causes gallery-dl to exit rather than continue trying.

This appears to be a form of anti-bot measure. It would also appear that similar to a few other services (such as Pixiv Fanbox), ci-en is using Cloudflare as their intermediary. At the moment, my config just uses username/password for login.

Is there a step I'm missing? Something I need to change in the config? Or is this a bug that can be fixed in gallery-dl?

Thanks as always, and again thank you for implementing ci-en support!

[mikf]

Sorry for not replying to your previous post (#2885 (comment)). I hoped you figured something out yourself or that the 1-2 second sleep-request default would help (d65fc24), but apparently not.

The thing is, during all my testing during development I've personally never gotten any 403 or network error. None at all. And to make sure (and potentially reproduce the issue), I've just now downloaded all free media from https://ci-en.net/creator/7491 (102 files, 650MB) with default settings and ... no 403 errors, nothing.

Maybe something in your config is overwriting the 1-2 second sleep default (check with -E), maybe some HTTP headers are different for you due to config settings, maybe Ci-en doesn't like downloading with gallery-dl while using authenticated cookies.

my config just uses username/password for login.

Login with username + password is currently not supported. You need to pass cookies to count as logged in.

[mikf]

Another thing to possibly use to debug the problem: You can tell gallery-dl to write the full 403 error response to disk with --write-pages or to print it to stderr with --print-traffic.

[nomakewan]

That was it. The gallery.conf template includes a value of 0 for sleep-request which was overriding the defaults. After removing that line, no more 403s on ci-en. I do have another issue but it's unrelated to this so I'll close this one.

Thank you!