Dockerfile: Use libcurl-impersonate in place of libcurl

[wrobelda]

Note that this is a similar fix to the one proposed in #2925, except it does so by using Docker multi-staging build.

Also note that this doesn't currently work for ARM, due to lwthiker/curl-impersonate#92, but that seems like an easy fix.

Also requires user_agent not to be set, otherwise curl_impersonate will honor it (lwthiker/curl-impersonate#96) and, as a result, ignore its own defaults.

Fixes #2547

EDIT: also requires lwthiker/curl-impersonate#93 to be fixed upstream now done.

[wrobelda]

Actually, it seems that the x64 has some issues, in fact. I'll get back on that.

EDIT: lwthiker/curl-impersonate#93

[dvikan]

As far as I can tell, this change does not set any useful http headers to mimic browsers? It modifies the tls behavior only?

#2927

[wrobelda]

As far as I can tell, this change does not set any useful http headers to mimic browsers? It modifies the tls behavior only?

#2927

The ENV CURL_IMPERSONATE ff91esr line is responsible for exactly that. If you look into the corresponding curl_ff91esr script for standalone curl, it's pretty obvious: https://github.com/lwthiker/curl-impersonate/blob/main/firefox/curl_ff91esr. Also see the README notes on that from the project itself: https://github.com/lwthiker/curl-impersonate#using-curl_impersonate-env-var

[dvikan]

Tested and seems to work fine. But getting this error on PatreonBridge. Still getting it when removing the custom ciphers that the bridge itself sets.

 Uncaught Exception HttpException: Unknown cipher: DEFAULT (59) at lib/contents.php line 229
#0 index.php:35
#1 actions/DisplayAction.php:134
#2 bridges/PatreonBridge.php:84
#3 bridges/PatreonBridge.php:181
#4 lib/contents.php:103
#5 lib/contents.php:229 

The base image php:7.4.29-fpm uses Debian 11 "bullseye" but the curl-impersonate is Debian 10 "buster". I don't know how that works out but seems to work.

[dvikan]

https://daniel.haxx.se/blog/2022/09/02/curls-tls-fingerprint/

[wrobelda]

I just updated rss-bridge and some of my more demanding bridged (#2942) stopped working with 429 error. I had to override user_agent in config.php and set it to an empty string, otherwise curl_impersonate would honor it and, as a result, ignore its own defaults. Somehow the current rssbridge's user-agent (recently changed by #2926) blocks curl-impersonate from working properly.

What I suggest is, if this gets merged, to detect curl-impersonate and ignore the config.php-provided user-agent, perhaps with an option to force it.

Also, what's the blocker for this to be merged? Is it bullseye vs buster issue? If so, then I suggested that the upstream moves to bullseye: lwthiker/curl-impersonate#118

[dvikan]

I am blocking this because I don't understand how it works. Also as I wrote earlier I got a TLS CIPHER error which must be related to this. And I don't like complexity. Maybe you can write a few sentences on how it works.

[wrobelda]

And I don't like complexity. Maybe you can write a few sentences on how it works.

Can you explain what specifically do you find complex? Is it the curl-impersonate itself or how it is integrated in the Dockerfile in this very PR?

[dvikan]

I think it's the docker multi-stage that irks me. I don't understand it. I don't want to maintain that.

Also I fear we will experience lots of TLS-CIPHER errors just like the one I described.

[wrobelda]

I think it's the docker multi-stage that irks me. I don't understand it. I don't want to maintain that.

All it does is literally copying that single libcurl binary file from their upstream image. The alternative is copy-pasting their Dockerfile into your own Dockerfile, but it takes quite a while to build all the dependencies.

Given that the viability of the project is limited by the ability to operate in stealth mode, I'd argue that it's at least worth taking it for a spin? Maybe create another Dockerfile.impersonate or sth like that? That one would actually be quite simple since it would use your vanilla as a source image.

As to Unknown cipher: DEFAULT (59), if that's coming from libcurl itself then I'd suggest asking the developer behind curl-impersonate, they're quite responsive and helpful.

[wrobelda]

Actually, I just realized that since LD_PRELOAD and CURL_IMPERSONATE are both env variables, they can be set outside of Dockerfile, thus making libcurl completely optional (and configurable). For anyone wanting to leverage it, they'd simply pass them with -e to command line (or appropriately in their docker-compose.yml).

[wrobelda]

ping @dvikan

[dvikan]

I tested it again for a few bridges and it seems good. How about we merge this as it is now and then you help me fix related issues if they occur?

[wrobelda]

I tested it again for a few bridges and it seems good. How about we merge this as it is now and then you help me fix related issues if they occur?

Happy to!