Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for HTTP2 requests in Curl #42

Closed
mnapoli opened this issue Jan 5, 2023 · 4 comments · Fixed by #43
Closed

Add support for HTTP2 requests in Curl #42

mnapoli opened this issue Jan 5, 2023 · 4 comments · Fixed by #43
Assignees

Comments

@mnapoli
Copy link
Member

mnapoli commented Jan 5, 2023

We had that issue in Bref 1 previously:

We need to fix that in the new layers as well.

Here is how I checked the problem:

  • Run bash in a Bref container
    • docker run --rm -it --entrypoint=bash bref/php-81:1.7.15
    • docker run --rm -it --entrypoint=bash bref/php-81:2
  • Check php -i | grep HTTP2 and php -r "var_dump(get_defined_constants());" | grep CURL_HTTP

I should see these constants:

  ["CURL_HTTP_VERSION_1_0"]=>
  ["CURL_HTTP_VERSION_1_1"]=>
  ["CURL_HTTP_VERSION_NONE"]=>
  ["CURL_HTTP_VERSION_2_0"]=>
  ["CURL_HTTP_VERSION_2"]=>
  ["CURL_HTTP_VERSION_2TLS"]=>
  ["CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE"]=>

But instead I see only these:

  ["CURL_HTTP_VERSION_1_0"]=>
  ["CURL_HTTP_VERSION_1_1"]=>
  ["CURL_HTTP_VERSION_NONE"]=>

In Bref 1, we compiled LIBNGHTTP2. In Bref 2 I'd love to find a solution that doesn't involve compiling (slow), if we can install a package that's better.

@shouze do you have any idea?

@mnapoli mnapoli added this to Bref 2.0 Jan 5, 2023
@github-project-automation github-project-automation bot moved this to Todo in Bref 2.0 Jan 5, 2023
@mnapoli
Copy link
Member Author

mnapoli commented Jan 5, 2023

What is weird is that libnghttp2 is already installed in AL2…

ldd /lib64/php/modules/curl.so
	libcurl.so.4 => /lib64/libcurl.so.4 (0x0000004001a45000)
	libc.so.6 => /lib64/libc.so.6 (0x0000004001cdc000)
	libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x0000004002089000)
	...

And /lib64/libnghttp2.so.14 is already here in AL2. We can even try to install it:

yum install libnghttp2

...

Package libnghttp2-1.41.0-1.amzn2.x86_64 already installed and latest version

So I'm not sure what's missing here. We don't copy libnghttp2.so.14 in the Lambda layer because that file already exists on Lambda/in AL2.

@mnapoli
Copy link
Member Author

mnapoli commented Jan 5, 2023

Forgot to mention: the CURL version in the new layers is 7.79.1, so it should be more than enough to have the HTTP 2 constants. Something is missing but I'm not sure what.

@shouze
Copy link

shouze commented Jan 5, 2023

So I'm not sure what's missing here. We don't copy libnghttp2.so.14 in the Lambda layer because that file already exists on Lambda/in AL2.

@mnapoli are you experiencing on x86, arm or both?
I've quickly checked and yes at least in the docker image looks like libnghttp2 is installed by default in AL2:

❯ docker run --rm -it --entrypoint= public.ecr.aws/lambda/provided:al2-x86_64 find / -iname 'libnghttp*'
/usr/lib64/libnghttp2.so.14.20.0
/usr/lib64/libnghttp2.so.14
/usr/share/licenses/libnghttp2-1.41.0

So... are you experiencing on the true runtime lambda layer, AL2 docker image or both? If only on rumtime lambda layer, worth to double check there's no diff between AL2 docker image & true lambda runtime layers.

Then, despite the libnghttp2 is provided, it then depends if installed libcurl has been compiled with libnghttp2 support or not. And it seems that libcurl is not provided by default in AL2 layer. And finally, it depends if php itself has been eventually compiled with libnghttp2.

And looks like... yes (at least on x86):

❯ docker run --rm -it --entrypoint=bash bref/php-81:2 ldd /opt/bref/extensions/curl.so | grep nghttp
        libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f8915550000)

So, maybe Remi compiles php with php support included instead of letting it as a separate extension? So even if the extension looks like to support http2, in fact this extension is unused? Looks like no, because the extension is loaded:

bash-4.2# php -m | grep curl
curl

I've also tried to not load the extension and then php -i | grep -i curl returns nothing, so no curl built into php binary by default.

I have to admit, this drives me nuts!

@mnapoli
Copy link
Member Author

mnapoli commented Jan 8, 2023

Some notes for today.

The ARM layers do have support for HTTP2.

So the problem is limited to x86 layers built from remi's repo.

Let's start from scratch (public.ecr.aws/lambda/provided:al2-x86_64) and do everything again:

$ curl --version
curl 7.79.1 (x86_64-koji-linux-gnu) libcurl/7.79.1 OpenSSL/1.0.2k-fips zlib/1.2.7 libidn2/2.3.0 libssh2/1.4.3 nghttp2/1.41.0 OpenLDAP/2.4.44
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL UnixSockets

$ yum install -y \
        https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \
        https://rpms.remirepo.net/enterprise/remi-release-7.rpm \
        yum-utils \
        epel-release \
        curl
...

$ yum-config-manager --enable remi-php81
...

$ yum update -y && yum upgrade -y
...

$ yum install php-cli
...

$ php -v
PHP 8.1.14 (cli) (built: Jan  4 2023 06:45:14) (NTS gcc x86_64)
Copyright (c) The PHP Group
Zend Engine v4.1.14, Copyright (c) Zend Technologie

$ php -i | grep HTTP2
$ php -r "var_dump(get_defined_constants());" | grep CURL_HTTP
  ["CURL_HTTP_VERSION_1_0"]=>
  ["CURL_HTTP_VERSION_1_1"]=>
  ["CURL_HTTP_VERSION_NONE"]=>

No luck.

Note
It seems curl is already installed, we can remove yum install curl from our Dockerfiles, but that's a minor detail.

I have opened remicollet/remirepo#228 to try and get some help from Remi.

@mnapoli mnapoli self-assigned this Jan 16, 2023
@mnapoli mnapoli moved this from Todo to In Progress in Bref 2.0 Jan 16, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in Bref 2.0 Jan 29, 2023
phoenix933 pushed a commit to phoenix933/aws-lambda-layers that referenced this issue Dec 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants