http-parser -> llhttp

[asvetlov]

The resurrection of #5364

Closes #3561

Co-authored-by: Dmitry Erlikh [email protected]

[asvetlov]

@derlih the initial #5364 had about 8 failed tests when merged to master.
I've reverted the merge and created this PR to don't miss your work.
Not sure when I can find time to work on failed tests; my current target is publishing 3.8 with python 3.10 support, llhttp is nice to have but not mandatory.

If we cannot find time to fit in 3.8 time window -- we can publish 3.9 relatively soon, there is no need to wait another year if we will get a value soon.

P.S.
The branch is available for pushing to every aiohttp committer, feel free to help. Please don't be confused with revert-6143-revert-5364-llhttp branch name (revert of reversion :)

[derlih]

I've updated the llhttp to the latest stable and fixed the spelling error in CI

[derlih]

test_http_request_parser_two_slashes fails for llhttp. Looks line the behavior has been changed in comparison to http_parser.
Do we really need this test? It checks that python parser behaves the same as a native one. But this case tests wrong path with // inside. Also it doesn't cover many other cases with //. For example llhttp for path //path/a returns /a. Should this strange result be also implemented by python http parser and covered by tests?

Update:
This behavior is not covered by llhttp tests.

[derlih]

The next problem is aiohttp.web.normalize_path_middleware.
This middleware has the merge_slashes parameter:

If merge_slashes is True, merge multiple consecutive slashes in the path into one.

But any middleware is called after the HTTP parser. And since the behavior of double slashes in path is changed in llhttp, it makes sense to remove this merge_slashes parameter from this middleware. It's an API change -> should be shifted to the next major release.

Also I think it means that we can't have a transition phase with both native parsers at the same time.

[derlih]

Also I found that llhttp fixes regression #5621 :-)

[asvetlov]

test_http_request_parser_two_slashes fails for llhttp...

We cannot change llhttp upstream (at least it requires long enough conversation I think).
But we can change the behavior of our pure-python parser to get the same result for both Python and llhttp parsing results.
Please update the failed test.

[asvetlov]

The next problem is aiohttp.web.normalize_path_middleware...

If we really cannot support merge_slashes=False let's make the argument no-op and raise DeprecationWarning if False was passed.

[asvetlov]

Also I found that llhttp fixes regression #5621 :-)

Awesome!

[asvetlov]

@derlih I think I've found the reason for broken double-slash tests.

llhttp is correct but URL("//abc") assumes that 'abc' is an authority, not path.
URL.build() should be used instead.
I'm working on it.

[codecov]

Codecov Report

Merging #6144 (e3e52ad) into master (0fef0a3) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #6144      +/-   ##
==========================================
- Coverage   93.31%   93.29%   -0.02%     
==========================================
  Files         102      102              
  Lines       30200    30238      +38     
  Branches     2706     2712       +6     
==========================================
+ Hits        28180    28212      +32     
- Misses       1843     1849       +6     
  Partials      177      177              

Flag	Coverage Δ
unit	93.23% <100.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
tests/test_http_parser.py	99.05% <100.00%> (+0.01%)	⬆️
aiohttp/worker.py	26.98% <0.00%> (-1.35%)	⬇️
aiohttp/client.py	24.88% <0.00%> (ø)
tests/test_client_ws.py	100.00% <0.00%> (ø)
aiohttp/test_utils.py	99.06% <0.00%> (+<0.01%)	⬆️
tests/test_test_utils.py	98.29% <0.00%> (+0.08%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0fef0a3...e3e52ad. Read the comment docs.

[derlih]

@asvetlov the middleware test is checking the security issue inside aiohttp. I don't want to reintroduce it )

[asvetlov]

I didn't touch tests except test_http_parser.py, where only fixture ids are added without changing the actual test code.

[asvetlov]

Tests are green, merging

[patchback]

Backport to 3.8: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 441c10e on top of patchback/backports/3.8/441c10eaf7ac9ded20aed376715b53dbac9cb3f1/pr-6144

Backporting merged PR #6144 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.8/441c10eaf7ac9ded20aed376715b53dbac9cb3f1/pr-6144 upstream/3.8

4. Now, cherry-pick PR http-parser -> llhttp #6144 contents into that branch:

   $ git cherry-pick -x 441c10eaf7ac9ded20aed376715b53dbac9cb3f1

   If it'll yell at you with something like fatal: Commit 441c10eaf7ac9ded20aed376715b53dbac9cb3f1 is a merge but no -m option was given., add -m 1 as follows intead:

   $ git cherry-pick -m1 -x 441c10eaf7ac9ded20aed376715b53dbac9cb3f1

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR http-parser -> llhttp #6144 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.8/441c10eaf7ac9ded20aed376715b53dbac9cb3f1/pr-6144

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

[aio-libs-github-bot]

💔 Backport was not successful

The PR was attempted backported to the following branches:

• ❌ 3.8: Commit could not be cherrypicked due to conflicts