Keepalive not working?

[jameslittle]

Testing with apachebench and curl; verbose output from curl, which you can see tries to reuse the connection:

curl -v http://192.168.1.190:8081 http://192.168.1.190:8081
* About to connect() to 192.168.1.190 port 8081 (#0)
*   Trying 192.168.1.190... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: 192.168.1.190:8081
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-type: text/plain
< Content-Length: 5
< 
* Connection #0 to host 192.168.1.190 left intact
* Connection #0 seems to be dead!
* Closing connection #0
* About to connect() to 192.168.1.190 port 8081 (#0)
*   Trying 192.168.1.190... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-unknown-linux-gnu) libcurl/7.22.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.18
> Host: 192.168.1.190:8081
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-type: text/plain
< Content-Length: 5
< 
* Connection #0 to host 192.168.1.190 left intact
* Closing connection #0
Pong!Pong

Note the 'connection seems to be dead' line, and that the content of two responses seems to come all at once at the end ("Pong!Pong!"), rather than displayed inline with the request as Curl would normally do.
apachebench with -k flag will just hang after making a connection (presumably after the first request is served).

Here's my test app:

#!/usr/bin/python 

import bjoern

def application(environ, start_response):
    status = '200 OK'
    output = 'Pong!'

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)
    return [output]

bjoern.run(application, '192.168.1.190', 8081)

Testing with the latest code base.

Worth noting that Curl does not pipeline requests, it's waiting for the response before sending the second request.

[jonashaag]

Thanks for the report! Can you please compile bjoern with debug output (make debug) and run your tests again?

[jameslittle]

Hi Jonas, I was actually doing that as you replied :) Anyway, after further investigation, this seems to be an issue of standards implementation (and I am certainly no expert). But this is what I've observed (with the help of debug mode and tcpdumps):

Curl (by default) sends an HTTP 1.1 request, but does not set a 'Connection: Keep-Alive' header, so bjoern closes the connection (confirmed by debug messages). Curl is designed to reuse connections, so it obviously expects them to be left open (when using HTTP1.1) without sending a the keep-alive header. I think keep-alive is supposed to be default in 1.1, so I'm not sure if this request header should be mandatory? Used pycurl to manually set this header and connections are reused successfully.

ApacheBench sends HTTP 1.0 requests with the keep-alive header (required by 1.0), but expects a "Connection: keep-alive" response header, and hangs on the connection until it gets one (confirmed bjoern keeps connection alive until it times out). So the apachebench scenario is solved by explicitly setting the response header, which you can obviously do in the wsgi app code, e.g. my app changes to:

import bjoern

def application(environ, start_response):
    status = '200 OK'
    output = 'Pong!'

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output))), 
                        ('Connection', 'keep-alive')]
    start_response(status, response_headers)
    return [output]

bjoern.listen(application, '192.168.1.190', 8081)
bjoern.run()

I guess the question is, does the implementation in Bjoern need to change to meet the relevant standard? (RFC 2616?)

[jameslittle]

Also, just confirmed the same issue exists when using httperf as the client (with http 1.1 and keep-alives). httperf does not consider setting the response header and so hits a connection reset error when bjoern closes. Command to test:

httperf --hog --timeout=5 --client=0/1 --server=192.168.1.190 --port=8081 --uri=/ --rate=50000 --send-buffer=4096 --recv-buffer=16384 --num-conns=1 --num-calls=3 --http-version="1.1"

^attempts 3 requests on the same socket.

[jonashaag]

Wow, thanks for the detailed analysis!

I think that most HTTP/1.1 implementations indeed default to Keep-Alive and if you don't want to use it you have to return Connection: close which - IIRC - bjoern currently doesn't do. Could you test the curl thing with manually settings Connection: close in the WSGI app?

(Also I agree that HTTP connection should be completely abstracted away from WSGI apps, i.e. no fiddling with the Connection header should be necessary. The server should take care of that.)

[jameslittle]

If I set "connection: close" then Curl works as expected, not attempting to reuse the socket. I noticed from earlier packet captures that bjoern was not erroneously setting 'Connection: close'.

The main issue seems to be that HTTP 1.1 connections aren't treated as persistent by default, even though the code in http-parser.c http_should_keep_alive suggests it will be, and the parser flags bitmask seems to be tested ok in bjoern/wsgi.c, so I can't spot the issue. Tests from http-parser/test.c seem to return ok too...