Client example hangs

[dignifiedquire]

Hey, I'm trying out the example in examples/client.rs. It's running fine against the google.com address, but when trying it against my own server it will just hang after printing out the headers:

cargo run
     Running `target/debug/ipfs-api`
----- Headers -----
Status: 200 OK
Access-Control-Allow-Headers: X-Stream-Output, X-Chunked-Output
Access-Control-Expose-Headers: X-Stream-Output, X-Chunked-Output
Content-Type: application/json
Server: go-ipfs/0.4.0-dev
Trailer: X-Stream-Error
Transfer-Encoding: chunked
Date: Thu, 10 Mar 2016 09:40:40 GMT
Transfer-Encoding: chunked
# hangs here

The request when I use curl looks like this:

curl -v localhost:5001/api/v0/version
*   Trying ::1...
* connect to ::1 port 5001 failed: Connection refused
*   Trying fe80::1...
* connect to fe80::1 port 5001 failed: Connection refused
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5001 (#0)
> GET /api/v0/version HTTP/1.1
> Host: localhost:5001
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: X-Stream-Output, X-Chunked-Output
< Access-Control-Expose-Headers: X-Stream-Output, X-Chunked-Output
< Content-Type: application/json
< Server: go-ipfs/0.4.0-dev
< Trailer: X-Stream-Error
< Transfer-Encoding: chunked
< Date: Thu, 10 Mar 2016 09:43:01 GMT
< Transfer-Encoding: chunked
<
{"Version":"0.4.0-dev","Commit":"6f8ea13","Repo":"3"}
* Connection #0 to host localhost left intact

[tailhook]

1. You have duplicate Transfer-Encoding, is it intended? (this might cause an issue, as must be interpreted as chunked, chunked by spec, i.e. a chunked encoding encoded by chunked encoding)
2. All that X-Stream-* headers make me suspicious of whether the response is not endless/streaming? (the example client buffers everything)

[dignifiedquire]

1. It's a known issue with the api, that I can't change atm though. But curl and other http clients seem to be able to handle it fine.
2. There are other endpoints that use this. But this version request is not streaming.

[dignifiedquire]

If I use hyper I get this as body:

36\r\n{\"Version\":\"0.4.0-dev\",\"Commit\":\"6f8ea13\",\"Repo\":\"3\"}\n\r\n0\r\n\r\n

it isn't either able to handle the double Transfer-Encoding (hyperium/hyper#683)

[tailhook]

Hm, so it's not a duplicate header, it's real nested chunked encoding? Well, I'm not sure how to handle that in any efficient way.

[pyfisch]

I suspect that the rotor client does not work with any chunked response. This is a chunked example page and it does also hang: https://www.httpwatch.com/httpgallery/chunked/chunkedimage.aspx

[tailhook]

It looks like we need better debugging info :) The buffer limit is 16384, the size of the chunked image is 32k.

[pyfisch]

Increasing the buffer size did not help.

Another page I found also does not work: https://jigsaw.w3.org/HTTP/ChunkedScript

If I switch to progressive mode the first two chunks are received and then the client again hangs.

[tailhook]

Okay, I've fixed chunked encoding. At least in buffered mode. (Will fix progressive mode soon).

As described in ipfs/kubo#2385 it's just wrong header, not wrong double character encoding. So I'm not sure why hyper fails.

rotor-http should probably reject wrong headers anyway, but it would be nice if you try (or is there some public place to send request to?)

[tailhook]

@dignifiedquire I've just checked, Transfer-Encoding: chunked, chunked works. So you can use it. While I'm not sure that it will stay this way, author of go-ipfs stated that he will fix the headers, so you will have no problems in future.

The reason it works, it's because here is stated that checking last transfer encoding is good enough (to withstand some kinds of attacks). But if we are decoding chunked transfer encoding, a user might assume that rotor-http decodes any transfer encoding. @pyfisch, any thoughts?

[dignifiedquire]

Thanks will check in the morning :)
If you want to test yourself you can doenload a version from dist.ipfs.io and run it on your machine.

[pyfisch]

@tailhook: I think it is ok if we just decode chunked encoding. Maybe we should add a note to the docs. Note also that currently rotor-http can't serialize messages with a Transfer-Encoding different from chunked. As far as I know Transfer-Encoding is only used for chunked encoding in practice all recipents support Content-Encoding instead.

[dignifiedquire]

Thanks, so I got a bit further, but no full success yet.

• Issue 1: localhost does not work, it just hangs, no request is made
• Issue 2: Instead trying to use 127.0.0.1 directly, panics, as url handles ip4 addresses differently than regular domains. Will see if I can make it work later

[dignifiedquire]

Okay I was able to implement a version that solves Issue2: https://gist.github.com/dignifiedquire/6aadd79d970198d56171 but it still just hangs and prints nothing. Not sure how to go from here :/

[dignifiedquire]

Okay #36 solves the issue of handling ip4 addresses in the client example :) So I can confirm now that the chunked encoding is properly parsed now and I get the expected output printed to the console when running it.

[tailhook]

Does it hangs, without printing anything? (but previously it printed headers, right?) I think that go-ipfs may use TLS by default?

[dignifiedquire]

This is what I get:

cargo run --example client http://localhost:5001/api/v0/version
   Compiling rotor-http v0.6.0 (file:///Users/dignifiedquire/opensource/rotor-http)
     Running `target/debug/examples/client http://localhost:5001/api/v0/version`

It seems independent of go-ipfs. I tried a simple static file server and the same hanging occurs. (And no go-ipfs does not use TLS by default as far as I know)

[dignifiedquire]

For comparison, using 127.0.0.1

cargo run --example client http://127.0.0.1:5001/api/v0/version
     Running `target/debug/examples/client http://127.0.0.1:5001/api/v0/version`
----- Headers -----
Status: 200 OK
Access-Control-Allow-Headers: X-Stream-Output, X-Chunked-Output
Access-Control-Expose-Headers: X-Stream-Output, X-Chunked-Output
Content-Type: application/json
Server: go-ipfs/0.4.0-dev
Trailer: X-Stream-Error
Transfer-Encoding: chunked
Date: Sun, 13 Mar 2016 20:22:02 GMT
Transfer-Encoding: chunked
----- Response -----
{"Version":"0.4.0-dev","Commit":"286a596","Repo":"3"}

[tailhook]

Hm, probably it's because localhost resolves to ipv6. I'm not sure why does it hang anyway. But that's the only difference I see.

[dignifiedquire]

So the ipv6 pointer was correct. I have in my hosts file

::1             localhost
fe80::1%lo0 localhost

for dealing with ipv6. When commenting those lines out everything works fine.

[tailhook]

Hm, so this is another case where error reporting is missing.

[tailhook]

Fixed progressive chunked encoding and error reporting on connection reset in ce45540