add extra checking of header buffer, to support multi line header value

[tedli]

#123

[erikdubbelboer]

Thank you for the pull request.
I'm afraid there is one issue. For multi line headers the \r\n that splits up the value should be removed.
See: https://play.golang.org/p/Kpmruh6fghm which is your test but then using net/http.
Can you please fix this?

[tedli]

@erikdubbelboer
Hi,
I'm glad to work on this. But since we need to modify the original value, it means we have to make a copy of a range of the original buffer, which will introduce some overhead.
If you have some ideas or thought about this, please share with me.

[erikdubbelboer]

Thanks for the changes. I'm afraid some more changes are needed before I can accept this.

[tedli]

@erikdubbelboer
Hi,
Thanks for reply. I'm working on this, but it's little complex, and will take some time.
Your point is that modify the request or response buffer in place, rather than alloc new space for multi line value. Am I got you wrong?
If so, we have to move the rest headers that haven't been parsed yet, and the body forward. If only one or two multi line value headers I think it's OK, but when several such header exists, we may move data many times.
And if just move content forward, it's not that harmless, but what if we have to move backward. I mean the net/http formatted the header value, like one=1;two=2 will be parsed into one=1; two=2, there is a space inserted, in this circumstances we need move backward, and we need extra space, what the buffer may don't have.

[erikdubbelboer]

I think if every time you get a newline within a header value you just skip that byte and copy everything after it back by one byte it should work.

[tedli]

@erikdubbelboer
Hi,
I've made new changes according to your suggestion.

[erikdubbelboer]

@tedli sorry for the slow reply. This is a big change with lots of edge cases and I haven't found the time to really review this yet. I'll try to do it this weekend.

[erikdubbelboer]

Sorry for the delay. Almost there, just some minor changes.

[erikdubbelboer]

@tedli not sure if you noticed but the Travis CI build failed because your code panics on certain inputs: https://play.golang.org/p/e7m-bwnCMzf

[tedli]

@erikdubbelboer
Thanks for notice. I will dig into it.
Only one question: Is this case a valid case, or just need extra checking to make it robust when getting invalid data?

[erikdubbelboer]

This is just some random input generated by https://fuzzit.dev/ but it still shouldn't cause a panic of course. It needs to be robust against invalid data yes.

[erikdubbelboer]

Thanks for fixing the panic. Now there is only one thing left to fix with the test.

[erikdubbelboer]

Thanks!

[erikdubbelboer]

@tedli I have made some more improvements, can you please check them? #708