in_http: Support for Content-Type application/x-ndjson

[chrisbertsch]

Is your feature request related to a problem? Please describe.

Moving away from a legacy application in which logs were sent over HTTP as newline delimited JSON objects. With the added support for newline delimited JSON added to the in_http plugin Fluentd would be a drop-in replacement.

I see that the out_http plugin supports application/x-ndjson so this enhancement to the HTTP input would be complementary to the HTTP output plugin's support of newline-delimited JSON objects.

https://docs.fluentd.org/output/http#content_type

Describe the solution you'd like

Implement support for Content-Type application/x-ndjson for newline-delimited JSON objects to be summited via the in_http input plugin.

https://docs.fluentd.org/input/http#how-to-use-http-content-type-header

Describe alternatives you've considered

N/A

Additional context

N/A

[fujimotos]

@chrisbertsch I implemented the "ndjson" support and just released
the new version (v1.14.5) that include that feature.

Plese refer to the release note for details:

https://github.com/fluent/fluentd/releases/tag/v1.14.5

Feel free to ask me if you have any feedback.

[chrisbertsch]

So I just tested this but when I attempt to send multiple lines I get a HTTP 400 response code with undefined method `[]=' for nil:NilClass but if I send a single line it works correctly.

Contents of test.ndjson with line endings shown.

{"message":"Test 1"}\n
{"message":"Test 2"}\n

$ cat -e test.ndjson
{"message":"Test 1"}$
{"message":"Test 2"}$
$ cat test.ndjson | curl -v -d @- -H 'Content-Type: application/x-ndjson' http://127.0.0.1:8080/test
* About to connect() to 127.0.0.1 port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> POST /test HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8080
> Accept: */*
> Content-Type: application/x-ndjson
> Content-Length: 40
>
* upload completely sent off: 40 out of 40 bytes
< HTTP/1.1 400 Bad Request
< Content-Type: text/plain
< Connection: Keep-Alive
< Content-Length: 56
<
400 Bad Request
undefined method `[]=' for nil:NilClass
* Connection #0 to host 127.0.0.1 left intact

[fujimotos]

@chrisbertsch Can you post in_http configuration?

I guess your coniguration somehow turns off the batch processing logic,
and is unable to handle an array of events.

[chrisbertsch]

@fujimotos My source config is as follows.

<source>
  @type http
  @id input_http
  @label @pipeline
  port 8080
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
  add_remote_addr true
</source>

[fujimotos]

@fujimotos My source config is as follows.

@chrisbertsch It turned out that curl -d automatically trims newlines
from the payload. According to the man page:

-d, --data <data>
  ...
  Posting data from a file named 'foobar' would thus be done with -d, --data @foobar.
  When --data is told to read from a file like that, carriage returns and newlines
  will be stripped out.

For this reason, invalid data (like {"message":"Test 1"}{"message":"Test 2"}) was
transferred to Fluentd. This was the reason why you encountered the error.

To post ndjson payloads correctly, you need to use --data-binary as follows:

$ cat test.ndjson | curl -v --data-binary @- -H 'Content-Type: application/x-ndjson' http://127.0.0.1:8080/test

I can confirm it works with Fluentd v1.14.5.

[chrisbertsch]

@fujimotos I can confirm this works now. Sorry I completely missed that during my testing.

Do you know wow long it takes for a new Fluentd version to show up in the latest td-agent package?

[fujimotos]

Ask @ashie. He is the downstream's manager!