[Enhancement] HTTP output with mutual TLS auth

[speedchair]

Problem

Referring to: fluent/fluentd#1879
I'd like to ask for similar functionality in out_http.

Steps to replicate

None (feature request).

Expected Behavior or What you need to ask

<match *>
  @type http
  ...
  tls_client_cert_path /etc/ssl/endpoint1.cert # default: ''
  tls_client_private_key_path /etc/ssl/endpoint1.pem # default: ''
  tls_client_private_key_passphrase topsecret # default: '', secret: true
</match>

Using Fluentd and out_http plugin versions

• OS version

  3.8.2
  NAME="Alpine Linux"
  ID=alpine
  VERSION_ID=3.8.2
  PRETTY_NAME="Alpine Linux v3.8"
  HOME_URL="http://alpinelinux.org"
  BUG_REPORT_URL="http://bugs.alpinelinux.org"
  

• Fluentd v0.12 or v0.14/v1.0
  fluentd 1.4.0
• out_http plugin 1.x.y or 0.x.y
  1.1.7

[cosmo0920]

@mkazlouski Could you take a look? #44

[speedchair]

@cosmo0920 Looking great to me. I've tried it using the following docker image:

FROM golang:1.11-alpine3.9 as builder
RUN \
    apk add --no-cache --update \
        git \
        openssl \
    && git clone https://github.com/joekir/mTLS-Tests \
    && cd mTLS-Tests \
    && GOPATH=$PWD go build -o /bin/server server.go \
    && mkdir /certs \
    && cd /certs \
    && openssl genrsa -out server.key 4096 \
    && openssl req -new -x509 -sha256 -key server.key -out server.crt -days 365 -subj "/C=NL/ST=Zuid Holland/L=Rotterdam/O=Sparkling Network/OU=IT Department/CN=127.0.0.1" \
    && openssl genrsa -out client.key 4096 \
    && openssl req -new -key client.key -out client.csr -subj "/C=NL/ST=Zuid Holland/L=Rotterdam/O=Sparkling Network/OU=Remote Location/CN=127.0.0.1" \
    && echo "00" > file.srl \
    && openssl x509 -req -in client.csr -CA server.crt -CAkey server.key -CAserial file.srl -out client.crt

FROM fluent/fluentd:v1.4.2-1.0
USER root
RUN \
    apk add --no-cache --update --virtual .build-deps \
        build-base \
        ruby-dev \
        git \
    && git clone https://github.com/fluent-plugins-nursery/fluent-plugin-out-http \
    && cd fluent-plugin-out-http \
    && git fetch origin pull/44/head:44 \
    && git checkout 44 \
    && fluent-gem build fluent-plugin-out-http.gemspec \
    && fluent-gem install --no-ri --no-rdoc \
        fluent-plugin-out-http-*.gem \
    && rm -rf fluent-plugin-out-http \
    && gem sources --clear-all \
    && apk del .build-deps \
    && rm -rf /home/fluent/.gem/ruby/2.5.0/cache/*.gem
COPY --from=builder /bin/server /bin/
COPY --from=builder /certs/* /certs/
COPY fluentd-mtls.conf /fluentd/etc/fluentd.conf
CMD [ \
    "/bin/sh", "-c", \
        "fluentd -c /fluentd/etc/fluentd.conf -vvv & server" \
]

with the following config called fluentd-mtls.conf:

<source>
    @type dummy
    tag to.success
    dummy {"foo": "bar"}
    auto_increment_key foo_sequence_number
</source>

<source>
    @type dummy
    tag to.failure
    dummy {"baz": "pew"}
    auto_increment_key foo_sequence_number
</source>

<match to.success>
    @type http
    endpoint_url https://127.0.0.1/
    cacert_file /certs/server.crt
    client_cert_path /certs/client.crt
    private_key_path /certs/client.key
</match>

<match to.failure>
    @type http
    endpoint_url https://127.0.0.1/
    cacert_file /certs/server.crt
</match>

<match **>
    @type stdout
</match>

and it works like a charm. What I'm doing there is I'm building the fluentd image with the set of certs: a self-signed server one and a client one signed with the server one as a CA, and a test HTTP server which only accepts connections from clients with valid client certificate. I'm then emitting dummy events with tags to.success and to.failure. The first ones are sent using the client cert specified via new parameters. The second ones use just the CA. As a result when running the container I should be seeing errors only from the to.failure match, which I do.

Thanks a lot for your time!
Mikhail