undefined method 'peeraddr' for #<Fluent::PluginHelper::HttpServer::Request

[dentonk]

Description:
I get the error undefined method 'peeraddr' for #<Fluent::PluginHelper::HttpServer::Request... when trying to use the protobuf-http input plugin. If I hardcode a peeraddr then it progresses to a similar message undefined method 'header'.... If I hardcode both the content_type and the peeraddr, thus removing both of those method calls, the plugin seems to work as intended, minus actually performing the checks against the actual request peeraddr and headers.

Reviewing both the documentation for the Fluentd HTTP Server plugin helper, as well as the code, I would agree with the error message, it does not look like those methods are available in the request object.

To Reproduce
Steps to reproduce the behavior:

1. I'm using the fluent/fluentd:v1.16-debian-1 Docker image
2. Added apt-get install -y protobuf-compiler and RUN gem install fluent-plugin-protobuf-http to the DockerFile
3. Using stdout for my output
4. POST to endpoint using Postman with a valid protobuf message

fluentd.conf

<source>
  @type       protobuf_http
  @id         protobuf_http_input
  @log_level  debug

  bind        0.0.0.0
  port        8081
  tag         debug.test

  proto_dir   /fluentd/etc
  in_mode     binary
  out_mode    json
</source>

<match **>
  @type stdout
</match>

[dentonk]

What's most confusing to me is I'm struggling to find an update that would have caused this to stop working. I'm not a Ruby developer so it might be obvious, but it looks like the HTTP Server plugin helper request object hasn't changed since 2019 so I feel like I must be missing something given the update to the plugin in 2021.

[dentonk]

While testing, I also tried adding attr_reader request to the request class so that I could call headers from the Protocol::HTTP::Request class and that worked, i.e. content_type = req.request.headers['content-type']

[iamazeem]

@dentonk
Thank you for reporting this issue!
I never tested this on Debian. I'll try to take a look at this soon.

Is this how your Dockerfile looks like?

FROM fluent/fluentd:v1.16-debian-1
RUN apt-get update && apt-get install -y protobuf-compiler
RUN gem install fluent-plugin-protobuf-http

If there's anything else, please do include that. Thanks!

[dentonk]

@iamazeem Thanks for the quick reply!

I was starting from this guide, specifically the section "How to build your own image > 3.1 For current images > Debian version" so I have a few more steps in my Dockerfile, but I think those are primarily for cleanup:

FROM fluent/fluentd:v1.16-debian-1

USER root

RUN buildDeps="sudo make gcc g++ libc-dev" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && apt-get install -y protobuf-compiler \
 && sudo gem install fluent-plugin-protobuf-http \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent

Then I'm running the container with:
docker run -p 8081:8081 -v ${PWD}:/fluentd/etc fluentd-proto

[dentonk]

Also, I just tested with the Alpine image following the same guide as above and I'm seeing the same error.

FROM fluent/fluentd:v1.16-1

USER root

RUN apk add --no-cache --update --virtual .build-deps \
        sudo build-base ruby-dev \
 && apk add --no-cache make protobuf-dev \
 && sudo gem install fluent-plugin-protobuf-http \
 && sudo gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

USER fluent

[iamazeem]

Tested locally on an Ubuntu 21.04 machine and fluentd 1.16.1 with:

• ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
• ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux] -- (as in fluent/fluentd:v1.16-debian-1)

and, it works fine with both Ruby versions.

This is the Dockerfile I used for this testing (using root and fluent users was causing a permissions denied issue while compiling the .proto file, though didn't look into it much, maybe it was imposed by the base image):

FROM fluent/fluentd:v1.16-debian-1

USER root

RUN \
  apt-get update \
  && apt-get install -y protobuf-compiler \
  && gem install fluent-plugin-protobuf-http \
  && gem sources --clear-all \
  && rm -rf /var/lib/apt/lists/* \
  && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

---

However, testing in the docker container did reproduce the issue that you're facing.

After further inspection of the installed gems in the container I found out that there were both async-http and webrick installed where async-http has the higher precedence for the fluent http server helper.

Uninstalling async-http in Dockerfile made this work:

gem uninstall async-http

IIRC, when I wrote this plugin, it was based on webrick and I never had to check for async-http.
I thought that both should be using the same APIs and are drop-in replacements for each other.
Looks like that's not the case.

Anyways, you may proceed and verify this on your side.
I'll see if I need to do something in code to specify the precedence of webrick.
Or, maybe, need to use the common APIs only. Later.

Here's the Dockerfile that worked for me for Debian:

FROM fluent/fluentd:v1.16-debian-1

USER root

RUN \
  apt-get update \
  && apt-get install -y protobuf-compiler \
  && gem install fluent-plugin-protobuf-http \
  && gem uninstall async-http \
  && gem sources --clear-all \
  && rm -rf /var/lib/apt/lists/* \
  && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

Let me know if this works.
Please do the test with alpine as well if you find some time.
Thanks and regards!

[dentonk]

Thank you so much for digging into this so quickly and your detailed explanation!! That seems to have done it! I missed that one little detail around async-http vs webrick and could not for the life of me figure out how this ever worked with the async-http stack. I knew I must be missing something, I was just looking in the wrong place. 🙂

I tested with both debian and alpine and was able to get it to work by uninstalling async-http as you suggested.

[iamazeem]

Awesome! You're welcome! 😄
Thank you for your verification! 👍

I looked at the implementation of this plugin and indeed it's using the webrick APIs.
That makes it more compatible with webrick and completely incompatible with async-http.
I'll add a new section in the README for known issues and list its hard dependency on webrick and how to resolve its conflict with async-http. I'll close this issue then. Cheers!

[iamazeem]

Updated README with #11.