@copy deep_copy does not work

[dvob]

Describe the bug
I would like to send data to multiple outputs(Loki and Splunk). From the data which I send to Splunk I want remove some metadata data. Despite I use deep_copy true the data I remove from the Splunk record is also removed from the Loki record.

Are there other ways to send records in different formats do different outputs?

To Reproduce

• config fluent.conf

<source>
  @type http
  port 9880
  bind 0.0.0.0
  body_size_limit 32m 
  keepalive_timeout 10s 
</source>

<match **> 
  @type copy
  deep_copy true
  <store>
    @type relabel
    @label @splunk
  </store>
  <store>
    @type relabel
    @label @loki
  </store>
</match>

<label @splunk>
  <filter **> 
    @type record_transformer
    remove_keys $.kubernetes.labels
    <record>
      type splunk
    </record>
  </filter>
  <match **> 
    @type stdout
  </match>
</label>

<label @loki>
  <filter **> 
    @type record_transformer
    <record>
      type loki
    </record>
  </filter>
  <match **> 
    @type stdout
  </match>
</label>

• Run fluentd 1.8

docker run -it --rm -p 9880:9880 -v $(pwd)/fluent.conf:/etc/fluent.conf fluent/fluentd:v1.8.0-1.0 fluentd -c /etc/fluent.conf

• Send log

curl -H 'Content-Type: application/json' -d '{"log":"foobar","kubernetes":{"pod":"bla bla", "labels":{"app":"foo"}}}' http://localhost:9880/mytag

• Output (labels are removed from both records)

... mytag: {"log":"foobar","kubernetes":{"pod":"bla bla"},"type":"splunk"}
... mytag: {"log":"foobar","kubernetes":{"pod":"bla bla"},"type":"loki"}

Expected behavior
Data is only removed from the Splunk record

• Expected output:

... mytag: {"log":"foobar","kubernetes":{"pod":"bla bla"},"type":"splunk"}
... mytag: {"log":"foobar","kubernetes":{"pod":"bla bla","labels":{"app":"foo"}},"type":"loki"}

[repeatedly]

deep_copy parameter uses ruby's dup method internally and it doesn't copy all objects recursively.
Maybe, we need to add copy_mode parameter to support true deep copy via marshal or something.

For your case, changing copy order is one ad-hoc approach.

<match **> 
  @type copy
  deep_copy true
  <store>
    @type relabel
    @label @loki
  </store>
  <store>
    @type relabel
    @label @splunk
  </store>
</match>

[repeatedly]

Patch: #2747

[dvob]

Thank you for your response and the patch.
Changing the order in copy works for me but I still try to understand what exactly copy is doing?

• Are all the stores in a copy run sequentially?
• What happens if the delivery of a record in the first store is bufferd (e.g. due to flush_interval 10s or so) and the second store changes the record? Does then the change in the second store also affect the record in the first store or is the record in the first store already waiting serialized in the buffer?
• Changes in the root of the record (no nested objects) do not affect the record in later stores even if deep_copy is set to false. So what does setting deep_copy to truereally change? Or why does this work?

I think I do not necessarily need the deep_copy. It seems that the things I would like to achieve can also be done with a different order of the stores in the copy.
I guss solving this without deep copying the record is even better since it uses less memory and CPU.