Skip to content

Conversation

@cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Jul 10, 2023

Currently, in_udp plugin does not offer adding source address feature.
This PR provides this feature.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[INPUT]
    Name        udp
    Listen      0.0.0.0
    Port        5170
    Chunk_Size  32
    Buffer_Size 64
    Format      json
    Source_Address_Key source_host

[OUTPUT]
    Name        stdout
    Match       *
  • Debug log output from testing the change
Fluent Bit v2.1.7
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/10 18:31:32] [ info] Configuration:
[2023/07/10 18:31:32] [ info]  flush time     | 1.000000 seconds
[2023/07/10 18:31:32] [ info]  grace          | 5 seconds
[2023/07/10 18:31:32] [ info]  daemon         | 0
[2023/07/10 18:31:32] [ info] ___________
[2023/07/10 18:31:32] [ info]  inputs:
[2023/07/10 18:31:32] [ info]      udp
[2023/07/10 18:31:32] [ info] ___________
[2023/07/10 18:31:32] [ info]  filters:
[2023/07/10 18:31:32] [ info] ___________
[2023/07/10 18:31:32] [ info]  outputs:
[2023/07/10 18:31:32] [ info]      stdout.0
[2023/07/10 18:31:32] [ info] ___________
[2023/07/10 18:31:32] [ info]  collectors:
[2023/07/10 18:31:32] [ info] [fluent bit] version=2.1.7, commit=48ffe6485a, pid=468270
[2023/07/10 18:31:32] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/07/10 18:31:32] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/10 18:31:32] [ info] [cmetrics] version=0.6.3
[2023/07/10 18:31:32] [ info] [ctraces ] version=0.3.1
[2023/07/10 18:31:32] [ info] [output:stdout:stdout.0] worker #0 started
[2023/07/10 18:31:32] [ info] [input:udp:udp.0] initializing
[2023/07/10 18:31:32] [ info] [input:udp:udp.0] storage_strategy='memory' (memory only)
[2023/07/10 18:31:32] [debug] [udp:udp.0] created event channels: read=21 write=22
[2023/07/10 18:31:32] [debug] [downstream] listening on 0.0.0.0:5170
[2023/07/10 18:31:32] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2023/07/10 18:31:32] [ info] [sp] stream processor started
[2023/07/10 18:31:35] [debug] [input chunk] update output instances with new chunk size diff=86, records=1, input=udp.0
[0] udp.0: [[1688981495.611518811, {}], {"key 1"=>123456789, "key 2"=>"abcdefg", "source_host"=>"udp://127.0.0.1:44745"}]
[2023/07/10 18:31:36] [debug] [task] created task=0x60fd020 id=0 OK
[2023/07/10 18:31:36] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/07/10 18:31:36] [debug] [out flush] cb_destroy coro_id=0
[2023/07/10 18:31:36] [debug] [task] destroy task=0x60fd020 (task_id=0)
[2023/07/10 18:31:38] [debug] [input chunk] update output instances with new chunk size diff=95, records=1, input=udp.0
[0] udp.0: [[1688981498.719706212, {}], {"msg"=>[{"key 1"=>123456789, "key 2"=>"abcdefg"}], "source_host"=>"udp://127.0.0.1:43725"}]
[2023/07/10 18:31:39] [debug] [task] created task=0x6164e50 id=0 OK
[2023/07/10 18:31:39] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/07/10 18:31:39] [debug] [out flush] cb_destroy coro_id=1
[2023/07/10 18:31:39] [debug] [task] destroy task=0x6164e50 (task_id=0)
^C[2023/07/10 18:31:40] [engine] caught signal (SIGINT)
[2023/07/10 18:31:40] [ warn] [engine] service will shutdown in max 5 seconds
[2023/07/10 18:31:41] [ info] [engine] service has stopped (0 pending tasks)
[2023/07/10 18:31:41] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/10 18:31:41] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found
==468270== 
==468270== HEAP SUMMARY:
==468270==     in use at exit: 95,170 bytes in 702 blocks
==468270==   total heap usage: 2,927 allocs, 2,225 frees, 1,346,894 bytes allocated
==468270== 
==468270== LEAK SUMMARY:
==468270==    definitely lost: 0 bytes in 0 blocks
==468270==    indirectly lost: 0 bytes in 0 blocks
==468270==      possibly lost: 0 bytes in 0 blocks
==468270==    still reachable: 95,170 bytes in 702 blocks
==468270==         suppressed: 0 bytes in 0 blocks
==468270== Reachable blocks (those to which a pointer was found) are not shown.
==468270== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==468270== 
==468270== For lists of detected and suppressed errors, rerun with: -s
==468270== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Initially added documentation which corresponds to fluent/fluent-bit-docs#1165

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 09:34 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 09:34 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 09:34 — with GitHub Actions Inactive
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 09:57 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 force-pushed the cosmo0920-append-source-address-in_udp branch from 7d6c6d1 to 3503b25 Compare July 10, 2023 09:57
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 09:58 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 09:58 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 09:58 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 10, 2023 10:19 — with GitHub Actions Inactive
@edsiper edsiper added this to the Fluent Bit v2.1.8 milestone Jul 18, 2023
@cosmo0920 cosmo0920 marked this pull request as ready for review July 19, 2023 02:12
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:37 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:37 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:37 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:50 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:50 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:50 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 16:23 — with GitHub Actions Inactive
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920 cosmo0920 temporarily deployed to pr July 20, 2023 06:56 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 20, 2023 06:56 — with GitHub Actions Inactive
@edsiper
Copy link
Member

edsiper commented Jul 24, 2023

@leonardo-albertovich pls review

@edsiper edsiper merged commit e886624 into master Aug 31, 2023
@edsiper edsiper deleted the cosmo0920-append-source-address-in_udp branch August 31, 2023 19:02
@edsiper
Copy link
Member

edsiper commented Aug 31, 2023

cc: @lecaros docs

@jvandenbroek
Copy link

This is great, but for some reason it only works with format json and not format none. Is there a way to also insert the source ip with format none set?

@leonardo-albertovich
Copy link
Contributor

You should either open a new improvement request issue or even better a pull request to add the missing part of the feature.

If you decide to give the PR and need some guidance feel free to contact me on slack. All of the required code is already in the plugin so it should be a matter of adding less than 10 lines total and it should be an easy entry point if you're interesting in contributing to the project in the future.

@jvandenbroek
Copy link

You should either open a new improvement request issue or even better a pull request to add the missing part of the feature.

If you decide to give the PR and need some guidance feel free to contact me on slack. All of the required code is already in the plugin so it should be a matter of adding less than 10 lines total and it should be an easy entry point if you're interesting in contributing to the project in the future.

You should either open a new improvement request issue or even better a pull request to add the missing part of the feature.

If you decide to give the PR and need some guidance feel free to contact me on slack. All of the required code is already in the plugin so it should be a matter of adding less than 10 lines total and it should be an easy entry point if you're interesting in contributing to the project in the future.

@leonardo-albertovich Thanks for your reply. Indeed it wasn't too difficult, although it's been a while since I've been looking at C code :-) Maybe you could review it? #11160

@leonardo-albertovich
Copy link
Contributor

Dang, as the MK2 announcer would say "Outstanding!"
I'll review your PR in a minute, thanks for taking the initiative!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants