Skip to content

[receiver/kafka] Pre-compute header attribute keys to avoid per-message string allocation#46828

Merged
andrzej-stencel merged 4 commits into
open-telemetry:mainfrom
paulojmdias:feat/46816_1st
Mar 11, 2026
Merged

[receiver/kafka] Pre-compute header attribute keys to avoid per-message string allocation#46828
andrzej-stencel merged 4 commits into
open-telemetry:mainfrom
paulojmdias:feat/46816_1st

Conversation

@paulojmdias

Copy link
Copy Markdown
Member

Description

getMessageHeaderResourceAttributes concatenates "kafka.header." + headerName on every message for each configured extraction header. Since the header list comes from the config and never changes, we can build the mapping once at startup and just look up the pre-computed attribute key per message.

I did this before starting the execution of #46816 to avoid conflicts and mixing topics in the same PR.

The results are below, and I feel it deserves a review.

Testing

Added benchmark tests to measure the change.

goos: darwin
goarch: arm64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver
cpu: Apple M3 Pro
                                                  │ header_attrs_main.txt │        header_attrs_fix.txt         │
                                                  │        sec/op         │   sec/op     vs base                │
GetMessageHeaderResourceAttributes/headers=1-12              46.24n ±  1%   55.35n ± 2%  +19.70% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=4-12              169.2n ± 22%   123.3n ± 1%  -27.15% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=8-12              340.6n ± 19%   221.0n ± 4%  -35.11% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=16-12             703.4n ±  1%   510.2n ± 3%  -27.48% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=32-12             1.838µ ±  2%   1.442µ ± 2%  -21.52% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=64-12             5.715µ ±  5%   4.930µ ± 4%  -13.72% (p=0.001 n=10)
GetMessageHeaderResourceAttributes/headers=128-12            15.71µ ± 11%   13.79µ ± 5%  -12.20% (p=0.000 n=10)
geomean                                                      845.6n         691.3n       -18.25%

                                                  │ header_attrs_main.txt │         header_attrs_fix.txt         │
                                                  │         B/op          │     B/op      vs base                │
GetMessageHeaderResourceAttributes/headers=1-12                56.00 ± 0%     32.00 ± 0%  -42.86% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=4-12               152.00 ± 0%     56.00 ± 0%  -63.16% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=8-12               280.00 ± 0%     88.00 ± 0%  -68.57% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=16-12               536.0 ± 0%     152.0 ± 0%  -71.64% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=32-12              1048.0 ± 0%     280.0 ± 0%  -73.28% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=64-12              2072.0 ± 0%     536.0 ± 0%  -74.13% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=128-12            4.242Ki ± 0%   1.224Ki ± 1%  -71.14% (p=0.000 n=10)
geomean                                                        531.9          172.3       -67.61%

                                                  │ header_attrs_main.txt │        header_attrs_fix.txt        │
                                                  │       allocs/op       │ allocs/op   vs base                │
GetMessageHeaderResourceAttributes/headers=1-12                3.000 ± 0%   2.000 ± 0%  -33.33% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=4-12                9.000 ± 0%   5.000 ± 0%  -44.44% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=8-12               17.000 ± 0%   9.000 ± 0%  -47.06% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=16-12               33.00 ± 0%   17.00 ± 0%  -48.48% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=32-12               65.00 ± 0%   33.00 ± 0%  -49.23% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=64-12              129.00 ± 0%   65.00 ± 0%  -49.61% (p=0.000 n=10)
GetMessageHeaderResourceAttributes/headers=128-12              257.0 ± 0%   129.0 ± 0%  -49.81% (p=0.000 n=10)
geomean                                                        31.77        17.08       -46.24%

…er-message string allocation

Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
Signed-off-by: Paulo Dias <paulodias.gm@gmail.com>
@paulojmdias paulojmdias added waiting-for-code-owners Skip Changelog PRs that do not require a CHANGELOG.md entry labels Mar 10, 2026
@paulojmdias paulojmdias changed the title [receiver/kafkareceiver] Pre-compute header attribute keys to avoid per-message string allocation [receiver/kafka] Pre-compute header attribute keys to avoid per-message string allocation Mar 10, 2026

@axw axw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@paulojmdias paulojmdias added ready to merge Code review completed; ready to merge by maintainers and removed waiting-for-code-owners labels Mar 11, 2026
@andrzej-stencel andrzej-stencel merged commit 1ceaf55 into open-telemetry:main Mar 11, 2026
210 of 218 checks passed
@paulojmdias paulojmdias deleted the feat/46816_1st branch March 11, 2026 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready to merge Code review completed; ready to merge by maintainers receiver/kafka Skip Changelog PRs that do not require a CHANGELOG.md entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants