Replace largeBuffer []byte with chunked LargeBuffer#1431
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1431 +/- ##
==========================================
+ Coverage 43.74% 43.86% +0.11%
==========================================
Files 312 313 +1
Lines 33911 34213 +302
==========================================
+ Hits 14834 15007 +173
- Misses 18122 18232 +110
- Partials 955 974 +19
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
mariomac
approved these changes
Mar 3, 2026
Contributor
mariomac
left a comment
There was a problem hiding this comment.
Loving the improvement. Thanks!
mmat11
reviewed
Mar 3, 2026
mmat11
reviewed
Mar 3, 2026
mmat11
approved these changes
Mar 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The previous
largeBuffertype was a plainstruct{ buf []byte }. When a TCP payload spans multiple ring-buffer events, eachlargeBufferActionAppendcall didappend(lb.buf, chunk...), which copies the entire accumulated buffer every time it grows past capacity.Replace it with LargeBuffer, a chunked store that keeps each ring-buffer chunk as an independent allocation. Appending is always a single copy into a freshly allocated slice plus one pointer append into the chunk index. Previously written data is never moved. Reading is handled by
LargeBufferReader(cursor-based,io.Reader-compatible) for sequential parsers, and byUnsafeViewAt/UnsafeView(zero-copy within a chunk, scratch-backed across boundaries) for parsers that need all bytes at once.Parsers that only need the full payload are changed to take
*LargeBufferand callUnsafeView()directly, eliminating the reader allocation at each call site. Parsers doing genuine sequential reads (handlePostgres,Kafka,http.ReadRequest/ReadResponse) keep*LargeBufferReader. Thekafkaparsersub-package is migrated from raw[]byteslices to abyteReaderinterface so it can consume aLargeBufferReaderwithout importing ebpfcommon.The new
large_buffer_test.gocovers construction, all cursor operations, random-access methods, scalar helpers, zero-alloc hot paths, and UnsafeView stale-slice semantics.This is the foundation PR. Once merged, individual parsers can be migrated in follow-up PRs to take advantage of chunked reads, for example, avoiding the full-buffer materialisation in Postgres, or streaming Kafka frames directly from a LargeBufferReader without copying into an intermediate
[]bytefirst. For now, I tried to keep the code somewhat consistent.I recommend reviewing
pkg/ebpf/common/large_buffer.goas that contextualises the rest of the PR.Checklist