pkg/sensors: reduce stack trace map memory footprint #2546
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mtardy
added
the
release-note/bug
✅ Deploy Preview for tetragon ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
mtardy
force-pushed
the
pr/mtardy/stack-trace-memory-reduce
branch
from
c6db970 to
3307f93
Compare
We stopped on a stack trace map that has a max_entries of 32768, which is 64 bits pointers * PERF_MAX_STACK_DEPTH (which is fixed at 127 for now), so 127*64/8=1016 bytes per entry + it's key_size of 32 bits (4 bytes) so 1020 bytes per entry. So 1020 * 32768 = 33,423,360 bytes. From bpftool, this map has a total bytes_memlock of 34,079,040 bytes. So for each stack trace map we load, we had 34MB of kernel memory, and it happened to be loaded many times when we were loading any tracing policy. Since the map is used by the generic program, the loader will allocate the memory needed for the map even if we don't create a reference from the agent side and create an anonymous map. So we end up allocating a small map of max_entries 1 by default and resize it when the tracing policy actually specifies a matchAction containing a kernelStackTrace or userStackTrace to true. This should drastically reduce the memory footprint of this feature when it's unused. Signed-off-by: Mahe Tardy <[email protected]>
mtardy
force-pushed
the
pr/mtardy/stack-trace-memory-reduce
branch
from
3307f93 to
4dbc45d
Compare
olsajiri
approved these changes
Jun 12, 2024
kkourt
approved these changes
Jun 12, 2024
mtardy
added
backport-pending/1.0
backport-done/1.1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We stopped on a stack trace map that has a max_entries of 32768, which is 64 bits pointers * PERF_MAX_STACK_DEPTH (which is fixed at 127 for now), so 127*64/8=1016 bytes per entry + it's key_size of 32 bits (4 bytes) so 1020 bytes per entry. So 1020 * 32768 = 33,423,360 bytes. From bpftool, this map has a total bytes_memlock of 34,079,040 bytes. So for each stack trace map we load, we had 34MB of kernel memory, and it happened to be loaded many times when we were loading any tracing policy.
Since the map is used by the generic program, the loader will allocate the memory needed for the map even if we don't create a reference from the agent side and create an anonymous map. So we end up allocating a small map of max_entries 1 by default and resize it when the tracing policy actually specifies a matchAction containing a kernelStackTrace or userStackTrace to true. This should drastically reduce the memory footprint of this feature when it's unused.