Skip to content

Switch from bpf_get_stackid() to bpf_get_stack()#1264

Merged
fabled merged 2 commits intoopen-telemetry:mainfrom
bobrik:ivan/bpf-get-stack
Mar 23, 2026
Merged

Switch from bpf_get_stackid() to bpf_get_stack()#1264
fabled merged 2 commits intoopen-telemetry:mainfrom
bobrik:ivan/bpf-get-stack

Conversation

@bobrik
Copy link
Copy Markdown
Contributor

@bobrik bobrik commented Mar 18, 2026

To use bpf_get_stack() we need a newer kernel.

Getting rid of bpf_get_stackid() also lets us get rid of the 16MiB map:

$ sudo bpftool map show name kernel_stackmap
31557211: stack_trace  name kernel_stackmap  flags 0x0
        key 4B  value 1016B  max_entries 16384  memlock 17039648B
        pids ebpf-profiler(3973147)

Plus the costs of reading the map for every sample with kernel compoments.

Instead of saving the stack id into the map, we now save the actual kernel frames into the same list of frames where all other frames go.

See: #1257.

@bobrik bobrik requested review from a team as code owners March 18, 2026 00:37
@bobrik bobrik force-pushed the ivan/bpf-get-stack branch 5 times, most recently from f8cfa7e to a55ceb2 Compare March 18, 2026 03:28
Copy link
Copy Markdown
Contributor Author

@bobrik bobrik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw two approaches to this:

  1. Save the kernel stack as an array of u64 addresses and do specialized parsing in userspace. It's kind of similar to how stack id was turned into actual frames.
  2. Produce proper kernel frames and do no specialized parsing in userspace. Kernel is just another kind of an interpreter.

I think the second approach is nicer, so I opted for that. I can be persuaded otherwise.

This doesn't work on v5.4. I'm not sure what the status of dropping support for that.

I have this running in production and it looks right compared to #1257:

Image

Comment thread support/ebpf/tracemgmt.h Outdated
Comment thread support/ebpf/tracemgmt.h Outdated
Comment thread support/ebpf/types.h Outdated
Copy link
Copy Markdown
Contributor

@fabled fabled left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this! Added some comments on the frame data format.

Comment thread support/ebpf/tracemgmt.h Outdated
Comment thread support/ebpf/tracemgmt.h Outdated
When one kernel fails, it cancels all the other tests, making it harder
to see where the fault line lies. Let's run them all to completion instead.
@bobrik bobrik force-pushed the ivan/bpf-get-stack branch 3 times, most recently from abcaca7 to dcf9a97 Compare March 20, 2026 00:06
@bobrik
Copy link
Copy Markdown
Contributor Author

bobrik commented Mar 20, 2026

I updated to push kernel frames as a list of u64 at the beginning of frame_data directly via bpf_get_stack(). This is the approach number 1 from here. It's a bit more work for userspace to separate and stitch them with the rest of the frames, but it is less data copied from the kernel and it's easier on the verifier.

I'll deploy this internally to make sure everything looks right.

Copy link
Copy Markdown
Contributor

@fabled fabled left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much! Looks very neat and clean. Few minor clean up comments still.

Comment thread tracer/tracer.go
Comment thread support/ebpf/tracemgmt.h
Comment thread doc/internals.md Outdated
To use `bpf_get_stack()` we need a newer kernel.

Getting rid of `bpf_get_stackid()` also lets us get rid of the 16MiB map:

```
$ sudo bpftool map show name kernel_stackmap
31557211: stack_trace  name kernel_stackmap  flags 0x0
        key 4B  value 1016B  max_entries 16384  memlock 17039648B
        pids ebpf-profiler(3973147)
```

Plus the costs of reading the map for every sample with kernel compoments.

Instead of saving the stack id into the map, we now save the actual
kernel frames into the same list of frames where all other frames go.
The format is different though: it's just addresses. That way it's
easier on the bpf verifier.
@bobrik bobrik force-pushed the ivan/bpf-get-stack branch from dcf9a97 to 2d22cfa Compare March 20, 2026 17:54
@bobrik
Copy link
Copy Markdown
Contributor Author

bobrik commented Mar 20, 2026

I incorporated all 3 suggestions.

Copy link
Copy Markdown
Contributor

@fabled fabled left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! lgtm! @florianl @christos68k can either of you take a second review?

Copy link
Copy Markdown
Member

@florianl florianl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works fine and thanks for the work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants