Skip to content

Pass non-zero mapID into UpdateStackDeltaPages to fix cleanups#1274

Merged
fabled merged 1 commit intoopen-telemetry:mainfrom
bobrik:ivan/mapref
Mar 23, 2026
Merged

Pass non-zero mapID into UpdateStackDeltaPages to fix cleanups#1274
fabled merged 1 commit intoopen-telemetry:mainfrom
bobrik:ivan/mapref

Conversation

@bobrik
Copy link
Copy Markdown
Contributor

@bobrik bobrik commented Mar 21, 2026

Previously it picked up zero from return values, which resulted in no deletes happening in practice, resulting in duplicate warnings and likely failure to profile for affected mappings.

Previously it picked up zero from return values, which resulted
in no deletes happening in practice, resulting in duplicate warnings
and likely failure to profile for affected mappings.
@bobrik
Copy link
Copy Markdown
Contributor Author

bobrik commented Mar 21, 2026

Logging before the bug was fixed:

$ sudo journalctl -u ebpf-profiler --since '2026-03-21 00:22:38' | grep 'c10a998c845402'
Mar 21 00:27:02 44m298 ebpf-profiler[1610714]: report executable: fileID = c10a998c8454026e44c511a78c0e5011, name = llvm-gsymutil, build id = 3b11b277b1178b0b8f7533469e3d71bb24e39492
Mar 21 00:27:02 44m298 ebpf-profiler[1610714]: time=2026-03-21T00:27:02.721Z level=INFO msg="async update: fileID = c10a998c8454026ec10a998c8454026e, outer = HashOfMaps(exe_id_to_9_stack_deltas)#42, inner = Array(exe_id_to_9_stack_deltas_inner)#984"
Mar 21 00:27:02 44m298 ebpf-profiler[1610714]: time=2026-03-21T00:27:02.721Z level=ERROR msg="loadDeltas: UpdateStackDeltaPages failed for fileID c10a998c8454026ec10a998c8454026e mapID 9, attempting rollback with ref.MapID=0 (bug: should be mapID=9): batch update: key too big for map: argument list too long"
Mar 21 00:28:33 44m298 ebpf-profiler[1610714]: time=2026-03-21T00:28:33.458Z level=ERROR msg="loadDeltas: UpdateStackDeltaPages failed for fileID c10a998c8454026ec10a998c8454026e mapID 9, attempting rollback with ref.MapID=0 (bug: should be mapID=9): batch update: key too big for map: argument list too long"
Mar 21 00:28:33 44m298 ebpf-profiler[1610714]: time=2026-03-21T00:28:33.458Z level=INFO msg="async update: fileID = c10a998c8454026ec10a998c8454026e, outer = HashOfMaps(exe_id_to_9_stack_deltas)#42, inner = Array(exe_id_to_9_stack_deltas_inner)#997"
Mar 21 00:28:33 44m298 ebpf-profiler[1610714]: time=2026-03-21T00:28:33.458Z level=ERROR msg="Outer map update failure: update: key already exists | fileID = c10a998c8454026ec10a998c8454026e, outer = HashOfMaps(exe_id_to_9_stack_deltas)#42, inner = Array(exe_id_to_9_stack_deltas_inner)#997"

I'm not exactly sure what's causing E2BIG here and whether that's also an issue.

[pid 1611513] bpf(BPF_MAP_UPDATE_BATCH, {batch={keys=0x27e65848a120, values=0x27e66090cfa0, count=2, map_fd=31, elem_flags=BPF_ANY, flags=0x1}} <unfinished ...>
[pid 1610983] <... tgkill resumed>)     = 0
[pid 1611513] <... bpf resumed> => {batch={count=0}}, 56) = -1 E2BIG (Argument list too long)

@bobrik
Copy link
Copy Markdown
Contributor Author

bobrik commented Mar 21, 2026

I think this was surfaced by #1144, but I'm not exactly sure on that point. It's good to fix it regardless.

Copy link
Copy Markdown
Contributor

@fabled fabled left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@fabled fabled merged commit 6ff0091 into open-telemetry:main Mar 23, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants