-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
panic on underflowing subtraction during recording #26
Comments
Note that usually samply is built and used in release mode, to avoid debug slowdown that leads to missed samples. In release mode, integer overflows are silently replaced with wrapping arithmetic, and most of the overflows in samply ends up being harmless. |
The unordered timestamps are probably due to the code around here, which gets records from the perf events of multiple CPUs and doesn't sort them: samply/samply/src/linux/perf_group.rs Line 262 in 3b5f5e1
It would be great if somebody could read up on what the guarantees are about the ordering of records from perf events from multiple CPUs. Is sorting just "the current batch" enough, or do new records from the current batch sometimes need to be ordered before records from the previous batch? I think the latter is the case, and I also needed to deal with it when reading records from a perf.data file. The code for the batch reordering is here: https://github.com/mstange/linux-perf-data/blob/f0c5b59781303eaf1c7e6aa523e711a4018f5c09/src/sorter.rs |
It looks like the reordering is also causing other troubles, for example, fork events can come before the execve event:
Which mess up memory map tracking and cause symbolization to fail. Not sure what to do with reordering though, it does seem to be a legitimately hard and annoying problem. |
I think it comes down to something similar to the "rounds" logic in linux-perf-data. There's no "guarantee" from perf, but for causally related events (e.g. fork → exec) it's possible to reason about conditions under the assumption that events are written synchronously wrt the syscall. When event B happens-after event A, and say we just read event B in one of the buffers, then acquire-release ordering guarantees we will be able to see (or already have seen) event A in one of the buffers as well. Hence, if we read in the order of (CPU 0, Round x), (CPU 1, Round x), ..., (CPU N, Round x), (CPU 0, Round x+1), then we can commit (CPU 0, Round x) as soon as we have examined records up to (CPU N, Round x) for re-sorting. This is a slightly tighter bound compared to the "rounds" formulation. This doesn't however guarantee anything about events that are not causally related to each other. In such case, event numbering can sometimes happen in different order than the samples write finishes, and timestamps have other reasons to drift (TSC synchronization issues can cause drift by a small amount). But those should never lead to logic errors and can be safely ignored. Right now, a blocker for live event reordering is that we only read a limited number of events per samply/samply/src/linux/perf_event.rs Lines 542 to 563 in 4ab060c
To exploit the causal relationship, all available events need to be pushed into the buffer at once. This will likely require some refactor. The bitmask tracking is the tricky part, so if we at least made the reordering part consume events in-order it will likely simplify tracking logic required for this. |
Thank you so much for investigating this. Please feel free to do whatever refactoring is needed, I am not attached to this code at all. I imported it from https://github.com/koute/not-perf/ just to get a basic working version of samply running - this code has already survived for longer than I expected. |
I've made some preliminary changes at #29 and #30. I have a binary heap based sorter at ishitatsuyuki@664e1ae, but I then realized that the sorter from So first, here's the question of which sorter to use. The pros of binary heap sorter is that it's more granular, but the con is that binary heap falls short when it comes to absolute performance (quicksort/pdqsort is extremely well optimized). Then there's the question of where the sorter should reside in. Right now, the sorter in |
Minimizing the number of lost perf events is more important than minimizing the CPU overhead of the sampler. So if the more granular releasing of the binary heap approach lets us advance the read position in |
cargo build
)Command that triggers the issue:
gdb
is from the Arch Linux repositories (version: 12.1-2).One particular panic:
Instrumenting all
ContextSwitchHandler::handle_{sample,context_in,context_out}
calls gives the following:call-log.txt
The text was updated successfully, but these errors were encountered: