interpreter/go: replace Rust with native Go symbolization

[florianl]

This is a follow up PR show casing the use of #455.

A side effect of this change is the reduced complexity of compilation, less cgo calls, reduction of binary size.

Fixes #457
Fixes #512

Benchmark results of current main vs this branch:

$ benchstat old.txt new.txt
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/ebpf-profiler/interpreter/go
cpu: 12th Gen Intel(R) Core(TM) i7-12700H
          │     old.txt     │               new.txt               │
          │     sec/op      │   sec/op     vs base                │
Golang-20   6375752.5n ± 6%   969.3n ± 2%  -99.98% (p=0.000 n=10)

          │  old.txt   │              new.txt               │
          │    B/op    │    B/op     vs base                │
Golang-20   848.5 ± 1%   632.0 ± 0%  -25.52% (p=0.000 n=10)

          │  old.txt   │              new.txt               │
          │ allocs/op  │ allocs/op   vs base                │
Golang-20   24.00 ± 0%   17.00 ± 0%  -29.17% (p=0.000 n=10)

I think it is important to state, that the Rust based symbolization is not the isse but the overhead introduced by cgo calling the Rust implementation in Go.

[tsint]

This copy is unnecessary because the SearchGoPclntab function has already loaded the data into a slice via p.Data(maxBytesGoPclntab). In fact, there's no need to save this data at all - we only need to keep the symTable returned by gosym.NewTable(nil, pcln).

[florianl]

These two lines of code were the result of tests. And in the case, where the executable is no longer available, but we still try to symbolize frames for it, it will run into a panic, if this copy does not exist.

[tsint]

As incredible as it sounds, isn't pclnData a slice with memory already allocated?

[fabled]

As incredible as it sounds, isn't pclnData a slice with memory already allocated?

It was mmapped byte slice. Which gets unmapped when the ELF is Closed. This can be fixed by adding a reference counting mechanism, or having this interpreter plugin steal the pfelf.File from pfelf.Reference to keep it open.

The other bigger problem is that debug/gosym on opening allocates huge amount of memory linear to the number of symbols/functions, and further may allocate more memory on each symbolization account.

I will likely extend our existing elfgopclntab.go code to expose a similar Go symbolization API as debug/gosym which this interpreter plugin can then use. I believe we can implement this in a way that there will be no dynamic memory allocations.

[tsint]

Got it, thanks for explaining!

[athre0z]

I believe we can implement this in a way that there will be no dynamic memory allocations.

Yes, it's definitely possible -- the Rust impl also doesn't alloc or copy anything, except for the symbol / file name strings when going through the C API / FFI. I documented the symbol / inline table format here; may be useful if you want to go down the route of teaching elfgopclntab to support that.

[fabled]

I documented the symbol / inline table format here; may be useful if you want to go down the route of teaching elfgopclntab to support that.

Yes, I saw that. Appreciated!

Though, currently it seems the Rust point resolver frontend for Golang does not support reporting inlining information. So just doing the basic non-inlined version to start with should be on par with the feature set. Right?

[athre0z]

Yes, that is correct. The (Rust) gosym library fully supports providing all the necessary data, though -- I think it wouldn't be more than 10-30 lines to actually also use that data in the Rust point resolver. With point resolving it's much simpler to use the data than when generating symbfiles for it, since the latter involves a lot of complicated range tree fumbling.

The inline structure stuff is a bit more painful to extract because it needs access to moduledata (runtime.firstmoduledata), which isn't exported, so the lib locates it with a pattern search.

So yeah, long story short: it would currently not be a reduction in the feature set to not supported it, but in the Rust impl it would be quite easy to add since all the annoying plumbing is already done.

[fabled]

I do have a strong preference to do this in Go if possible. We've already seen the overhead of bringing the Rust wold in (executable size larger), having two runtimes with two memory subsystems, bundling also the Rust .a in Git and making sure it runs on glibc and musl OSes, managing the new Rust dependencies (lot of PR cruft), the memory management issues, the CGO overhead of calling Rust, and the allocation overhead of just moving strings from Rust to Go all add up to a bundle of overhead.

I think I can implement this in Go in a small enough time to warrant doing this. One time development overhead would also mean saved hours in the long term to not needing to update the Rust dependencies. Even if its automated, its still manual review and fixing if something breaks...

[athre0z]

Yeah, fair. Especially having the static libs in git and all the linker foo is indeed rather atrocious for maintenance.

[fabled]

Looks mostly good to me. Some observations listed - mostly related to code I wrote :)

[fabled]

LGTM. Thanks!