Extract reporter data generation, and use pdata #245

dmathieu · 2024-11-19T11:28:16Z

This is a proposal to help reduce the code repetition in #208
It extracts the data generation into its own internal module, and switches from OTLP to pdata (which is an OTLP abstraction), so both reporters can rely on the same data generator.

dmathieu · 2024-11-19T13:10:15Z

This is blocked on open-telemetry/opentelemetry-collector#11706, since pdata currently uses a map for the attribute table.

dmathieu · 2024-11-29T09:59:15Z

@open-telemetry/ebpf-profiler-approvers this change would greatly reduce the size of the OTLP and Collector reporters, as both would use pdata rather than OTLP and pdata.

florianl

Just some first spots. At the moment it is hard to review, as there is no backend that accepts this data. I will try to provide a devfiler version supporting v1.4.0 to continue here.

reporter/internal/pdata/generate.go

reporter/otlp_reporter.go

reporter/internal/pdata/generate.go

reporter/otlp_reporter.go

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>

florianl · 2024-11-29T16:02:13Z

Here is a version of devfiler with OTel Profiling v1.4.0 support:

curl -L -H 'Authorization: ca14712d70ea6f6e' -o 'devfiler-v0.9.1-beta-v1.4.0.tar.gz' https://upload.elastic.co/d/b55a069766c48b8c0d4c0f2b854b0e846ee4deaea75596b68a8e1d88172dd0ca

or

https://upload.elastic.co/d/b55a069766c48b8c0d4c0f2b854b0e846ee4deaea75596b68a8e1d88172dd0ca
Authentication token: ca14712d70ea6f6e

dmathieu · 2024-12-04T08:08:02Z

@open-telemetry/ebpf-profiler-approvers this is ready for review.

florianl

Just a first impression. Will have a deeper look again.

reporter/internal/pdata/generate.go

reporter/internal/pdata/pdata.go

florianl

Just a minor last comment. Approving as merging this allows to unblock other work. Minor things can still be changed in subsequent steps.

reporter/internal/pdata/generate.go

Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co>

rockdaboot

LGTM

* Bump golangci-lint to 1.63.4 (open-telemetry#311) Author: Tim Rühsen <tim.ruhsen@elastic.co> * CI: reduce boiler plate of environment (open-telemetry#312) * Consistent format of the eBPF code (open-telemetry#310) * Off CPU profiling (open-telemetry#196) * CI: Check for differences in the eBPF binary blobs (open-telemetry#305) * README: drop make target docker-image as pre requirement (open-telemetry#304) Signed-off-by: Florian Lehner <florian.lehner@elastic.co> * Setup collector receiver (open-telemetry#160) Author: Damien Mathieu <42@dmathieu.com> Co-authored-by: Tim Rühsen <tim.ruhsen@elastic.co> * Send process name as process.executable.name (/proc/PID/comm) (open-telemetry#298) Author: Christos Kalkanis <christos.kalkanis@elastic.co> Co-authored-by: Tim Rühsen <tim.ruhsen@elastic.co> * Remove host metadata collector (open-telemetry#299) Author: Damien Mathieu <42@dmathieu.com> * apmint: return error if Attach() fails (open-telemetry#271) * docker: build and push docker builder image (open-telemetry#297) Signed-off-by: Florian Lehner <florian.lehner@elastic.co> * Update golang.org/x/net to v0.33.0 (Fix CVE-2024-45338) (open-telemetry#294) Author: Tim Rühsen <tim.ruhsen@elastic.co> * metrics: update package documentation (open-telemetry#295) Signed-off-by: Florian Lehner <florian.lehner@elastic.co> * Fix pdata function table order (open-telemetry#286) Author: Tolya Korniltsev <korniltsev.anatoly@gmail.com> * Fix cross-compilation (open-telemetry#282) (open-telemetry#290) Author: Christos Kalkanis <christos.kalkanis@elastic.co> * Use clang-17 for building tracers (open-telemetry#270) Author: Mathieu Bressolle-Chataigner <mathieu@konvu.com> Co-authored-by: Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> * Make the CollAgentAddr config optional in controller (open-telemetry#279) Author: Damien Mathieu <42@dmathieu.com> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> Co-authored-by: Florian Lehner <florian.lehner@elastic.co> * Remove unused type libpf.TraceAndCounts (open-telemetry#283) Author: Tim Rühsen <tim.ruhsen@elastic.co> * Collector Reporter (open-telemetry#208) Author: Damien Mathieu <42@dmathieu.com> Co-authored-by: Tim Rühsen <tim.ruehsen@gmx.de> Co-authored-by: Florian Lehner <florian.lehner@elastic.co> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> * proc: handle nvidia kernel modules (open-telemetry#274) * README: provide new version of devfiler (open-telemetry#275) Author: Florian Lehner <florian.lehner@elastic.co> * Extract reporter data generation, and use pdata (open-telemetry#245) Author: Damien Mathieu <42@dmathieu.com> Co-authored-by: Florian Lehner <florian.lehner@elastic.co> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> * apmint: warn user about changes in the APM service (open-telemetry#269) Author: Florian Lehner <florian.lehner@elastic.co> Co-authored-by: Joel Höner <joel@elastic.co> * coredump: no need for full dump with bundled files (open-telemetry#213) Author: Timo Teräs <timo.teras@iki.fi> * Send executable path with every stack trace (open-telemetry#253) * Send thread name as ThreadNameKey not ProcessCommandKey (open-telemetry#265) Author: Christos Kalkanis <christos.kalkanis@elastic.co> * proc: Fix parsing Kernel Module lines where refcount is - (open-telemetry#263) Author: Frederic Branczyk <fbranczyk@gmail.com> * reporter: extend lifetime for cached elements (open-telemetry#260) Author: Florian Lehner <florian.lehner@elastic.co> * Reporter: allow extending samples with extra attributes (open-telemetry#237) Author: Joel Höner <joel@elastic.co> Co-authored-by: Tim Rühsen <tim.ruehsen@gmx.de>

#245 refactored code and introduced a configuration option to an internal element. Promote package samples from an internal package to public again, so that ExtraSampleAttrProd can be configured again. Fixes #280 Signed-off-by: Florian Lehner <florian.lehner@elastic.co>

* Bump golangci-lint to 1.63.4 (open-telemetry#311) Author: Tim Rühsen <tim.ruhsen@elastic.co> * CI: reduce boiler plate of environment (open-telemetry#312) * Consistent format of the eBPF code (open-telemetry#310) * Off CPU profiling (open-telemetry#196) * CI: Check for differences in the eBPF binary blobs (open-telemetry#305) * README: drop make target docker-image as pre requirement (open-telemetry#304) Signed-off-by: Florian Lehner <florian.lehner@elastic.co> * Setup collector receiver (open-telemetry#160) Author: Damien Mathieu <42@dmathieu.com> Co-authored-by: Tim Rühsen <tim.ruhsen@elastic.co> * Send process name as process.executable.name (/proc/PID/comm) (open-telemetry#298) Author: Christos Kalkanis <christos.kalkanis@elastic.co> Co-authored-by: Tim Rühsen <tim.ruhsen@elastic.co> * Remove host metadata collector (open-telemetry#299) Author: Damien Mathieu <42@dmathieu.com> * apmint: return error if Attach() fails (open-telemetry#271) * docker: build and push docker builder image (open-telemetry#297) Signed-off-by: Florian Lehner <florian.lehner@elastic.co> * Update golang.org/x/net to v0.33.0 (Fix CVE-2024-45338) (open-telemetry#294) Author: Tim Rühsen <tim.ruhsen@elastic.co> * metrics: update package documentation (open-telemetry#295) Signed-off-by: Florian Lehner <florian.lehner@elastic.co> * Fix pdata function table order (open-telemetry#286) Author: Tolya Korniltsev <korniltsev.anatoly@gmail.com> * Fix cross-compilation (open-telemetry#282) (open-telemetry#290) Author: Christos Kalkanis <christos.kalkanis@elastic.co> * Use clang-17 for building tracers (open-telemetry#270) Author: Mathieu Bressolle-Chataigner <mathieu@konvu.com> Co-authored-by: Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> * Make the CollAgentAddr config optional in controller (open-telemetry#279) Author: Damien Mathieu <42@dmathieu.com> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> Co-authored-by: Florian Lehner <florian.lehner@elastic.co> * Remove unused type libpf.TraceAndCounts (open-telemetry#283) Author: Tim Rühsen <tim.ruhsen@elastic.co> * Collector Reporter (open-telemetry#208) Author: Damien Mathieu <42@dmathieu.com> Co-authored-by: Tim Rühsen <tim.ruehsen@gmx.de> Co-authored-by: Florian Lehner <florian.lehner@elastic.co> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> * proc: handle nvidia kernel modules (open-telemetry#274) * README: provide new version of devfiler (open-telemetry#275) Author: Florian Lehner <florian.lehner@elastic.co> * Extract reporter data generation, and use pdata (open-telemetry#245) Author: Damien Mathieu <42@dmathieu.com> Co-authored-by: Florian Lehner <florian.lehner@elastic.co> Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co> * apmint: warn user about changes in the APM service (open-telemetry#269) Author: Florian Lehner <florian.lehner@elastic.co> Co-authored-by: Joel Höner <joel@elastic.co> * coredump: no need for full dump with bundled files (open-telemetry#213) Author: Timo Teräs <timo.teras@iki.fi> * Send executable path with every stack trace (open-telemetry#253) * Send thread name as ThreadNameKey not ProcessCommandKey (open-telemetry#265) Author: Christos Kalkanis <christos.kalkanis@elastic.co> * proc: Fix parsing Kernel Module lines where refcount is - (open-telemetry#263) Author: Frederic Branczyk <fbranczyk@gmail.com> * reporter: extend lifetime for cached elements (open-telemetry#260) Author: Florian Lehner <florian.lehner@elastic.co> * Reporter: allow extending samples with extra attributes (open-telemetry#237) Author: Joel Höner <joel@elastic.co> Co-authored-by: Tim Rühsen <tim.ruehsen@gmx.de>

dmathieu added 2 commits November 19, 2024 12:26

extract reporter data generation, and use pdata

Verified

This commit was signed with the committer’s verified signature.

KyleFromNVIDIA Kyle Edwards

SSH Key Fingerprint: HCOKmVHDGRyoBbxy3oHe2oM/H/VI74oAiMzA3TKFtmM
Verified
Learn about vigilant mode

848dd3a

run make legal

3b36dab

dmathieu force-pushed the otlp-reporter-pdata branch from 3d87afb to 3b36dab Compare November 27, 2024 14:14

dmathieu added 4 commits November 27, 2024 16:20

Merge branch 'main' into otlp-reporter-pdata

d48c17e

run make legal

6352757

rely on pdata from the collector main branch

73de398

update pdata generation to the latest pdata

d087971

dmathieu marked this pull request as ready for review November 29, 2024 09:58

dmathieu requested review from a team as code owners November 29, 2024 09:58

florianl reviewed Nov 29, 2024

View reviewed changes

reporter/internal/pdata/generate.go Outdated Show resolved Hide resolved

reporter/otlp_reporter.go Outdated Show resolved Hide resolved

reporter/internal/pdata/generate.go Outdated Show resolved Hide resolved

reporter/otlp_reporter.go Outdated Show resolved Hide resolved

dmathieu and others added 6 commits November 29, 2024 11:33

Update reporter/internal/pdata/generate.go

555ec51

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>

Update reporter/otlp_reporter.go

fb35ec4

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>

Update reporter/internal/pdata/generate.go

7c4ae61

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>

Update reporter/otlp_reporter.go

8210f2c

Co-authored-by: Florian Lehner <florianl@users.noreply.github.com>

properly apply changes from open-telemetry#256

13e046c

add otlp reporter test

12515e5

dmathieu force-pushed the otlp-reporter-pdata branch from d936928 to d4cf5d9 Compare November 29, 2024 13:34

fix lint

76589ca

dmathieu force-pushed the otlp-reporter-pdata branch from d4cf5d9 to 76589ca Compare November 29, 2024 13:37

dmathieu added 2 commits December 2, 2024 09:30

Merge branch 'main' into otlp-reporter-pdata

420566c

add tests to the helper methods

fd73c65

dmathieu force-pushed the otlp-reporter-pdata branch from 13500da to 76e623a Compare December 2, 2024 09:35

bring back attribute to the dummy mapping entry

c2737da

dmathieu force-pushed the otlp-reporter-pdata branch from 76e623a to c2737da Compare December 2, 2024 09:35

This comment was marked as resolved.

Sign in to view

Merge branch 'main' into otlp-reporter-pdata

5247e9b

florianl reviewed Dec 5, 2024

View reviewed changes

reporter/internal/pdata/generate.go Show resolved Hide resolved

reporter/internal/pdata/generate.go Show resolved Hide resolved

reporter/internal/pdata/pdata.go Outdated Show resolved Hide resolved

dmathieu mentioned this pull request Dec 5, 2024

sample.LocationsLength is mis calculated #262

Closed

use long variable name in pdata.New

a32a28a

florianl approved these changes Dec 6, 2024

View reviewed changes

reporter/internal/pdata/generate.go Show resolved Hide resolved

dmathieu mentioned this pull request Dec 6, 2024

Handle new default_sample_type_strindex and period_type fields #266

Open

Merge branch 'main' into otlp-reporter-pdata

9fde270

christos68k approved these changes Dec 6, 2024

View reviewed changes

reporter/internal/pdata/generate.go Outdated Show resolved Hide resolved

Update reporter/internal/pdata/generate.go

1fe30b4

Co-authored-by: Christos Kalkanis <christos.kalkanis@elastic.co>

dmathieu force-pushed the otlp-reporter-pdata branch 6 times, most recently from 6d451a4 to bd69069 Compare December 6, 2024 17:34

Merge branch 'main' into otlp-reporter-pdata

e1ef19b

dmathieu force-pushed the otlp-reporter-pdata branch from bd69069 to e1ef19b Compare December 6, 2024 17:36

rockdaboot approved these changes Dec 9, 2024

View reviewed changes

Merge branch 'main' into otlp-reporter-pdata

3fa924b

rockdaboot merged commit 42bf4f5 into open-telemetry:main Dec 10, 2024
23 checks passed

dmathieu deleted the otlp-reporter-pdata branch December 10, 2024 09:57

florianl mentioned this pull request Dec 10, 2024

Off CPU profiling #196

Merged

dmathieu mentioned this pull request Dec 10, 2024

Collector Reporter #208

Merged

athre0z mentioned this pull request Dec 13, 2024

ExtraSampleAttrProd reporter config can no longer be used externally #280

Closed

florianl mentioned this pull request Jan 20, 2025

reporter: move pkg samples from internal to public #314

Merged

gnurizen mentioned this pull request Feb 25, 2025

Merge upstream into parca fork parca-dev/opentelemetry-ebpf-profiler#44

Closed

gnurizen mentioned this pull request Mar 6, 2025

Merge from upstream parca-dev/opentelemetry-ebpf-profiler#50

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extract reporter data generation, and use pdata #245

Extract reporter data generation, and use pdata #245

dmathieu commented Nov 19, 2024

dmathieu commented Nov 19, 2024

dmathieu commented Nov 29, 2024

florianl left a comment

florianl commented Nov 29, 2024

dmathieu commented Dec 4, 2024

This comment was marked as resolved.

florianl left a comment

florianl left a comment

rockdaboot left a comment

Extract reporter data generation, and use pdata #245

Extract reporter data generation, and use pdata #245

Conversation

dmathieu commented Nov 19, 2024

dmathieu commented Nov 19, 2024

dmathieu commented Nov 29, 2024

florianl left a comment

Choose a reason for hiding this comment

florianl commented Nov 29, 2024

dmathieu commented Dec 4, 2024

This comment was marked as resolved.

florianl left a comment

Choose a reason for hiding this comment

florianl left a comment

Choose a reason for hiding this comment

rockdaboot left a comment

Choose a reason for hiding this comment