[WIP] tracing: replacing trace_events with perfetto

[jasnell]

This is a work in progress preview of integrating perfetto as a replacement
for the trace events subsystem.

This is no where near complete. I'm opening it early to start getting feedback.

Perfetto (https://perfetto.dev) is a tracing subsystem created by Google and used
in Android and Chrome. It supports both in-process and out-of-process with the
ability to integrate application and system traces.

The traces generated are emitted as encoded protobufs rather than JSON, and the
subsystem is much more stable and robust than the existing unfinished tracing
support we currently have.

Unfortunately, it's also much more complex, with large new dependencies and a
complicated configuration model.

Perfetto will allow richer and more flexible diagnostic tracing use cases,
including the ability to stream tracing data in-process (something we currently
cannot do easily). The traces are more efficient at runtime, take less memory
and disk space, and generally work better.

Currently, this will likely only build on Linux systems (I've only tested on
Ubuntu to this point). It will definitely fail on Windows, and DEFINITELY
will not pass CI in any way shape or form. There's still a ton of work to do
here (including reducing the size of the added dependencies)

Notes

To build with perfetto:

$ ./configure --use-perfetto
$ make -j4

(there's currently a bug in the building step, first time through, it will likely fail, just run make -4 again and it should be ok. It's an open todo to fix it)

Once built, running with the regular command line options (e.g. --trace-event-categories=v8,node) will cause the trace event file to be emitted. Currently, perfetto is being invoked with verbose logging enabled to allow easier debugging, that will be turned off a bit later once I'm sure everything is working right.

The trace event file will be a binary sequence of serialized protobufs rather than a json document. It needs to be loaded into the perfetto UI to be processed. The legacy chrome://tracing will not work

Traces from JavaScript (e.g. trace events for AsyncResource and console.time() are emitted as legacy trace events. I will be converting those over into the new model soon.

The entire current trace_events module will be completely redone. The existing API no longer makes sense and is not supportable. Fortunately, it's still considered experimental but we should still consider it a breaking change. A new API is being implemented that allows for some very interesting options... this is not yet fully implemented but here's an example of what will be possible:

const { TracingSession } = require('trace_events');

const session = new TracingSession({
  categories: 'v8,node.async_hooks',
  duration: 10, * 1000 /* Trace for 10 seconds */
  flushPeriod: 200  /* Flush traces every 200 ms */
});

for await (const chunk of session) {
  // Process chunk of trace event data
}

This will create a new tracing session that pipes traces into a stream rather than out to file. The file trace and the stream trace are independent of each other but can have overlapping information if both are enabled at the same time.

There are a number of other features that are planned as this progresses, such as the ability to limit trace file size, apply compression to the trace output automatically, support tracing via the Inspector protocol, provide trace triggers, provide a user-accessible JavaScript API for emitting traces, and more.

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

[jasnell]

Ping @nodejs/diagnostics ... any thoughts on this? It's still definitely a work in progress and has a ways to go but it would be good to get some input on it before I dedicate much more time to it.

[Flarna]

I have a few questions to get this into the right viewpoint:

• What will be traced?
  Low level details of V8/deps/core?
  High level API (entry points) of core?
  Is it open to any node module?
• Who is the target audience of these traces?
• How does this relate to the recently added diagnostics channel?

I assume the main target audience for this are the current users of trace-events. I have never used trace-events by myself and I have never seen anyone asking here for traces to analyze an issue.
Do we know who/where trace-events are used currently as I guess these people will provide the best input.

[RafaelGSS]

It looks very promising, as previously said would nice some input to check the workflow.

Currently, this will likely only build on Linux systems

If landed, we'll create a specific mirror for it?

[jasnell]

@Flarna

What will be traced?
Low level details of V8/deps/core?
High level API (entry points) of core?

This is a replacement to the existing trace events subsystem in core. See trace_events.md for details on what is currently traced.

Is it open to any node module?

The current trace events mechanism is not exposed to end users except to allow users to enable/disable them. This will be similar at first but I do have plans to expose a user-facing API.

Who is the target audience of these traces?

Diagnostic tooling. For example, our clinic.js tooling uses the trace events generated for async hooks. It would be the same.

How does this relate to the recently added diagnostics channel?

They are separate. Trace events long predate the newly added diagnostic channel. Perfetto is just a replacement for the underlying engine that supports those trace events.

@RafaelGSS:

If landed, we'll create a specific mirror for it?

No, we wouldn't land this until it supports all Node.js supported platforms. This is just an early work in progress.

[cjihrig]

Realistically, what are the chances of perfetto being abandoned by Google? I'd like to avoid a repeat of gyp if possible.

[jasnell]

I have concerns there also @cjihrig but the fact that it's a dependency of android and there are ABI stability guarantees, I'd say it's a fairly safe bet. It is also significantly better and more robust than the partial existing trace events implementation that was written by folks at google but never finished.

[Qard]

Not a huge fan of how much ifdef stuff this piles onto the code everywhere. Hopefully we can stabilize this and delete those relatively quickly. 😬

Also, might be a good idea to keep the old and new tracing files separate. The file structure is not super clear where the division is.

Other than that, seems like you're on track to something usable. :)