PerfCollect does not collect all trace

[mallikarjunraju]

Hi,

I am trying to collect perfcollect trace on my .net core 3.1 application.

My application is running in dockerswarm. I did docker exec --privileged -u 0 -it {containter_id} /bin/bash and ran below commands.

export COMPlus_PerfMapEnabled=1
export COMPlus_EnableEventLog=1
export COMPlus_ZapDisable=1

apt-get update
curl -OL http://aka.ms/perfcollect
chmod +x perfcollect
./perfcollect install
./perfcollect collect sampleTrace

After copying the trace from the container, I used perfview to see the results. Below is the screenshot.

perfdata.txt: (same content)

........................
docker     0/0     [000] 162199.074442:    1000000 cpu-clock: 
	ffffffffb5fcccf2 native_safe_halt ([kernel.kallsyms])
	ffffffffb5fcc9b0 __cpuidle_text_start ([kernel.kallsyms])
	ffffffffb56396c5 arch_cpu_idle ([kernel.kallsyms])
	ffffffffb5fcce83 default_idle_call ([kernel.kallsyms])
	ffffffffb56d6c52 do_idle ([kernel.kallsyms])
	ffffffffb56d6eb3 cpu_startup_entry ([kernel.kallsyms])
	ffffffffb5fbf3ae rest_init ([kernel.kallsyms])
	ffffffffb6ca607c start_kernel ([kernel.kallsyms])
	ffffffffb6ca5478 x86_64_start_reservations ([kernel.kallsyms])
	ffffffffb6ca54ee x86_64_start_kernel ([kernel.kallsyms])
	ffffffffb56000d5 secondary_startup_64 ([kernel.kallsyms])

docker     0/0     [000] 162199.075551:    1000000 cpu-clock: 
	ffffffffb5fcccf2 native_safe_halt ([kernel.kallsyms])
	ffffffffb5fcc9b0 __cpuidle_text_start ([kernel.kallsyms])
	ffffffffb56396c5 arch_cpu_idle ([kernel.kallsyms])
	ffffffffb5fcce83 default_idle_call ([kernel.kallsyms])
	ffffffffb56d6c52 do_idle ([kernel.kallsyms])
	ffffffffb56d6eb3 cpu_startup_entry ([kernel.kallsyms])
	ffffffffb5fbf3ae rest_init ([kernel.kallsyms])
	ffffffffb6ca607c start_kernel ([kernel.kallsyms])
	ffffffffb6ca5478 x86_64_start_reservations ([kernel.kallsyms])
	ffffffffb6ca54ee x86_64_start_kernel ([kernel.kallsyms])
	ffffffffb56000d5 secondary_startup_64 ([kernel.kallsyms])
.......................

Why am I seeing docker and docker-init Threads but not my application classes and their hierarchy?

I even installed dot-net trace and collected trace logs.

dotnet-trace collect -p 1 --format Speedscope -o speedscope.trace
My trace view using speedscope.

I know I am doing some rookie mistake.. I am completely new to this, I have spent so much to resolve this but was unsuccessful.

Any input is appreciated. Thanks.

[brianrob]

You are very close. The key is that these commands must be run in the same shell that starts the app, so that the app gets these environment variables set in its environment:

export COMPlus_PerfMapEnabled=1
export COMPlus_EnableEventLog=1
export COMPlus_ZapDisable=1

These environment variables control runtime behavior and not perfcollect behavior.

As far as dotnet-trace what you are showing looks right to me. One of the big differences for dotnet-trace is that it only shows managed frames, and so you shouldn't expect to see any native or runtime stack frames there.

[mallikarjunraju]

Thank you for the reply.

I have set those env variables at the start of the app. But still do not see my complete code trace.
One good thing is, I can see dotnet process :

When I end my perfcollect action (ctrl + c), I see crossgen warning. Does this need to be fixed for a complete trace?

[mallikarjunraju]

I was able to solve it by following below workaround found at dotnet/corefx-tools#80
:)

curl -OL https://www.nuget.org/api/v2/package/runtime.linux-x64.Microsoft.NETCore.Runtime.CoreCLR/2.0.8
unzip -j 2.0.8 tools/crossgen
cp crossgen /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.21

Now I am seeing issues while opening the perfdata.txt in prefview tool. It says [Error: exception while opening CTF (lttng) data.]
I am able to see the code trace when I open perfdata.txt file in notepad.

It looks like a tool issue.

Any idea on this?

[mallikarjunraju]

My bad.
I unzipped the contents and tried to use the perfview tool.

I am able to see the trace now in perfview tool when I load the .zip file!!!

I really appreciate your help.

[brianrob]

Yup, you got it. Glad that it's working!