QEMUv8: C++ exception test fails when TA is built with -pg (ftrace)

[jforissier]

When OP-TEE is built with function tracing enabled and the TA is instrumented (-pg compiler flag), the C/C++ mixed frame exception test fails with an abort() triggered by the C++ runtime.

$ make CFG_FTRACE_SUPPORT=y CFLAGS_ta_arm64="-pg -O0"

$ xtest 1031
...
* regression_1031 Test C++ features
o regression_1031.1 Global object constructor (main program)
  regression_1031.1 OK
o regression_1031.2 Global object constructor (shared library)
  regression_1031.2 OK
o regression_1031.3 Global object constructor (dlopen()ed lib)
  regression_1031.3 OK
o regression_1031.4 Exceptions (simple)
  regression_1031.4 OK
o regression_1031.5 Exceptions (mixed C/C++ frames)
regression_1000.c:2302: TEEC_InvokeCommand(&session, 29, ((void *)0), &ret_orig) has an unexpected value: 0xffff3024 = TEE_ERROR_TARGET_DEAD, expected 0x0 = TEEC_SUCCESS
  regression_1031.5 FAILED
  regression_1031 FAILED

[Secure console]
terminate called after throwing an instance of 'MixedFrameException'
Abort!
E/TC:? 0 
E/TC:? 0 TA panicked with code 0x0
E/LD:  Status of TA 5b9e0e40-2636-11e1-ad9e-0002a5d5c51b
E/LD:   arch: aarch64
E/LD:  region  0: va 0x40004000 pa 0x0e300000 size 0x002000 flags rw-s (ldelf)
E/LD:  region  1: va 0x40006000 pa 0x0e302000 size 0x008000 flags r-xs (ldelf)
E/LD:  region  2: va 0x4000e000 pa 0x0e30a000 size 0x001000 flags rw-s (ldelf)
E/LD:  region  3: va 0x4000f000 pa 0x0e30b000 size 0x004000 flags rw-s (ldelf)
E/LD:  region  4: va 0x40013000 pa 0x0e30f000 size 0x001000 flags r--s
E/LD:  region  5: va 0x40014000 pa 0x0e421000 size 0x003000 flags rw-s (stack)
E/LD:  region  6: va 0x40022000 pa 0x00000000 size 0x002000 flags r-xs [1] .hash .gnu.hash .dynsym .dynstr .rela.dyn .rela.plt .plt .text .rodata .eh_frame
E/LD:  region  7: va 0x40024000 pa 0x00001000 size 0x002000 flags rw-s [1] .tdata .tbss .init_array .dynamic .got .got.plt .bss
E/LD:  region  8: va 0x40070000 pa 0x00000000 size 0x002000 flags r-xs [2] .hash .gnu.hash .dynsym .dynstr .rela.dyn .rela.plt .plt .text .rodata .eh_frame
E/LD:  region  9: va 0x40072000 pa 0x00001000 size 0x002000 flags rw-s [2] .init_array .dynamic .got .got.plt .bss
E/LD:  region 10: va 0x4008b000 pa 0x00001000 size 0x02a000 flags r-xs [0] .ta_head .text .plt .eh_frame_hdr .eh_frame .gcc_except_table .rodata .gnu.hash .dynsym .dynstr .hash .rela.dyn
E/LD:  region 11: va 0x400b5000 pa 0x0002b000 size 0x0e7000 flags rw-s [0] .dynamic .tdata .tbss .got .rela.got .rela.plt .data .init_array .bss
E/LD:   [0] 5b9e0e40-2636-11e1-ad9e-0002a5d5c51b @ 0x4008b000 (out-br/build/optee_test_ext-1.0/ta/os_test/out/5b9e0e40-2636-11e1-ad9e-0002a5d5c51b.elf)
E/LD:   [1] ffd2bded-ab7d-4988-95ee-e4962fff7154 @ 0x40022000 (out-br/build/optee_test_ext-1.0/ta/os_test_lib/out/libos_test.so)
E/LD:   [2] b3091a65-9751-4784-abf7-0298a7cc35ba @ 0x40070000 (out-br/build/optee_test_ext-1.0/ta/os_test_lib_dl/out/libos_test_dl.so)
E/LD:  Call stack:
E/LD:   0x00000000400a1e20 abort at optee_os/lib/libutee/abort.c:14
E/LD:   0x000000004008cdd8 _ZN9__gnu_cxx27__verbose_terminate_handlerEv at /tmp/dgboter/bbs/rhev-vm8--rhe6x86_64/buildbot/rhe6x86_64--aarch64-linux-gnu/build/src/gcc/libstdc++-v3/libsupc++/vterminate.cc:90
E/LD:   0x000000004008b290 _ZN10__cxxabiv111__terminateEPFvvE at /tmp/dgboter/bbs/rhev-vm8--rhe6x86_64/buildbot/rhe6x86_64--aarch64-linux-gnu/build/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:44
E/LD:   0x000000004008b2dc _ZSt9terminatev at /tmp/dgboter/bbs/rhev-vm8--rhe6x86_64/buildbot/rhe6x86_64--aarch64-linux-gnu/build/src/gcc/libstdc++-v3/libsupc++/eh_terminate.cc:57
E/LD:   0x000000004008c6e8 __cxa_throw at /tmp/dgboter/bbs/rhev-vm8--rhe6x86_64/buildbot/rhe6x86_64--aarch64-linux-gnu/build/src/gcc/libstdc++-v3/libsupc++/eh_throw.cc:94
E/LD:   0x000000004009dd68 throw_mfe at :?
E/LD:   0x000000004009dd78 _ZN23MixedFrameExceptionTest4testEv at :?
E/LD:   0x000000004009ddb8 ta_entry_cxx_exc_mixed at :?
E/LD:   0x00000000400a1de0 entry_invoke_command at optee_os/lib/libutee/arch/arm/user_ta_entry.c:357
E/LD:   0x000000004009de24 __ta_entry at optee_os/out/arm/export-ta_arm64/src/user_ta_header.c:48

[jforissier]

After some investigation and reading the ftrace code again, the error is not surprising. The C++ runtime needs to unwind the stack to propagate exceptions. When doing so, it assumes a "standard" stack layout. When ftrace is enabled however, the stack is modified which is why the unwinder can't find an exception handler and aborts the program.

The linux kernel ftrace uses a similar mechanism to intercept function returns, see this patch: [1] https://lore.kernel.org/patchwork/patch/709315/ which is part of a series that makes the unwinder ftrace-aware. Our unwinder in ldelf seems to be aware of ftrace, too (ftrace_map_lr()).

Unfortunately, modifying the C++ unwinder to make it ftrace-aware is impossible in our case. It seems our only option is to document that function tracing and C++ exception handling are incompatible, and skip the test 😢

[jforissier]

@b49020 FYI and valuable comments 😉

[b49020]

Thanks @jforissier for the notification. I will try to look into this issue.