riscv: fault handler violates PMP when RISCV_ALWAYS_SWITCH_THROUGH_ECALL is enable

[jimmyzhe]

Describe the bug

When both CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL and CONFIG_RISCV_PMP are enabled, some test cases fail because PMP error is triggered again during a do_swap() from the _Fault.

Fail test case:

1. tests/kernel/fatal/exception/kernel.common.stack_protection
2. tests/kernel/threads/thread_apis/kernel.threads.apis

To Reproduce

1. add CONFIG_RISCV_ALWAYS_SWITCH_THROUGH_ECALL=y to boards/qemu/riscv32/qemu_riscv32_defconfig
2. $ ./scripts/twister -c -v -p qemu_riscv32 --integration -T tests/kernel/fatal/exception/ -T tests/kernel/threads/thread_apis/

Expected behavior

Executing do_swap() successfully from _Fault.

Impact

When a kernel thread triggers a stack overflow or a user thread violates memory protection, the _Fault handler cannot switch out correctly.

Logs and console output

1. tests/kernel/fatal/exception/kernel.common.stack_protection

...
test stack HW-based overflow - supervisor 1
E: 
E:  mcause: 7, Store/AMO access fault
E:   mtval: 800243fc
E:      a0: 00000000    t0: 80025ff8
E:      a1: 00000000    t1: 80000ca4
E:      a2: 00000000    t2: 00000000
E:      a3: 00000000    t3: 80000aea
E:      a4: 00000002    t4: 00021880
E:      a5: 00000000    t5: 00000000
E:      a6: 00000000    t6: 0000002d
E:      a7: 00000000
E:      sp: 800243e0
E:      ra: 8000093e
E:    mepc: 80000922
E: mstatus: 00021880
E: 
E: call trace:
E: 
E: >>> ZEPHYR FATAL ERROR 2: Stack overflow on CPU 0
E: Current thread: 0x800208d0 (unknown)
Caught system error -- reason 2
E: 
E:  mcause: 5, Load access fault
E:   mtval: 800242a4
E:      a0: 80020a20    t0: 80007686
E:      a1: 800208d0    t1: 00020000
E:      a2: 00000000    t2: 7fffffff
E:      a3: 80020a20    t3: 00000010
E:      a4: 00000000    t4: 00000005
E:      a5: 80020f98    t5: 00000001
E:      a6: 00000000    t6: 00000000
E:      a7: 0000002d
E:      sp: 800242a0
E:      ra: 8000763c
E:    mepc: 800000f8
E: mstatus: 00021800
E: 
E: call trace:
E: 
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x80020a20 (test_fatal)
Caught system error -- reason 0
Was not expecting a crash
===================================================================
RunID: 60a8560517e7b74d5930206c340eba1c
PROJECT EXECUTION FAILED

2. tests/kernel/threads/thread_apis/kernel.threads.apis

...
 PASS - test_threads_suspend_timeout in 0.270 seconds
===================================================================
START - test_user_mode
E: 
E:  mcause: 5, Load access fault
E:   mtval: 80020898
E:      a0: 00000000    t0: 00000091
E:      a1: 80015338    t1: 00000000
E:      a2: 00000000    t2: aaaaaaaa
E:      a3: 00000000    t3: aaaaaaaa
E:      a4: 80021000    t4: aaaaaaaa
E:      a5: 8000167e    t5: aaaaaaaa
E:      a6: 00000010    t6: aaaaaaaa
E:      a7: 200080ff
E:      sp: 8002efb0
E:      ra: 80006f8c
E:    mepc: 80001682
E: mstatus: 00000080
E: 
E: call trace:
E: 
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x80021990 (test_user_mode)
E: 
E:  mcause: 5, Load access fault
E:   mtval: 80034ec4
E:      a0: 80021c90    t0: 8000be6e
E:      a1: 80021990    t1: 00020000
E:      a2: 00000000    t2: 7fffffff
E:      a3: 80021c90    t3: 0000002a
E:      a4: 00000000    t4: 0000002e
E:      a5: 80021e10    t5: 0000007f
E:      a6: 00000068    t6: 00000010
E:      a7: 0000006a
E:      sp: 80034ec0
E:      ra: 8000be24
E:    mepc: 800000f8
E: mstatus: 00021800
E: 
E: call trace:
E: 
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x80021c90 (main)
E: 
E:  mcause: 5, Load access fault
...

Environment (please complete the following information):

• OS: linux
• Toolchain: zephyr-sdk-0.16.8
• Commit SHA: fd0e465 / tag: v3.7.0-rc3

Additional context