Cannot invoke pthread_jit_write_protect_np

[krauthaufen]

Description

In an effort to make our rendering-engine run on Apple M1 machines we need to create a minimal ARM64 JIT running in our dotnet program (we have the same thing for x64 architectures).

I read a little about protection of executable code on M1 machines and have a running C++ example which uses pthread_jit_write_protect_np(false) to allow writing to the memory (obtained via mmap).

While porting that to our dotnet project I ran into an issue with calling pthread_jit_write_protect_np:

Whenever I call it the whole program exits instantly without any further information (maybe a signal-handler or something installed by dotnet itself?)

Reproduction Steps

Just try to call pthread_jit_write_protect_np from a dotnet program.

Here's my F# code:

module JIT =

    [<DllImport("libc")>]
    extern void* mmap(nativeint addr, unativeint len, int prot, int flags, int fd, nativeint offset);

    [<DllImport("pthread")>]
    extern void pthread_jit_write_protect_np(int enabled)
    

let mem = JIT.mmap(0n, 32un <<< 10, 0x7, 0x1000 ||| 0x0002 ||| 0x0800, -1, 0n)
printfn "got memory: %A" mem

JIT.pthread_jit_write_protect_np(0)
printfn "writable"
JIT.pthread_jit_write_protect_np(1)
    

which will never reach the "writeable" print

Expected behavior

The call should work

Actual behavior

Whole Program just exits with code 0

Regression?

Can't really tell since dotnet 6 is the first one to run natively on M1 right?

Known Workarounds

None

Configuration

• dotnet 6.0.101
• MacOS 12.1
• MacBook Air M1

Other information

No response

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[krauthaufen]

Due to nesting problems it would be ideal if I could call PAL_JitWriteProtect(bool) from here although I don't intend to call back into managed code from my assembled code.

is there any way to achieve this? (I'm not asking for a "clean" way to do this) but not being able to generate code would significantly affect our project.

EDIT: after some fiddling with lldb I found that the function fails with:

Process 52926 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x28140f82c)
    frame #0: 0x000000028140f82c
->  0x28140f82c: ldr    x0, [x29, #0x60]
    0x28140f830: mov    w1, #0x1
    0x28140f834: strb   w1, [x0, #0xc]
    0x28140f838: mov    x0, #0xcb60

[krauthaufen]

Update (which confuses me tbh.)

I tried allocating memory via mmap that initially has protection PROT_READ | PROT_WRITE and write some instructions to it. After that I changed the protection (using mprotect) to PROT_EXEC, wrap it in a delegate and invoke it.
To my astonishment this works when running it via dotnet run or Rider's "run" button but does not work (just exiting without any info) when starting the program from Visual Studio Code (via launch.json)

Also I can change the protection back to rw, write new code to it, make it executable again and invoke it.

So it seems to me that I don't need to call pthread_jit_write_protect_np at all (everything still crashes when trying to do call it)

I would be glad for some help and/or hints on how to make this work properly

[janvorli]

It seems that the crash happens because calling pthread_jit_write_protect_np(0) effectively changes protection of all memory of generated code (that was mapped with MAP_JIT flag set) for the current thread from readable-executable to readable-writeable. So you are basically pulling a rug under your feet. When the call from pthread_jit_write_protect_np returns, it cannot continue executing the calling managed code, as it is no longer executable.

[janvorli]

To my astonishment this works when running it via dotnet run

I believe the reason is that dotnet run uses a special (apphost based) host for executing the application (which I've learned recently here: #63952 (comment)). That host is only adhoc signed without any entitlements and so the system doesn't guard it against the executable mapping flipping.

[krauthaufen]

Hey, thanks for the insights, so effectively this means i can't write to executable memory in dotnet using this mechanism (using the pthread-function) since the dotnet-generated code would then also become non-executable.

However I validated that the mprotect-pattern (described above, switching between RW and EXEC) works also in a plain c project (without any special settings).

My current suspicion is that pages can be marked executable without the pthread-function but they cannot be writeable at the same time. Can anyone shed some light on this?

[janvorli]

However I validated that the mprotect-pattern (described above, switching between RW and EXEC) works also in a plain c project (without any special settings).

Yes, that's like the case of the apphost I've talked about - the case when the application is not signed with enlistments that would request the necessity of using the MAP_JIT flag and pthread_jit_write_protect_np.

i can't write to executable memory in dotnet using this mechanism (using the pthread-function)

You could create a native function that would do the

pthread_jit_write_protect_np(0)
// do your writes here
pthread_jit_write_protect_np(1)

And pinvoke it from the managed code. The writing would need to be purely native, of course.

[krauthaufen]

The whole pthread_jit_write_protect_np API seems pretty ad-hoc to me since it basically prohibits creating a non-native JIT.

I just figured I could create a native pthread worker that just handles copies to executable code and all the other threads would remain unaffected, so we could assemble our code to some non-executable region and then copy it using a special thread, but sadly that might be inefficient due to some fragmentation in our code

[krauthaufen]

Thanks, your idea is actually better than mine using the thread, but nonetheless the whole point of our assembler is to batch together thousands of native calls and some minimal logic in one managed-native transition. (While the real assembler remained completely managed)

So i could essentially create a native function execcpy(dst, src, size) that just copies some data to executable memory (using your idea)

This leaves me with several options (all of which are suboptimal):

1. I encode instructions in managed code and "flush" them to executable memory directly after encoding them
2. I encode everything to a "normal" memory and copy everything once before actually using it (the problem here is that we have relatively large programs that link internal fragments via jmp instructions) so i potentially would copy lots of unused data
3. I find some more appropriate granularity for copying that sadly will change our API quite a bit

[agocke]

The protection bit here that matters for Mac is that you have the JIT entitlement. dotnet itself doesn't have that entitlement because it doesn't need it. If you want to add that entitlement to your app, you need to publish with an apphost and add that entitlement during signing/notarization (which you'll have to do to distribute your app).

[krauthaufen]

Hey, I'm aware that i need to sign things when distributing it, however dotnet itself (to my knowledge) needs this entitlement itself, so it shouldn't change much there...
After some thought I can work around all the problems with @janvorli s suggestion 👍

[janvorli]

@krauthaufen can this issue be closed?

[AaronRobinsonMSFT]

After some thought I can work around all the problems with @janvorli s suggestion 👍

Closing issue based on the above.