16. August 2022

[dimakuv]

Agenda

(please write your proposed agenda items in comments under this discussion)

• High-level description of Borys's [LibOS,PAL] Bookkeep PAL memory in LibOS #839
• Borys: RFC: Add POSIX shared untrusted memory support #757 @llly started implementing it, but we must first reach consensus on the approach

High-level description of Borys's #839

• There is a simple memory allocator for PAL boostrapping phase. This is used for example for parsing the TOML manifest file.

• The rest PAL memory allocations go to LibOS's bookkeeping memory allocator.

• PalVirtualMemoryAlloc() is now much simpler, it doesn't contain the checks and assumes caller-specified address.

• Main process of Gramine can allocate anywhere; the child process can allocate anywhere except the set of memory ranges received from the parent.

• When LibOS is bootstrapped, the memory that is already allocated by PAL (during its own bootstrapping) is given to LibOS as "already reserved".

• In the Linux PAL, the sending of "reserved memory ranges" from parent process to child process is done via memfd (kinda like shared memory). In the Linux-SGX PAL, these memory ranges are located in the untrusted part of memory and are read by the enclave at PAL startup.

Li Xun's description of #757

• There is a need for shared memory that is accessible to the SGX enclave, to the host kernel and to the hardware device (e.g., GPU TEE).

• Gramine implementation of shared memory starts with "direct" (aka "transparent") mode:

  • User app will know about untrusted shared memory.
  • User app will be responsible to NOT save any secrets into this shared memory.

• Li Xun suggests to use /dev/shmem mmap instead of device-backed mmap. Because the device-driver software stack can do smth like fd = open("/dev/shmem"); addr = mmap(fd); ioctl(DEVICE_USE_SHARED_MEM, addr), which is enough (and we assume that the driver stack can be slightly modified to accommodate this flow).

• Proposal for shared memory: use POSIX shmem API -- shm_open, shm_unlink. This implementation actually uses a tmpfs FS that is normally mounted under /dev/shm on Linux.

  • Important: the user must explicitly ask for untrusted shared memory.
  • Otherwise there is a high risk of a vulnerability when the app accidentally allocates untrusted memory.

• Description of the flow... shm_open() -> ftruncate() -> mmap(MAP_SHARED) -> ioctl(DEVICE_USE_SHARED_MEM, addr).

  • Mona: why not just mmap(MAP_SHARED)? Because this is hacky, unintuitive and error-prone by app developers.
  • Borys: likes this approach, it is clear what happens and hard to accidentally abuse.

• Changes to Gramine:

  • New manifest option sys.insecure__shared_memory = "none|passthrough".
  • New FS shm_builtin_fs (similar to chroot FS; mounts /dev/shm on the host).
  • New range for LibOS memory allocator: "shared memory range" (must be outside of enclave).

• This may conflict with Borys's PR [LibOS,PAL] Bookkeep PAL memory in LibOS #839.

  • Probably the only clash will be in renamed/restructured "PAL-preallocated memory ranges" and also in the removed PalGetAvailableUserAddressRange() API.
  • Borys's PR has higher priority than Li Xun's PR.

• The PR already has the standalone regression test. But we'll need one more PR with a test that has ioctl(DEVICE_USE_SHARED_MEM, addr) -- this will be implemented after we merge Dmitrii's PR on IOCTLs ([LibOS,PAL] Add flexible device-specific IOCTL support #671).

Worries about too many big PRs at once

Michal worries about too many big changes (PRs) that Gramine sees recently. This may lead to slower review process and creeping bugs.

Mona agrees that we need to prioritize the big changes and move through them slowly and profoundly.

[mkow]

too many big changes (PRs)

Small clarification: I was mostly talking about design proposals which we're working on at the same time, not only PRs (e.g. KSS, LA-TLS, untrusted shared memory, ...). These are quite big and user-facing, so fixing them after doing them badly will be quite painful.