[RFC] PAL memory bookkeeping

[boryspoplawski]

Description

For description of the issue and current workaround please check: #50

Solution

The idea is to use an "upcall" mechanism (similar to what we have for exceptions handling). We would have a function pointer in PAL, that would be used to request a memory range (e.g. request_memory(size_t size)), this pointer would be set by LibOS. This would work differently, depending on the runtime phase:

1. Initial PAL bootstrapping - LibOS is not yet loaded. PAL allocates memory as it wishes and just adds everything to g_pal_public_state->preloaded_ranges.
2. PAL calls LibOS entrypoint - from this point PAL uses "upcall" mechanism. The pointer to the function to call is now NULL, so no additional memory can be allocated by PAL. We would have some memory overallocated in step 1, but this phase is really short, so we won't need much (if any).
3. LibOS VMA subsystem initialization sets the request_memory function pointer. From now on if PAL requires new memory, it first requests a range where it should be allocated (using request_memory upcall).

Note that the above mechanism is rather simple - we already have all building blocks (i.e. VMA bookkeeping should be ready for such changes and has all primitives already implemented).
Also note that freeing memory is not described here. The idea is that PAL would rarely call it (as the mechanism above is only for allocating whole pages, PAL has a dedicated memory allocator implemented on top of it anyway) and creating similar interface for freeing would be cumbersome (it would need to happen in two steps, for details please check bkeep_munmap). If it turns out we need it after all, we can implement it then (there are no fundamental blockers here).

With the above changes, we could remove VMA tracking from PAL completely.

Potential issues

1. Exceptions/interrupts - this memory request upcall mechanism is thread safe, but not reentrant. This means that no PAL exception handler can allocate dynamic memory. Fortunately this is already the case, since malloc has some internal locking. I've verified we mostly adhere to this, with two exceptions: possible malloc in snprintf (called by log_*) - this would require using precision in format string ("%.3d", notice the .) which we never do - and describing fault location - this would require the fault to happen inside malloc itself. There might be more places we malloc in exception handlers, but I couldn't find any.
   This also requires that LibOS VMA bookkeping does not call any PAL function, but it's already the case.
2. No flexibility on allocated memory. I assumed that PAL would allocate only RW "heap" memory via this mechanism (all other memory is already allocated upfront in preloaded_ranges). Actually this is not a problem, if needed we can augment the upcall with requested memory permissions.
3. Differences in virtual addresses layout between child and parent process after fork. Due to the dynamic nature of this mechanism, there might be differences between addresses allocated by PAL depending on whether it's the first (initial) process. The only problem that could happen here is that PAL gets initially some memory that would later need to be allocated for user when it's restored from checkpointing blob. < begin of hand-waving > If the child process uses less memory than the initial until the point of checkpoint restore, then we should be all fine, thanks to the top-down algorithm of VMA allocation (basically they are allocated from the highest address down to the lowest) and the lack of ASLR for internal allocations (which is the case, I believe). If the child process uses more memory, then we are kind of doomed, but intuitively it should use less? This part is tricky...

[mkow]

Another small issue: Any idea how to make a test which tests this logic? I.e. uses huge amounts of PAL internal memory.

[boryspoplawski]

Another small issue: Any idea how to make a test which tests this logic? I.e. uses huge amounts of PAL internal memory.

Nothing clean and robust. I think we could leverage the LibOS testing function calling mechanism. We would test some normal stuff, call that function which would allocate a lot of PAL objects (handles? anything else?) and then test some more normal stuff, then check these objects still work. No other idea.

[dimakuv]

This was fixed by #839. Closing.