18. June 2024

[dimakuv]

Agenda

(please write your proposed agenda items in comments under this discussion)

• Dmitrii: how to rework [LibOS] Fix dentry of open file handles related to a rename #1874

MichaelS described the #PR1874 which is trying to address a problem when dentry is updated, file handles are not updated and this is a problem for roll back protection PR. He has been discussing some updates and refactoring/updates around some locking strategies with Dmitrii. Dmitrii is suggesting to refactor of PR into multiple PRs and MichaelS will need to work with Dmitrii to better understand that. MichałK mentioned that 100 lines of code are ok for a single PR.

• Mona: Discuss status and plan for Gramine reproducible builds

Reproducible build works for the most part, but we do not test in our CI. Will need to revisit this issue -- #153

• Dmitrii/Mona: Stack overflows in non-EDMM cases are silent and hard to debug
  • Recently had a tough debug session with strange segfaults, turned out it was a stack overflow because by default stack size is 256KB
  • See https://gramine.readthedocs.io/en/stable/manifest-syntax.html#stack-size
  • In particular, Python apps typically require 2MB - 8MB sized stacks
  • The problem is that a stack overflow is not detectable in SGXv1, as all stack pages -- including the guard page -- are marked as RWX (because the app stack is allocated on the enclave heap)
  • Can we come up with some trick to detect a stack overflow in SGXv1 (w/o EDMM and page protections)? Alternatively, can we issue a log_error() if we detect a potential for stack overflow during startup (like if the binary name has python and the stack size is less than 2MB).

Michał:

• this is actually related to rlimit:

  gramine/libos/src/libos_init.c

  Lines 279 to 316 in 974f2c6

  	int init_stack(const char* const* argv, const char* const* envp, char*** out_argp,
  	elf_auxv_t** out_auxv) {
  	int ret;
  	assert(g_manifest_root);
  	uint64_t stack_size;
  	ret = toml_sizestring_in(g_manifest_root, "sys.stack.size", get_rlimit_cur(RLIMIT_STACK),
  	&stack_size);
  	if (ret < 0) {
  	log_error("Cannot parse 'sys.stack.size'");
  	return -EINVAL;
  	}
  	stack_size = ALLOC_ALIGN_UP(stack_size);
  	set_rlimit_cur(RLIMIT_STACK, stack_size);
  	struct libos_thread* cur_thread = get_cur_thread();
  	if (!cur_thread || cur_thread->stack)
  	return 0;
  	void* stack = allocate_stack(stack_size, ALLOC_ALIGNMENT, /*user=*/true);
  	if (!stack)
  	return -ENOMEM;
  	log_debug("Allocated stack at %p (size = %#lx)", stack, stack_size);
  	/* if there is envp inherited from parent, use it */
  	envp = migrated_envp ?: envp;
  	ret = populate_stack(stack, stack_size, argv, envp, out_argp, out_auxv);
  	if (ret < 0)
  	return ret;
  	cur_thread->stack_top = stack + stack_size;
  	cur_thread->stack = stack;
  	cur_thread->stack_red = stack - ALLOC_ALIGNMENT;
  	return 0;
  	}

• no way to fix this w/o EDMM, suggest to try running the same w/ EDMM (an orthogonal discussion was using EDMM may bring performance impacts, e.g., in cases where apps entirely demand pages)
• fine w/ doc update

Mona:

• opt#1: increase the default stack size: https://gramine.readthedocs.io/en/stable/manifest-syntax.html#stack-size to e.g., 2MB -- what about older platforms w/ limited EPC (still in use?)
• opt#2: update manifest doc to mention that python (or any) application use large stack may need to adjust the value; otherwise it can crash -- this might not be helpful as no direct/specific information (related to stack) shown when the crash happens (only instruction pointer in the failed case).
• why the issue is not reported earlier in e.g., CI: the CI examples set default to 2MB (

  gramine/CI-Examples/python/python.manifest.template

  Line 34 in 974f2c6

  sys.stack.size = "2M"

  ), similarly there are LTP, blender etc. which do the same

Anjo:

• add spaces in between (possible for Gramine stack; not app stack)
• no space for the stack to ever grow
• remap last page as guard page -- might be heuristical

MichaelS:

• to also mention in the manifest that the default stack size is lower than the Linux default (8MB)
• warning at startup -- which can be always printed (thus can be easily ignored, or maybe not)

Chia-Che:

• actively checking stack usage at every system call -- may incur overhead
• use EDMM to protect the page below the stack

Conclusion: update the manifest doc as a minimal solution

NOTE: Dmitrii may not be able to attend the meeting due to personal reasons.

[monavij]

Discuss status and plan for Gramine reproducible builds