Mountpoint order is not preserved in current manifest syntax

[mkow]

Description of the problem

Currently we don't use proper TOML syntax for declaring mountpoints, instead, we use a syntax which resembles the pre-TOML one used in Graphene. As a result, the entries are not ordered, but Graphene actually relies on the specific mounting order (e.g. you can't mount /lib/asdf first and then /lib, but the other way around works). The problem is, that TOML structure is just a dictionary, so the order of keys is not preserved.

This will be solved by finally migrating to a proper TOML syntax for everything, which will be eventually done (after this, mount points list should be a TOML array). Cleaning up mounting code should also help here - I think that in Graphene mount order shouldn't matter, we only need to ensure that there are no duplicated mounts.

Steps to reproduce

Swap the order of entries in e.g. manifest.template in LibOS regression, so that "$(ARCH_LIBDIR)" is listed before "/lib".

Expected results

Everything still works.

Actual results

[P6098:T1:bootstrap_cpp] Mount /lib already exists, verify that there are no duplicate mounts in manifest
[P6098:T1:bootstrap_cpp] (note that /proc and /dev are automatically mounted in Graphene).
[P6098:T1:bootstrap_cpp] Mounting file:../../../../Runtime on /lib (type=chroot) failed (17)
[P6098:T1:bootstrap_cpp] Error during shim_init() in init_mount (-17)

Additional information

gramineproject/graphene#2210 contains an ugly workaround, but we need a proper solution.

[dimakuv]

True. We will need to change the manifest syntax for this...

Notice that this only happens on gramineproject/graphene#2210, which adds TOML parsing to our Python scripts. The default TOML parser in Python doesn't preserve the order.

Interestingly, the TOML parser that we use in Graphene itself (in our C codebase) preserves the order of TOML entries. That's why we haven't seen this error manifesting itself in Graphene itself.

UPDATE: Ok, so I looked at gramineproject/graphene#2210. My comment above is wrong. We do modify the C code. But I'm not sure why this is needed currently, @mkow what tests actually break? I understand that our current code is brittle, but the C implementation worked, so what has changed?

[mkow]

I answered the same question under gramineproject/graphene#2210. TL;DR: migrating Python scripts to a proper TOML library shuffled the order (which is perfectly correct) and broke the incorrect assumptions our C code has.

[dimakuv]

This should be fixed now, as we moved to a proper TOML syntax with fs.mounts being a TOML array (which preserves the order).