Impact of Compat default in OCI mode on User Experience

[marcodelapierre]

Is your feature request related to a problem? Please describe.

Hey team, I have been deploying Singularity as the HPC container runtime as staff at the Pawsey national centre (AUS) for 4+ years. Now I am maintaining its interface with it within the workflow engine Nextflow as part of my new role at Seqera.
I have started to test the new OCI functionalities in V4.
I am a bit concerned about --oci implying --compat, as it may disrupt user experience quite a bit. This is particularly true if a system admins decide to enable it by default in singularity.conf.

Describe the solution you'd like

I am raising the two following solutions for discussion:

1. adding a toggle in singularity.conf for compat mode, to enable sys admins to control the feature alongside oci mode; corresponding PR : Adding singularity.conf option "compat mode" #2306
2. considering disabling the behaviour where --oci implies --compat, to minimise disruption of user experience.

Thoughts?

Thank you!

[dtrudg]

Hi @marcodelapierre and thanks for opening this issue / discussion.

With the native runtime (i.e. SingularityCE 3.x or SingularityCE 4.x without --oci) the --compat flag was added to turn on features that bring behaviour as close as possible to Docker and other OCI runtimes, to fix some issues commonly experienced when people were directly running containers from Docker hub. The OCI compatibility that --compat can achieve is limited due to the nature of the native runtime and container format. For example, containers that use USER in their Dockerfile don't run exactly as under Docker.

The new --oci mode in SingularityCE 4.x is also explicitly about OCI compatibility, but now we use a modified OCI-SIF image format, and runc / crun as a true OCI low-level runtime under the hood. It's our specific intention that singularity run --oci ... is very close to docker run -it --rm ... and does not mirror historic Singularity behaviour.

I am a bit concerned about --oci implying --compat, as it may disrupt user experience quite a bit. This is particularly true if a system admins decide to enable it by default in singularity.conf.

As above, the difference in behaviour is intentional. It would, in our view, be strange to have an --oci that implied --no-compat by default, since then it would be an --oci mode, that is by default not providing expected OCI container behaviour.

I am raising the two following solutions for discussion:

1. adding a toggle in singularity.conf for compat mode, to enable sys admins to control the feature alongside oci mode; corresponding PR : Adding singularity.conf option "compat mode" #2306

I'm afraid I'm not convinced this is a good idea for the following reasons:

• OCI mode is not the default on installation. It must be enabled, so we look at enabling OCI-mode as being an explicit decision that the user or administrator does not want the Singularity native mode behaviour, but does want more OCI compatible / docker like behaviour.
• I'm not clear why an administrator would want to default to OCI mode if they want the older Singularity native runtime behaviour.
• The combination of --oci and --no-compat does not completely replicate the older native runtime behaviour. There are some behaviours that cannot be replicated when a container is run with an OCI low-level runtime e.g. crun/runc.
• We would be creating an inverse consistency issue, where if an admin specified compat = yes & oci mode = yes a user would need --no-oci --no-compat to get traditional singularity behaviour... where it would be reasonable to expect that --no-oci alone means 'non-OCI traditional Singularity behaviour'.

2. considering disabling the behaviour where --oci implies --compat, to minimise disruption of user experience.

I'm afraid this option is at odds with the intent of --oci mode, as above. It is also a breaking change that could not be implemented except in a major version - i.e. SingularityCE v5.0.0.

I think there's a mismatch between how we view OCI-Mode, and how you are thinking about it in terms of Nextflow executing containers?

From our point of view, if Singularity is configured for OCI-Mode then it would be preferable if workflow systems used it as if it were a docker -it --rm, and did not attempt to force the non-OCI-Mode behaviours.

Alternatively, if the workflow system finds Singularity >=4 and really wants the older behaviour, it could just add --no-oci to the command line always?

I think before proceeding with any code, it'd be good to discuss what Nextflow needs / expects out of Singularity versus how we see native vs OCI modes.

[marcodelapierre]

Hi Dave, many thanks for taking the time to share your articulated and insightful message, I have deeply appreciated.

I think there's a mismatch between how we view OCI-Mode, and how you are thinking about it in terms of Nextflow executing containers?

The mismatch is indeed the key and final point; while reading your reply I have realised I hadn't grasped the philosophy behind the OCI mode, probably mostly due to my historic HPC focus, but also as I hadn't followed the discussion path to Singularity 4 in full.

Thanks to you I now have a much clearer and sensible picture.

No worries, as regards interfacing with Nextflow, we will proceed by complying with the principles of OCI mode in Singularity 4, as we want to provide compatibility in the appropriate way.

I will make sure to follow up the future developments of Singularity more closely 😉

[dtrudg]

Glad the notes above were useful. Let us know if you find anything that doesn't seem to work quite right in the OCI mode, or would like to discuss anything Nextflow <-> Singularity related in general.