Converted images show random diffs in rootfs

[tianouya-db]

We found sometimes there can be random diffs in the rootfs between the original and converted image:

Running diff on container rootfs: /run/containerd/io.containerd.runtime.v2.task/default/2783b311f0831276-1682629450/rootfs, /run/containerd/io.containerd.runtime.v2.task/default/2783b311f0831276-1682629450-overlaybd/rootfs
Only in /run/containerd/io.containerd.runtime.v2.task/default/2783b311f0831276-1682629450/rootfs/etc/alternatives: python3
Only in /run/containerd/io.containerd.runtime.v2.task/default/2783b311f0831276-1682629450/rootfs/etc/apt/preferences.d: blocked_packages
Symbolic links /run/containerd/io.containerd.runtime.v2.task/default/2783b311f0831276-1682629450/rootfs/usr/bin/python3 and /run/containerd/io.containerd.runtime.v2.task/default/2783b311f0831276-1682629450-overlaybd/rootfs/usr/bin/python3 differ
Only in /run/containerd/io.containerd.runtime.v2.task/default/2783b311f0831276-1682629450/rootfs/var/lib/dpkg/alternatives: python3
Binary files /run/containerd/io.containerd.runtime.v2.task/default/2783b311f0831276-1682629450/rootfs/var/log/alternatives.log and /run/containerd/io.containerd.runtime.v2.task/default/2783b311f0831276-1682629450-overlaybd/rootfs/var/log/alternatives.log differ

These diffs can disappear the next time we convert the image. Is this an expected behavior?

[WaberZhuang]

This appears to be a bug and not the behavior we expected. Would you mind providing more information?

1. Which image format are you using, fastoci or overlaybd?
2. How did you do the image conversion, was it via ctr obdconv or convertor(i.e. userspace image-convertor)?
3. Did you use layer deduplication?

[tianouya-db]

@WaberZhuang Thanks for the reply.

1. We use overlaybd
2. We run sudo /opt/overlaybd/snapshotter/ctr obdconv
3. Not sure about it - how can I find out?

[WaberZhuang]

Layer deduplication is enabled if the --dbstr option is used with ctr obdconv. It doesn't seem to be used, I guess.
If your image is publicly available, would you be willing to provide us with the original image you used to check it, preferably both correct and incorrect converted images?

[tianouya-db]

Layer deduplication is enabled if the --dbstr option is used with ctr obdconv. It doesn't seem to be used, I guess.

Right it's not used. Would it make a difference?

provide us with the original image you used to check it

Unfortunately we use private images and are not able to share them. Is there anything else we can try out to narrow down the issue?

[WaberZhuang]

Would layer deduplication make a difference?

Some complex or incorrect operations with deduplication may lead to unexpected situations, but since you don't use it, the problem is not caused by it.

Is there anything else we can try out to narrow down the issue?

1. You could try another way we have provided to convert the image and see if the converted image still has the same problems as before. I will refer to it in the following as userspace convertor.
   userspace convertor is used differently than ctr obdconv. When using userspace convertor, you do not need to do ctr pull or ctr push, userspace convertor includes downloading and uploading, so all you need to do is /opt/overlaybd/snapshotter/convertor ... and /opt/overlaybd/snapshotter/ctr rpull .... See this manual for more details.
2. Which version of snapshotter and overlaybd are you using? I suggest you try to use the latest version, especially if you want to use userspace convertor.
3. What is the size of your original image? If it exceeds 64GB it may cause some problems.

[liulanzheng]

@tianouya-db It seams some error occurs with randomness. you may try userspace convertor as Waber said. ctr obdconv has dependences on your system, eg, tcmu, loop, and mount. Some users have encountered some unexpect situations under some kernel versions. userspace convertor has less dependences and has more controllability.