N4 cross-platform-version reproducibility

[ahoopes]

Hi,

We've been running freesurfer reproducibility tests since we're in the process of upgrading our lab from centos 7 to 8, and we noticed an interesting issue with the bias correction. All of our centos7-compiled binaries generate reproducible results across centos 7 and 8 except for N4BiasFieldCorrection. The differences are generally very small (on the order of 1e-5), but it's consistent.

I've observed this with our FS-specific binaries, locally built ANTs 2.2.0 binaries, and the prebuilt 2.1.0 binaries available on github (all ensuring ITK threads set to 1). Has anyone ever reported anything like this? Again, the differences are tiny, but it seems to be the main source of recon-all discrepancies across linux platform versions, which we'd like to avoid.

System information

So far we've only tested this across centos versions:

• centos 7.7
• centos 8.2

[ntustison]

Thanks @ahoopes . Just to be clear, you haven't tested this on other platforms such as MacOS, correct?

[ahoopes]

Right. I can also test across ubuntu versions if that's helpful.

[ntustison]

Yeah, that would be helpful. If you print the output to the screen, how do the convergence numbers compare?

[ahoopes]

I observed the same differences across ubuntu 16.04 and 20.04. I imagine it's something like a very low-level floating point error accumulation. The differences, although very small, are noticeable immediately in the convergence log:

                         centos 7 |     centos 8
Iteration 1 (of 50):  0.000122689 |  0.000122687
Iteration 2 (of 50):  0.000106237 |  0.000106238
Iteration 3 (of 50):  9.52423e-05 |  9.52433e-05
...

Adding @rpwang17, who's been helping debug this on the FS end.

[gdevenyi]

Different default cmake flags? please share your build logs

[ntustison]

Hey @ahoopes , @gdevenyi and others here are a great resource but you might also might want to start a parallel conversation over at the ITK discourse forum. There are several people over there who have fine-tuned certain computational aspects of N4 and might be able to provide additional insight. Please tag me so that I can weigh in if warranted.

[ahoopes]

Sounds good, thanks @ntustison, we'll open another conversation there.

[asmaharry]

Have you noticed any difference in N3 bias correction (using ants function) for different versions of Ubuntu? (@ahoopes, @ntustison)
I have observed that N3 bias correction on Unbuntu 20 and on Ubuntu 22 produces different images.

[ntustison]

Perhaps the same issue. Make sure to test using single threaded and see if you can determine any patterns in the discrepancy. I don't have those same platforms and I'm pretty sure the FS don't use the ANTs version of N3 so you're probably on your own for trying to figure this out.

[asmaharry]

Thank you for the reply, Could you please guide me how to set the threads for ants N3 bias correction function, so that the resultant images are reproducible?

[asmaharry]

I have found
export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=$threads
But how we can set it in antspy. I have no idea.

[ntustison]

import os

os.environ["ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS"] = "1"

In the future, please post ANTsPy-specific questions on the ANTsPy repository.