Questions regarding the FORM setup parameters

[Matthias945]

Hello,

I am currently working on a 4-loop calculation using FORM and am encountering difficulties in selecting the appropriate setup parameters.
I would greatly appreciate your help in this matter.

Using the default parameters, some of my calculations crash on the computer cluster.
I am running TFORM with w=4 workers, and I am allocating 5CPUs with 96G to 128GB RAM (in total), depending on the diagram.
In certain Feynman diagrams, the number of terms after tensor reduction exceeds 1.5 billion (some even exceed 2 billion terms), leading to the following error message:

"Time = 42490.02 sec Expr586 Terms active = 1556852625 Bytes used =16758292834 0FillInputGZIP: Error in gzip handling of input. zerror = -3 PutIn: We have RetCode = FFFFFFFFFFFFFFFF while reading 1E125C bytes Program terminating in thread 0 at TensorReduction Line 311 --> 49487.66 sec + 98432.06 sec: 147919.73 sec out of 101893.85 sec"

In an analysis of the job, I noticed that only ~ 30GB of RAM was utilised out of the allocated 128 GB.
For this reason, I tried to change some of the setup parameters.
In particular, I tried to change the following parameters:

largesize 50G
maxtermsize 10K
smallextension 4G
smallsize 2G
scratchsize 8G
termsinsmall 5M
threadbucketsize 2K
threadscratchoutsize 10M
threadscratchsize 2M
totalsize ON
workspace 8G
SortIOSize 100M

I tested 15 different configurations with different values, but in all cases, the jobs failed again --- this time with an "Out of Memory" error.
Indeed, further analysis showed that the memory usage quickly reached the allocated limit.

I am struggling to understand why this happens, as the adjusted parameters should still be within the 128 GB limit.
I studied the FORM reference manual (in particular chapters 16 to 18), but I was not able to fully understand the dynamics of these setup parameters.
Could you please clarify how these values influence FORM’s behavior?
More importantly, how can I optimise FORM’s setup to fully utilise the available hardware without running into excessive or insufficient memory usage?

I would be very grateful for any guidance you can provide.
Thank you in advance for your time and help.

Yours sincerely,

Matthias

[jodavies]

This is yet another instance of the gzip errors, see here #596 .

So far, I have not been able to reproduce any reported zerror = -3 crash using code sent to me by users. If you have a "fairly small" example which crashes reliably on your systems I would be very interested to test it. I've never been able to rule out genuine disk errors causing FORM to read invalid zlib data, for eg, so it is very unclear if these cases are even FORM bugs.

[Matthias945]

Thank you very much for your quick response.
Unfortunately, this was the first time I encountered it and it only occurs in some diagrams which huge expressions (several more than 1.5 billion terms).
So, I do not have a small example which crashes reliably, but I will keep investigating it and come back to you if I can tell you more.

[jodavies]

Regarding your allocations, which is a separate issue, it certainly doesn't help to have such a large workspace (each thread has a workspace, you should leave this as the default unless your program really requires a larger workspace) and the large sortiosize might be making the larger buffer larger than you expect (there is a relation between sortiosize, maxtermsize and maxfilepatches which must be satisfied).

[Matthias945]

Thank you for these information.
However, if I use 4 workers, 4*8GB does not seem to be large to me (regarding the workspace) or do I missunderstand how these parameters interact / how the demanded memory adds up?
Regarding sortiosize and the relation you mentioned:
Is there some source where I can read more about these relations?

[jodavies]

There is some information about the buffer settings here https://www.nikhef.nl/~form/maindir/documentation/reference/html/manual.html#x1-34700017 but this particular relation is not documented, no.

Regarding the workspace, it is not a case of "more is better" but rather it only needs to be big enough for your form script. If it doesn't need to be as large as 8GB (very heavy use of repeat for eg) then the memory is better spent on other buffers, such as large or small.

[Matthias945]

Ok, thank you for your help.

[tueda]

To adjust the setup parameters, I would try the parameters printed by formset, for example,

formset --human-readable --ncpus 4 --total-cpus 4 --total-memory 128G

# formset 1.0.0 --human-readable --ncpus 4 --total-cpus 4 --total-memory 128G (cpu: 4, mem: 96G; total cpu: 4, total mem: 128G; 1x4)
compresssize 320K
largesize 24600M
scratchsize 8200M
smallextension 7380M
smallsize 4920M
sortiosize 31100K
termsinsmall 49200K
threadscratchoutsize 2560K
threadscratchsize 2560K

Note that this program shows, for your parameters:

formset --human-readable --ncpus 4 --total-cpus 4 --total-memory 128G LargeSize=50G MaxTermSize=10K SmallExtension=4G
SmallSize=2G ScratchSize=8G TermsInSmall=5M ThreadBucketSize=2K ThreadScratchOutSize=10M ThreadScratchSize=2M WorkSpace=
8G SortIOSize=100M

failed to find parameters: 125G bytes shortage

[Matthias945]

Thank you very much for your quick response and your help.
This looks very promising.
I will test formset and the recommended parameters in my calculations and check their performance.