Silent crash when using factarg on a large expression

[vsht]

Hi,

here's what I get with the latest repo snapshot when I try to factorize to the coefficients of the master integrals when assembling the final amplitude

on HighFirst;

#define lsclDenNumFactorizeArguments "{gs\,la}"
#define lsclPprApplyPolyRatFunVerbosity "0"
#define lsclPprNumDenFactorizeVerbosity "0"
S gs,la;
Auto I mu;
Auto S lsclS;
Auto T lsclT;
Auto V lsclP;
Auto CF topology,lsclWrapFun,lsclF;
Load ampL1From1To29.res;
CF lsclNum,lsclDen,lsclRat;

G amp = ampL1;

print;

b,
#do i=1, 72
topology`i',
#enddo
;
.sort
collect lsclWrapFun1,lsclWrapFun2;
.sort

#call lsclApplyPolyRatFun(lsclNum,lsclDen,lsclRat,lsclWrapFun1,lsclWrapFun2);
.sort

#call lsclNumDenFactorize(lsclNum,lsclDen,lsclRat,`lsclDenNumFactorizeArguments');
.sort

b,
#do i=1, 72
topology`i',
#enddo
;
print[];

.end

Program terminating in thread 0 at FORM-test-script.frm Line 26 -->

Here are my procedures for the factorization

#procedure lsclApplyPolyRatFun(NUM,DEN,RAT,WRAP1,WRAP2)

* lsclApplyPolyRatFun() applies PolyRatFun to the expressions inside WRAP1 and WRAP2
* At the end PolyRatFun is deactivated but the expressions still remain wrapped in RAT

* Change the denominator function to DEN, but only inside WRAP1 and WRAP2
argument `WRAP1',`WRAP2';
denominators `DEN';
endargument;

* Factorize the arguments of WRAP1 and WRAP2 so that e.g. WRAP1(c1+c2+....) -> WRAP1(d1,d2,d3,...), 
* where the original result would be d1*d2*d3* ...
 factarg,`WRAP1',`WRAP2';

* Split arguments of WRAP1 and WRAP2 into products WRAP1(a,b,c) -> WRAP1(a)*WRAP1(b)*WRAP1(c).
* Now each WRAP1 and WRAP2 contains only one argument
 chainout `WRAP1';
 chainout `WRAP2';

* Another splitting of the arguments of WRAP1 and WRAP2 to decompose the sums into smaller 
* pieces: WRAP1(a+b+c) -> WRAP1(a,b,c). This way we again end up with having WRAP1 and WRAP2 
* having multiple arguments
 splitarg `WRAP1';
 splitarg `WRAP2';

* Now each argument is just a single term, not a sum of terms, so we can split them into
* proper sums: WRAP1(a,b,c) -> WRAP1(a)+WRAP1(b)+WRAP1(c)
 repeat;
 id lsclF?{`WRAP1',`WRAP2'}(lsclS1?,lsclS2?,?a) =  lsclF(lsclS1) + lsclF(lsclS2) + lsclF(?a);
* Very important, without this id statement the final result will be wrong
 id lsclF?{`WRAP1',`WRAP2'}() = 0;
 endrepeat;

* Each term is still a product of terms, so we use factarg again to 
* get WRAP1(a*b*c) -> WRAP1(a,b,c)
 factarg `WRAP1',`WRAP2';

* Finally, we are in the position to pull out the denominators
 repeat id lsclF?{`WRAP1',`WRAP2'}(?a,`DEN'(?c), ?b) =  `DEN'(?c)*lsclF(?a,?b);
 repeat id lsclF?{`WRAP1',`WRAP2'}() = 1;

* Repeat the splitting WRAP1(a,b,c) -> WRAP1(a)*WRAP1(b)*WRAP1(c). This recovers 
* the original expression if each WRAP1 is set to a unit function
 chainout `WRAP1';
 chainout `WRAP2';

* Check that all denominators were succesfully pulled out
 argument `WRAP1',`WRAP2';
  if (occurs(`DEN')) exit "lsclApplyPolyRatFun: Something went wrong while factoring out the denominators";
 endargument;

 id lsclF?{`WRAP1',`WRAP2'}(lsclS?) = `NUM'(lsclS);

* Sometimes the input expression may already contain NUM and DEN

* Remove tensor functions from numerators
 repeat id `NUM'(lsclT?(?a)) = lsclT(?a);

* Remove scalar functions from numerators, but not other NUM functions!
* repeat id `NUM'(lsclF?!{`NUM'}(?a)) = lsclF99(lsclF(?a));

* Remove scalar products from numerators
 repeat id `NUM'(lsclP1?.lsclP2?^lsclS?!{,0}) = lsclP1.lsclP2^lsclS;

* Remove nested numerators
 repeat id `NUM'(`NUM'(?a)) = `NUM'(?a);


#if (`lsclPprApplyPolyRatFunVerbosity'>1)
  print;
  .sort
#endif


#if (`lsclPprApplyPolyRatFunVerbosity'>0)
  #message lsclApplyPolyRatFun: Activating PolyRatFun : `time_' ...
#endif

.sort: lsclApplyPolyRatFun 1;

PolyRatFun `RAT';
id `NUM'(lsclS?) = `RAT'(lsclS,1);
id `DEN'(lsclS?) = `RAT'(1,lsclS);

#if (`lsclPprApplyPolyRatFunVerbosity'>1)
  print;  
#endif

#if (`lsclPprApplyPolyRatFunVerbosity'>0)
  #message lsclApplyPolyRatFun: Deactivating PolyRatFun : `time_' ...
#endif

.sort: lsclApplyPolyRatFun 2;

PolyRatFun;

#endprocedure

and

#procedure lsclNumDenFactorize(NUM,DEN,RAT,PULLOUTSYMBOLS)

* lsclNumDenFactorize() applies FactArg to the expressions inside NUM, DEN and RAT
* PULLOUTSYMBOLS must be either an empty set {\,0} or a set of variables that should
* be pulled out of NUM and DEN functions, e.g. {x} or {x\,y}
* The commas have to be escaped because of <https://github.com/vermaseren/form/issues/545>


repeat id `RAT'(lsclS1?,lsclS2?) = `NUM'(lsclS1)*`DEN'(lsclS2);

* Factorize the arguments of NUM and DEN so that e.g. NUM(c1+c2+....) -> NUM(d1,d2,d3,...), 
* where the original result would be d1*d2*d3* ...
 factarg,`NUM',`DEN';

#if (`lsclPprNumDenFactorizeVerbosity'>1)
  print;
  .sort
#endif


* Split arguments of NUM and DEN into products NUM(a,b,c) -> NUM(a)*NUM(b)*NUMs(c).
* Now each NUM and DEN contains only one argument
 chainout `NUM';
 chainout `DEN';

#if (`lsclPprNumDenFactorizeVerbosity'>1)
  print;
  .sort
#endif

* Simplify the output
repeat id `NUM'(lsclS?`PULLOUTSYMBOLS') = lsclS;
repeat id `DEN'(lsclS?`PULLOUTSYMBOLS') = 1/lsclS;
repeat id `NUM'(lsclS?number_) = lsclS;
repeat id `DEN'(lsclS?number_) = 1/lsclS;
repeat id `NUM'(1/lsclS?) = `DEN'(lsclS);
repeat id `NUM'(lsclS?)*`DEN'(lsclS?) = 1;
 
#endprocedure

It looks like the code is crashing during collect, but the real problem actuall appears here

* Factorize the arguments of NUM and DEN so that e.g. NUM(c1+c2+....) -> NUM(d1,d2,d3,...), 
* where the original result would be d1*d2*d3* ...
 factarg,`NUM',`DEN';

code.zip

I'm a bit surprised that there are no error messages whatsoever, though.

Of course, it might also be that my procedures are not really well optimized...

[jodavies]

I think this is the same deal as #516

The EndSort here:
https://github.com/vermaseren/form/blob/4fc8e4047a2678c3d0d264f29866491107f0a63c/sources/argument.c#L2158
should be par=1.

Your example runs for me if I set this.

[vsht]

Thanks for looking into it.

In general, the factorization in FORM seems to be a bit tricky, in the sense that large expressions are quite likely
to trigger either the maxtermsize error or some unknown bugs which require manual interaction.

Are there some general recommendations to avoid such issues? At least I already see that factorizing prefactors
of masters in a really big amplitude at once is like explicitly asking for troubles.

Is it better to use argtoextrasymbol and process integral by integral? Or rather reorganize the calculation in a different way?

[jodavies]

I am not sure I follow: if the coefficients fit in a term as polyratfun, surely the factorized versions fit also? Do you have a good example of what you mean?

[vsht]

I mean that it's difficult to estimate a value for maxtermsize that won't exceed the available RAM and make sure that
all diagrams run through.

In practice, I often end up checking failing diagrams and then bumping maxtermsize until they also can be calculated.
In many cases the expressions in question are not even properly factorizable, just some very longish numerators made out different variable combinations.

So it kind of requires a lot of micromanagement as compared to avoiding factorization in most steps.

[jodavies]

Here you could perhaps get things a little smaller by not putting all symbols inside the prf and later pulling them out (last argument of lsclNumDenFactorize). Why not include those symbols in the bracketing in the first place?

Obviously I am not sure of the format of your expressions in general, but it looks like everything you want to have in num,den functions already are in them in the input. In this case the procedures could be simplified a bit, for eg I can reproduce your output (more quickly) with

#call numden2prf(lsclNum,lsclDen,lsclRat)
#call prf2numden(lsclRat,lsclNum,lsclDen)

which you can find for eg, here: https://github.com/vermaseren/form/blob/4fc8e4047a2678c3d0d264f29866491107f0a63c/check/user.frm#L170

However things are coded, they should of course not cause crashes with no error... I will look for more places where EndSort is called with the wrong parameter.

[jodavies]

Your expression generates a lsclNum function which contains a factor with 11,527 terms. It fits in your large MaxTermSize but when FORM sorts the final expression (with a par=0 EndSort) it wants to check the existence of a sort file here:
https://github.com/vermaseren/form/blob/4fc8e4047a2678c3d0d264f29866491107f0a63c/sources/sort.c#L960
because 11,527 > SubTermsInSmall and if par=0 the filehandle is not created (newout is null).

Here is a simple test which causes (not exactly the same) crash. It crashes at an earlier EndSort in ArgFactorize, but for the same reason:
https://github.com/vermaseren/form/blob/4fc8e4047a2678c3d0d264f29866491107f0a63c/sources/argument.c#L2097

#: MaxTermSize 1M

CFunction f;
Symbol a,b,c,d,e;

* Generate a function arg with more than SubTermsInSmall (10K) terms:
Local test = f((a+b+c+d+e)^21);

FactArg f;

.end

Presumably every EndSort in ArgFactorize needs to be par=1. I will make a patch and test.

[vsht]

Would you say that there's a general recommendation not to make the numerator too large (>10K terms is large for FORM, I guess)?

That's something I could control with nterms_ to prevent such issues from happening in the first place?

[vsht]

Obviously I am not sure of the format of your expressions in general, but it looks like everything you want to have in num,den functions already are in them in the input. In this case the procedures could be simplified a bit, for eg I can reproduce your output (more quickly) with

I was trying to write something similar to Mathematica's Collect and Factor where I can factorize expressions that are already partially (or completely) factorized without generating errors.

[jodavies]

Not particularly, provided your buffer settings allow for it. The behaviour you have here is a bug rather than your expression being too big. I always run FORM with larger-than-default sub* buffer sizes which helps performance and works around many problems like the one here (your example runs fine, with my usual settings).

SubSmallSize      10M
SubSmallExtension 15M
SubTermsInSmall   500K

SubLargeSize      100M
SubLargePatches   512

SubFilePatches    512
SubSortIOSize     1M

[vsht]

I see, thanks. I think that in my case (and for my future calculations) it's easier to avoid running into such issues by using
something like

repeat id `RAT'(lsclS1?,lsclS2?) = `HOLDNUM'(lsclS1,(nterms_(lsclS1)))*`HOLDDEN'(lsclS2,nterms_(lsclS2));
repeat id `HOLDNUM'(lsclS1?,lsclS2?{,<5000}) = `NUM'(lsclS1);
repeat id `HOLDDEN'(lsclS1?,lsclS2?{,<5000}) = `DEN'(lsclS1);

in lsclNumDenFactorize and the alerting the user if the numerators/denominators start getting too big.

Big numerators are most likely signalling that the expression hasn't been collected w.r.t the right variables.

Even though this behavior is a bug, it got me to rethink my original strategy for factorizing the coefficients :)

[jodavies]

https://github.com/jodavies/form/tree/issue-563

Could you try this with your original code,, if you have more and larger amplitudes?

[vsht]

https://github.com/jodavies/form/tree/issue-563

Could you try this with your original code,, if you have more and larger amplitudes?

Thanks, it works with my example. I need this calculation only at 1L, so there are currently no more complicated amplitudes where this issue could arise.