gzip errors

[jodavies]

Several times, I have seen errors like

1FillInputGZIP: Error in gzip handling of input.
PutIn: We have RetCode = FFFFFFFFFFFFFFFF while reading 124F80 bytes
Program terminating in thread 1 at replace Line 18 -->

They are not reproducible, generally I can re-run the calculation and everything goes through without problems. It is hard to know what is to blame here, perhaps they are genuine read errors from the disk (although I have seen this issue on more than one machine).

I don't know if there would be much point, upon receiving an error from zlib, retrying the inflate?

Have you seen this before? Andreas says he does not recall ever seeing a crash like this. Not a very useful report I suppose, but I thought I would mention it in case you have any ideas.

[vermaseren]

Hi Josh,

For now it does not ring a bell.
Problem with retrying is that compressing/decompressing is something with memory. That means you
cannot jump into the middle, and you can also not just retry. You would have to start from the ‘beginning’
again. This could be possible only if the compression would be arranged as separate entities and not
as one complete object for a whole patch. Unless of course the error occurs at the start of a patch.

How often has this occurred to you?

There is a still undocumented feature (I have not yet tested it. It was built by Ali Mirsoleimani) to use
BZIP2 instead of GZIP. This could give shorter output and depending of the level it could also be
a little bit faster (definitely compared to GZIP n with n >6). If you try this, let me know the results.

Jos

On 20 mei 2016, at 12:23, jodavies notifications@github.com wrote:

Several times, I have seen errors like

1FillInputGZIP: Error in gzip handling of input.
PutIn: We have RetCode = FFFFFFFFFFFFFFFF while reading 124F80 bytes
Program terminating in thread 1 at replace Line 18 -->
They are not reproducible, generally I can re-run the calculation and everything goes through without problems. It is hard to know what is to blame here, perhaps they are genuine read errors from the disk (although I have seen this issue on more than one machine).

I don't know if there would be much point, upon receiving an error from zlib, retrying the inflate?

Have you seen this before? Andreas says he does not recall ever seeing a crash like this. Not a very useful report I suppose, but I thought I would mention it in case you have any ideas.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub #95

[jodavies]

I have seen this maybe 5 times over the last few months. They were all in cases with rather large (O(TB)) files.

It looks like the branch with the bzip code is too old to run forcer so I guess that would need to be merged, or, I could set up some tests with mincer.

Out of interest, are the patches in the large buffer compressed, or just after sorting them and writing to disk? (Another thing one could try is to disable FORM's zlib compression entirely and leave it all to the btrfs filesystem).

[vermaseren]

Hi Josh,

gzip compression is only used for the sort files. In the large buffer only the standard form compression
is used which is something rather simple.

Jos

On 20 mei 2016, at 12:54, jodavies notifications@github.com wrote:

I have seen this maybe 5 times over the last few months. They were all in cases with rather large (O(TB)) files.

It looks like the branch with the bzip code is too old to run forcer so I guess that would need to be merged, or, I could set up some tests with mincer.

Out of interest, are the patches in the large buffer compressed, or just after sorting them and writing to disk? (Another thing one could try is to disable FORM's zlib compression entirely and leave it all to the btrfs filesystem).

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub #95 (comment)

[jodavies]

I have also seen, twice, errors like

Time =  108303.46 sec    Generated terms =   71246936
             d5d90522070 Terms left      =   70733861
     dotrewrite-d89-0-25 Bytes used      =15614237544
Read error in SetScratch
Program terminating in thread 3 at replace Line 18 -->

It is possible that these are real hardware errors. I will try to investigate whether this is the case.

Would you anticipate any issues with the following setup parameters? I run a very large TermsInSmall as otherwise the serial sections of the reduction become very slow with much master sorting into a .sor file, since it hits a smaller TermsInSmall limit very often and produces a lot of patches.

I also allow very large numbers of LargePatches and FilePatches. For disk capacity reasons, it is important not to go into stage4 sorting.

Perhaps at some point, some of these parameters are multiplied together, and could exceed a 32bit int? I don't know how such errors would manifest.

* 192GB
#: MaxTermSize 300K
#: WorkSpace 400M
#: LargeSize 44G
#: SmallSize 6G
#: ScratchSize 16G
#: HideSize 8G
#: TermsInSmall 50M
#: LargePatches 16384
#: FilePatches 16384

[vermaseren]

Hi Josh,

A good setting for TermsInSmall is SmallSize/(typical term size).
In your crash that is in form compressed form about 200 bytes. In reality in the small buffer the terms are
not compressed and hence about 400 bytes is more realistic. 6Gbytes/50Mbytes is 120, hence there should
be no problem there.
It is possible that the above is a very rare error in FORM of course. I have had some of that in the past
that only in some very special cases something went wrong and only when things were very big.
And then, when this happens with tform, it may occur only once out of many runs.
This is very hard to debug as you can imagine.
The setup parameters look rather fine to me. If the ratio between LargeSize and SmallSize is not very
large, you may not need so many LargePatches, unless sorting the small buffer gives consistently a very
large collapse in the number of terms. This is different with the file patches. But indeed, it is important to
avoid stage4, because that is never fast (I have been in stage5 only once, but it worked).

If you can find out more about these crashes, maybe we can sooner or later make them deterministic and
have a chance to find the cause.

Jos

On 24 mei 2016, at 13:04, jodavies notifications@github.com wrote:

I have also seen, twice, errors like

Time = 108303.46 sec Generated terms = 71246936
d5d90522070 Terms left = 70733861
dotrewrite-d89-0-25 Bytes used =15614237544
Read error in SetScratch
Program terminating in thread 3 at replace Line 18 -->
It is possible that these are real hardware errors. I will try to investigate whether this is the case.

Would you anticipate any issues with the following setup parameters? I run a very large TermsInSmall as otherwise the serial sections of the reduction become very slow with much master sorting into a .sor file, since it hits a smaller TermsInSmall limit very often and produces a lot of patches.

I also allow very large numbers of LargePatches and FilePatches. For disk capacity reasons, it is important not to go into stage4 sorting.

Perhaps at some point, some of these parameters are multiplied together, and could exceed a 32bit int? I don't know how such errors would manifest.

• 192GB
  #: MaxTermSize 300K
  #: WorkSpace 400M
  #: LargeSize 44G
  #: SmallSize 6G
  #: ScratchSize 16G
  #: HideSize 8G
  #: TermsInSmall 50M
  #: LargePatches 16384
  #: FilePatches 16384
  —
  You are receiving this because you commented.
  Reply to this email directly or view it on GitHub gzip errors #95 (comment)

[burp]

I have exactly the same issue (with default form settings), and since it is reproducible for me I did a bisection to track it down from a good 4.1 tag and a bad head to: " Fixed the compress/gzip/hide bug." 43a5b1e. Hope this helps!

Cheers, Tobias

[jodavies]

@burp , are you able to provide a script for your reproducible example, if it is small and crashes quickly?

I have a calculation which has crashed the last 4 times I have run it, but it produces 2TB of scratch files and takes more than a day to crash.

[burp]

It crashes within 1-2 minutes, that's why I was able to do a quick bisection. Unfortunately it is part of a bigger setup with tons of includes etc. I will try to boil it down to one simple script/file.

[jodavies]

@burp , were you able to create a simple example which crashes like this in the end?

I have had this crash on a second machine now, making genuine disk errors less likely.

Should we also have && fi->zsp != 0 in the if statement of line 1266, sort.c, in the function Sflush ?

[burp]

@joedavies: Have not had time for it yet, maybe I can just send you the setup in private. You will have to adjust a few include paths etc., but it's probably simpler for me than trying to minimize the reproducible example.

[jodavies]

That works, if it is OK with you!

Probably it is best if you can send Jos a copy also, he is much more likely to be able to find possible bugs.

[jodavies]

I had a quick look at this example. With GZIPDEBUG enabled at various places, during normal operation, one sees output like the following,

Preparing z-stream 0 with compression 1
...
Preparing z-stream 4 with compression 1

-+Reading 160000 bytes in stream 0 at position          0; stop at    5320619
Want to read in stream 0 at position     160000
--Reading 160000 bytes in stream 0 at position     160000
...
Closing stream 0
--Reading 160000 bytes in stream 1 at position    5480619
...
Closing stream 1
--Reading 160000 bytes in stream 2 at position   10310142
...
...
Closing stream 4
etc

In the example which crashes, we find

Preparing z-stream 0 with compression 1
Preparing z-stream 1 with compression 1
-+Reading 160000 bytes in stream 0 at position          0; stop at    5486806
--Reading 160000 bytes in stream 0 at position     160000
...
Closing stream 0
  -Last words: 1 1 2 -3 0
 zerror = -2 in stream 0. At position    5486806

It seems like FORM is trying to read from stream 0, which was already "closed"?

The offending call of PutIn() is the one below the label NextTerm: in sort.c. It seems ki has the value 0 when it should be 1?

[jodavies]

This crash does not occur if one changes MaxTermSize (and thus the buffersize passed to FillInputGZIP).

The crash scenario seems to be:

• finish a stream (print "Closing stream 0")
• call PutIn (from sort.c:4068) and then FillInputGZIP with the same stream number
• enter if branch in compress.c:490
• enter else branch in compress.c:497. Obtain value toread = 0.
• attempt to inflate at compress.c:545. Get zerror = -2.

EDIT: there are no valgrind errors

[tueda]

@jodavies Is there any public test code to reproduce the bug?

[jodavies]

I am using the code from @burp , of which I think Jos also has a copy. I have not been able to reduce it to a particularly minimal crashing example.