SIGABRT for FITS web service on MOV file

[bbpennel]

We are running into a case where a particular .mov file is causing the jetty instance running the FITS web service to crash. However, it only causes this crash if the web service is processing other file as the same time. The .mov itself is ~250mb, while the jetty instance has -Xmx2g so it seems unlikely it is maxing out memory. To be safe I tried doubling the max heap, but had the same problem.

The logs from jetty don't show anything at the time of the crash, and if I enable a heap dump for running out of memory it does not perform the dump. We do see this error in the FITS log:

2021-07-21 12:16:17 - ERROR - Fits:677 - Tool error processing file: IMG_1396.MOV
null
java.lang.NullPointerException
        at edu.harvard.hul.ois.fits.tools.mediainfo.MediaInfoUtil.reviseIdentification(MediaInfoUtil.java:560)
        at edu.harvard.hul.ois.fits.tools.mediainfo.MediaInfoUtil.reviseXmlData(MediaInfoUtil.java:430)
        at edu.harvard.hul.ois.fits.tools.mediainfo.MediaInfo.extractInfo(MediaInfo.java:228)
        at edu.harvard.hul.ois.fits.tools.ToolBase.run(ToolBase.java:284)
        at java.lang.Thread.run(Thread.java:748)

This is happening with FITS version 1.5.0 and the web service at version 1.2.1.

Its also worth noting that I can process the same file multiple times concurrently with the FITS CLI with no problems. But If I do the same thing with the web service it dies. All requests to the server get empty responses.

We do get a core backtrace from system logs, which mostly lists a lot of waiting threads, but does show one thread where libmediainfo is getting aborted:

reason:         java killed by SIGABRT
cmdline:        /bin/java -Dfile.encoding=UTF-8 -Dfits.home=/opt/fits -DFITS_SERVICE_PROPS=/etc/opt/fits/fits-service.properties -Dlog4j.configuration=/etc/opt/fits/log4j.properties -Xmx2g -Xms512m -Djetty.home=/opt/jettys/fits/jetty -Djetty.base=/opt/jettys/fits/web/mybase -Djava.io.tmpdir=/opt/jettys/fits/temp -jar /opt/jettys/fits/jetty/start.jar jetty.state=/opt/jettys/fits/web/mybase/fits.state jetty-started.xml start-log-file=/opt/jettys/fits/web/mybase/logs/fits_start.log
executable:     /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/bin/java
package:        java-1.8.0-openjdk-headless-1.8.0.292.b10-1.el7_9
component:      java-1.8.0-openjdk
count:          1
abrt_version:   2.1.11
analyzer:       CCpp
architecture:   x86_64
event_log:     
global_pid:     18286
kernel:         3.10.0-1160.31.1.el7.x86_64
last_occurrence: 1626457704
os_release:     Red Hat Enterprise Linux Server release 7.9 (Maipo)
pkg_arch:       x86_64
pkg_epoch:      1
pkg_fingerprint: 199E 2F91 FD43 1D51
pkg_name:       java-1.8.0-openjdk-headless
pkg_release:    1.el7_9
pkg_vendor:     Red Hat, Inc.
pkg_version:    1.8.0.292.b10
runlevel:       N 3
time:           Fri 16 Jul 2021 01:48:24 PM EDT
type:           CCpp

core_backtrace:
:{   "signal": 6
:,   "executable": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/bin/java"
:,   "stacktrace":
:      [ {   "crash_thread": true
:        ,   "frames":
:              [ {   "address": 139946836280199
:                ,   "build_id": "e6847a931dd483773bab779dd3985b17c11caab2"
:                ,   "build_id_offset": 222087
:                ,   "function_name": "raise"
:                ,   "file_name": "/lib64/libc.so.6"
:                }
:              , {   "address": 139946836286072
:                ,   "build_id": "e6847a931dd483773bab779dd3985b17c11caab2"
:                ,   "build_id_offset": 227960
:                ,   "function_name": "abort"
:                ,   "file_name": "/lib64/libc.so.6"
:                }
:              , {   "address": 139946828279561
:                ,   "build_id": "cf5a5a4c57ad4f474d05b80cb8ee0856e5783741"
:                ,   "build_id_offset": 9694985
:                ,   "function_name": "os::abort(bool)"
:                ,   "file_name": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so"
:                }
:              , {   "address": 139946830536794
:                ,   "build_id": "cf5a5a4c57ad4f474d05b80cb8ee0856e5783741"
:                ,   "build_id_offset": 11952218
:                ,   "function_name": "VMError::report_and_die()"
:                ,   "file_name": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so"
:                }
:              , {   "address": 139946828320949
:                ,   "build_id": "cf5a5a4c57ad4f474d05b80cb8ee0856e5783741"
:                ,   "build_id_offset": 9736373
:                ,   "function_name": "JVM_handle_linux_signal"
:                ,   "file_name": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so"
:                }
:              , {   "address": 139946828267720
:                ,   "build_id": "cf5a5a4c57ad4f474d05b80cb8ee0856e5783741"
:                ,   "build_id_offset": 9683144
:                ,   "function_name": "signalHandler(int, siginfo_t*, void*)"
:                ,   "file_name": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.292.b10-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so"
:                }
:              , {   "address": 139946846570032
:                ,   "build_id": "e10cc8f2b932fc3daeda22f8dac5ebb969524e5b"
:                ,   "build_id_offset": 63024
:                ,   "file_name": "/lib64/libpthread.so.0"
:                }
:              , {   "address": 139945614467302
:                ,   "build_id": "ccc291383c9948af95be7f7367e9b718fbc5031f"
:                ,   "build_id_offset": 360678
:                ,   "file_name": "/opt/fits/tools/mediainfo/linux/libmediainfo.so.0"
:                }
:              , {   "address": 139945615015078
:                ,   "build_id": "ccc291383c9948af95be7f7367e9b718fbc5031f"
:                ,   "build_id_offset": 908454
:                ,   "file_name": "/opt/fits/tools/mediainfo/linux/libmediainfo.so.0"
:                }
:              , {   "address": 139945617581829
:                ,   "build_id": "ccc291383c9948af95be7f7367e9b718fbc5031f"
:                ,   "build_id_offset": 3475205
:                ,   "file_name": "/opt/fits/tools/mediainfo/linux/libmediainfo.so.0"
:                }
:              , {   "address": 139945617577280
:                ,   "build_id": "ccc291383c9948af95be7f7367e9b718fbc5031f"
:                ,   "build_id_offset": 3470656
:                ,   "file_name": "/opt/fits/tools/mediainfo/linux/libmediainfo.so.0"
:                }
:              , {   "address": 139945617579606
:                ,   "build_id": "ccc291383c9948af95be7f7367e9b718fbc5031f"
:                ,   "build_id_offset": 3472982
:                ,   "file_name": "/opt/fits/tools/mediainfo/linux/libmediainfo.so.0"
:                }
:              , {   "address": 139945615039880
:                ,   "build_id": "ccc291383c9948af95be7f7367e9b718fbc5031f"
:                ,   "build_id_offset": 933256
:                ,   "file_name": "/opt/fits/tools/mediainfo/linux/libmediainfo.so.0"
:                }
:              , {   "address": 139945615026747
:                ,   "build_id": "ccc291383c9948af95be7f7367e9b718fbc5031f"
:                ,   "build_id_offset": 920123
:                ,   "file_name": "/opt/fits/tools/mediainfo/linux/libmediainfo.so.0"
:                }
:              , {   "address": 139945618194075
:                ,   "build_id": "ccc291383c9948af95be7f7367e9b718fbc5031f"
:                ,   "build_id_offset": 4087451
:                ,   "function_name": "MediaInfo_Open"
:                ,   "file_name": "/opt/fits/tools/mediainfo/linux/libmediainfo.so.0"
:                }
:              , {   "address": 139945624287136
:                ,   "build_id_offset": 75680
:                }
:              , {   "address": 139945362302360
:                ,   "build_id_offset": 139945362302360
:                } ]
:        }
...

This is being performed with the libmediainfo.so file that comes with FITS, which appears to be from 2015. We tried updating to a current version libmediainfo, but it seemed like FITS didn't know how to interact with it, so we'd see <tool toolname="MediaInfo" toolversion="21.03" status="failed" /> in the result. Judging by the regular output, it looks like FITS ships with version 0.7.75. The version we install were rpms from rhel, I'm not sure what the original source of the files that ship with FITS is.

[pwinckles]

@bbpennel Are you aware of this PR: #209? I haven't tried it yet myself.

[bbpennel]

I tested PR209 locally. It appears to work in general, but has not resolved this particular problem. Its possible some later version of libmediainfo has a bug fix, since 209 is close to a year old.

When running #209 I did get responses from the server which contained the failure, which was a new behavior. It looks like it may be a memory allocation problem with libmediainfo when multiple scans are running in the same jvm. I actually got 3 different error messages back. The first was:

*** Error in `/bin/java': double free or corruption (fasttop): 0x00007f4504133910 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f45d91d6329]
/opt/fits/tools/mediainfo/linux/libmediainfo.so.0(+0x53730)[0x7f457d187730]
/opt/fits/tools/mediainfo/linux/libmediainfo.so.0(+0x53c49)[0x7f457d187c49]
/opt/fits/tools/mediainfo/linux/libmediainfo.so.0(+0xe620c)[0x7f457d21a20c]
/opt/fits/tools/mediainfo/linux/libmediainfo.so.0(+0x4efddf)[0x7f457d623ddf]
/opt/fits/tools/mediainfo/linux/libmediainfo.so.0(+0x10ddad)[0x7f457d241dad]
/opt/fits/tools/mediainfo/linux/libmediainfo.so.0(+0x109b7c)[0x7f457d23db7c]
/opt/fits/tools/mediainfo/linux/libmediainfo.so.0(MediaInfo_Open+0x84)[0x7f457d708d54]
/opt/jettys/fits/temp/jna-3143298/jna3091214734966084280.tmp(ffi_call_unix64+0x4c)[0x7f45a83837a0]
/opt/jettys/fits/temp/jna-3143298/jna3091214734966084280.tmp(ffi_call+0x20b)[0x7f45a838329b]
/opt/jettys/fits/temp/jna-3143298/jna3091214734966084280.tmp(+0x5e41)[0x7f45a8376e41]
/opt/jettys/fits/temp/jna-3143298/jna3091214734966084280.tmp(Java_com_sun_jna_Native_invokeInt+0x22)[0x7f45a83796f2]
[0x7f45c1016087]

The other two were similar, with the following top level messages:

*** Error in `/bin/java': corrupted double-linked list: 0x00007f10b8000ea0
*** Error in `/bin/java': double free or corruption (fasttop): 0x00007f4504133910 ***

[pwinckles]

@bbpennel Have tried running MediaInfo directly on the file?

[bbpennel]

I think I may have at the time, but it was probably whatever version was bundled with FITS. I could try it again once a new version of FITS is released with #209 included.