crash: v2 module syment space malloc (-1037442748 symbols): Cannot allocate memory

[tdhammer]

I am working with a target device having an i386 CPU and Gentoo-based GNU/Linux OS.
I have enabled kdump/kexec to attempt capture of an issue customers are encountering but cannot reproduce in-house. In anticipation of collecting dump files, I would like to be able to utilize the crash utility. I was able to "cross" compile the utility to run on our 64-bit Ubuntu 18.04 workstations but read the 32-bit files from our device.

However, when attempting to run the utility, I encounter this error.
Complete output:

$ ~/Downloads/crash-7.2.7_X86/crash vmlinux vmcore-201909301633_SOFTLOCKUP 

crash 7.2.7
Copyright (C) 2002-2019  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=i686-pc-linux-gnu"...

WARNING: kernel version inconsistency between vmlinux and dumpfile

please wait... (gathering module symbol data)   
crash: v2 module syment space malloc (-1037442748 symbols): Cannot allocate memory

This message is printed from symbols.c::store_modules_symbols_v2()

        if ((st->ext_module_symtable = (struct syment *)
             calloc(total, sizeof(struct syment))) == NULL)
                error(FATAL, "v2 module syment space malloc (%ld symbols): %s\n",
			total, strerror(errno));

I will attempt to do some additional investigation, but am hoping someone can identify something I did incorrectly in my build process to get past this or a fix that is not yet available in the repo.

Thanks!
.Tim

[crash-utility]

----- Original Message -----

I am working with a target device having an i386 CPU and Gentoo-based GNU/Linux OS. I have enabled kdump/kexec to attempt capture of an issue customers are encountering but cannot reproduce in-house. In anticipation of collecting dump files, I would like to be able to utilize the crash utility. I was able to "cross" compile the utility to run on our 64-bit Ubuntu 18.04 workstations but read the 32-bit files from our device.

Right, so I'm presuming that you ran "make TARGET=x86" in a virgin source tree on an x86_64 host, and that you're running the resultant binary on the x86_64 host.

However, when attempting to run the utility, I encounter this error. Complete output: ``` $ ~/Downloads/crash-7.2.7_X86/crash vmlinux vmcore-201909301633_SOFTLOCKUP crash 7.2.7 Copyright (C) 2002-2019 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=i686-pc-linux-gnu"... WARNING: kernel version inconsistency between vmlinux and dumpfile

This shouldn't happen. Run "crash -d4 ..." and see why it doesn't believe that the vmlinux file and the dumpfile are from the same kernel version. If the vmlinux and dumpfile are from different kernel versions, then it's likely that it is related to any subsequent errors.

please wait... (gathering module symbol data) crash: v2 module syment space malloc (-1037442748 symbols): Cannot allocate memory ``` This message is printed from `symbols.c::store_modules_symbols_v2()` ``` if ((st->ext_module_symtable = (struct syment *) calloc(total, sizeof(struct syment))) == NULL) error(FATAL, "v2 module syment space malloc (%ld symbols): %s\n", total, strerror(errno)); ``` I will attempt to do some additional investigation, but am hoping someone can identify something I did incorrectly in my build process to get past this or a fix that is not yet available in the repo.

You didn't mention what kernel version the vmcore/dumpfile is, but w/respect to the crash utility sources, there's nothing newer than what was just released in crash-7.2.7. However, since I don't even bother testing crash against 32-bit x86 kernels anymore (since Red Hat stopped supporting them in RHEL7), it's also entirely possible that something in the 32-bit x86 kernel has changed. Anyway, if it's not related to the vmlinux/dumpfile inconsistency, then you'll have to figure out why it's getting the bogus "total" from module_init(), which it determines by looping through the installed modules. Dave

…

Thanks! .Tim -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #40

[crash-utility]

BTW, if your embedded kernel doesn't load any kernel modules, you could try
using the "crash --no_modules ..." argument and see if it manages to make it
through the initialization sequence. But it still would be worth it to determine why
the kernel versions don't match and why module_init() is coming up with the invalid
total.