memory corruption

[leilchen]

I'm building and running crash-7.1.9. But the tool itself crashes because of memory overflow. The core dump point is at free time, which makes it a little tricky to debug. The error message looks like this:

crash 7.1.9
Copyright (C) 2002-2016 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

*** Error in `/tmp/crash': free(): invalid pointer: 0x0000000001022270 ***
Aborted (core dumped)

The following patch can fix the problem:

`--- /tmp/crash/crash-7.1.9/filesys.c 2017-04-21 02:54:33.000000000 +0800
+++ crash-7.1.9/filesys.c 2017-08-31 06:18:32.000040000 +0800
@@ -308,6 +308,7 @@
#define CREATE 1
#define DESTROY 0
#define DEFAULT_SEARCHDIRS 5
+#define EXTRA_SEARCHDIRS 5

static char **
build_searchdirs(int create, int *preferred)
@@ -340,11 +341,11 @@
*preferred = 0;

    /*

•    *  Allow, at a minimum, the defaults plus an extra three directories
  

•    *  Allow, at a minimum, the defaults plus EXTRA_SEARCHDIRS directories
     *  for the two possible /usr/src/redhat/BUILD/kernel-xxx locations
     *  plus the Red Hat debug directory.
     */
  

•   cnt = DEFAULT_SEARCHDIRS + 3;
  

•   cnt = DEFAULT_SEARCHDIRS + EXTRA_SEARCHDIRS;
  
     if ((dirp = opendir("/usr/src"))) {
             for (dp = readdir(dirp); dp != NULL; dp = readdir(dirp))
  

`

[crash-utility]

----- Original Message -----

I'm building and running crash-7.1.9. But the tool itself crashes because of memory overflow. The core dump point is at free time, which makes it a little tricky to debug. The error message looks like this:

You should be able to run gdb on the core file and get a backtrace. But anyway, with your patch applied, can you show me what "crash -d1" shows? Thanks, Dave

…

crash 7.1.9 Copyright (C) 2002-2016 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. *** Error in `/tmp/crash': free(): invalid pointer: 0x0000000001022270 *** Aborted (core dumped) The following patch can fix the problem: `--- /tmp/crash/crash-7.1.9/filesys.c 2017-04-21 02:54:33.000000000 +0800 +++ crash-7.1.9/filesys.c 2017-08-31 06:18:32.000040000 +0800 @@ -308,6 +308,7 @@ #define CREATE 1 #define DESTROY 0 #define DEFAULT_SEARCHDIRS 5 +#define EXTRA_SEARCHDIRS 5 static char ** build_searchdirs(int create, int *preferred) @@ -340,11 +341,11 @@ *preferred = 0; /* - * Allow, at a minimum, the defaults plus an extra three directories + * Allow, at a minimum, the defaults plus EXTRA_SEARCHDIRS directories * for the two possible /usr/src/redhat/BUILD/kernel-xxx locations * plus the Red Hat debug directory. */ - cnt = DEFAULT_SEARCHDIRS + 3; + cnt = DEFAULT_SEARCHDIRS + EXTRA_SEARCHDIRS; if ((dirp = opendir("/usr/src"))) { for (dp = readdir(dirp); dp != NULL; dp = readdir(dirp)) ` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #20

[crash-utility]

----- Original Message -----

----- Original Message ----- > I'm building and running crash-7.1.9. But the tool itself crashes because of > memory overflow. The core dump point is at free time, which makes it a > little tricky to debug. The error message looks like this: You should be able to run gdb on the core file and get a backtrace. But anyway, with your patch applied, can you show me what "crash -d1" shows? Thanks, Dave

Also, can you show me the output of "ls /usr/src" on your host machine where the crash occurs? I'm trying to figure out how the malloc'd searchdirs[] memory gets corrupted, and I'm thinking that just bumping the extras from 3 to 5 is papering over the problem. Thanks, Dave

…

> > crash 7.1.9 > Copyright (C) 2002-2016 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005, 2011 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for > details. > > *** Error in `/tmp/crash': free(): invalid pointer: 0x0000000001022270 *** > Aborted (core dumped) > > The following patch can fix the problem: > > `--- /tmp/crash/crash-7.1.9/filesys.c 2017-04-21 02:54:33.000000000 > +0800 > +++ crash-7.1.9/filesys.c 2017-08-31 06:18:32.000040000 +0800 > @@ -308,6 +308,7 @@ > #define CREATE 1 > #define DESTROY 0 > #define DEFAULT_SEARCHDIRS 5 > +#define EXTRA_SEARCHDIRS 5 > > static char ** > build_searchdirs(int create, int *preferred) > @@ -340,11 +341,11 @@ > *preferred = 0; > > /* > - * Allow, at a minimum, the defaults plus an extra three > directories > + * Allow, at a minimum, the defaults plus EXTRA_SEARCHDIRS > directories > * for the two possible /usr/src/redhat/BUILD/kernel-xxx locations > * plus the Red Hat debug directory. > */ > - cnt = DEFAULT_SEARCHDIRS + 3; > + cnt = DEFAULT_SEARCHDIRS + EXTRA_SEARCHDIRS; > > if ((dirp = opendir("/usr/src"))) { > for (dp = readdir(dirp); dp != NULL; dp = readdir(dirp)) > ` > > -- > You are receiving this because you are subscribed to this thread. > Reply to this email directly or view it on GitHub: > #20

[crash-utility]

----- Original Message -----

----- Original Message ----- > > > ----- Original Message ----- > > I'm building and running crash-7.1.9. But the tool itself crashes because > > of > > memory overflow. The core dump point is at free time, which makes it a > > little tricky to debug. The error message looks like this: > > You should be able to run gdb on the core file and get a backtrace. > But anyway, with your patch applied, can you show me what "crash -d1" > shows? > > Thanks, > Dave Also, can you show me the output of "ls /usr/src" on your host machine where the crash occurs? I'm trying to figure out how the malloc'd searchdirs[] memory gets corrupted, and I'm thinking that just bumping the extras from 3 to 5 is papering over the problem. Thanks, Dave

And I'm also wondering whether somehow the 1500 bytes in dirbuf[BUFSIZE] in build_searchdirs() is getting overrun and spilling into the searchdirs allocated memory? Instead of bumping the count from 3 to 5, what happens if you patch the dirbuf[] size, something like this: - char dirbuf[BUFSIZE]; + char dirbuf[BUFSIZE*2]; I just don't understand how you came to the conclusion to add the 3-to-5 change? That code hasn't changed in years, and I can't figure out what's different about your host machine setup. Dave

…

> > > > > > crash 7.1.9 > > Copyright (C) 2002-2016 Red Hat, Inc. > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > > Copyright (C) 1999-2006 Hewlett-Packard Co > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > > Copyright (C) 2005, 2011 NEC Corporation > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > This program is free software, covered by the GNU General Public License, > > and you are welcome to change it and/or distribute copies of it under > > certain conditions. Enter "help copying" to see the conditions. > > This program has absolutely no warranty. Enter "help warranty" for > > details. > > > > *** Error in `/tmp/crash': free(): invalid pointer: 0x0000000001022270 > > *** > > Aborted (core dumped) > > > > The following patch can fix the problem: > > > > `--- /tmp/crash/crash-7.1.9/filesys.c 2017-04-21 02:54:33.000000000 > > +0800 > > +++ crash-7.1.9/filesys.c 2017-08-31 06:18:32.000040000 +0800 > > @@ -308,6 +308,7 @@ > > #define CREATE 1 > > #define DESTROY 0 > > #define DEFAULT_SEARCHDIRS 5 > > +#define EXTRA_SEARCHDIRS 5 > > > > static char ** > > build_searchdirs(int create, int *preferred) > > @@ -340,11 +341,11 @@ > > *preferred = 0; > > > > /* > > - * Allow, at a minimum, the defaults plus an extra three > > directories > > + * Allow, at a minimum, the defaults plus EXTRA_SEARCHDIRS > > directories > > * for the two possible /usr/src/redhat/BUILD/kernel-xxx > > locations > > * plus the Red Hat debug directory. > > */ > > - cnt = DEFAULT_SEARCHDIRS + 3; > > + cnt = DEFAULT_SEARCHDIRS + EXTRA_SEARCHDIRS; > > > > if ((dirp = opendir("/usr/src"))) { > > for (dp = readdir(dirp); dp != NULL; dp = readdir(dirp)) > > ` > > > > -- > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly or view it on GitHub: > > #20 >

[leilchen]

Hi Dave,
I'm running a customized distro which doesn't have /usr/src. The rational behind EXTRA=5 is:

• line 401, the cnt is set to DEFAULT_SEARCHDIRS
• from line 404, kernel/redhat kernel v1/redhat kernel v2/redhat debug directories are added, i.e. 4 new items get added.
• Besides, line 454 actually accesses DEFAULT_SEARCHDIRS+5 item.

But why there's no bug report on other distro? I have no idea.

The backtrace is as below:
(gdb) bt
#0 0x00007ffff6cd5da5 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff6cd7558 in __GI_abort () at abort.c:90
#2 0x00007ffff6d16d7b in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff6e21de8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007ffff6d1e192 in malloc_printerr (ptr=0xfca270, str=0x7ffff6e1f4de "free(): invalid pointer", action=3) at malloc.c:4964
#4 _int_free (av=0x7ffff7044720 <main_arena>, p=0xfca260, have_lock=0) at malloc.c:3796
#5 0x00000000004aa159 in ?? ()
#6 0x00000000004aa97a in fd_init ()
#7 0x0000000000464350 in main ()

Thanks,
Lei Chen

[crash-utility]

Ok thanks -- committed!

BTW, in the future, please post patches on the crash utility mailing list:
[email protected]
It's a moderated list, but if you choose not to enroll, I will see any legitimate patch postings among the spam, and I will accept it.

[crash-utility]

If you want to join the list, it's here:
https://www.redhat.com/mailman/listinfo/crash-utility