Use After Free Error // Heap Overflow

[dsmithington]

Describe the bug
When using a debug build (-DCMAKE_BUILD_TYPE=Debug) and a recursive directory iterator. If the directory to iterate is fairly large, a use after free or heap overflow occurs

To Reproduce

1. Debug Build - Enable address sanitizer
2. Use a recursive directory iterator on a directory that has at least 12 levels of sub
   directories

Expected behavior
After some execution you should see a crash located at filesystem.hpp:4570:

==32507==ERROR: AddressSanitizer: unknown-crash on address 0x62d0011f6328 at pc 0x555e71d4f2d5 bp 0x7ffd5f191200 sp 0x7ffd5f1911f0
READ of size 280 at 0x62d0011f6328 thread T0
.
.
.
0x62d0011f6430 is located 0 bytes to the right of 32816-byte region [0x62d0011ee400,0x62d0011f6430)
allocated by thread T0 here:
    #0 0x7f25c1f1db50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f25c064c9b5 in opendir (/lib/x86_64-linux-gnu/libc.so.6+0xdf9b5)

SUMMARY: AddressSanitizer: unknown-crash {PATH}/filesystem.hpp:4570 in i_readdir_r
Shadow bytes around the buggy address:
  0x0c5a80236c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c5a80236c60: 00 00 00 00 00[00]00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c5a80236c80: 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa fa
  0x0c5a80236c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80236ca0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c5a80236cb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==32507==ABORTING

Additional context
For some reason it is listed as an unknown-crash but it has all the symptoms of a heap overflow or a use-after-free error.

[gulrak]

Yeah, sorry, accidentally I yesterday recognized, that the recursive_directory_iterator has some more issues, and since then work on a fix. I will use this ticket to track it.

[gulrak]

Did run a test with the fixes over a tree with depth 16, ~700k files in ~5k directories with no error. Still will run more tests and will add a bunch of additional tests to the testsuite before I close this and tag the release on sunday.

[gulrak]

Okay, adding more test showed another issue when using pop(), that I just fixed. The version on master looks good and I ran sanitizer tests on macOS and Ubuntu with huge directory trees.

[gulrak]

Released with v1.1.2

[gulrak]

Sadly I found an additional issue that could lead to a crash on Linux. All has to do with my old code trying to use readdir_r or readdir in a unified way depending on the system. That code has some issues and readdir_r is deprecated anyways, so I rework that code and add some performance boost for deep iterations on the way.

Sorry for the short lived v1.1.2, expect an additional v1.1.3 at latest tomorrow.

[gulrak]

Reworked POSIX implementation with (tagged with wrong issue number, my bad): cc476ac

[gulrak]

Released with v1.1.4