Messed up output when using -d:useMalloc on program with block statement, options, and json-parsing #21540

tsoj · 2023-03-18T01:43:55Z

Description

When running the following program with nim r -d:useMalloc main.nim it produces unexpected results with the newest version, but it works fine with Nim 1.6.12.

import std/[
    json,
    options
]

let initialFEN = block:
    let initialFEN = some("""{"initialFen": "startpos"}""".parseJson{"initialFen"}.getStr)
    echo "<", initialFEN.get, ">"
    initialFEN.get

echo "<", initialFEN, ">"

Nim Version

Nim Compiler Version 1.9.1 [Linux: amd64]
Compiled at 2023-03-17
Copyright (c) 2006-2023 by Andreas Rumpf

git hash: b5ee81f
active boot switches: -d:release

Current Output

<startpos>
<�§�â*>

Expected Output

<startpos>
<startpos>

Possible Solution

No response

Additional Information

Workaround:

import std/[
    json,
    options
]

let initialFEN = block:
    let initialFEN = some("""{"initialFen": "startpos"}""".parseJson{"initialFen"}.getStr)
    echo "<", initialFEN.get, ">"
    let tmp = initialFEN.get # this temporary variable seems to mitigate the issue
    tmp

echo "<", initialFEN, ">"

The text was updated successfully, but these errors were encountered:

ghost · 2023-03-18T04:22:21Z

Simplified a bit (x2, edited), seems to be related to get of Option returning a lent + copyMem

type
  Option = object
    val: string
    has: bool

proc some(val: string): Option =
  result.has = true
  result.val = val

# Remove lent and it works
proc get(self: Option): lent string =
  result = self.val

type
  StringStream = ref object
    data: string
    pos: int

proc readAll(s: StringStream): string =
  result = newString(s.data.len)
  copyMem(addr(result[0]), addr(s.data[0]), s.data.len)

proc newStringStream(s: string = ""): StringStream =
  new(result)
  result.data = s

proc parseJson(s: string): string =
  let stream = newStringStream(s)
  result = stream.readAll()

proc main = 
  let initialFEN = block:
    let initialFEN = some parseJson("startpos")
    initialFEN.get

  echo initialFEN

main()

ringabout · 2023-04-12T04:15:53Z

Here is what the valgrind says

==19365== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)
==19365== 
==19365== 1 errors in context 1 of 3:
==19365== Invalid read of size 1
==19365==    at 0x4852A10: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19365==    by 0x10F7F0: memcpy (string_fortified.h:29)
==19365==    by 0x10F7F0: nimCopyMem (memory.nim:10)
==19365==    by 0x10F7F0: copyMem__system_1752 (system.nim:1966)
==19365==    by 0x10F7F0: nimAsgnStrV2 (strs_v2.nim:156)
==19365==    by 0x10F8AB: eqcopy___stdZassertions_33 (assertions.nim:25)
==19365==    by 0x1111DA: main__test50_59 (assertions.nim:25)
==19365==    by 0x11131E: NimMainModule (test2.nim:38)
==19365==    by 0x11136C: NimMainInner (assertions.nim:49)
==19365==    by 0x111383: NimMain (assertions.nim:60)
==19365==    by 0x1113A9: main (assertions.nim:68)
==19365==  Address 0x4a92110 is 16 bytes inside a block of size 17 free'd
==19365==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19365==    by 0x10AB3D: deallocImpl__system_1772 (malloc.nim:28)
==19365==    by 0x10AB4F: deallocSharedImpl__system_1785 (malloc.nim:45)
==19365==    by 0x10AB61: deallocShared (memalloc.nim:308)
==19365==    by 0x10AB8C: eqdestroy___stdZassertions_30 (assertions.nim:25)
==19365==    by 0x110FE9: eqdestroy___test50_65 (test2.nim:33)
==19365==    by 0x1111BB: main__test50_59 (test2.nim:33)
==19365==    by 0x11131E: NimMainModule (test2.nim:38)
==19365==    by 0x11136C: NimMainInner (assertions.nim:49)
==19365==    by 0x111383: NimMain (assertions.nim:60)
==19365==    by 0x1113A9: main (assertions.nim:68)
==19365==  Block was alloc'd at
==19365==    at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19365==    by 0x10AB19: alloc0Impl__system_1770 (malloc.nim:11)
==19365==    by 0x10AB2B: allocShared0Impl__system_1783 (malloc.nim:37)
==19365==    by 0x10F7B0: nimAsgnStrV2 (strs_v2.nim:151)
==19365==    by 0x10F8AB: eqcopy___stdZassertions_33 (assertions.nim:25)
==19365==    by 0x11085C: some__test50_4 (assertions.nim:25)
==19365==    by 0x11112C: main__test50_59 (test2.nim:33)
==19365==    by 0x11131E: NimMainModule (test2.nim:38)
==19365==    by 0x11136C: NimMainInner (assertions.nim:49)
==19365==    by 0x111383: NimMain (assertions.nim:60)
==19365==    by 0x1113A9: main (assertions.nim:68)
==19365== 
==19365== 
==19365== 1 errors in context 2 of 3:
==19365== Invalid read of size 8
==19365==    at 0x485298D: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19365==    by 0x10F7F0: memcpy (string_fortified.h:29)
==19365==    by 0x10F7F0: nimCopyMem (memory.nim:10)
==19365==    by 0x10F7F0: copyMem__system_1752 (system.nim:1966)
==19365==    by 0x10F7F0: nimAsgnStrV2 (strs_v2.nim:156)
==19365==    by 0x10F8AB: eqcopy___stdZassertions_33 (assertions.nim:25)
==19365==    by 0x1111DA: main__test50_59 (assertions.nim:25)
==19365==    by 0x11131E: NimMainModule (test2.nim:38)
==19365==    by 0x11136C: NimMainInner (assertions.nim:49)
==19365==    by 0x111383: NimMain (assertions.nim:60)
==19365==    by 0x1113A9: main (assertions.nim:68)
==19365==  Address 0x4a92108 is 8 bytes inside a block of size 17 free'd
==19365==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19365==    by 0x10AB3D: deallocImpl__system_1772 (malloc.nim:28)
==19365==    by 0x10AB4F: deallocSharedImpl__system_1785 (malloc.nim:45)
==19365==    by 0x10AB61: deallocShared (memalloc.nim:308)
==19365==    by 0x10AB8C: eqdestroy___stdZassertions_30 (assertions.nim:25)
==19365==    by 0x110FE9: eqdestroy___test50_65 (test2.nim:33)
==19365==    by 0x1111BB: main__test50_59 (test2.nim:33)
==19365==    by 0x11131E: NimMainModule (test2.nim:38)
==19365==    by 0x11136C: NimMainInner (assertions.nim:49)
==19365==    by 0x111383: NimMain (assertions.nim:60)
==19365==    by 0x1113A9: main (assertions.nim:68)
==19365==  Block was alloc'd at
==19365==    at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19365==    by 0x10AB19: alloc0Impl__system_1770 (malloc.nim:11)
==19365==    by 0x10AB2B: allocShared0Impl__system_1783 (malloc.nim:37)
==19365==    by 0x10F7B0: nimAsgnStrV2 (strs_v2.nim:151)
==19365==    by 0x10F8AB: eqcopy___stdZassertions_33 (assertions.nim:25)
==19365==    by 0x11085C: some__test50_4 (assertions.nim:25)
==19365==    by 0x11112C: main__test50_59 (test2.nim:33)
==19365==    by 0x11131E: NimMainModule (test2.nim:38)
==19365==    by 0x11136C: NimMainInner (assertions.nim:49)
==19365==    by 0x111383: NimMain (assertions.nim:60)
==19365==    by 0x1113A9: main (assertions.nim:68)
==19365== 
==19365== 
==19365== 1 errors in context 3 of 3:
==19365== Invalid read of size 8
==19365==    at 0x10F723: nimAsgnStrV2 (strs_v2.nim:26)
==19365==    by 0x10F8AB: eqcopy___stdZassertions_33 (assertions.nim:25)
==19365==    by 0x1111DA: main__test50_59 (assertions.nim:25)
==19365==    by 0x11131E: NimMainModule (test2.nim:38)
==19365==    by 0x11136C: NimMainInner (assertions.nim:49)
==19365==    by 0x111383: NimMain (assertions.nim:60)
==19365==    by 0x1113A9: main (assertions.nim:68)
==19365==  Address 0x4a92100 is 0 bytes inside a block of size 17 free'd
==19365==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19365==    by 0x10AB3D: deallocImpl__system_1772 (malloc.nim:28)
==19365==    by 0x10AB4F: deallocSharedImpl__system_1785 (malloc.nim:45)
==19365==    by 0x10AB61: deallocShared (memalloc.nim:308)
==19365==    by 0x10AB8C: eqdestroy___stdZassertions_30 (assertions.nim:25)
==19365==    by 0x110FE9: eqdestroy___test50_65 (test2.nim:33)
==19365==    by 0x1111BB: main__test50_59 (test2.nim:33)
==19365==    by 0x11131E: NimMainModule (test2.nim:38)
==19365==    by 0x11136C: NimMainInner (assertions.nim:49)
==19365==    by 0x111383: NimMain (assertions.nim:60)
==19365==    by 0x1113A9: main (assertions.nim:68)
==19365==  Block was alloc'd at
==19365==    at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==19365==    by 0x10AB19: alloc0Impl__system_1770 (malloc.nim:11)
==19365==    by 0x10AB2B: allocShared0Impl__system_1783 (malloc.nim:37)
==19365==    by 0x10F7B0: nimAsgnStrV2 (strs_v2.nim:151)
==19365==    by 0x10F8AB: eqcopy___stdZassertions_33 (assertions.nim:25)
==19365==    by 0x11085C: some__test50_4 (assertions.nim:25)
==19365==    by 0x11112C: main__test50_59 (test2.nim:33)
==19365==    by 0x11131E: NimMainModule (test2.nim:38)
==19365==    by 0x11136C: NimMainInner (assertions.nim:49)
==19365==    by 0x111383: NimMain (assertions.nim:60)
==19365==    by 0x1113A9: main (assertions.nim:68)
==19365== 
==19365== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)

ringabout · 2023-04-12T05:05:38Z

related: #20107

ringabout · 2023-04-18T14:35:57Z

var
  initialFEN
  :tmpD
try:
  `=copy`(initialFEN, block :tmp:
    var
      seed
      :tmpD_1
    :tmpD =
      seed = some do:
        :tmpD_1 = parseJson("startpos")
        :tmpD_1
      get(seed)
    `=destroy`(:tmpD_1)
    `=destroy_1`(seed)
    :tmpD)
  echo [initialFEN]
finally:
  `=destroy`(initialFEN)

The problem is that :tmpD borrows from seed, which is destroyed before:tmpD is assigned to variables outside the block.

ringabout · 2023-04-18T14:47:14Z

A possible soultion I have in mind is somehow to deref the block at the sempass/transf, not in the codegen phase so that the tmp will get the derefed value without worrying about the borrowed objects.

…unction properly

…unction properly (#21688) * fixes #21540; deref block at transf phase to make injectdestructors function properly * add a test case * add one more test * fixes the type of block * transform block

…uctors function properly (nim-lang#21688) * fixes nim-lang#21540; deref block at transf phase to make injectdestructors function properly * add a test case * add one more test * fixes the type of block * transform block

…unction properly (#21688) * fixes #21540; deref block at transf phase to make injectdestructors function properly * add a test case * add one more test * fixes the type of block * transform block (cherry picked from commit 9cb06d3)

…uctors function properly (nim-lang#21688) * fixes nim-lang#21540; deref block at transf phase to make injectdestructors function properly * add a test case * add one more test * fixes the type of block * transform block

tsoj changed the title ~~Think about the title, twice.~~ Garbled output when using -d:useMalloc on program with block statement, options, and json-parsing Mar 18, 2023

tsoj changed the title ~~Garbled output when using -d:useMalloc on program with block statement, options, and json-parsing~~ Messed up output when using -d:useMalloc on program with block statement, options, and json-parsing Mar 18, 2023

ringabout added the ARC/ORC Memory Management label Apr 12, 2023

ringabout self-assigned this Apr 12, 2023

Araq added the Showstopper label Apr 14, 2023

ringabout added a commit that referenced this issue Apr 19, 2023

fixes #21540; deref block at transf phase to make injectdestructors f…

3cfc7ee

…unction properly

ringabout mentioned this issue Apr 19, 2023

fixes #21540; deref block at transf phase to make injectdestructors function properly #21688

Merged

Araq closed this as completed in #21688 Apr 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Messed up output when using -d:useMalloc on program with block statement, options, and json-parsing #21540

Messed up output when using -d:useMalloc on program with block statement, options, and json-parsing #21540

tsoj commented Mar 18, 2023 •

edited

Loading

ghost commented Mar 18, 2023 •

edited by ghost

Loading

ringabout commented Apr 12, 2023

ringabout commented Apr 12, 2023

ringabout commented Apr 18, 2023

ringabout commented Apr 18, 2023 •

edited

Loading

Messed up output when using -d:useMalloc on program with block statement, options, and json-parsing #21540

Messed up output when using -d:useMalloc on program with block statement, options, and json-parsing #21540

Comments

tsoj commented Mar 18, 2023 • edited Loading

Description

Nim Version

Current Output

Expected Output

Possible Solution

Additional Information

ghost commented Mar 18, 2023 • edited by ghost Loading

ringabout commented Apr 12, 2023

ringabout commented Apr 12, 2023

ringabout commented Apr 18, 2023

ringabout commented Apr 18, 2023 • edited Loading

tsoj commented Mar 18, 2023 •

edited

Loading

ghost commented Mar 18, 2023 •

edited by ghost

Loading

ringabout commented Apr 18, 2023 •

edited

Loading