[WIP] fix #13790 #13792

timotheecour · 2020-03-28T12:15:45Z

WIP, stay tuned

fixes ptr char implicitly converts to cstring, resulting in undefined behavior #13790
bootstrap works
some speedups in affected areas of code

TODO

fix tests
maybe add --legacy:implicitPtrCharToCstring
maybe generate warning instead of error for now

TODO after PR

get completely rid of nimNoArrayToCstringConversion branches in code (but leave the symbol defined) which was introduced in 3308d26; it was introduced in 2017 so not needed for bootstrap anymore and it will simplify a lot of code; that nimNoArrayToCstringConversion was in same spirit as this PR and is related
(EDIT: i did that in another PR in the meantime)

krux02

Conversions are done with cstring(expr) or cast[cstring](expr). Don't introduce new identifiers for conversion.

krux02 · 2020-03-28T12:53:50Z

lib/system.nim

+template toCstring*(a: ptr char): cstring = cast[cstring](a)
+template toCstring*[N](a: ptr array[N, char]): cstring = cast[cstring](a)
+template toCstring*[N](a: array[N, char]): cstring = cast[cstring](a.addr)
+template toCstring*[N](a: ptr UncheckedArray[char]): cstring = cast[cstring](a)
+template toCstring*(a: UncheckedArray[char]): cstring = cast[cstring](a.addr)


Remove all of this. Do not add new stuff to system.nim unless it is required. And if it is required, justify why it needs to be in system.nim.

Clyybber · 2020-03-28T14:19:52Z

compiler/lexer.nim

@@ -915,7 +915,7 @@ proc getSymbol(L: var TLexer, tok: var TToken) =
 proc endOperator(L: var TLexer, tok: var TToken, pos: int,
                 hash: Hash) {.inline.} =
  var h = !$hash
-  tok.ident = L.cache.getIdent(addr(L.buf[L.bufpos]), pos - L.bufpos, h)
+  tok.ident = L.cache.getIdent(addr(L.buf[L.bufpos]).toCstring, pos - L.bufpos, h)


Suggested change

tok.ident = L.cache.getIdent(addr(L.buf[L.bufpos]).toCstring, pos - L.bufpos, h)

tok.ident = L.cache.getIdent(L.buf[L.bufpos].addr.toCstring, pos - L.bufpos, h)

to be consistent with the line in 904. Also I agree with krux02 here, instead of toCstring you should use casts manually and remove toCstring.

Clyybber · 2020-03-28T14:22:41Z

lib/system.nim

+proc setLen*(s: var string, newlen: Natural, isInit = true) {.
  magic: "SetLengthStr", noSideEffect.}
-  ## Sets the length of string `s` to `newlen`.
+  ## Sets the length of string `s` to `newlen`. If `isInit == true` and
+  ## If `newlen > s.len`, when `isInit == true`, new entries are '\0' including
+  ## the unreachable terminator n[s.len]. when `isInit == false`, only the
+  ## terminator is '\0' (for optimization). TODO: implement this.


Please don't change setLen's semantics in this PR.

Clyybber · 2020-03-28T14:23:06Z

lib/system/dollars.nim

@@ -1,3 +1,5 @@
+const notJSnotNims = not defined(js) and not defined(nimscript)


Inline this const please

Clyybber · 2020-03-28T14:23:46Z

lib/system/dollars.nim

+proc strAppend*(result: var string, a: ptr char, n: int) {.inline.} =
+  ## appends `n` `char`'s from `a` to `result`, efficiently
+  ## note: should use a Slice[char]
+  # D20200328T022947
+  let old = result.len
+  result.setLen(old + n, isInit = false) # optimized here
+  when notJSnotNims:
+    copyMem(result[old].addr.toCstring, a, n)
+  else:
+    let a2 = cast[cstring](a)
+    for i in 0..<n: result[old+i] = a2[i]
+
+proc strAppend*(result: var string, a: cstring) {.inline.} =
+  # TODO: replace `CStrToStr` ?
+  strAppend(result, cast[ptr char](a.unsafeAddr), a.len)


Doesn't belong in this PR either.

lib/system/repr.nim

Clyybber · 2020-03-28T14:26:21Z

lib/system/repr.nim

-    return $buf
+  const N = (pointer.sizeof * 2) + 2 + 1 # 0x + hex addr + '\0'
+  var buf {.noinit.}: array[N, char]
+  result.strAppend buf[0].addr, c_sprintf(buf.toCstring, "%p", x)


Suggested change

result.strAppend buf[0].addr, c_sprintf(buf.toCstring, "%p", x)

discard c_sprintf(buf, "%p", x)

return $buf

As strAppend really doesn't belong in this PR

Clyybber

Agree with krux02 in that the toCstring templates should be inlined and removed from system.nim
Also please remove the unrelated strAppend and the setLen change from this PR.
Other than that nice :)

juancarlospaco · 2020-03-29T14:22:14Z

lib/system/dollars.nim

+  ## append `n` `char`'s from `a` to `result`, efficiently;
+  ## `\0` are insignificant.
+  ## note: should use a Slice[char]
+  # D20200328T022947


juancarlospaco · 2020-03-29T14:37:18Z

I do not understand why all those templates need to be public,
even template names are longer than its code body itself.

stale · 2021-04-01T09:19:27Z

This pull request has been automatically marked as stale because it has not had recent activity. If you think it is still a valid PR, please rebase it on the latest devel; otherwise it will be closed. Thank you for your contributions.

…to cstring

stale · 2022-04-17T10:06:48Z

This pull request has been automatically marked as stale because it has not had recent activity. If you think it is still a valid PR, please rebase it on the latest devel; otherwise it will be closed. Thank you for your contributions.

krux02 suggested changes Mar 28, 2020

View reviewed changes

Clyybber reviewed Mar 28, 2020

View reviewed changes

lib/system/repr.nim Outdated Show resolved Hide resolved

Clyybber reviewed Mar 28, 2020

View reviewed changes

timotheecour force-pushed the pr_dollar_ptr_char branch from 76619b8 to 6d331bf Compare March 28, 2020 21:50

juancarlospaco reviewed Mar 29, 2020

View reviewed changes

timotheecour mentioned this pull request Mar 30, 2020

[duplicate] assigning var to local let does not properly copy in default gc #13771

Closed

timotheecour mentioned this pull request Jun 25, 2020

Add byte-array-to-string conversion to system module #14810

Open

stale bot added the stale Staled PR/issues; remove the label after fixing them label Apr 1, 2021

timotheecour mentioned this pull request Apr 16, 2021

Fix buffer-overrun bug in net #17728

Merged

_

5f96df0

stale bot removed the stale Staled PR/issues; remove the label after fixing them label Apr 16, 2021

timotheecour added 12 commits April 16, 2021 11:44

fix nim-lang#13790 ptr char (+friends) should not implicitly convert …

2332b7f

…to cstring

_

e922fc9

bootstrap works

bfedaeb

fix tests

6f76e1c

fix tests

ca0ecc9

fix tests; address some comments

5fe7106

fixup

fa34d85

fix test

1a39c8b

add tmisc_imports for lazy testing of modules without real tests yet

5276e68

fix test

7c23658

fix tests; introduce toString0

bd37982

fixup

835f135

PRTEMP

4f34842

timotheecour force-pushed the pr_dollar_ptr_char branch from 488e35c to 4f34842 Compare April 16, 2021 19:23

timotheecour added 2 commits April 16, 2021 14:54

fix tests

66c2d70

fixup

c890423

stale bot added the stale Staled PR/issues; remove the label after fixing them label Apr 17, 2022

stale bot closed this May 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] fix #13790 #13792

[WIP] fix #13790 #13792

timotheecour commented Mar 28, 2020 •

edited

Loading

krux02 left a comment

krux02 Mar 28, 2020

Clyybber Mar 28, 2020

Clyybber Mar 28, 2020

Clyybber Mar 28, 2020

Clyybber Mar 28, 2020

Clyybber Mar 28, 2020 •

edited

Loading

Clyybber left a comment •

edited

Loading

juancarlospaco Mar 29, 2020 •

edited

Loading

juancarlospaco commented Mar 29, 2020 •

edited

Loading

stale bot commented Apr 1, 2021

stale bot commented Apr 17, 2022

	tok.ident = L.cache.getIdent(addr(L.buf[L.bufpos]).toCstring, pos - L.bufpos, h)
	tok.ident = L.cache.getIdent(L.buf[L.bufpos].addr.toCstring, pos - L.bufpos, h)

		@@ -1,3 +1,5 @@
		const notJSnotNims = not defined(js) and not defined(nimscript)

	result.strAppend buf[0].addr, c_sprintf(buf.toCstring, "%p", x)
	discard c_sprintf(buf, "%p", x)
	return $buf

[WIP] fix #13790 #13792

[WIP] fix #13790 #13792

Conversation

timotheecour commented Mar 28, 2020 • edited Loading

TODO

TODO after PR

krux02 left a comment

Choose a reason for hiding this comment

krux02 Mar 28, 2020

Choose a reason for hiding this comment

Clyybber Mar 28, 2020

Choose a reason for hiding this comment

Clyybber Mar 28, 2020

Choose a reason for hiding this comment

Clyybber Mar 28, 2020

Choose a reason for hiding this comment

Clyybber Mar 28, 2020

Choose a reason for hiding this comment

Clyybber Mar 28, 2020 • edited Loading

Choose a reason for hiding this comment

Clyybber left a comment • edited Loading

Choose a reason for hiding this comment

juancarlospaco Mar 29, 2020 • edited Loading

Choose a reason for hiding this comment

juancarlospaco commented Mar 29, 2020 • edited Loading

stale bot commented Apr 1, 2021

stale bot commented Apr 17, 2022

timotheecour commented Mar 28, 2020 •

edited

Loading

Clyybber Mar 28, 2020 •

edited

Loading

Clyybber left a comment •

edited

Loading

juancarlospaco Mar 29, 2020 •

edited

Loading

juancarlospaco commented Mar 29, 2020 •

edited

Loading