std/lists: O(1) concatenation of singly- and doubly linked lists. #16362

salvipeter · 2020-12-15T12:45:20Z

There is also new toSinglyLinkedList and toDoublyLinkedList
functions for conversion from openArrays, similarly
to toHashSet or toTable.

Araq · 2020-12-15T13:50:01Z

Thanks! Please add .since annotations, tests (edit tests/stdlib/tlists.nim) and a changelog entry.

Clyybber · 2020-12-15T14:08:36Z

I think these should be overloads of append/prepend.

salvipeter · 2020-12-15T14:22:44Z

Added the annotations (I hope they're OK), the tests, and the changelog.
As for the naming... I also first searched for this functionality as append (and yes, a prepend version can also be done), but on the other hand, sequtils has concat.

Clyybber · 2020-12-15T14:26:37Z

Yes but sequtils.concat is not an inplace operation (and has varargs).

salvipeter · 2020-12-15T14:47:38Z

Okay, I renamed it. But thinking about it, I found that prepend is not really a good match, because always the first member changes, and it would be strange to say l1.prepend(l2), modifying l2 while l1 stays the same. We should just use l2.append(l1).

Araq · 2020-12-15T14:48:53Z

Instead of "concat" you can use "&", but I don't like using prepend/append for it.

salvipeter · 2020-12-15T14:52:39Z

I personally like the append idea, it is very straightforward naming, and consistent with other languages. But I'm open to other suggestions, and an operator (even if only as an alias) is useful. How are these questions decided? By votes?

Araq · 2020-12-15T14:56:09Z

Well & is Nim's concat operator, nothing controversial about it. We tend to vote for controversial things.

salvipeter · 2020-12-15T15:01:34Z

The problem with & that it gives the wrong impression - that it is a pure function returning a new instance. I would certainly not expect l1 & l2 to change l1 (but I would expect it to return a value!).

timotheecour · 2020-12-15T18:27:56Z

Instead of "concat" you can use "&", but I don't like using prepend/append for it.

& is outplace, not inplace, so you probably mean &=, but it's much less standard than add and an operator has no benefit here.

Almost all of stdlib uses add for inplace append, so we should use that instead of this PR's append.

ropes: add(a: var Rope; b: Rope)
ropes: add(a: var Rope; b: string)
system: add[T](x: var seq[T]; y: openArray[T])
system: add(x: var string; y: cstring)
unicode: add(s: var string; c: Rune)
macros: add(father, child: NimNode): NimNode
macros: add(father: NimNode; children: varargs[NimNode]): NimNode
...

it's also what's recommended in nep1 explicitly: https://nim-lang.github.io/Nim/nep1.html
use add instead of append

(note: append is currently only used inside lists module and the (AFAIK unused) std/chains module, we can {.deprecated: [append: add].} in future work)

salvipeter · 2020-12-15T19:16:45Z

@timotheecour has a point there... if the others agree, I'll rename it to add then.

Araq · 2020-12-16T10:00:14Z

& is outplace, not inplace, so you probably mean &=, but it's much less standard than add and an operator has no benefit here.

Indeed. However, why not instead add & which is outplace. People who are into lists are likely more into FP so "consistency with the rest of Nim" where we have inplace + dup doesn't really convince me.

salvipeter · 2020-12-16T10:04:48Z

why not instead add & which is outplace

Because the whole point of this PR is to create an O(1) concatenation :)
We can add an outplace & operator, but it will be O(n).

salvipeter · 2020-12-16T11:43:29Z

By the way, dup does not work on lists, because it is just an assignment under the hood, IIUC, not a deepCopy, but we could have

import lists, sugar
func `&`(L1, L2: SinglyLinkedList): SinglyLinkedList = L1.deepCopy.dup(add(L2.deepCopy))

... or L2 may be without deepCopy, which would be what I would expect (as someone coming from Lisp), but it does not match the functionality of & elsewhere in Nim.

Araq · 2020-12-16T19:32:04Z

Because the whole point of this PR is to create an O(1) concatenation :)
We can add an outplace & operator, but it will be O(n).

Ah, good one.

salvipeter · 2020-12-16T20:45:00Z

I've just realized that this operation is not really meaningful for doubly linked lists, as it will always leave one in an inconsistent state. Consider

var
  a = [1, 2, 3].toDoublyLinkedList
  b = [4, 5, 6].toDoublyLinkedList
  c = a.add(b)

If I set the prev pointer of the first node in b to the last node of a, after the add operation c will be consistent, but b won't be. OTOH if b is not changed (as in my implementation), then c will be inconsistent.

I can't see a really good solution, so I removed the DoublyLinkedList implementation.

timotheecour · 2020-12-16T23:58:43Z

tests/stdlib/tlists.nim

@@ -81,3 +81,39 @@ block tlistsToString:
    l.append('2')
    l.append('3')
    doAssert $l == """['1', '2', '3']"""
+
+block SinglyLinkedListConversion:


name the block after the symbols you're testing: toSinglyLinkedList

it makes it easier to search for tests for a given symbol
(note that merely using a symbol in some test isn't the same as a dedicated block testing it)

if a block tests multiple symbols, use:

block: # foo1, foo2

timotheecour · 2020-12-16T23:59:55Z

tests/stdlib/tlists.nim

@@ -81,3 +81,39 @@ block tlistsToString:
    l.append('2')
    l.append('3')
    doAssert $l == """['1', '2', '3']"""
+
+block SinglyLinkedListConversion:
+  let l: seq[int] = @[]


let l = seq[int].default

timotheecour · 2020-12-17T00:01:19Z

tests/stdlib/tlists.nim

+
+block SinglyLinkedListConversion:
+  let l: seq[int] = @[]
+  doAssert $l.toSinglyLinkedList == "[]"


use toSeq for tests; in general it's more robust and independent of how something is rendered

doAssert [1].toSinglyLinkedList.toSeq == [1]

timotheecour · 2020-12-17T00:54:02Z

I've just realized that this operation is not really meaningful for doubly linked lists, as it will always leave one in an inconsistent state.

Actually there's a similar problem with SinglyLinkedList: b is left in inconsistent state after being added twice, and other problems, see RFC nim-lang/RFCs#303

I can't see a really good solution, so I removed the DoublyLinkedList implementation.

I have one, see RFC

But thinking about it, I found that prepend is not really a good match, because always the first member changes, and it would be strange to say l1.prepend(l2), modifying l2 while l1 stays the same. We should just use l2.append(l1).

see RFC, which explains that can't work and prepend still needs to be implemented

salvipeter · 2020-12-17T08:29:06Z

Revised the tests according to @timotheecour's comments.

timotheecour · 2020-12-17T10:52:51Z

lib/pure/collections/lists.nim

-    var a = [1, 2, 3].toSinglyLinkedList
-    let b = [4, 5].toSinglyLinkedList
+    let a = [1, 2, 3].toSinglyLinkedList
+    assert $a == $a.copy


can you expand on this example to illustrate the semantics of shallow copying? eg using a ref object as in nim-lang/RFCs#303 (comment) or something better

timotheecour · 2020-12-17T10:55:32Z

lib/pure/collections/lists.nim

+
+proc add*[T](L1, L2: var SinglyLinkedList[T]) {.since: (1, 5).} =
+  ## Moves `L2` to the end of `L1`. Efficiency: O(1).
+  ## Note that `L2` becomes empty after the operation.


needs to explain what happens for a.add a

It becomes empty, with the current implementation.

empty sucks though;
how about raising instead? ideally would be disallowed at CT (like rust) but for now raising might be best we can do

We can do that, but I don't know ... somehow I feel that we concentrate on a special case too much. A lot worse than empty can happen when someone starts shuffling heads and tails, even when sticking with the API's append. (You can do a.append(a.head), after all, which results in a list of just the first element.)
I'm just wondering how far we should intervene... I'm all for liberty :)

tests/stdlib/tlists.nim

timotheecour · 2020-12-17T17:26:13Z

lib/pure/collections/lists.nim

+      b = a.copy
+    f.x = 42
+    assert a.head.value.x == 42
+    assert b.head.value.x == 42


add this or similar to emphasize the copy semantics. Makes behavior even more crystal clear.
(requiers changing a to var)

a.add [f].toSinglyLinkedList doAssert a.toSeq == [f, f] doAssert b.toSeq == [f] # b isn't modified... f.x = 42 assert a.head.value.x == 42 assert b.head.value.x == 42 # ... but elements are not deep copied

If you insist... but (at least for me) the immutable argument in the declaration is enough to be sure that b is not modified.

the immutable argument in the declaration is enough to be sure that b is not modified

unfortunately that's not the case, see #16312 (comment)

I'm not sure I understand ... from that thread I see that an assignment can make a (bad) shallow copy, and mess up things. But surely a func such as this copy cannot do that?
... or maybe it can. But - to quote you - this is really bad.

timotheecour · 2020-12-17T19:37:25Z

tests/stdlib/tlists.nim

+      l2.add(l1)
+      doAssert l3.toSeq == [4, 5, 6]
+      doAssert l2.toSeq == [2, 3, 1]
+  block: # DoublyLinkedList


there's a ton of duplication here. instead use this:

template testCommon(initList, toList) = var l0 = initDoublyLinkedList[int]() l1 = [1].toList l2 = [2, 3].toList l3 = [4, 5, 6].toList l4 = l3.copy l5 = l3.copy l0.add(l3) l1.add(l4) l2.add(l5) testCommon(initDoublyLinkedList, toDoublyLinkedList) testCommon(initSinglyLinkedList, toSinglyLinkedList)

it guarantees equal coverage, and simplifies maintenance etc

tests specific to doubly or singly can be in a different block, or, if it's a small diff, you can always use:

when type(l3) is DoublyLinkedList: ...

Thanks, that will improve the code a lot, I was also bothered by the code duplication, but templates didn't occur to me.

timotheecour · 2020-12-17T22:13:30Z

tests/stdlib/tlists.nim

      doAssert l3.toSeq == [4, 5, 6]
      doAssert l2.toSeq == [2, 3, 1]
+  testCommon(initSinglyLinkedList, toSinglyLinkedList)


nit: you'd have less boilerplate with a single testCommon, and sub-blocks for each common proc

block: # SinglyLinkedList, DoublyLinkedList template testCommon(initList, toList) = block copy: ... block add: ... ... testCommon(initSinglyLinkedList, toSinglyLinkedList)

I also considered that, but I assumed that a single top-level block for each tested procedure was the standard :)

…ove`)

…tyle guide.

timotheecour · 2020-12-18T22:37:17Z

changelog.md

@@ -65,6 +65,27 @@

 - `echo` and `debugEcho` will now raise `IOError` if writing to stdout fails.  Previous behavior
  silently ignored errors.  See #16366.  Use `-d:nimLegacyEchoNoRaise` for previous behavior.
+- Added `lists.toSinglyLinkedList` and `lists.toDoublyLinkedList` for conversion from


changelog needs to be fixed (reason for duplication is /changelog.md merge=union, but it's there for a good reason)

group everything for this PR under just 1 entry

newline between preceding entry (echo and debugEcho...) and your new entry

timotheecour · 2020-12-18T22:41:14Z

lib/pure/collections/lists.nim

@@ -178,6 +179,28 @@ proc newSinglyLinkedNode*[T](value: T): <//>(SinglyLinkedNode[T]) =
  new(result)
  result.value = value

+func toSinglyLinkedList*[T](elems: openArray[T]): SinglyLinkedList[T] {.since: (1, 5, 1).} =
+  ## Creates a new `SinglyLinkedList` with the members of the
+  ## given collection (seq, array, or string) `elems`.


that's not the place to describe what openArray is; it would violate "one definition rule" for docs; note that openArray definition is 1 click away thanks to docgen
=> simply:

Creates a new `SinglyLinkedList` from members of `elems`

(+ elsewhere)

timotheecour · 2020-12-18T22:50:35Z

tests/stdlib/tlists.nim

+    doAssert [1].toList.toSeq == [1]
+    doAssert [1, 2, 3].toList.toSeq == [1, 2, 3]
+
+  block copy:


this test doesn't actually test correctness of copy (2 parts: source container is not modified; but elements are references)

(Remember that runnableExamples is only run for 1 specific backend, unlike this which will be tested on all backends (eventually, once #16384 is fixed))

EDIT: how about this:

type Foo = ref object x: int let f0 = Foo(x: 0) let f1 = Foo(x: 1) var a = [f0]. toList var b = a.copy b.append f1 doAssert a.toSeq == [f0] doAssert b.toSeq == [f0, f1]

timotheecour · 2020-12-18T23:23:07Z

lib/pure/collections/lists.nim

+  a.tail = b.tail
+  if a.head == nil:
+    a.head = b.head
+  b.head = nil


change to:

if a.addr != b.addr: b.head = nil b.tail = nil

and add this test:

import lists, sequtils import std/enumerate import std/sugar var a = [0,1].toSinglyLinkedList # actualy, `toList` in your test a.addMoved a let s = collect: for i, ai in enumerate(a): if i >= 6: break ai doAssert s == [0,1,0,1,0,1]

and edit doc comment to:

## Note that `b` becomes empty after the operation. ## Self-adding results in an empty list.

=>

## Note that `b` becomes empty after the operation unless it has same address as `a`.

that's IMO a saner behavior

ditto with SinglyLinkedList

timotheecour · 2020-12-18T23:40:35Z

lib/pure/collections/lists.nim

+    let c = [1, 2, 3].toSinglyLinkedList
+    assert $c == $c.copy
+  result = initSinglyLinkedList[T]()
+  for x in a:


a.items is needed until for loop invoked from generic procedure defined in another module cant find items iterator #11167 is fixed
(ie same workaround as done in packedsets fix regression introduced in #15564 #16060)

copy also needs this fix

Added some extra tests.

salvipeter · 2020-12-19T08:52:06Z

Done... btw, why can't I write a.enumerate instead of enumerate(a) ? This seems to be against the general logic of the language.

timotheecour · 2020-12-19T19:19:58Z

btw, why can't I write a.enumerate instead of enumerate(a) ? This seems to be against the general logic of the language.

=> followup in timotheecour#473 to avoid hijacking this

timotheecour

LGTM. thanks for your patience!

salvipeter · 2020-12-19T19:27:40Z

And thank you for your comments, I learned a lot in the process :)

* O(1) concatenation of singly- and doubly linked lists. There is also new `toSinglyLinkedList` and `toDoublyLinkedList` functions for conversion from `openArray`s, similarly to `toHashSet` or `toTable`. * Add `sequtils` import to runnable examples with `toSeq`. * Added missing call to runnable examples. * Add .since annotation, changelog, and tests. * Rename `lists.concat` as an overload to `lists.append`. * Renamed `append` to `add` in lists. * Remove faulty `add` for `DoublyLinkedList`s. * Improved tests for lists. * `lists.add` moves the second list; added `lists.copy` for shallow copy. * More tests for `lists.add` and `lists.copy`. * More compact tests for lists with templates. * List concatenation with copying (`add`) and moving (tentatively `addMove`) * Renamed `addMove` to `addMoved`; renamed arguments according to the style guide. * Added extended example to `lists.copy`. * Corrected .since annotations to 1.6 * Add .since annotation, changelog, and tests. * Rename `lists.concat` as an overload to `lists.append`. * Renamed `append` to `add` in lists. * Remove faulty `add` for `DoublyLinkedList`s. * `lists.add` moves the second list; added `lists.copy` for shallow copy. * More tests for `lists.add` and `lists.copy`. * List concatenation with copying (`add`) and moving (tentatively `addMove`) * Renamed `addMove` to `addMoved`; renamed arguments according to the style guide. * Since declarations changed to (1,5,1). * Add .since annotation, changelog, and tests. * Rename `lists.concat` as an overload to `lists.append`. * Renamed `append` to `add` in lists. * Remove faulty `add` for `DoublyLinkedList`s. * `lists.add` moves the second list; added `lists.copy` for shallow copy. * More tests for `lists.add` and `lists.copy`. * List concatenation with copying (`add`) and moving (tentatively `addMove`) * Renamed `addMove` to `addMoved`; renamed arguments according to the style guide. * Changelog update. * Fix rebasing errors. * Self-adding with `lists.addMove` results in a cycle now. Added some extra tests.

timotheecour reviewed Dec 16, 2020

View reviewed changes

timotheecour reviewed Dec 17, 2020

View reviewed changes

timotheecour mentioned this pull request Dec 17, 2020

std/lists: support destructive add/prepend for SinglyLinkedList, DoublyLinkedList + other lists improvements nim-lang/RFCs#303

Open

timotheecour mentioned this pull request Dec 17, 2020

misc lists timotheecour/Nim#464

Open

timotheecour reviewed Dec 17, 2020

View reviewed changes

tests/stdlib/tlists.nim Show resolved Hide resolved

timotheecour reviewed Dec 17, 2020

View reviewed changes

timotheecour mentioned this pull request Dec 17, 2020

regression since 1.4.2: vm crash with lists.SinglyLinkedRing #16384

Closed

salvipeter added 7 commits December 18, 2020 23:31

Rename lists.concat as an overload to lists.append.

363571d

Renamed append to add in lists.

e20d684

Remove faulty add for DoublyLinkedLists.

1495139

lists.add moves the second list; added lists.copy for shallow copy.

a9e0b5f

More tests for lists.add and lists.copy.

a4fba89

List concatenation with copying (add) and moving (tentatively `addM…

7538244

…ove`)

Renamed addMove to addMoved; renamed arguments according to the s…

37f6299

…tyle guide.

salvipeter force-pushed the list-concat branch from 6b1a49b to 37f6299 Compare December 18, 2020 22:34

timotheecour reviewed Dec 18, 2020

View reviewed changes

Changelog update.

7cb28fe

timotheecour reviewed Dec 18, 2020

View reviewed changes

Fix rebasing errors.

7eb491a

timotheecour reviewed Dec 18, 2020

View reviewed changes

timotheecour closed this Dec 18, 2020

timotheecour reopened this Dec 18, 2020

timotheecour reviewed Dec 18, 2020

View reviewed changes

Self-adding with lists.addMove results in a cycle now.

5305d46

Added some extra tests.

timotheecour mentioned this pull request Dec 19, 2020

can we support UFCS/MCS for ForLoopStmt macros? timotheecour/Nim#473

Closed

timotheecour approved these changes Dec 19, 2020

View reviewed changes

Araq merged commit 051477b into nim-lang:devel Dec 20, 2020

Hassuna123 referenced this pull request Dec 20, 2020

properly deprecate parseopt2 (#10452)

268197a

timotheecour mentioned this pull request Dec 21, 2020

Add jsheaders nim-lang/fusion#56

Merged

timotheecour changed the title ~~O(1) concatenation of singly- and doubly linked lists.~~ std/lists: O(1) concatenation of singly- and doubly linked lists. Jan 6, 2021

salvipeter deleted the list-concat branch February 8, 2021 21:27

std/lists: O(1) concatenation of singly- and doubly linked lists. #16362

std/lists: O(1) concatenation of singly- and doubly linked lists. #16362

Conversation

salvipeter commented Dec 15, 2020

Araq commented Dec 15, 2020

Clyybber commented Dec 15, 2020

salvipeter commented Dec 15, 2020

Clyybber commented Dec 15, 2020 • edited Loading

salvipeter commented Dec 15, 2020

Araq commented Dec 15, 2020

salvipeter commented Dec 15, 2020

Araq commented Dec 15, 2020

salvipeter commented Dec 15, 2020

timotheecour commented Dec 15, 2020 • edited Loading

salvipeter commented Dec 15, 2020

Araq commented Dec 16, 2020

salvipeter commented Dec 16, 2020

salvipeter commented Dec 16, 2020 • edited Loading

Araq commented Dec 16, 2020

salvipeter commented Dec 16, 2020

timotheecour Dec 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timotheecour Dec 17, 2020 • edited Loading

Choose a reason for hiding this comment

timotheecour commented Dec 17, 2020

salvipeter commented Dec 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timotheecour Dec 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timotheecour Dec 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timotheecour Dec 17, 2020 • edited Loading

Choose a reason for hiding this comment

salvipeter Dec 17, 2020 • edited Loading

Choose a reason for hiding this comment

timotheecour Dec 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timotheecour Dec 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timotheecour Dec 18, 2020 • edited Loading

Choose a reason for hiding this comment

timotheecour Dec 18, 2020 • edited Loading

Choose a reason for hiding this comment

timotheecour Dec 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

salvipeter commented Dec 19, 2020

timotheecour commented Dec 19, 2020

timotheecour left a comment

Choose a reason for hiding this comment

salvipeter commented Dec 19, 2020

Clyybber commented Dec 15, 2020 •

edited

Loading

timotheecour commented Dec 15, 2020 •

edited

Loading

salvipeter commented Dec 16, 2020 •

edited

Loading

timotheecour Dec 16, 2020 •

edited

Loading

timotheecour Dec 17, 2020 •

edited

Loading

timotheecour Dec 17, 2020 •

edited

Loading

timotheecour Dec 17, 2020 •

edited

Loading

timotheecour Dec 17, 2020 •

edited

Loading

salvipeter Dec 17, 2020 •

edited

Loading

timotheecour Dec 17, 2020 •

edited

Loading

timotheecour Dec 17, 2020 •

edited

Loading

timotheecour Dec 18, 2020 •

edited

Loading

timotheecour Dec 18, 2020 •

edited

Loading

timotheecour Dec 18, 2020 •

edited

Loading