Make IntSet a generic ordinal set PackedSet[A] + new module packedsets

[landerlo]

This makes intsets generic for ordinal types like enums or distinct of ints. For cases when multiple intsets with complex logic making them generic boosts their usability and type safety.
E.g. iqqn my use case I need to distinguish different ids making them distinct and get compilation errors if
I do any set operations with incompatible ids.

The PR achieves this maintaining full backwards compatibility and same performance.
As a drop in replacement, the current api is unchanged and IntSet is aliased to OrdSet[int].
OrdSet is for ordinal set. I realise there might be a confusion with ordered sets, so naming suggestions welcome.

Better error messages on compilation errors will be achieved when the concepts are ready, but I didn't want to tie it to a future roadmap, and the current solution achieves the stated goals satisfactorily IMHO.

[disruptek]

I liked it better before it touched the compiler, but whatever. I think we have to wait until Ordinal is a concept before it will work for distinct types.

[timotheecour]

I liked it better before it touched the compiler, but whatever.

/cc @landerl
I agree with @disruptek , I don't think there's a need to change client code, see my answer here: #15564 (comment)

I think we have to wait until Ordinal is a concept before it will work for distinct types.

what do you mean? this passes:

type int2 = distinct int
doAssert int2 is Ordinal

[landerlo]

I liked it better before it touched the compiler, but whatever.

/cc @landerl
I agree with @disruptek , I don't think there's a need to change client code, see my answer here: #15564 (comment)

I think we have to wait until Ordinal is a concept before it will work for distinct types.

what do you mean? this passes:

type int2 = distinct int
doAssert int2 is Ordinal

See my comment above, making all the type params A: Ordinal didn't work for me, but just making OrdSet[A: Ordinal] and the api procs [A] seems to work and has now better compilation errors, so yes, concepts are not necessary for this.

[disruptek]

I mean that Ordinal doesn’t capture a type than can nominally behave as an ordinal because it’s magic and not a concept.

[landerlo]

I mean that Ordinal doesn’t capture a type than can nominally behave as an ordinal because it’s magic and not a concept.

At the moment is magic but there could be a concept modeling Ordinal with ord(a: A): A, fromOrd(o: int): A, and an implicit assumption that the constraint: forall x: int ord(fromOrd(int)) == x && forall a: A fromOrd(ord(a)) == a, that's left to an axiom for ints, enums and distincts of an ordinal. But maybe user could define other ordinal types to benefit from things like ordsets.

That Ordinal concept could be further generalised to Isomorphic. so concept Ordinal[A] = Isomorphic[A, int] with the above signature.

[timotheecour]

LGTM; good to merge?

[timotheecour]

landerlo requested a review from timotheecour 16 minutes ago

I've already did a full review and gave my LGTM ...

@Araq or other nim member can i merge this?

[Araq]

Sorry, I thought it already was merged.

[landerlo]

I do realise now that sed doesn't go through the commit messages, so the commit log still talks about OrdSet and not the renamed PackedSet. Nevermind.

[disruptek]

Thanks for all your work on this, @landerlo -- I know it was a slog but this is a nice contribution. ❤️

[disruptek]

I hate to say it, but this is causing the items iterator against an IntSet to fail:

https://github.com/disruptek/gram/runs/1419861309?check_suite_focus=true

/home/adavidoff/nims/lib/std/packedsets.nim(370, 14) Error: type mismatch: got <PackedSet[system.int]>
but expected one of: 
iterator items(a: cstring): char
  first type mismatch at position: 1
  required type for a: cstring
  but expression 's' is of type: PackedSet[system.int]
iterator items(a: string): char
  first type mismatch at position: 1
  required type for a: string
  but expression 's' is of type: PackedSet[system.int]
iterator items(n: NimNode): NimNode
  first type mismatch at position: 1
  required type for n: NimNode
  but expression 's' is of type: PackedSet[system.int]
iterator items[IX, T](a: array[IX, T]): T
  first type mismatch at position: 1
  required type for a: array[IX, T]
  but expression 's' is of type: PackedSet[system.int]
iterator items[N, E; F: static[GraphFlags]](graph: Graph[N, E, F]): N
  first type mismatch at position: 1
  required type for graph: Graph[items.N, items.E, items.F]
  but expression 's' is of type: PackedSet[system.int]
iterator items[T: char](a: openArray[T]): T
  first type mismatch at position: 1
  required type for a: openArray[T: char]
  but expression 's' is of type: PackedSet[system.int]
iterator items[T: enum](E: typedesc[T]): T
  first type mismatch at position: 1
  required type for E: type T: enum
  but expression 's' is of type: PackedSet[system.int]
iterator items[T: not char](a: openArray[T]): lent2 T
  first type mismatch at position: 1
  required type for a: openArray[T: not char]
  but expression 's' is of type: PackedSet[system.int]
iterator items[T](a: seq[T]): lent2 T
  first type mismatch at position: 1
  required type for a: seq[T]
  but expression 's' is of type: PackedSet[system.int]
iterator items[T](a: set[T]): T
  first type mismatch at position: 1
  required type for a: set[T]
  but expression 's' is of type: PackedSet[system.int]
iterator items[T](s: HSlice[T, T]): T
  first type mismatch at position: 1
  required type for s: HSlice[items.T, items.T]
  but expression 's' is of type: PackedSet[system.int]

expression: items(s)

[timotheecour]

@disruptek can you provide a minimized reproducible example?

[disruptek]

Is gram too complex for you?

[timotheecour]

that's basically what everyone has to go through when submitting bug reports, providing minimized reproducing examples. More work for the reporter, less work for everyone else looking at it.

[disruptek]

Yeah, but only if they didn't manage to get their package into important_packages first. If they were able to do that, then for some reason the developers are willing to do the same work themselves. Weird.

[timotheecour]

• if its's a compiler bug: minimize with 0 stdlib module dependencies, when possible
• if it's a stdlib bug: minimize with as few stdlib module dependencies, ideally just the module in question
  it's about minimizing total work = 1 * work(bug reporter) + N * work(people looking at issue)

[disruptek]

Let's agree that you'll be the 1 to do the work and all N of us will reap the benefits.

[Araq]

As a compromise, both modules could expand a common template implementation -- then there cannot be regressions like this.

[landerlo]

@disruptek Sorry for the regression. Tests test int iterators but there must be some issue why the iterator is not properly inferred.
I've created a PR that simply reverts IntSet to previous impl rather than as an alias for PackedSet[int], that should do the trick for now. I want to dig deeper into the issue anyway at some point.
Rollbak IntSet as before:
#16059

[disruptek]

Can't we just fix it instead?

[timotheecour]

As a compromise, both modules could expand a common template implementation -- then there cannot be regressions like this.

=> #16060, much cleaner

[disruptek]

Nice work! Another good reason to be explicit with iterators, at least until this is fixed.