Cannot mix generic with typedesc

[bluenote10]

I would have expected that mixing generics and typedescs is possible.

Example

proc f[T](X: typedesc) =
  echo "test"

f[int](string)

Current Output

test.nim(4, 1) Error: cannot instantiate: 'f[int]'; got 1 type(s) but expected 2

Expected Output

Should compile.

Additional Information

$ nim -v
Nim Compiler Version 0.19.9 [Linux: amd64]
Compiled at 2019-04-13
Copyright (c) 2006-2019 by Andreas Rumpf

git hash: f6ad071a46a2bec57db453343d8d8b75d3d16ac2
active boot switches: -d:release

[krux02]

@zah I heared that you had a lot of influence on design of typedesc as it currently is. What do you think goes wrong here, and how would be a clean way to fix it.

[bluenote10]

Some observations: It works if there is connection between the typedesc argument and the generic argument, but it still depends on how the proc is called:

type
  Generic[T] = object

proc f[T](X: typedesc[Generic[T]]) = discard

Generic[int].f()        # works
f(Generic[int])         # works
f[int](Generic[int])    # fails, probably should work

[krux02]

I would assume it has something to do with the hidden generic argument when you use typedesc. I always try to avoid using any hidden generic arguments, as it basically lies to the reader that the function is non-generic when it actually is generic: a hidden generic. Passing typedesc is such a hidden generic parameter.

Here is an example:

type
  Vec4[T] = object
    data: array[0..3, T]

# this doesn't look generic, but it is
proc foo(arg: Vec4): Vec4 =
  result.data[0] = arg.data[0] + 2
  result.data[1] = arg.data[1] - 2
  result.data[2] = arg.data[2] * 2
  result.data[3] = 2

var tmp = Vec4[int](data: [1,2,3,4])

tmp = foo(tmp)

echo tmp

# when you now think you set a single generic argument, you actually
# create a second generic argument.
proc bar[T](arg: Vec4): Vec4 =
  echo T
  result.data[0] = arg.data[0] + 2
  result.data[1] = arg.data[1] - 2
  result.data[2] = arg.data[2] * 2
  result.data[3] = 2


# cannot instantiate: 'bar[float32]'; got 1 type(s) but expected 2
#let z = bar[float32](tmp)

# now when you actually pass two generic parameters, it still fails,
# but this time without explanation:
# cannot instantiate: 'bar[float32, int]'
#let z = bar[float32, int](tmp)


# if you avoid these hidden generics, then you don't have any problems:
proc baz[T,U](arg: Vec4[U]): Vec4[U] =
  echo T
  result.data[0] = arg.data[0] + 2
  result.data[1] = arg.data[1] - 2
  result.data[2] = arg.data[2] * 2
  result.data[3] = 2

# this works as you expect.
let z = baz[float32, int](tmp)


# as I explained before typedesc has a hidden generic as well, but
# when I try to apply this trick here, it get's even worse:

proc f[T,U](t: typedesc[U]) =
  echo "test"

# ``string`` is implicitly converted to ``typedesc[string]``, so this
# should work, but this is the error:
# cannot instantiate: 'f[int, string]'; got 2 type(s) but expected 3
f[int,string](string)

BTW: These workarounds do work:

proc f1[T,X](t1: typedesc[T]; t2: typedesc[X]) =
  echo "test"

f1(int, string)

proc f2[T,X]() =
  echo "test"

f2[int, string]()

[zah]

Yes, this is a more general problem with explicit generic instantiation. Implicit generics create hidden parameters and the instantiation logic doesn't try to compensate for this, neither it allows partial instantiation as supported in C++.

For this reason, I usually recommend creating APIs that don't rely on explicit generic parameters. I've written more about this here:
nim-lang/RFCs#40 (comment)

When your API requires non-inferred types to be passed explicitly, just use typedesc parameters. In return, you get concise code, the ability to overload, the ability to use default values, etc.

proc f(T, X: typedesc) =
  echo "test"

[bluenote10]

@zah In fact the reason I was mixing them was because I was following your object construction RFC nim-lang/RFCs#48 which seems to have problems with the explicit form init[K, V](T: Table[K, V]).

[zah]

If you follow my proposal, you would implement the init proc like this:

proc init[K, V](T: type Table[K, V]) = initTable[K, V]()

This is slightly better, because it allows you to create an alias for the table type:

type
  UserMap = Table[UserId, User]

...

var m = UserMap.init

[krux02]

@zah I agree the pattern for init is a good pattern, except that I would use the typedesc syntax. But I am surprised that proc f(T,X: typedesc) works, because the rules for implicit generics would not allow this:

type
  Vec4[T] = object
    data: array[0..3, T]

# two arguments, implicitly generic.
proc foo(arg1, arg2: Vec4) =
  discard

var v1: Vec4[float32]
var v2: Vec4[int32]

foo(v1,v2) # does not not work because they are not the same instance of Vec4

# two arguments, implicity generic. But "special"
proc f(T, X: typedesc) =
  echo "test"

f(int, string) # does work even though they are not the same instace of typedesc

This is far too much special treatment of typesdesc for my personal taste.

[mratsim]

Actually the "specialness" is because in the second case, the types passed to typedesc arguments are "values" similar to how passing different int to proc f(T, X: int) = will be OK.

As @Araq, often repeated, there is a mix of types as types and types as values in the typedesc/generics/static handling that makes this area of the compiler the most tricky and bug prone.

Also, static, concepts (and typedesc?) were bolted on the generics codepath very late so I guess the design is poor and in great need of a refactor.

All in on all, I think we will have to pay the technical debt at one point because the current generics/static/typedesc are creating recurrent issues ({.this.}, strformat, async nested in generics - #8677).

They are also quite hard to handle in macros nim-lang/RFCs#44, and triggered lots of discussion about container/generics initialization and constructor API:

• [RFC] partial generic deduction RFCs#86 (partial generics deduction)
• [RFC] default type parameter for generics RFCs#85 (default typedesc param)
• [RFC] universal constructor syntax: a={} to mean: a=defaultInit(typeof(a)), eg replaces initTable[K,V](), HashSet[string].init, newSeqOfCap[T](c)` etc  RFCs#64 (universal constructor syntax)
• [RFC] Another proposal for a standardized object construction/initialization. RFCs#48 (standardized object construction)
• [meta issue] make all Nim APIs generic: ctor, init, to, streams, types, etc RFCs#41 (All Nim API generic: ctor, streams, init)
• [RFC] guidelines for when to use typedesc vs generics RFCs#40 (typedesc vs generics)

All the way up to generic generics/higher kinded types nim-lang/RFCs#5

I'm pretty sure this rework could also benefits concepts. Given the scope, I think this should target 1.X rather than 1.0, the status quo is OK for the short term (but probably not in the long term).

[krux02]

Actually the "specialness" is because in the second case, the types passed to typedesc arguments are "values" similar to how passing different int to proc f(T, X: int) = will be OK.

yes they are value, but values of different types, typedesc[int] and typedesc[string]. Not the same type at all. Here is another example of this:

static:
  var t1: typedesc[int]
  var t2: typedesc[int]
  var t3: typedesc[float]

  t1 = t2 # ok
  t1 = t3 # type mismatch: got <type float> but expected 'type int'

typedesc[float] isn't just a different value of typedesc, like 1 and 2 are different values of int, They are different values of different types.

[mratsim]

In your original example they are values of the typedesc type:

i.e.

proc f(T, X: typedesc)

and not

proc f(T: typedesc[int], X: typedesc[string])

# or

proc f[U](T: typedesc[U], X: typedesc[U])

[krux02]

proc f(T, X: typedesc) was exactly identical to proc f[U](T: typedesc[U], X: typedesc[U]), as proc f(v1,v2: Vec4) is exactly identical to proc f[T](v1, v2: Vec4[T]). I don't know what has been changed, but I could live a very good live so far and deny any existance of a bare typedesc that does not have a (hidden) generic parameter. This is also what is reflected in the error message from the issue: test.nim(4, 1) Error: cannot instantiate: 'f[int]'; got 1 type(s) but expected 2. There is this expected but hidden generic parameter. Any usage of typedesc without specifying the generic parameter is a lie from the compiler that lets you think that such a thing as typedesc without generic parameter exists. But it doesn't, and there are many hacks in the compiler that allow you to think in such a way.

[zah]

In general, Nim allows you to define both "bind once" type classes and "bind many" type classes as explained the type classes section of the manual.

It's up to the standard library to decide which (if any) of the standard types will get the bind-many treatment. Notable examples for this are auto and typedesc. If you use just type, you get the bind-once behavior by accident - I consider this a bug.

[krux02]

In general, Nim allows you to define both "bind once" type classes and "bind many" type classes as explained the type classes section of the manual.

This is all I can find in the type classes section:

Alternatively, the distinct type modifier can be applied to the type class to allow each param matching the type class to bind to a different type. Such type classes are called bind many types.

I would not call this "explained" it is barely mentioned. There is no example of bind many outside of the concepts section.

It's up to the standard library to decide which (if any) of the standard types will get the bind-many treatment. Notable examples for this are auto and typedesc. If you use just type, you get the bind-once behavior by accident - I consider this a bug.

Where in the standard library is declared that typedesc has bind-many treatment? All I can find is
type typedesc* {.magic: TypeDesc.} it even hides that it has a generic argument at all. I would rather call it a compiler surprise if types get the bind-many treatment or not.

[zah]

In the initial versions of the implicit generics, it used to be the case that auto was defined as a bind-once magic type and we had a separate any type defined like this:

type any* = distinct auto

This reflects the time when the manual was written. It would be my preference as well that we use such explicit distinct non-magic types to define auto and typedesc, but Araq didn't try to preserve this property while making various fixes related to auto in the past and we ended up with the current status quo.

BTW, whether the type is considered bind-once or bind-many is relatively easy to fix - it all happens in the liftParamType proc in semtypes.nim

[zevv]

I just ran into a problem when mixing generics and typedesc:

import macros

macro foo(name: string, n1: untyped) =
  echo "one"

macro foo(name: string, n1: typedesc, n2: untyped) = 
  echo "two" 

foo("hello"):
  a = b

I'd expect foo() to invoke the first macro, but instead it seems to match the second, and then fail compilation with Error: undeclared identifier: 'a'

Is this problem related to this issue?

[Araq]

@zevv no, yours is just the old untyped parameters don't work well with overloading. (Not that of a suprise really, if you think about it...)

[krux02]

We really should restrict untyped macros and templates, so that they may not be overloaded. Overloading with both typed and untyped never really worked.

[mratsim]

The error message should be improved, but sometimes in the far future I'd like to see untyped overloading possible. Anyway, untyped overloading should be discussed in the corresponding issue: #9414

[krux02]

@mratsim: The problem is the following, when you have a proc with untyped argument, then you tell the compiler, please don't semcheck this AST. When you have another overload that needs this argument to be semchecked, you have a conflict. You have an AST that needs to be both semchecked and not semchecked at the same time. This can only cause problems and should be reported to the programmer as a compilation error.

[zevv]

Araq already explained on IRC there are certain subleties involved, but what confused me that the compiler fails to differentiate between the two macros, even though they have a different number of arguments. This does work for two macros with only untyped args, but fails when I introduce the typedesc. This works just fine, in contrast to my above example:

macro foo(a, b: untyped): untyped =
  echo "one"
  
macro foo(a, b, c: untyped): untyped =
  echo "two"
  
foo(1, 2)
foo(1, 2, 3)