Implement interfaces/traits instead of concepts

[tulayang]

The longer I use Nim, the more I feel the seriousness that Nim lacks Java-like interfaces.

Nim is already a better C, but Nim is not C. The reason why Nim and C are different is that there is no concept of encapsulation in C. There is no information hiding in C and you can freely access any field of any struct. However, it is different in Nim because of encapsulation. The result of encapsulation is that you cannot access the fields of an object as flexibly as C.

Here is a common example. Object M is created in module A to handle certain things. Later, object N was created in module B. object N needs some interoperation with object M. However, because of encapsulation, N cannot access some fields of M, which hinders this interoperability. To support this inter-module interoperability while keeping information hiding, interface or abstract class is a commonly used method. They provide a declaration to defer certain operations on object M to the implementation objects.

I think one of the main reason that Nim's lack of popularity is lack of interfaces. This makes it difficult for people who use Nim to write software to work together, and it is difficult to produce things such as plug-ins, middleware, and so on, which are an important factor for the prosperity of an industry.

I think that interface is not just a matter of style choice for Nim, but an urgent and important thing that can have a key impact on Nim's popularity and growth. In addition, I have some ideas about how to implement the interface, that is, to adopt the trait scheme of rust. Below I use a program to explain this.

Original solution - none interface

The example here describes a multi-producer-single-consumer task queue. Each thread acts as a producer or a consumer. Producers produce data quickly, and consumers consume data slightly slower. Producer and consumer cooperate through a notification mechanism. When a producer makes a new piece of data in the queue, it signal() the consumer; the consumer receives the producer's signal through wait().

Based on different implementation strategies, the behavior of signal() and wait() will be different:

• In the Linux environment, using eventfd as the notification mechanism, signal() sends a count signal, and wait() receives the counter
• signal() does nothing; wait() checks the read position and write position, and then returns the amount that can be consumed
• others

In order to make these strategies flexible, the implementation of signal() and wait() was deferred until the queue was instantiated.

1. Declares a memory allocator for shared heap:

type
  PurePointer* = ptr | pointer 

  Alloctor*[T] = object
    allocImpl: proc (): T {.nimcall, gcsafe.}
    deallocImpl: proc (p: T) {.nimcall, gcsafe.}

proc initAlloctor*[T: PurePointer](
  allocImpl: proc (): T {.nimcall, gcsafe.}, 
  deallocImpl: proc (p: T) {.nimcall, gcsafe.}
): Alloctor[T] =
  result.allocImpl = allocImpl
  result.deallocImpl = deallocImpl

proc alloc*[T: PurePointer](a: var Alloctor[T]): T {.inline.} =
  a.allocImpl() 

proc dealloc*[T: PurePointer](a: var Alloctor[T], p: T) {.inline.} =
  a.deallocImpl(p) 

2. Declares a signal counter to act as the underlying notification mechanism:

type
  SigCounter* = ptr object of RootObj
    signalImpl*: proc (c: SigCounter) {.nimcall, gcsafe.}
    waitImpl*: proc (c: SigCounter): Natural {.nimcall, gcsafe.}

proc signal*(c: SigCounter) {.inline.} =
  c.signalImpl(c) 

proc wait*(c: SigCounter): Natural {.inline.} =
  c.waitImpl(c) 

3. The multi-producer-single-consumer task queue:

type
  MpscQueue*[T] = object 
    data: ptr UncheckedArray[T]
    ... other fields
    counter: SigCounter                    # the underlying notification mechanism
    counterAlloctor: Alloctor[SigCounter]  # the allocator for ``SigCounter``

proc `=destroy`*[T](x: var MpscQueue[T]) = 
  if x.data != nil:
    deallocShared(x.data)
    x.data = nil
    x.counterAlloctor.dealloc(x.counter)
    x.counter = nil

proc initMpscQueue*[T](counterAlloctor: Alloctor[SigCounter]): MpscQueue[T] =
  ...
  result.counterAlloctor = counterAlloctor
  result.counter = result.counterAlloctor.alloc()

proc add*[T](x: var MpscQueue[T], item: sink T) = 
  # produce a item
  ... add a new item
  x.counter.signal()

proc sync*[T](x: var MpscQueue[T]) = 
  x.len.inc(x.counter.wait())

proc take*[T](x: var MpscQueue[T]): T = 
  # consume a item
  result = ... remove a item
  x.len.dec()

4. A use case testing:

import std/os
import std/posix

proc eventfd*(initval: cuint, flags: cint): cint {.
  importc: "eventfd", 
  header: "<sys/eventfd.h>"
.}

type 
  MySigCounter = ptr object of SigCounter
    efd: cint

proc signalMySigCounter(c: SigCounter) = 
  var buf = 1'u64
  if cast[MySigCounter](c).efd.write(buf.addr, sizeof(buf)) < 0:
    raiseOSError(osLastError())

proc waitMySigCounter(c: SigCounter): Natural = 
  var buf = 0'u64
  if cast[MySigCounter](c).efd.read(buf.addr, sizeof(buf)) < 0:
    raiseOSError(osLastError())
  result = buf 

proc allocMySigCounter(): SigCounter = 
  let p = cast[MySigCounter](allocShared0(sizeof(MySigCounter)))
  p.signalImpl = signalMySigCounter
  p.waitImpl = waitMySigCounter
  p.efd = eventfd(0, 0)
  if p.efd < 0:
    raiseOSError(osLastError())
  result = p

proc deallocMySigCounter(c: SigCounter) =
  deallocShared(cast[MySigCounter](c))

var rcounter = 0
var rsum = 0
var mq = initMpscQueue[int](
  initAlloctor(allocMySigCounter, deallocMySigCounter))

proc producerFunc() {.thread.} =
  for i in 1..1000:
    mq.add(i) 

proc consumerFunc() {.thread.} =
  while rcounter < 4000:
    mq.sync()
    while mq.len > 0:
      rcounter.inc()
      var val = mq.take()
      rsum.inc(val)

proc test() = 
  var producers: array[4, Thread[void]]
  var comsumer: Thread[void]
  for i in 0..<4:
    createThread(producers[i], producerFunc)
  createThread(comsumer, consumerFunc)
  joinThreads(producers)
  joinThreads(comsumer)
  doAssert rsum == ((1 + 1000) * (1000 div 2)) * 4 # (1 + n) * n / 2

test()

The following is my vision of the Nim interface/trait

type
  PurePointer* = ptr | pointer 

  Alloctor*[T: PurePointer] = trait
    alloc: proc (a: Alloctor[T]): T {.nimcall, gcsafe.}
    dealloc: proc (a: Alloctor[T], p: T) {.nimcall, gcsafe.}

  SigCounter* = trait
    signal: proc (c: SigCounter) {.nimcall, gcsafe.}
    wait: proc (c: SigCounter): Natural {.nimcall, gcsafe.}

  Printer* = trait
    echo: proc (e: Printer) {.nimcall, gcsafe.}

type 
  MySigCounter = ptr object
    efd: cint

  MySigCounterAllocator = object

impl SigCounter for MySigCounter:
  proc signal(c: MySigCounter) = 
    var buf = 1'u64
    if c.efd.write(buf.addr, sizeof(buf)) < 0:
      raiseOSError(osLastError())

  proc wait(c: MySigCounter): Natural = 
    var buf = 0'u64
    if c.efd.read(buf.addr, sizeof(buf)) < 0:
      raiseOSError(osLastError())
    result = buf 

impl Printer for MySigCounter:
  proc echo(c: MySigCounter) = 
    echo "Hello MySigCounter!"

impl Alloctor[MySigCounter] for MySigCounterAllocator:
  proc alloc(a: var MySigCounterAllocator): MySigCounter = 
    let p = cast[MySigCounter](allocShared0(sizeof(MySigCounter)))
    p.efd = eventfd(0, 0)
    if p.efd < 0:
      raiseOSError(osLastError())
    result = p

  proc dealloc(a: var MySigCounterAllocator, c: MySigCounter) =
    deallocShared(c)

impl Printer for MySigCounterAllocator:
  proc echo(a: MySigCounterAllocator) = 
    echo "Hello MySigCounterAllocator!"

Testing:

var mq = initMpscQueue[int](MySigCounterAllocator())

[alaviss]

Nim have a documentation for an unimplemented interface design built on top of concepts known as vtref: https://raw.githubusercontent.com/nim-lang/Nim/devel/doc/manual_experimental.rst (Ctrl + F and search for vtref. It's commented so won't show up on the final render).

EDIT: Provided a render of the docs below:

VTable types

Concepts allow Nim to define a great number of algorithms, using only static polymorphism and without erasing any type information or sacrificing any execution speed. But when polymorphic collections of objects are required, the user must use one of the provided type erasure techniques - either common base types or VTable types.

VTable types are represented as "fat pointers" storing a reference to an object together with a reference to a table of procs implementing a set of required operations (the so called vtable).

In contrast to other programming languages, the vtable in Nim is stored externally to the object, allowing you to create multiple different vtable views for the same object. Thus, the polymorphism in Nim is unbounded - any type can implement an unlimited number of protocols or interfaces not originally envisioned by the type's author.

Any concept type can be turned into a VTable type by using the vtref or the vtptr compiler magics. Under the hood, these magics generate a converter type class, which converts the regular instances of the matching types to the corresponding VTable type.

type
  IntEnumerable = vtref Enumerable[int]

  MyObject = object
    enumerables: seq[IntEnumerable]
    streams: seq[OutputStream.vtref]

proc addEnumerable(o: var MyObject, e: IntEnumerable) =
  o.enumerables.add e

proc addStream(o: var MyObject, e: OutputStream.vtref) =
  o.streams.add e

The procs that will be included in the vtable are derived from the concept body and include all proc calls for which all param types were specified as concrete types. All such calls should include exactly one param of the type matched against the concept (not necessarily in the first position), which will be considered the value bound to the vtable.

Overloads will be created for all captured procs, accepting the vtable type in the position of the captured underlying object.

Under these rules, it's possible to obtain a vtable type for a concept with unbound type parameters or one instantiated with metatypes (type classes), but it will include a smaller number of captured procs. A completely empty vtable will be reported as an error.

The vtref magic produces types which can be bound to ref types and the vtptr magic produced types bound to ptr types.

It's shelved due to lack of interest (in implementing) IIRC, but I think it's time we should revisit the idea.

/cc @Araq since he said he has a concepts RFC pending.

[timotheecour]

The longer I use Nim, the more I feel the seriousness that Nim lacks Java-like interfaces.

can't you use nim's OOP features for that? eg see std/streams FileStream inheriting Stream and using interface (eg proc close*(s: Stream))
(see also methods)

[Araq]

/cc @Araq since he said he has a concepts RFC pending.

My RFC is here, #168 and is not about interfaces at all.

[github-actions]

This RFC is stale because it has been open for 1095 days with no activity. Contribute a fix or comment on the issue, or it will be closed in 30 days.