Add optionals module (Maybe[T])

[oprypin]

Superseded by #2762

• Documentation
• Code

• This module implements types which encapsulate an optional value.

  A value of type ?T (Maybe[T]) either contains a value x (represented as just(x)) or is empty (nothing[T]).

  This can be useful when you have a value that can be present or not. The absence of a value is often represented by nil, but it is not always available, nor is it always a good solution.

• Discussion in IRC

This pull request is not final. I am looking for feedback.

I am planning to squash the commits at the end, but for now I will keep the history.

[ReneSac]

I still think that the or operator:

var x = y or z

is better than:

var x = y.get z

When you want y or z depending if y is is nothing or not, regardless on what python does with it's dictionaries. Even better if the z is an Maybe too, as then the get name becomes even less adequate. I think get() should be reserved simply for an`one parameter exception throwing value access.

And will the Maybe module really be named "optionals"?

[oprypin]

var x = y or z

Both of these options don't really belong in the language. But seeing as there is a precedent, I am inclined to prefer or.

I took this opportunity to also replace x.get() with x[].

---

really be named "optionals"?

Even Haskell says:

The Maybe type encapsulates an optional value

I don't see a better name for the module. Besides, it is an opportunity to talk to more people in terms familiar to them: have both "optional" and "maybe" in the names.

[oprypin]

I've overhauled the module. The most important change:

Now just and nothing are not simply procedures that return one of the two implementations of Maybe, but actual types. Just[T] = distinct T, which always has a value. Nothing[T] never has a value. From there they are converted to an appropriate Maybe implementation when needed. This means that one doesn't need to use the ?T operator and depend on its choice. People who don't like the nil-based implementation can simply ignore it in their code.

Basic usage remained almost the same.

[flaviut]

I'd like to object to this module for several reasons:

• Using ? as a type is a poor idea. A little extra typing isn't going to kill anyone, especially with type inference. Using Maybe[Foo] will also improve readability significantly.

• MaybeDistinct and MaybeObj do not need to be public, they are internal optimization details. The client does not need to know about them.

  • The following types can be used to allow same optimizations while at the same time creating a nicer public API:

  Nillable = concept x
    isNil(x) is bool
    x = nil
  Ref = concept x
    isNil(x)

  Doing this would however be blocked upon When expansion is too eager #2002.

[oprypin]

https://github.com/BlaXpirit/nre/commits/optionals
Case study. I replaced optional_t with this library in two steps:

• Switched the API.
• Implemented isNil operations in order to switch from MaybeObj to MaybeDistinct (and save a couple of bytes of memory 😐).

[oprypin]

Using ? as a type is a poor idea

It is not compulsory to use this. First of all, one can specify concrete types like MaybeObj. Now let's think what would be the cases where one would use ?:

• Procedure argument type: it is actually best to accept the generic Maybe[T] and not just one concrete type.
• Procedure return type:
  Nim has concrete return types and auto. If Nim supported generic return types (which is basically auto with additional checks), one could write Maybe[T].

---

MaybeDistinct and MaybeObj do not need to be public

The module lets you make a conscious choice. If you have a type that is not nullable, you can make it nullable to take advantage of MaybeDistinct. If you have a nullable type but hate the idea of MaybeDistinct, or need the nil value, you can use MaybeObj directly.

Another important thing to notice is, these types are inevitably going to be present everywhere, even if it's through ?. I don't think it's good to limit access to them.

---

I really don't think it's possible to decide whether to include the boolean value or not inside the type declaration itself, thus the need for two types. I don't understand what problem your code snippet tries to solve.

[flaviut]

• Procedure argument type: it is actually best to accept the generic Maybe[T] and not just one concrete type.
• Procedure return type: Nim has concrete return types and auto. If Nim supported generic return types (which is basically auto with additional checks), one could write Maybe[T].

These would not be a problem if there was only a single Maybe[T] implementation type.

MaybeDistinct and MaybeObj do not need to be public

The module lets you make a conscious choice. If you have a type that is not nullable, you can make it nullable to take advantage of MaybeDistinct. If you have a nullable type but hate the idea of MaybeDistinct, you can use MaybeObj directly.

Another important thing to notice is, these types are inevitably going to be present everywhere, even if it's through ?. I don't think it's good to limit access to them.

Same as above.

I really don't think it's possible to decide whether to include the boolean value or not inside the type declaration itself, thus the need for two types. I don't understand what problem your code snippet tries to solve.

Please excuse the sloppy code, this is just some train-of-thought stuff:

import macros

type
  Foo = ref object
    a: int
  NonNilFoo = Foo not nil
  Ref = concept x
    isNil(x)
  Nillable = concept x
    isNil(x) is bool
    x = nil
proc isOptimizable(T: typedesc): bool =
  return T is Ref and T isnot Nillable
type Option[T] = object
    val: T
    when isOptimizable(T):
      isSome: bool

proc isNillable(T: typedesc): bool =
  when T is Ref and T isnot Nillable:
    return true
  else:
    return false
proc box*[T](val: T): ref T not nil =
  new result
  result[] = val

doAssert false == isNillable(Foo)
doAssert true == isNillable(NonNilFoo)
doAssert false == isNillable(int)
doAssert false == isNillable(ref int)


type NNRI = ref int8 not nil
echo repr(Option[NNRI](val:box(1i8)))
echo repr(Option[int8](val:1, isSome: true)) # fails, isSome doesn't exist because of #2002

[oprypin]

If it is possible to use when like that inside a type, then yes, this could all be made obsolete. The code seems to crash the compiler, though.

[ReneSac]

Just to bikeshed a little: I prefer .get over [], as it is like .val, but I don't fell so strong about this. I too hope the module can be simplified like flaviu is proposing.

[oprypin]

People seem to be in favor of dropping the nil-based implementation. So that's what I did...

[flaviut]

This is too easy to use IMO, I'd like something long and verbose like "unsafeVal" better.

[Araq]

I agree with flaviut. People who want Maybe[T] also obviously want the extra typing.

[flaviut]

LGTM 👍

A couple minor, non-blocking notes:

• var x: Maybe[int]; assert(not x.has) is true. This could be mentioned in the docs, but I'm not sure if encouraging this is a good idea.
• As I've said, I'm really not a fan of ?Foo. Unfortunately, it looks like I'm alone there.

[dom96]

I prefer .get too. Let's not give two meanings to [], you're not dereferencing a Maybe[T].

I would also prefer an optional default value for .get instead of the or operator overload. This has the additional benefit of being consistent with the json module.

I dislike the has proc and would prefer something more explicit like isJust and isNothing perhaps.

[dom96]

Is using val possible? It's not exported, so the docs shouldn't imply this.

[oprypin]

Woops, forgot to change that.

[oprypin]

We're slowly converging to https://github.com/flaviut/optional_t

[dom96]

You're right. Perhaps we should just adopt it.

[oprypin]

Maybe type has these operations: get-with-exception, unsafe-get, get-or-default, has.
Let's just pick one of these naming options:

• [], unsafeVal, a or b, has
• get, unsafeGet, a.get(b), isJust

I think that I like the 2nd ones better.

---

By the way, "just" adopting optional_t is not so easy. In any case, I already have docgen docs and tests here.

[dom96]

I like the second one better too. I am sceptical about the need for unsafeGet.

[refi64]

@dom96 I like an unsafeGet for those times when you know more than the compiler.

[dom96]

Alright, leave it in then.

[ReneSac]

I like better the third option:

• get, unsafeGet, a or b, has

"a.get(b)" don't reads well in my opinion. I don't like much isJust nor has. Rust uses Some. Swift uses simply a != nil, and then ! operator as an unsafeGet (but there are many other methods of treating optionals there too). I don't know what is better, all names sound strange for me, so I just picked the shorter one.

[bluenote10]

May I suggest to use the naming convention Option/Some/None instead of Maybe/Just/Nothing. In fact, my first functional language was Haskell, so I was actually a fan of Maybe/Just/Nothing for a long time. However, I have changed my mind, maybe because:

• Linguistically, "some" and "none" are (rather) clear opposites. In contrast to this, it is difficult to see that "just" is the opposite of "nothing".
• Both constructors are four-letter words => nice symmetry + more concise.
• Arguably, some/none has advantages regarding readability. After all, "just(5)" is literally not just five, since it has a different type than "just five". In Haskell, this also made it a bit hard for me to understand the monoid structure behind it e.g. when it comes to flatmapping. To me, "some(5)" puts a slight emphasis on the type difference (I have a collection which contains some value) which imho improves the readability of a flatmap.
• It looks like the general trend is to go for Option/Some/None, even for languages which are inspired by Haskell. With the exception of Idris, all other languages I'm aware of (Scala, F#, Ocaml, Rust) avoid the Maybe/Just/Nothing convention -- probably motivated by a similar reasoning.

The most concise naming pattern (Maybe + Some/None) would obviously work as well, but it is probably better to sacrifice one character in order to follow the most common convention.

[bluenote10]

Hm, this is weird. Traditionally, if we already have a value, injecting a default should not have any effect. I would expect just(5) or 2 == 5.

[flaviut]

Maybe it's getting parsed as assert just(5) or (2 == 2), although that should be a type error.
I'll look into it once I get to a computer.

[oprypin]

Must be assert just(5) or (2 == 2). Don't forget converter toBool. 😟
This is a huge argument against naming this operation as or.

[flaviut]

Ah, good point. That's how it typechecks.

assert just(5) or 2 == 2
assert just(5) or (2 == 2)
assert toBool(just(5)) or (2 == 2)
assert true or true

Isn't it possible to just get rid of toBool()? It's pretty much useless with ?= anyway.

[dom96]

Converters are usually a bad idea. Remove the converter instead of renaming or.

[bluenote10]

I spend a bit more time to make up my mind about the open questions here. @BlaXpirit I have made a few changes to this PR, which would be available here if you want to use any of this.

What I changed, or was thinking about changing (apart from the name change mentioned above):

• I added map and flatMap which are crucial imho when chaining optional computations in the traditional functional fashion.
• I added an items iterator, since Options should have a symmetrical design to collections (i.e., allowing monadic for comprehension).
• I use isSome and isNone (for symmetry) instead of has.
• I tried to provide o.isSome(x) similar to the ?= macro, because the result would be extremely close to pattern matching: if o.isSome(x): ... else: .... However I could not get it to work since it looks like the template must be immediate, and therefore overloading resolution is a problem.
• I had to make the ?= template immediate to use it in the map/flatMap functions. Obviously, I could avoid using them, but maybe this means the template should be immediate in general? (I haven't fully figured out when immediate is required and when not)
• I would prefer get and unsafeGet instead of [] + unsafeVal (probably because the two functions are syntactically too different).
• I'm torn regarding the or syntax. On the one hand, it is nice, because it avoids parentheses. On the other hand, I think it is important to see the resulting type of "chained options", because it is such a big difference if you inject a default or another optional. For instance with a or b or c it is unclear if the result is still optional or has a default injected. Therefore, I suggest to go the Scala/Rust way, to make it explicit at the expense of being parenthesis-free. In my implementation getOr means "still optional" and getOrElse means "definitely has a default". Maybe there are better names for them... Using the or also has other issues when it comes to Option[bool]. For instance, one may want to overload or for the boolean operations themselves, and some(false) or some(true) == some(false) looks unintuitive. Without distinguishing the two different types of or one may also accidentally chain a default twice opt or default or default -- the first injects the default, the second is actual or.
• I'm also undecided regarding the ? template. Nice abbreviation but maybe the language should provide a unique syntax? Could it be an issue regarding namespace pollution?

Sorry for the lengthy comment!

[oprypin]

@bluenote10

Great points about Option/Some/None. Agree with everything. This is not Haskell.
Does anyone except Araq like Maybe/Just/Nothing?

About the names -- yes, I like some of those better as well:

• [], unsafeVal, a or b, has
• get, unsafeGet, a.get(b), isJust

I think that I like the 2nd ones better.

---

Agree with everything except:

In my view, flatMap is just a shorthand with little use.

getOrElse looks ugly to me. And I don't see a real need for an operation like getOr. Tempted to drop it while dropping or operator.

I don't see a need to cater for chaining of these calls because optionals are just going to be rarely used in Nim.

name pollution

What else are you gonna use ?(typedesc) for?

You're the first one to suggest an iterator. I don't see a need for it. It's weird.

---

All in all, you're going too far seeking to borrow elegance from functional languages. Nim's standard library has neither the elegance, nor much else in common with them.

[flaviut]

May I suggest to use the naming convention Option/Some/None instead of Maybe/Just/Nothing

I'd like that too.

I added map and flatMap which are crucial imho when chaining optional computations in the traditional functional fashion.
I added an items iterator, since Options should have a symmetrical design to collections (i.e., allowing monadic for comprehension).

👍

For instance, one may want to overload or for the boolean operations themselves

It's a container, it should not special case any type.

Without distinguishing the two different types of or one may also accidentally chain a default twice opt or default or default -- the first injects the default, the second is actual or.

or(opt: Option[T], or(default: T, default: T)) will fail to typecheck. Although behavior might be weirder for booleans in or, where some(true) or false or true might act in an unexpected way

I'm also undecided regarding the ? template. Nice abbreviation but maybe the language should provide a unique syntax? Could it be an issue regarding namespace pollution?

I dislike ?. Identifiers should be longer and more descriptive.

---

just a quick remark:

for v in some(1):
  ...

if v ?= some(1):
  ...

Main advantage of if ... ?= ... is that an else clause is possible.

[oprypin]

The pull request started huge and complicated. Back then I wanted to have all the features one could possibly want.

Then Araq shared his view. The main point of having this in the standard library is to avoid countless incompatible implementations.

It is nice to be able to return an option from a function, instead of nil or -1 or tuple-with-bool. I don't see much other uses for optionals. They are not going to become ubiquitous in Nim, they'll stay a black sheep. So I want to define the base intuitive set of operations, very quick to learn, and not turn this into a haven of functional programming.

Then, if people insist, each individual additional operation's merits would be discussed separately, because there are just too many conflicting opinions here.

Besides, it is easy to add more operations in your own code. Again, the main point is compatibility.

[flaviut]

Then, if people insist, each individual additional operation's merits would be discussed separately, because there are just too many conflicting opinions here.

In that case, all we need to get merged in this PR are the minimal operations for this type:

get(Option)
get(Option, default)
unsafeGet(Option)
isSome(Option)
isNone(Option)
some(T)
none[T]()

Note that I didn't include ?=. I think that should wait until we come up with a consistent policy for pattern matching that can also be used elsewhere.

We can bikeshed over the rest later and figure things out.

---

I'd also like to point out that this is a blocker for #2511, so the faster we get something merged, the faster we can move on to more exciting stuff.

[oprypin]

Dropping ?= would really hurt, but it's something to consider.

Maybe we should draw the line at "features that will obviously be useful all the time"? 😛

[bluenote10]

Nim has a huge potential to cater for the needs of functional programming as well. I don't see the benefit of leaving out common functionality. A functional programmer new to Nim will look for the "option monad", and may be disappointed that it is only semi-usable for their needs. With just a few additions, we could satisfy all typical use patterns of optionals in functional programming.

@BlaXpirit Maybe a simple example helps to see why flatmapping is important and beautiful when it comes to option chaining. As soon "it is nice to be able to return an option from a function" it becomes "nice to easily combine these results". The question is not about whether or not optionals will be ubiquitous in Nim. The problem simply is there as well. The main point of having optionals is to provide a beautiful solution to the latter, not just wrapping a value.

Regarding overloading get: I still think that this is not the place for overloading simply because the semantics are so different.

[Araq]

Tell me when you guys figured it all out and want my review. ;-) IMO it should be as slim as possible and not support any FP feature under the sun just to show off that Nim can do it.

[bluenote10]

IMO it should be as slim as possible and not support any FP feature under the sun just to show off that Nim can do it.

This is pretty much what I tried to achieve today. I went back and forth from the interface definitions of Haskell/Scala/Rust, and reduced it to their bare minimum (e.g. Rust has more combination functions, which are somewhat uncommon anyways; Scala offers the whole "Traversable" stuff, which in Nim comes down to just providing an iterator, being the common denominator of all collections -- for that reason I also left out any "toContainer" conversion). In the end I added only 2 procs and 1 iterator (everything else was just renaming), so I really don't think that this is "going too far". Also, the names of the added functions (map and flatMap) are pretty much standard and non-critical. I'll have to sleep on it, but I think the main functional use patterns should already be covered.

[lokulin]

Was just wishing that Nim had an Option/Maybe today! 👍

[dom96]

Adding extra functionality is easy, removing it is not. For this reason I think we need the slimmest implementation possible for now, until we can fully find out which functionality is missing and the best way to find that out is to start using this module.

[bluenote10]

👍 for not having a converter and keeping it simple for the time being.

Btw: I really think it should be just the items iterator instead of the ?= template, since the template definitely has a bug potential:

if x ?= opt or true: # where "or true" is hidden in a more complex expression
  # executes code which should never be reached since x may not be valid

# in contrast:
for x in opt:
   # no way to reach this code if opt is empty

[reactormonk]

Closed because #2762