Produce static/const initializations for variables when possible

[exelotl]

A top-level initialization in Nim produces C code which first declares the variable, then later assigns a value in the module's Init function.

This is troublesome for embedded programming, because there is no way to make a variable 'const' in the generated C code.

As a result, there is no sensible way to put an object/array in ROM and be able to take a pointer to it using unsafeAddr. See my forum post for why this is important.

Example

The following Nim code:

type Point = object
  x, y: int

let foo* = Point(x:10, y:20)

currently leads to the following C code for our variable 'foo':

// ...

tyObject_Point_R2WHeB5ABtUUFMDs9bCVSuA foo_pp3aIVrMTc9cwqFMF9btpcoQ;

// ...

N_LIB_PRIVATE N_NIMCALL(void, unknown_mymoduleInit000)(void) {
{
	nimZeroMem((void*)(&foo_pp3aIVrMTc9cwqFMF9btpcoQ), sizeof(tyObject_Point_R2WHeB5ABtUUFMDs9bCVSuA));
	foo_pp3aIVrMTc9cwqFMF9btpcoQ.x = ((NI) 10);
	foo_pp3aIVrMTc9cwqFMF9btpcoQ.y = ((NI) 20);
}
}

Proposed Solution

After some further dicussion on IRC, we decided that the best way to handle this would be to make the codegen for let smarter, so that it will produce a fully initialized global or static const in cases where that is possible.

In other words, the previous C code would become:

const tyObject_Point_R2WHeB5ABtUUFMDs9bCVSuA foo_pp3aIVrMTc9cwqFMF9btpcoQ = { ((NI) 10), ((NI) 20) };

I believe this can be done for any top-level let variable as long as the initialization only contains the following:

• literals / constants
• addresses of other top-level variables / procedures
• cast[], sizeof, and any other operations that are acceptable in a static initialization context

The same rules can apply to var too (though I suppose it's less important). The 'const' modifier should be omitted in that case.

However, the codegen should not be applied for any initialization that depends on the result of a procedure or the value of a variable at runtime.

More Examples

Visibility/mutability rules

Nim code:

let a = 1
let b* = 2
var c = 3
var d* = 4

Desired C initializers:

static const NI a_xxx = 1;
const NI b_xxx = 2;
static NI c_xxx = 3;
NI d_xxx = 4;

Using C-style arrays (pointer + size)

Nim code:

type AnimData = object
  loop: bool
  frames: ptr UncheckedArray[int]
  len: int

let walkFrames = [2,3,4]

let walkAnimation = AnimData(
  loop: true,
  frames: cast[ptr UncheckedArray[int]](unsafeAddr walkFrames),
  len: walkFrames.len
)

Desired C initializers:

static const tyArray_xxx walkFrames_xxx = {((NI)2), ((NI)3), ((NI)4)};
static const tyObject_AnimData_xxx walkAnimation_xxx = {
	NIM_TRUE,
	(&walkFrames_xxx),
	((NI) 3)
};

Linked list of procedures

Nim code:

type Action = object
  body: proc () {.nimcall.}
  next: ptr Action

proc fn1() = echo "first"
proc fn2() = echo "second"

let second = Action(body: fn2, next: nil)
let first = Action(body: fn1, next: unsafeAddr second)

Desired C initializers:

static const tyObject_Action_xxx second_xxx = { fn2_xxx, NIM_NIL };
static const tyObject_Action_xxx first_xxx = { fn1_xxx, (&second_xxx) };

Other considerations

• This change may mess with some people's assumptions about the behavior of unsafeAddr. Currently let variables are still mutable under the hood, but with these changes that would not always be the case.
• I suspect this wouldn't play nicely with every imported C variable. (See the EACCES example in the manual).
• It might be worth introducing some pragma to control which kind of codegen is used, in case you (a) really want to guarantee static const (b) really want to avoid it

[krux02]

Well, I looked at the generated code, when I change the let into a const. And this is what the code generator spits out:

NIM_CONST tyObject_Point__r9bGBjWagGEgqMgn2hQfycQ TM__X9bGGOoNyKehOTF7f2xys3g_2 = {((NI) 10), ((NI) 20)};

For some reason the name foo got lost in translation, I don't know yet why that is the case. But this is already almost what you want. The only problem is, that the nim compiler doesn't let you take the address of a const. Well, I tested what happens when I remove the check, and it already works.

I tested what happens when I take the address from the same const from different modules, and it is a different address each time. Constants are instantiated once every time they are used. For small constants, such as PI this is probably not a problem at all, but considering your use case where you want to put your resources into constants, it might become a problems.

So what I am proposing is, that instead of making let work like const, I suggest to make unsafeAddr work on const values, because I think const is much closer to what you want than let. On top, it won't break anybody's assumptions about let.

I don't think it would cover the linked list use case with the pointers. Please elaborate if you really think that one is important.

Then since you want to use constant sections in your binary for resources, there is another idea in my head that I never got time to implement. And that is a feature I know from BlitzBasic, it is explicit resource bundling. Here is an example of that feature in the Monkey2 language. The big advantage here is, the resource data is not generated into C source code. Instead the linker is instructed to bundle the binary data into a section of the binary file. I don't think that you would want to generate hundreds of megabytes of data into C literals and then compile this with a C compiler, when you can just link the data.

I think though that the constant duplication should be removed though. Getting the addres from the same symbol but within different modules should not generated different addresses. This doesn't seem to be a hard problem though. I think we can then also revert to use the original symbol name. People using GDB will be grateful.

[exelotl]

I think allowing unsafeAddr for consts would be fine too, as long as it doesn't create opportunities for bugs (e.g. how would consts containing pointers to other consts work at compile time?). I mentioned something similar in my forum post, but I think Araq is more in favour of solutions around let, I guess for semantic reasons?

For my own use case, I don't particularly need a linked list of pointers to functions (though I wouldn't rule out such a structure being useful), but I do need all the individual parts of that example for different reasons.

1. I need constants that hold pointers to functions, to implement dispatch tables for scenes and entities in my game.
2. As far as I'm aware, I need constants with pointers to other constants, in order to have a type like AnimData in my examples (i.e. pointer to array + length of array)

As a separate thing, binary resources sounds like a nice idea too! For my project, the devkitARM makefiles already allow that (I put files in a 'data' folder and it generates .o files). It's handy for raw image data and other stuff that doesn't come from Nim, but somehow their 'bin2o' tool is slower than just using staticRead, so I switched to that instead where possible.

[mratsim]

I also prefer adding an {.addressable.} pragma to const.

However I don't see how you can have const pointers even to other consts.

I have 2 use-case for addressable consts:

• LUT Table, for example to reimplement transcendental functions like exp without having the compiler annoying me with side effects:

const
  ExpBits* = 10'i32
  ExpBitsMask* = 1'i32 shl ExpBits

  MantissaBits* = 23'i32
  MantissaBitsMask* = 1'i32 shl 23

  ln2 = ln(2'f32)

  ExpMax* = 88'i32
  ExpMin* = -88'i32
  ExpA* = ExpBitsMask.float32 / ln2
  ExpB* = ln2 / ExpBitsMask.float32

func initExpLUT(): array[ExpBitsMask, int32] =
  for i, val in result.mpairs:
    let y = pow(2'f32, i.float32 / ExpBitsMask.float32)
    val = cast[int32](y) and (MantissaBitsMask - 1)

# We need ExpLUT in the BSS so that we can take it's address so it can't be const
let ExpLUT* = initExpLUT()

• Storing static big ints in the BSS, only once, when doing modular arithmetic repeatedly on the same prime known at compile-time for elliptic curve cryptography

type
  Fp*[P: static BigInt] = object
    ## P is a prime number
    ## All operations on a field are modulo P
    value: type(P)

  Montgomery*[M: static BigInt] = object
    ## All operations in the Montgomery domain
    ## are modulo M. M **must** be odd
    value: type(M)

[cooldome]

IMO, it is easier to implement addressable const but it is a step back. It would help with this use case but others will complain as it is not always a win. While using const for let would some be something everyone will benefit.

[mratsim]

Those are not mutually exclusive though.

On another note, with let foo {.compileTime.} being visible at runtime (which I don't really agree with but that's another issue), is the data inlined at use site or is it stored in the binary and then a pointer is used, for example for an array.

[arnetheduck]

fwiw, nlvm already does this for let globals, and it generates code similar to const <type> = <init> for all let. It doesn't use the equivalent of static because afair that may cause two symbols with the same content within a compilation unit share address - this is fine for const, but not for let (ie that's one significant difference between then).

there are other differences: const-style stuff is generally expected to be inlined at compile time while let should be linked - I don't think I've seen this used practically in nim, but basically if you compile a static library and change a let value, the value from the static library should be used. this is similar to putting a constant in the header vs creating a symbol for it and linking it in, in C.

thus for let *, it seems that it should use const in the declaring module and extern const; in others to ensure it's the same symbol across the compilation (nlvm compiles everything as a single module so it doesn't have this issue), depending on desired address nuances.

[arnetheduck]

http://llvm.org/docs/LangRef.html#linkage-types has a good reference on the various linking options and what they mean / imply. here's the corresponding llvm IR that gets generated for c - can play around with different options easily: https://gcc.godbolt.org/z/1VS0vJ