High level design documentation?

[madsmtm]

I'm currently reading the excellent documentation on rustc's query system, and wondered what your plan for Wild is regarding achieving incremental linking? Do you have some documentation or design drafts, or is this all still far out in the future?

[davidlattimore]

I am getting closer to the point where I'll probably start looking at implementing incremental compilation soon. I haven't decided on a concrete plan as yet. I do have two possible ways that I'm considering.

One is to keep the linker in memory. Effectively have a cargo-watch style command, that runs cargo, wraps rustc, then watches for changes to any source files. When it sees a source file change, reruns the relevant rustc command directly (without invoking cargo), then takes the new outputs from rustc and integrates them into the existing in-memory link.

The other approach is to persist bits of state to disk, then when invoked, load that state, see what input files have changed and reprocess those. I wouldn't want to use any serialisation for persisting state, since that would be too slow. Rather I'd just make the on-disk format be the same as the in-memory format, so that the files can just be mmapped. That would require that the in-memory format not use any references. Most of Wild's in-memory structures already don't use any references. The biggest exception is the symbol name map, which borrows string slices from the original input files. So that would need to change. The tricky part is changing it in a way that doesn't kill performance. I suspect I should continue to reference the original input files, but just do so in a way that doesn't use references.

[madsmtm]

Interesting, thanks for the answer!

Just curious now, would there be any idea in merging the linker into a fork of rustc itself, probably with cranelift too, so that all the parts of the compiler can share the query system, and we can have "truly" incremental compilation? But that'd probably be too big of an undertaking.