Garbage Collection

[ChrisRackauckas]

It would be good to garbage collect the functions after the handle is lost, since it should essentially be only related to that one object that created it. That would be done by using a finalizer that deletes the function cache from the dictionary. Though that might cause unsafety if someone creates the same function twice. @c42f is there a way to do this safely? The reason we'd want this is because we're looking to build some million like expressions (and chop them up).

[simeonschaub]

It sounds like this should be pretty straightforward, if RuntimeGeneratedFunction was made mutable. Creating a new RuntimeGeneratedFunction could just increase a counter that's cached with the id; that way, it can be tracked how many times the same function is created. It wouldn't be threadsafe of course, but I'm pretty sure the current approach isn't either.

[c42f]

Implementing this with refcounting would work I suppose. But I think it makes more sense to just let the GC do the work. That's what I did in #9 by making the cache only hold a WeakRef rather than rooting the function body.

[willow-ahrens]

I'm hesitant to ask because I'm not totally sure if/how Julia handles garbage collection of overwritten or deleted methods, but since this is a library for "data as code," I feel the need to clarify: Does it make sense to add a finalizer that deletes the compiled method (Base.delete_method(m)) when the runtime generated function becomes unreachable? I understand that the gc can handle deleting the exprs in the cache, but I'm not convinced yet that the gc would recognize that the compiled method will never be called again, either.

[ChrisRackauckas]

A generated function fakes purity and the Julia compiler just slurps it up. So once it's compiled for a dispatch, it won't ever check the generated function again since it'll just use the cached compiled code. This has the weird side effect that if you modify the expression in the dictionary it won't actually update a RuntimeGeneratedFunction, but I think that same process is what makes this safe.

[willow-ahrens]

I'm asking more about what happens to the cached compiled code after generation. If I were to make a really big (like 100MB) expression, when I runtime generate it and call it, that 100MB expression gets saved in the RuntimeGeneratedFunctions.jl cache, but it also gets loaded into the generated function, compiled, turned into Julia IR, LLVM, and eventually assembly. Presumably julia caches at least the assembly somewhere, and it's probably also 100MB or so, right? I'm wondering if we need to delete that as well so we don't leak memory.

[willow-ahrens]

Essentially, I'm concerned that there will be a bunch of specializations of RuntimeGeneratedFunctions.generated_callfunc for unreachable ids. Is the memory which stores the code for these methods ever reclaimed?

[ChrisRackauckas]

I haven't seen that leak in practice at least, and it was a major issue with GeneralizedGenerated that caused the creation of this repo so I assume it's okay? But it's hard to know.

[willow-ahrens]

Memory usage of this simple loop stays static at about 400 MB:

while true
    rand(10000)
end

However, the following code explodes the memory usage of Julia 1.5.3:

RuntimeGeneratedFunctions.init(Main)
while true
    ex = :(function _f(x)
        return $(rand(10000))
    end)
    f = @RuntimeGeneratedFunction(ex)
    f(nothing)
end

That said, I'm not sure if it is the intention of this package to avoid such an outcome.

[willow-ahrens]

Digging a bit, no code that Julia compiles is ever garbage collected.
JuliaLang/julia#18446
JuliaLang/julia#14495
JuliaLang/julia#19013
JuliaLang/julia#20755

[willow-ahrens]

I understand now I was wrong earlier when I suggested this might be solved by Base.delete_method. Even if that function reclaimed memory (which it does not), there's only one method of the generated function. I think we may want the ability to delete/remove compiled code from the internal code cache?