Kaleido process in package initialisation hangs

[jd-foster]

Described: https://discourse.julialang.org/t/plotlyjs-0-18-11-hangs-when-trying-using-but-only-the-first-time/105636

I am experiencing a strange behavior with PlotlyJS 0.18.11.
The first time I try “using” the package, Julia hangs. If I interrupt the process (CTRL-C, I am on Windows 10) I get error different error messages in VSCode or in the REPL launched from a command prompt (see below). But in both cases, after interrupting the process, a second attempt to “using” works, and I haven’t noticed any problems with plots. I am on Julia 1.9.3, Windows 10, and just downgrading to PlotlyJS 0.18.10 fixes the issue in the environments I tried.

Version info
Julia 1.9.3, Windows 10, and happens on upgrading PlotlyJS 0.18.10 to PlotlyJS 0.18.11.

Appears to be related to the recent change in #469 (possibly evident more often on Windows?).

[empet]

I have Julia 1.9.0, Windows 10, and PlotlyJS 0.18.11 works fine. I suspect that there is an incompatibility with Julia 1.9.3.

[jd-foster]

Thanks for the information. I can't reproduce this myself under the same versions, but the original poster of the issue can reproduce the hanging behaviour reliably inside VS Code & julia extension, but not so reliably from command-line REPL use.

[BeastyBlacksmith]

that might be fixed by replacing the kaleido setup here with https://github.com/JuliaPlots/PlotlyKaleido.jl

[cserteGT3]

I also have similar issue. Sometimes I can load PlotlyJS, but what never works is Dash.jl+PlotlyJS (the extension precompilation always hangs). I'm on Windows 11, Julia v1.9.3, Dash v1.4.0 and PlotlyJS v0.18.11.
I am happy to test any code that helps debugging. (I can use the 0.18.10 version until the julia PR mentioned in the discourse thread is merged and released.)

[jd-foster]

@cserteGT3 Thanks for reporting this. Do you have any steps you can share that reproduces this, even "sometimes"? If you observe those times that PlotlyJS loads, what does PlotlyJS._kaleido_running() give you?

[cserteGT3]

I have the following environment:

(dashtemp) pkg> st
Status `C:\Users\username\dashtemp\Project.toml`
  [1b08a953] Dash v1.4.0
  [f0f68f2c] PlotlyJS v0.18.11

If I start a new julia session, the following happens (just updated to julia v1.9.4, therefore the recompiling and also possibly this was todays first call of PlotlyJS):

julia> using PlotlyJS
[ Info: Precompiling PlotlyJS [f0f68f2c-4968-5e81-91da-67840de0976a]

julia> PlotlyJS._kaleido_running()
true

julia> using Dash
[ Info: Precompiling Dash [1b08a953-4be3-4667-9a23-3db579824955]
[ Info: Precompiling DashBasePlotlyJSExt [1bcbdc22-5094-5a35-94fb-8d851f9434fe]

And the last line hangs.
If I exit and start a new session, then using PlotlyJS in itself hangs. Sending ctrl+c will cause the following:

julia> using PlotlyJS
# pressing ctrl+c after a few minutes
┌ Warning: Kaleido is not available on this system. Julia will be unable to save images of any plots.
└ @ PlotlyJS C:\Users\username\.julia\packages\PlotlyJS\qhYQ5\src\kaleido.jl:65
┌ Warning: InterruptException()
└ @ PlotlyJS C:\Users\username\.julia\packages\PlotlyJS\qhYQ5\src\kaleido.jl:66

julia> PlotlyJS._kaleido_running()
true

If I remove the C:\Users\username\.julia\compiled\v1.9\PlotlyJS folder (thus forcing PlotlyJS to recompile next time) and start a fresh session using PlotlyJS hangs:

julia> using PlotlyJS
[ Info: Precompiling PlotlyJS [f0f68f2c-4968-5e81-91da-67840de0976a]
# pressing ctrl+c after a few minutes
┌ Warning: Kaleido is not available on this system. Julia will be unable to save images of any plots.
└ @ PlotlyJS C:\Users\csert\.julia\packages\PlotlyJS\qhYQ5\src\kaleido.jl:65
┌ Warning: InterruptException()
└ @ PlotlyJS C:\Users\csert\.julia\packages\PlotlyJS\qhYQ5\src\kaleido.jl:66

julia> PlotlyJS._kaleido_running()
true

The same happens if:

• removing PlotlyJS, PlotlyBase and Kaleido_jll from the compiled folder.
• removing the whole C:\Users\username\.julia\compiled\v1.9 folder.

Then I removed the whole compiled folder again and killed the command line I was using, and started a new command line and a new julia session. using PlotlyJS still hangs as above.

I guess I should restart Windows, but can't do that now. Later I will try, if the first use of PlotlyJS works after restarting consistently.

I hope that this is useful in some way.

[jd-foster]

Thanks for all the extra detail.

[disberd]

that might be fixed by replacing the kaleido setup here with JuliaPlots/PlotlyKaleido.jl

I have to say that I experienced this issue (on Windows only) also when using PlotlyKaleido by itself, so it might not be fully solved by switching to PlotlyKaleido (which I believe is a worthy change in any case as it now has more features than the baked-in kaleido we have here)

[make296]

Downgrading Julia to 1.5.4 solves the problem. Now can save plot to static image.

[jd-foster]

Downgrading Julia to 1.5.4 solves the problem.

@make296 I don't think that is a good idea to change your Julia version, especially anything less that the LTS version 1.6 . You're better off downgrading the package itself in most cases to the last version that worked for you.

[make296]

Downgrading Julia to 1.5.4 solves the problem.

@make296 I don't think that is a good idea to change your Julia version, especially anything less that the LTS version 1.6 . You're better off downgrading the package itself in most cases to the last version that worked for you.

I reverted back to Julia 1.9.4 and followed step 2 in sglyon/PlotlyBase.jl#40 (comment). It worked for me.

[mattcbro]

Well I can verify that importing PlotlyJS hangs on my windows laptop running Julia 1.9.4 within VScode. On Linux Mint it's problem free. On Linux the Julia libraries all load very quickly in general, whereas windows lags for some reason.

Hmm. I tried downgrading to PlotlyJS 0.18.10 and my code still hangs every time calling PlotlyJS.savefig(). So the downgrading PlotlyJS does not seem to be working for me.

[mattcbro]

Downgrading Julia to 1.5.4 solves the problem.

@make296 I don't think that is a good idea to change your Julia version, especially anything less that the LTS version 1.6 . You're better off downgrading the package itself in most cases to the last version that worked for you.

I reverted back to Julia 1.9.4 and followed step 2 in sglyon/PlotlyBase.jl#40 (comment). It worked for me.

This didn't work for me. Maybe I did things in the wrong order. Does PlotlyJS have to be installed first?

[disberd]

@mattcbro did you try forcing Kaleido_jll to v0.1.0? I found in my experiments with PlutoPlotly that it seems to fix the hanging problem on windows for me (and doesn't break saving plots from my short tests).

[make296]

@mattcbro yes, install PlotlyJS and then replace Kaleido

[mattcbro]

@mattcbro yes, install PlotlyJS and then replace Kaleido

Whoops, did it in the wrong order. But it doesn't matter now. I install Linux as a windows subsystem to get a side by side Ubuntu distro with Win 11, and then installing julia and PlotlyJS allowed everything to work using the latest versions. It's curious that the linux version just works and MS windows, not so much.

[beorostica]

I created a small repository specifically for testing this issue, and it seems that the problem can be replicated in the GitHub CI. In that environment, I added PlotlyJS 0.18.11 in a subdirectory, allowing me to log some messages.

You can find all the experiments and output messages in the GitHub Actions at beorostica/TestPlotlyJS.jl#2. For example, refer to https://github.com/beorostica/TestPlotlyJS.jl/actions/runs/7357215267/job/20028500523.

The _start_kaleido_process function gets stuck when executing:

...
res = readline(P.stdout)    # here hangs
...

I don't know how to address the problem from here, but if someone has any pointers, I would be happy to help 🙂.

[cncastillo]

Just want to add that what @beorostica describes is very similar to plotly/Kaleido#134 (comment). Maybe it helps.

[disberd]

This should now be solved after JuliaPlots/PlotlyKaleido.jl#17 and JuliaPlots/PlotlyKaleido.jl#18.

The kaleido process should not hang anymore but the kaleido library itself will hang in many windows machines for versions of Kaleido_jll > 0.1.
This is unfortunately a problem that has nothing to do with julia but that is caused by the C++ library that is simply called by julia.
It is also a problem that will likely not be solved soon as it's an open issue in https://github.com/plotly/Kaleido since years.

What JuliaPlots/PlotlyKaleido.jl#17 did is simply wait for a timeout and kill the process if it appears to be hanging. It also gives an hopefully more informative warning with a suggested fix when this happens.

The simple fix is to force Kaleido_jll to version 0.1 on machines where this hanging occur (only windows and not all of windows machines). There was another issue causing problems with Windows and Kaleido only in julia 1.10 but that was also fixed in JuliaPlots/PlotlyKaleido.jl#17 so now downgrading Kaleido_jll to 0.1 should effectively solve the problem on windows and allow to save figures.

I did not check in depth what downgrades in saving figures functionality using version 0.1 of the kaleido library brings compared to version 0.2, but all figures in my tests/use cases are fine with version 0.1

NOTE:

The hanging is effectively solved only while using version 2.2.4 of the PlotlyKaleido dependency, and only on PlotlyJS versions which actually use PlotlyKaleido as dependency (which is minimum version 0.18.3 of PlotlyJS). This should anyhow be what happens automatically if you add PlotlyJS in an environment from now on.

Due to this, I am now closing this issue as solved. We can re-open if this problem seems to persist for someone using the latest versions of the packages as explained above.

[BeastyBlacksmith]

Seems like calling PlotlyKaleido.start in __init__ is still causing the precompilation hang in julia 1.10.2 when using PlotlyJS in a package extension.

E.g. on the branch for Plots v2:

using Plots
Precompiling Plots
  79 dependencies successfully precompiled in 40 seconds. 44 already precompiled.
Precompiling PlotsPlotlyJSExt
        Info Given PlotsPlotlyJSExt was explicitly requested, output will be shown live 

[pid 52547] waiting for IO to finish:
 Handle type        uv_handle_t->data
 pipe[17]           0x312cde0->0x7f1f15246860
 pipe[11]           0x310bdc0->0x7f1f15246900
 process[52621]     0x3109f40->0x7f1f1524ff10
 timer              0x3046560->0x7f1f150f0e50
This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.

[disberd]

@BeastyBlacksmith could this be related to Plots being only compatible with PlotlyKaleido@v1 based on the compat in the Project.toml of Plots?

[BeastyBlacksmith]

On JuliaPlots/Plots.jl#4866 I set the compat to 2.2.2.
Maybe it would make sense to start kaeleido in init to check if it works and kill it right after and then restart it when calling savefig?

[disberd]

On JuliaPlots/Plots.jl#4866 I set the compat to 2.2.2.
Maybe it would make sense to start kaeleido in init to check if it works and kill it right after and then restart it when calling savefig?

I assumed we were calling start in init to save the time to start the process, which won't happen if we also kill it in init.
If it's just for precompilaton why don't we use PrecompileTools instead of init?

It would be cleaner that way

[BeastyBlacksmith]

To me that is a side benefit and its not about precompiling it. To me its about being able to warn the user something is off before you do any work, that you possibly can't save at the end.

The quest is to do that in a way that does not precompilation of the package itself and of dependent packages.

[disberd]

Ok I see, the easy way would be to just start and stop the kaleido process inside of init, but the strange thing is that start shouldn't be called during precompilation already because of this if:

PlotlyJS.jl/src/PlotlyJS.jl

Lines 102 to 105 in 75eacf4

	if ccall(:jl_generating_output, Cint, ()) != 1
	# ensure precompilation of packages depending on PlotlyJS finishes
	PlotlyKaleido.start()
	end

[BeastyBlacksmith]

In theory that shold work. Maybe that function returns 1 when precompiling extensions?