Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package doesn't work with even demo.jl #5

Open
PallHaraldsson opened this issue Aug 8, 2019 · 30 comments
Open

Package doesn't work with even demo.jl #5

PallHaraldsson opened this issue Aug 8, 2019 · 30 comments

Comments

@PallHaraldsson
Copy link

PallHaraldsson commented Aug 8, 2019

I'm looking into debugging this, I thought it might be the new improved PyCall not working with this package, but even forcing the rev. you used didn't work:

[I doubt it should matter, but maybe me using Julia Version 1.2.0-DEV.55 is the problem? At least Julia 1.1.1 (or 1.0.x) should work, see next comment. Have not yet checked 1.0.x]

julia> py2jl"""
       def sum_by(f, seq):
           s = 0
           for e in seq:
               s = s + f(e)
           return s
       result = sum_by(lambda x: x * 10, [1, 2, 3])
       """
ERROR: LoadError: "class: Module, attributes: Symbol[:class, :body]."
Stacktrace:
 [1] (::getfield(Py2Jl.ASTGen, Symbol("##12#122")){Nothing,getfield(Py2Jl.ASTGen, Symbol("#trans_block#121")){getfield(Py2Jl.ASTGen, Symbol("##7#114")){Nothing}}})(::Dict{Symbol,Any}) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:392
 [2] to_ast(::Nothing, ::Dict{Symbol,Any}) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:398
 [3] py2jl(::Nothing, ::String) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/Py2Jl.jl:43
 [4] @py2jl_str(::LineNumberNode, ::Module, ::Any) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/Py2Jl.jl:51
in expression starting at REPL[5]:1

(v1.2) pkg> st
Status ~/.julia/environments/v1.2/Project.toml
[537997a7] AbstractPlotting v0.9.5
[6e4b80f9] BenchmarkTools v0.4.2
[a93c6f00] DataFrames v0.17.1
[864edb3b] DataStructures v0.15.0
[e9467ef8] GLMakie v0.0.4
[54eb57ff] InteractiveCodeSearch v0.3.0
[6042db11] JDBC v0.4.1
[682c06a0] JSON v0.21.0
[d8e11817] MLStyle v0.3.0
[ee78f7c6] Makie v0.9.1
[dbd62bd0] MakieGallery v0.0.4
[c020b1a1] NaturalSort v0.0.1+ #fix_for_1_0_0 (https://github.com/endrebak/NaturalSort.jl)
[9b87118b] PackageCompiler v0.6.3
[91a5bcdd] Plots v0.26.0
[e53ffe30] Py2Jl v0.1.0 #master (https://github.com/JuliaCN/Py2Jl.jl)
[438e738f] PyCall v1.91.2+ #1309efc (https://github.com/JuliaPy/PyCall.jl.git)
[d330b81b] PyPlot v2.7.0
[bf110322] Python v0.1.0 #master (https://github.com/229668880/Python.jl)
[1a8c2f83] Query v0.11.0
[e6cf234a] RandomNumbers v1.2.0
[f2b01f46] Roots v0.8.1
[3646fa90] ScikitLearn v0.5.0
[992d4aef] Showoff v0.2.1
[98e33af6] SimpleDirectMediaLayer v0.1.0
[276daf66] SpecialFunctions v0.7.2
[65254759] StatsMakie v0.0.2
[9a3f8284] Random

@PallHaraldsson
Copy link
Author

I installed the latest Julia 1.1.1 and just this package and dependencies. It doesn't work either (1.0.x only?).

By default I get the expected:
┌ Warning: getindex(o::PyObject, s::Symbol) is deprecated in favor of dot overloading (getproperty) so elements should now be accessed as e.g. o.s instead of o[:s].
│ caller = process(::String) at Process.jl:57
└ @ Py2Jl.Process ~/.julia/packages/Py2Jl/G6hOG/src/Process.jl:57
┌ Warning: getindex(o::PyObject, s::Symbol) is deprecated in favor of dot overloading (getproperty) so elements should now be accessed as e.g. o.s instead of o[:s].
│ caller = to_dict(::PyCall.PyObject) at Process.jl:19
└ @ Py2Jl.Process ~/.julia/packages/Py2Jl/G6hOG/src/Process.jl:19

but even with older the older PyCall then ending with the same (and beginning there) with the same ERROR.

@PallHaraldsson
Copy link
Author

The plot thickens.

I tried installing Julia 1.0.4 to be sure, but now I get a segfault unlike for e.g. Julia 1.1:

julia> using Py2Jl
[ Info: Precompiling Py2Jl [e53ffe30-a369-11e8-316f-1171168155ae]

signal (11): Segmentation fault
in expression starting at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:29
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1191
emit_invoke at /buildworker/worker/package_linux64/build/src/codegen.cpp:3094
emit_expr at /buildworker/worker/package_linux64/build/src/codegen.cpp:3893
emit_ssaval_assign at /buildworker/worker/package_linux64/build/src/codegen.cpp:3615
emit_stmtpos at /buildworker/worker/package_linux64/build/src/codegen.cpp:3801 [inlined]
emit_function at /buildworker/worker/package_linux64/build/src/codegen.cpp:6262
jl_compile_linfo at /buildworker/worker/package_linux64/build/src/codegen.cpp:1159
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1774
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
#1 at /home/qwerty/.julia/packages/MLStyle/ksIXg/src/Internal/Toolz.jl:23
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1809
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
#1 at /home/qwerty/.julia/packages/MLStyle/ksIXg/src/Internal/Toolz.jl:23
$ at /home/qwerty/.julia/packages/MLStyle/ksIXg/src/Internal/Toolz.jl:8
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1809
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
gen_match at /home/qwerty/.julia/packages/MLStyle/ksIXg/src/MatchCore.jl:220
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1809
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
@match at /home/qwerty/.julia/packages/MLStyle/ksIXg/src/MatchCore.jl:238
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1809
jl_invoke_julia_macro at /buildworker/worker/package_linux64/build/src/ast.c:987
jl_expand_macros at /buildworker/worker/package_linux64/build/src/ast.c:1049
jl_expand_macros at /buildworker/worker/package_linux64/build/src/ast.c:1092
jl_expand_macros at /buildworker/worker/package_linux64/build/src/ast.c:1092 [inlined]
jl_expand_stmt at /buildworker/worker/package_linux64/build/src/ast.c:1138
jl_eval_module_expr at /buildworker/worker/package_linux64/build/src/toplevel.c:230
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:655
jl_eval_module_expr at /buildworker/worker/package_linux64/build/src/toplevel.c:232
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:655
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:838
jl_load at /buildworker/worker/package_linux64/build/src/toplevel.c:839
include at ./boot.jl:317 [inlined]
include_relative at ./loading.jl:1044
include at ./sysimg.jl:29 [inlined]
include at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/Py2Jl.jl:1
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:324
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:430
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:363 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:682
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:806
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7ff25e6fdc7f)
unknown function (ip: 0xffffffffffffffff)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:815
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:805
jl_eval_module_expr at /buildworker/worker/package_linux64/build/src/toplevel.c:232
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:655
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:838
jl_load at /buildworker/worker/package_linux64/build/src/toplevel.c:839
include at ./boot.jl:317 [inlined]
include_relative at ./loading.jl:1044
include at ./sysimg.jl:29
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:324
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:430
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:363 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:682
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:613
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:806
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7ff25f30acbf)
unknown function (ip: 0x2)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:815
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:805
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319 [inlined]
eval at ./client.jl:393
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
top-level scope at ./none:3
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1809
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:799
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/builtins.c:622
eval at ./boot.jl:319
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
exec_options at ./client.jl:243
_start at ./client.jl:425
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2162
jl_apply at /buildworker/worker/package_linux64/build/ui/../src/julia.h:1537 [inlined]
true_main at /buildworker/worker/package_linux64/build/ui/repl.c:112
main at /buildworker/worker/package_linux64/build/ui/repl.c:233
__libc_start_main at /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
_start at /home/qwerty/julia-1.0.4/bin/julia (unknown line)
Allocations: 7301695 (Pool: 7300382; Big: 1313); GC: 16
ERROR: Failed to precompile Py2Jl [e53ffe30-a369-11e8-316f-1171168155ae] to /home/qwerty/.julia/compiled/v1.0/Py2Jl/hBweI.ji.
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1203
[3] _require(::Base.PkgId) at ./loading.jl:960
[4] require(::Base.PkgId) at ./loading.jl:858
[5] require(::Module, ::Symbol) at ./loading.jl:853

@thautwarm
Copy link
Member

Don't know why either but the Segmentation fault cannot be my fault... I'll do a quick patch.

@PallHaraldsson
Copy link
Author

PallHaraldsson commented Aug 8, 2019

What exact version should work (e.g. some 1.0.x prior to 1.0.4, or 1.1.y for some y, or did you use some 1.2 later?)?

I got (v1.1) pkg> add https://github.com/JuliaCN/Py2Jl.jl#72aaa82D

to get the same error in 1.0.4, but at least not a segfault.

@thautwarm
Copy link
Member

It should be 1.1.x, but I think it should be okay to work in 1.0.x.
The error class: Module, attributes: Symbol[:class, :body] is caused for === didn't work for String comparisons in 1.0.x, which raises a bug of the older MLStyle.jl. I'm now working on this, don't worry.

@thautwarm
Copy link
Member

Sorry, I've just got passed the tests... What I've done is simply updating MLStyle.jl dependency..

@PallHaraldsson
Copy link
Author

PallHaraldsson commented Aug 8, 2019

Ok, so you always used 1.1, when I suppose it wan not yet released. This could be very helpful for me (for work on Polyglot.jl) discussed here:

https://discourse.julialang.org/t/compatibility-across-julia-1-versions/27196/15#

@thautwarm
Copy link
Member

Some bugs in 1.0.x has been already resolved several months ago, but I don't sync this repo's dependencies.

@PallHaraldsson
Copy link
Author

PallHaraldsson commented Aug 8, 2019

You mean you used 1.0.x then changed to 1.1.y, and assumed all 1.0.z versions would still work (as you should)?

[I just need to know what commit of your code worked in some exact Julia version, e.g. some exact y (or x). Then I could try that combination of exact Julia version and older commit of your code. Or wait for you to fix, not pressuring to fix or answer soon, just curious to try your package.]

@thautwarm
Copy link
Member

No, this package is developed and tested in 1.1+, I've just tried v1.0, and got segmentation fault.
IIRC I never ran this in 1.0.x before.

@thautwarm
Copy link
Member

The Py2Jl.jl for 1.1 has just got fixed. Are you still getting errors?

@PallHaraldsson
Copy link
Author

PallHaraldsson commented Aug 8, 2019

Yes, still the same error in Julia 1.1.1.

I tried to drop the packages and do e.g. (to see if matters):

julia> ENV["PYTHON"] = raw"/usr/bin/python"  # not sure if needed?
"/usr/bin/python"

(v1.1) pkg> build PyCall

@PallHaraldsson
Copy link
Author

Yes, I mean I'm getting an error for what you have in the test cases... Did it work for you?! Am I maybe doing something wrong, or do you not have something to run the test (automatically) prior to committing?

@thautwarm
Copy link
Member

Did it work for you?! Am I maybe doing something wrong, or do you not have something to run the test (automatically) prior to committing?

Yes it does work for me... However I think it's not your fault. Did you update the dependency? Or is your PyCall version exact to my Project.toml?

@thautwarm
Copy link
Member

or do you not have something to run the test (automatically) prior to committing?
No need before running the test?

Did you try using package mode to test it directly?

julia1.1+ > ]
julia1.1+ > activate . 
julia1.1+ > test

@thautwarm
Copy link
Member

thautwarm commented Aug 8, 2019

Hi, I found that although in Julia 1.0.4 running the test will raise segmentation faults, if I paste all codes into julia shell, I can run it correctly(with many warnings though).

@PallHaraldsson
Copy link
Author

PallHaraldsson commented Aug 8, 2019

Yes, I get segfault for "using Py2Jl" on 1.0.4; and thus for "test Py2Cl" (which implies "using"; good to know about that command).

On 1.1.1 however, no segfault, so since I've reported segfault (and you too for same?), will concentrate on 1.1.1 for the time being.

Note, not just warnings, but an error at the end.

You can disable all the warnings with pkg> add PyCall#v1.18.0

Since those are less important, for now, I would concentrate on the error first, then tag maybe when working, then fix warnings and tag when that works without then.

@thautwarm
Copy link
Member

Yes, in my local environment for this repo,

(Py2Jl) pkg> status
Project Py2Jl v0.1.0
    Status `~/github/Py2Jl/Py2Jl.jl/Project.toml`
  [682c06a0] JSON v0.19.0
  [d8e11817] MLStyle v0.3.0
  [438e738f] PyCall v1.18.0
  [8dfed614] Test

It seems a pretty simple dependency case, I don't know why you got the error:

(Py2Jl) pkg> test
   Testing Py2Jl
 Resolving package versions...
Test Summary: | Pass  Total
py2jl mod gen |    4      4
   Testing Py2Jl tests passed 

@thautwarm
Copy link
Member

My global status:

(v1.1) pkg> st
    Status `~/.julia/environments/v1.1/Project.toml`
  [7d9fca2a] Arpack v0.3.1
  [6e4b80f9] BenchmarkTools v0.4.2
  [7057c7e9] Cassette v0.2.5
  [864edb3b] DataStructures v0.17.0
  [c91e804a] Gadfly v1.0.1
  [7073ff75] IJulia v1.18.1
  [682c06a0] JSON v0.21.0
  [0a2635e0] MIPSLang v0.1.0 [`~/github2/MIPSLang`]
  [d8e11817] MLStyle v0.3.0 #master (https://github.com/thautwarm/MLStyle.jl.git)
  [9b87118b] PackageCompiler v0.6.4
  [14b8a8f1] PkgTemplates v0.6.1
  [58dd65bb] Plotly v0.2.0
  [54e16d92] PrettyPrinting v0.1.0 [`~/.julia/dev/PrettyPrinting`]
  [438e738f] PyCall v1.91.2
  [dbe2f901] QASM2Jl v0.1.0 [`../../../github/QASM-Julia/QASM2Jl`]
  [83ef0002] RBNF v0.1.0 [`~/.julia/dev/RBNF`]
  [295af30f] Revise v2.1.6

@thautwarm
Copy link
Member

I've reproduced your result in my CentOS server:

ERROR: LoadError: "class: Module, attributes: Symbol[:class, :body]."

That's a good start. It's a bit late in my timezone and I'll make it tomorrow.

@thautwarm
Copy link
Member

I noticed that I was using julia 1.1.0-rc2.0 as my local distribution, you might check whether this package can work in 1.1.0.

Current state:

Julia1.0.4 Julia 1.1.0(rc2) Julia 1.1.1
Local Segment fault test pass ?
My server Segment fault ? "class: Module, attributes: Symbol[:class, :body]."
Your computer Segment fault ? "class: Module, attributes: Symbol[:class, :body]."

@thautwarm
Copy link
Member

thautwarm commented Aug 8, 2019

I've figured out what the problem it exactly is now.
It's caused by MLStyle compiles string literal "Module" into something like a -> a === "Module". In julia 1.1.0-rc2, "Module" is still typed as "String", thus if I pass a = "Module" I will get a true, but in Julia 1.1.1 things are in a different situation.

A concerned issue could be found at thautwarm/MLStyle.jl#62.

As a workaround, we can replace each string pattern with another one, for instance,

https://github.com/JuliaCN/Py2Jl.jl/blob/master/src/ASTGen.jl#L197

changing Record(:class => "Module", body) -> to Record(:class => &"Module", body) -> will be sufficiently okay, but there're indeed a lot of tasks to perform the modifications in the whole codebase.

@thautwarm
Copy link
Member

thautwarm commented Aug 9, 2019

@PallHaraldsson
Hi bro, I've positioned the bug...
The String comparisons by === should work like == since 1.1, (EDIT) but for 1.1.1,

let z = Dict{Symbol, Any}(:class=>"Module"), Main855_1218 = z
                   function Main858_1221(Main856_1219::Dict)
                       Main857_1220 = (get)(Main856_1219, :class) do 
                           nothing
                       end
                       @info :bug Main857_1220 typeof(Main857_1220) Main857_1220 === "Module"
                       if Main857_1220 === "Module"
                           1
                       else
                           nothing
                       end
                   end
                   function Main858_1221(Main856_1219)
                       nothing
                   end
                   Main854_1217 = Main858_1221(Main855_1218)
                   if Main854_1217 === nothing
                       throw("failed")
                   else
                       Main854_1217
                   end
end
┌ Info: bug
│   Main857_1220 = "Module"typeof(Main857_1220) = String
└   Main857_1220 === "Module" = false
ERROR: "failed"
Stacktrace:
 [1] top-level scope at REPL[60]:19

@thautwarm
Copy link
Member

@PallHaraldsson
Copy link
Author

PallHaraldsson commented Aug 9, 2019

"I noticed that I was using julia 1.1.0-rc2.0 [works with]" "test pass"

Are you sure? I've been testing, Julia 1.0.0, and some later versions. "test Py2Jl" strangely works in some versions, but when you actually go further then you get a segfault, with "using Py2Jl" (even though I would have thought it implied by test.

[I also got further in using /usr/bin/python3 some of the time when testing. I was following your instructions to the letter, and assumed python 2. Do you know if it should [NOT] work?]

@thautwarm
Copy link
Member

I was following your instructions to the letter, and assumed python 2.

Python 2 shouldn't work at all for it has different AST shapes against those of Python 3. Currently I just extract Julia ASTs from Python ASTs(exactly python 3.6).

I can support Python3.5+ by compating them via https://github.com/thautwarm/moshmosh/blob/master/moshmosh/ast_compat.py

@thautwarm
Copy link
Member

Are you sure? I've been testing, Julia 1.0.0, and some later versions. "test Py2Jl" strangely works in some versions, but when you actually go further then you get a segfault, with "using Py2Jl" (even though I would have thought it implied by test.

I'm pretty sure it works in several sub-versions of v1.1.0.

Will the segmentation fault occur in 1.1.x?

I'll at once try "using Py2Jl" in my remote server, to check if any inconsistent behaviours.

@thautwarm
Copy link
Member

thautwarm commented Aug 9, 2019

Note, even in Julia 1.0.x can use this package in a strange way, say, paste all codes of this package (without putting them into Modules but to the top level) to Julia shell will work.

thautwarm added a commit that referenced this issue Aug 9, 2019
@thautwarm
Copy link
Member

I've just fixed the unmatched problems in 1.1.0+, have a check?
I'm now working to resolve the issues of translating while constructs.

@PallHaraldsson
Copy link
Author

I'm pretty sure I had to only do (in other versions, at least rc2):

(v1.1) pkg> add Py2Jl
Updating registry at ~/.julia/registries/General
Updating git-repo https://github.com/JuliaRegistries/General.git
ERROR: The following package names could not be resolved:

  • Py2Jl (not found in project, manifest or registry)
    Please specify by known name=uuid.

i.e. not:

(v1.1) pkg> add https://github.com/JuliaCN/Py2Jl.jl
Updating git-repo https://github.com/JuliaCN/Py2Jl.jl
Resolving package versions...
Updating ~/.julia/environments/v1.1/Project.toml
[e53ffe30] + Py2Jl v0.1.0 #master (https://github.com/JuliaCN/Py2Jl.jl)
Updating ~/.julia/environments/v1.1/Manifest.toml
[e53ffe30] + Py2Jl v0.1.0 #master (https://github.com/JuliaCN/Py2Jl.jl)

Your nr. 5 update fixes 1.1.0 (and I guess 1.1.1) so README could be modified. I had just tested it and said otherwise in Discourse, but before the "(v1.1) pkg> update".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants