Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lein uberjar fails with type hints in graph.clj #34

Closed
tirkarthi opened this issue May 22, 2018 · 12 comments
Closed

lein uberjar fails with type hints in graph.clj #34

tirkarthi opened this issue May 22, 2018 · 12 comments
Labels

Comments

@tirkarthi
Copy link

I am using byte-streams library for clj_fdb. When I try to create a uberjar on the master I have the below error. I tried with the latest alpha release and also it throws the same error. I think it's due to some type hinting at graph.clj . I cloned the repo and removed the type hints and did lein install. Then I used the local version which works fine during uberjar creation.

The strange part was that it used to work fine except when the relevant file was moved not to another folder. I figured out the bad commit at vedang/clj_fdb@459071e where I move the namespace to another internal folder. When I revert the folder organization it works fine. I don't why moving it inside to internal folder causes the issue with type hinting while moving it one above works fine. Maybe I am missing something here.

Another caveat is that compiling the file as a separate file works fine and starting the repl also works fine. This happens only when I have the namespace inside the folder while moving it out works fine.

Exception

➜  clj_fdb git:(upstream/master) lein uberjar
Compiling clj-fdb.core
Compiling clj-fdb.FDB
Compiling clj-fdb.internal.byte-conversions
nil
Exception in thread "main" java.lang.ExceptionInInitializerError, compiling:(/private/var/folders/2b/mhgtnnpx4z943t4cc9yvw4qw0000gn/T/form-init7346111847608064189.clj:1:125)
	at clojure.lang.Compiler.load(Compiler.java:7526)
	at clojure.lang.Compiler.loadFile(Compiler.java:7452)
	at clojure.main$load_script.invokeStatic(main.clj:278)
	at clojure.main$init_opt.invokeStatic(main.clj:280)
	at clojure.main$init_opt.invoke(main.clj:280)
	at clojure.main$initialize.invokeStatic(main.clj:311)
	at clojure.main$null_opt.invokeStatic(main.clj:345)
	at clojure.main$null_opt.invoke(main.clj:342)
	at clojure.main$main.invokeStatic(main.clj:424)
	at clojure.main$main.doInvoke(main.clj:387)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:702)
	at clojure.main.main(main.java:37)
Caused by: java.lang.ExceptionInInitializerError
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at clojure.lang.RT.classForName(RT.java:2204)
	at clojure.lang.RT.classForName(RT.java:2213)
	at clojure.lang.RT.loadClassForName(RT.java:2232)
	at clojure.lang.RT.load(RT.java:450)
	at clojure.lang.RT.load(RT.java:426)
	at clojure.core$load$fn__6548.invoke(core.clj:6046)
	at clojure.core$load.invokeStatic(core.clj:6045)
	at clojure.core$load.doInvoke(core.clj:6029)
	at clojure.lang.RestFn.invoke(RestFn.java:408)
	at clojure.core$load_one.invokeStatic(core.clj:5848)
	at clojure.core$compile$fn__6553.invoke(core.clj:6056)
	at clojure.core$compile.invokeStatic(core.clj:6056)
	at clojure.core$compile.invoke(core.clj:6048)
	at user$eval164$fn__173.invoke(form-init7346111847608064189.clj:1)
	at user$eval164.invokeStatic(form-init7346111847608064189.clj:1)
	at user$eval164.invoke(form-init7346111847608064189.clj:1)
	at clojure.lang.Compiler.eval(Compiler.java:7062)
	at clojure.lang.Compiler.eval(Compiler.java:7052)
	at clojure.lang.Compiler.eval(Compiler.java:7052)
	at clojure.lang.Compiler.load(Compiler.java:7514)
	... 12 more
Caused by: java.lang.ClassCastException: byte_streams.graph.Type cannot be cast to byte_streams.graph.Type
	at byte_streams.graph.ConversionGraph.assoc_conversion(graph.clj:122)
	at byte_streams.graph$fn__2606$G__2568__2612.invoke(graph.clj:91)
	at byte_streams.graph$fn__2606$G__2567__2619.invoke(graph.clj:91)
	at clojure.lang.AFn.applyToHelper(AFn.java:171)
	at clojure.lang.AFn.applyTo(AFn.java:144)
	at clojure.lang.Atom.swap(Atom.java:79)
	at clojure.core$swap_BANG_.invokeStatic(core.clj:2347)
	at clojure.core$swap_BANG_.doInvoke(core.clj:2337)
	at clojure.lang.RestFn.invoke(RestFn.java:529)
	at clj_fdb.internal.byte_conversions$fn__3638.invokeStatic(byte_conversions.clj:10)
	at clj_fdb.internal.byte_conversions$fn__3638.invoke(byte_conversions.clj:10)
	at clj_fdb.internal.byte_conversions__init.load(Unknown Source)
	at clj_fdb.internal.byte_conversions__init.<clinit>(Unknown Source)
	... 34 more
Compilation failed: Subprocess failed

byte-streams type-hint change that makes uberjar work

diff --git a/src/byte_streams/graph.clj b/src/byte_streams/graph.clj
index fc62f1d..3c94b41 100644
--- a/src/byte_streams/graph.clj
+++ b/src/byte_streams/graph.clj
@@ -119,10 +119,10 @@
   (assoc-conversion [_ src dst f cost]
     (let [m' (assoc-in m [src dst] (Conversion. f cost))
           m' (if (and
-                   (nil? (.wrapper ^Type src))
-                   (nil? (.wrapper ^Type dst)))
-               (let [src (.type ^Type src)
-                     dst (.type ^Type dst)]
+                   (nil? (.wrapper src))
+                   (nil? (.wrapper dst)))
+               (let [src (.type src)
+                     dst (.type dst)]
                  (-> m'
                    (assoc-in [(Type. 'seq src) (Type. 'seq dst)]
                      (Conversion. (fn [x options] (map #(f % options) x)) cost))

Thanks a lot for the library.

@gsnewmark
Copy link
Contributor

Here's a minimal reproducible case https://github.com/gsnewmark/byte-streams-cnf

It seems like under some conditions Leiningen compiles namespaces in incorrect order causing expanded part of byte-streams/def-conversion to be executed twice. Second execution fails with the reported exception, because class of src / dst (byte_streams.graph.Type) is really a different one than the currently compiled one (not sure why/how though).

In case we have two namespaces core and conversions, conversions contains def-conversion's, and core requires it, lein compile :all firstly compiles conversions, and then core, expanded part of def-conversion is executed once, and compilation finishes correctly. But if we add a third namespace lib which also dependes on conversions for some reason lein compiles this new ns lib firstly, loads conversions during its compilation and executes expanded part of def-conversion successfully, but then lein tries to compile conversions, executes expanded part once more and fails with ClassCastException 🤷‍♀️

Following change avoids the exception, but it's clearly just a crude workaround, not a proper solution:

1 file changed, 19 insertions(+), 17 deletions(-)
src/byte_streams.clj | 36 +++++++++++++++++++-----------------

modified   src/byte_streams.clj
@@ -91,7 +91,7 @@
     :else
     (g/type (class x))))
 
-(defn- normalize-type-descriptor [x]
+(defn normalize-type-descriptor [x]
   (cond
     (instance? Type x)
     x
@@ -106,22 +106,24 @@
   "Defines a conversion from one type to another."
   [[src dst :as conversion] params & body]
   (let [^Type src (normalize-type-descriptor src)
-        dst (normalize-type-descriptor dst)]
-    `(let [f#
-           (fn [~(with-meta (first params)
-                   {:tag (when (and (instance? Class (.type src)) (not (.wrapper src)))
-                           (if (= src (normalize-type-descriptor 'bytes))
-                             'bytes
-                             (.getName ^Class (.type src))))})
-                ~(if-let [options (second params)]
-                   options
-                   `_#)]
-             ~@body)
-
-           cost#
-           ~(get (meta conversion) :cost 1)]
-       (swap! conversions g/assoc-conversion ~src ~dst f# cost#)
-       (swap! inverse-conversions g/assoc-conversion ~dst ~src f# cost#))))
+        dst       (normalize-type-descriptor dst)]
+    `(when (= ~src (normalize-type-descriptor ~src))
+       (let [f#
+             (fn [~(with-meta (first params)
+                     {:tag (when (and (instance? Class (.type src)) (not (.wrapper src)))
+                             (if (= src (normalize-type-descriptor 'bytes))
+                               'bytes
+                               (.getName ^Class (.type src))))})
+                  ~(if-let [options (second params)]
+                     options
+                     `_#)]
+               ~@body)
+
+             cost#
+             ~(get (meta conversion) :cost 1)]
+
+         (swap! conversions g/assoc-conversion ~src ~dst f# cost#)
+         (swap! inverse-conversions g/assoc-conversion ~dst ~src f# cost#)))))
 
 (defmacro def-transfer
   "Defines a byte transfer from one type to another."

@gsnewmark
Copy link
Contributor

Probably related https://dev.clojure.org/jira/browse/CLJ-1741

@ztellman
Copy link
Collaborator

That's weird, the deftype+ macro being used in byte-streams is supposed to avoid this exact issue, at least in REPL-based situations. I'll look into it.

@gsnewmark
Copy link
Contributor

@ztellman it's also reproducible in REPL using the repo I've linked in the previous message 😞 (cc @serzh):

➜  byte-streams-cnf git:(master) lein repl
Retrieving byte-streams/byte-streams/0.2.4/byte-streams-0.2.4.pom from clojars
Retrieving primitive-math/primitive-math/0.1.6/primitive-math-0.1.6.pom from clojars
Retrieving manifold/manifold/0.1.8/manifold-0.1.8.pom from clojars
Retrieving manifold/manifold/0.1.8/manifold-0.1.8.jar from clojars
Retrieving primitive-math/primitive-math/0.1.6/primitive-math-0.1.6.jar from clojars
Retrieving byte-streams/byte-streams/0.2.4/byte-streams-0.2.4.jar from clojars
nREPL server started on port 53229 on host 127.0.0.1 - nrepl://127.0.0.1:53229
REPL-y 0.3.7, nREPL 0.2.12
Clojure 1.9.0
Java HotSpot(TM) 64-Bit Server VM 1.8.0_131-b11
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e




user=> (compile 'byte
byte         byte-array   bytes        bytes?
user=> (compile 'byte-streams-cnf.lib)
byte-streams-cnf.lib
user=> (compile 'byte-streams-cnf.conversions)

ClassCastException byte_streams.graph.Type cannot be cast to byte_streams.graph.Type  byte-streams.graph.ConversionGraph (graph.clj:122)
user=>

It works completely fine if compile is not called though.

@gsnewmark
Copy link
Contributor

gsnewmark commented Jun 12, 2018

Minor addition: both compilation using lein and REPL work with with following dependencies in the test project

:dependencies [[org.clojure/clojure "1.6.0"]
               [byte-streams "0.2.0-alpha1"]]

but stops working starting from the [org.clojure/clojure "1.7.0"], so it's definitely caused by some changes in compiler introduced in 1.7.0 😞

Newer byte-streams versions don't work with 1.6.0 because they (transitively) rely on new features from 1.7.0.

@gsnewmark
Copy link
Contributor

Correction to my initial comment: after additional investigation it looks like the expanded part of deftype+ is not actually executed twice, but class is still somehow different, most probably it's really loaded by a different class loader (precisely what's described here).

@KingMob KingMob added the hard label Feb 7, 2021
@KingMob
Copy link
Collaborator

KingMob commented May 26, 2022

I can't replicate this, neither with current clj_fdb, nor with the byte-streams-cnf test case. I tried downgrading the Clojure versions to 1.7.0 and 1.9.0, but still couldn't trigger the error. I also tried to verify it with the listed clj_fdb commit, 459071e, but one of the deps, fdb-java 5.1.7, is no longer available from Maven.

@tirkarthi @gsnewmark I'm inclined to close it, unless you think it should remain open.

@gsnewmark
Copy link
Contributor

I'd say go for it 🙂

@KingMob
Copy link
Collaborator

KingMob commented May 26, 2022

Closing for now, but @tirkarthi lmk if it needs to be reopened.

@KingMob KingMob closed this as completed May 26, 2022
@tirkarthi
Copy link
Author

I am okay with closure as well since I am not using the library currently to replicate the issue. Thanks @KingMob

@vedang
Copy link

vedang commented May 26, 2022 via email

@KingMob
Copy link
Collaborator

KingMob commented Jul 7, 2023

For posterity:

it's definitely caused by some changes in compiler introduced in 1.7.0

Probably CLJ-1650, which was introduced in 1.7 and fixed in 1.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants