duckling: jpype1 error: Chaquopy cannot compile native code

[argideritzalpea]

This may be similar to #78:

I have added
install "duckling"
to the pip section in build.gradle. It appears that duckling requires jpype1.

I am getting this error upon building:

  running build_ext
  /private/var/folders/2x/c0mhw9716bj8zt68qvdk3dg40000gp/T/pip-install-17cdbb3f/jpype1/setup.py:147: FeatureNotice: Turned ON Numpy support for fast Java array access
    FeatureNotice)
  building '_jpype' extension
  error: CCompiler.compile: Chaquopy cannot compile native code

[mhsmith]

It might be possible for us to support jpype. However, there's a more serious obstacle to using duckling: if you look inside the whl file on pypi, you'll see that the underlying Java libraries are packaged as JARs. Android doesn't support loading JAR files at runtime: they have to be converted to DEX format during the app build process.

So I think the best approach is to add these JARs to your app as local binary dependencies, and then access them through the Chaquopy Python API.

The cleanest way to do this would be to modify python-duckling to use Chaquopy rather than jpype. I think the changes required would be relatively small. For example, in these lines, all you would have to do is change jpype.JClass to java.jclass.

Alternatively, rather than porting the whole of python-duckling to Chaquopy, you might be able to just port the parts of it you need, and copy them into your own app.

[mhsmith]

I should mention that you wouldn't need any of the parts of python-duckling which are concerned with setting up the classpath and starting the JVM, because on Android you would already be running within an active JVM.

[argideritzalpea]

@mhsmith Thanks for the reply. I have imported all the jars in app/lib and added implementation fileTree(dir: 'lib', include: ['*.jar']) to my app-level build.gradle to get them to be recognized.

Calling duckling imported into my app with chaquopy, I received the following error:

Execution failed for task ':app:transformResourcesWithMergeJavaResForDebug'.
> More than one file was found with OS independent path 'project.clj'

I assume this has to do with multiple JARS containing a project.clj file that I assume defines dependencies for clojure projects. Used the JARS here: https://github.com/FraBle/python-duckling/releases/tag/v1.8.0

I added

        exclude 'project.clj'
        exclude 'META-INF/INDEX.LIST'
    }

to see what would happen. When I attempted to run again, I received an error that the last line printed below doesn't work: duckling_load.invoke()
Error:

java.lang.RuntimeException: Unable to start activity ComponentInfo{}: com.chaquo.python.PyException: java.lang.IllegalStateException: Attempting to call unbound fn: #'duckling.core/load!

I modified the code to support java.jclass instead of jpype and commented out the jvm threading code and lock_release clause. Here are the first several lines of Duckling.py from python-duckling that I have in my app/src/main/python/duckling directory:

from java import jvoid, Override, static_proxy, jclass
import os
import imp
import socket
import threading
from six import string_types
from distutils.util import strtobool
from dateutil import parser
from .dim import Dim
from .language import Language

socket.setdefaulttimeout(15)


class Duckling(static_proxy()):

    """Python wrapper for Duckling by wit.ai.

    Attributes:
        jvm_started: Optional attribute to specify if the JVM has already been
            started (with all Java dependencies loaded).
        parse_datetime: Optional attribute to specify if datetime string should
            be parsed with datetime.strptime(). Default is False.
        minimum_heap_size: Optional attribute to set initial and minimum heap
            size. Default is 128m.
        maximum_heap_size: Optional attribute to set maximum heap size. Default
            is 2048m.
    """

    def __init__(self,
                 jvm_started=False,
                 parse_datetime=False,
                 minimum_heap_size='128m',
                 maximum_heap_size='2048m'):
        """Initializes Duckling.
        """

        self.parse_datetime = parse_datetime
        self._is_loaded = False
        self._lock = threading.Lock()

        #if not jvm_started:
        #    self._classpath = self._create_classpath()
        #    self._start_jvm(minimum_heap_size, maximum_heap_size)

        try:
            """
            # make it thread-safe
            if threading.activeCount() > 1:
                if not jpype.isThreadAttachedToJVM():
                    jpype.attachThreadToJVM()
            self._lock.acquire()
            """
            self.clojure = jclass('clojure.java.api.Clojure')
            # require the duckling Clojure lib
            require = self.clojure.var("clojure.core", "require")
            require.invoke(self.clojure.read("duckling.core"))
        except Exception as e:
            print(e)
        #finally:
        #    self._lock.release()

    def _start_jvm(self, minimum_heap_size, maximum_heap_size):
        jvm_options = [
            '-Xms{minimum_heap_size}'.format(minimum_heap_size=minimum_heap_size),
            '-Xmx{maximum_heap_size}'.format(maximum_heap_size=maximum_heap_size),
            '-Djava.class.path={classpath}'.format(
                classpath=self._classpath)
        ]
        """
        if not jpype.isJVMStarted():
            jpype.startJVM(
                jpype.getDefaultJVMPath(),
                *jvm_options
            )
        """

    def _create_classpath(self):
        jars = []
        for top, dirs, files in os.walk(os.path.join(imp.find_module('duckling')[1], 'jars')):
            for file_name in files:
                if file_name.endswith('.jar'):
                    jars.append(os.path.join(top, file_name))
        return os.pathsep.join(jars)

    def load(self, languages=[]):
        """Loads the Duckling corpus.

        Languages can be specified, defaults to all.

        Args:
            languages: Optional parameter to specify languages,
                e.g. [Duckling.ENGLISH, Duckling.FRENCH] or supported ISO 639-1 Codes (e.g. ["en", "fr"])
        """
        duckling_load = self.clojure.var("duckling.core", "load!")
        clojure_hashmap = self.clojure.var("clojure.core", "hash-map")
        clojure_list = self.clojure.var("clojure.core", "list")

        if languages:
            # Duckling's load function expects ISO 639-1 Language Codes (e.g. "en")
            iso_languages = [Language.convert_to_iso(lang) for lang in languages]

            duckling_load.invoke(
                clojure_hashmap.invoke(
                    self.clojure.read(':languages'),
                    clojure_list.invoke(*iso_languages)
                )
            )
        else:
            duckling_load.invoke()

        self._is_loaded = True

[mhsmith]

I'm not familiar with Clojure so I don't know what's going on here. However, I don't see any reason to comment out the code which uses _lock. I also don't see any reason to catch and ignore exceptions in __init__, because if that doesn't succeed, then I don't expect anything else will work.

If __init__ did succeed without throwing an exception, then please post the full stack trace from load.

[mhsmith]

If this is still a problem, please provide the requested information and I'll reopen the issue.

[mhsmith]

Since only one person has requested this package, we won't be adding it in the foreseeable future. If anyone else is interested, please post a comment and I'll reopen the issue.