Support numpy literals

[karenfeng]

What changes are proposed in this pull request?

We use DenseMatrix in our regression methods, but this is not a commonly used format to store matrix data in Python. This PR introduces support for numpy literals from double-typed numpy ndarrays under the hood with py4j once glow is imported. With lit():

• For 1-dimensional arrays, the conversion is to Java arrays. (This allows the user to bypass the conversions to Python lists and Java lists.)
• For 2-dimensional arrays, the conversion is to DenseMatrix.

How is this patch tested?

• Unit tests
• Integration tests
• Manual tests

Python tests.

[codecov]

Codecov Report

Merging #213 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #213   +/-   ##
=======================================
  Coverage   93.65%   93.65%           
=======================================
  Files          87       87           
  Lines        4226     4226           
  Branches      384      384           
=======================================
  Hits         3958     3958           
  Misses        268      268           

Impacted Files	Coverage Δ
core/src/main/scala/io/projectglow/functions.scala	32.25% <ø> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17d94ec...d6d1c08. Read the comment docs.

[henrydavidge]

The code looks fine. I'm a little confused about the circumstances when you can use these conversions. Is it only in a literal, or is it anywhere that you construct a DataFrame manually?

[henrydavidge]

Hm, it seems a little weird that we explicitly wrap in a Literal here. Does that mean that this conversion doesn't work with spark.createDataFrame? Not sure I would expect that as a user.

[karenfeng]

Unfortunately the PySpark pathways for createDataFrame and lit are very different. Passing this as a Literal allows us to bypass the usual type-checking and conversion logic.

[henrydavidge]

Just a comment. It might be possible to skip the catalyst data converter stuff and rely on Literal.create(java_obj). Also, have you verified that this works in DBR?

[henrydavidge]

It might work to call Literal.create directly on the Java object without creating a type converter manually.

[karenfeng]

We can't use create[T : TypeTag](v: T) due to type tags being filled in by the compiler. However, I believe we also can't natively use Literal.create(v, dataType), which depends on the limited types offered in convertToCatalyst versus those in createToCatalystConverter. I changed this to directly call the constructor for Literal to match the type-tagged implementation.

[karenfeng]

Just a comment. It might be possible to skip the catalyst data converter stuff and rely on Literal.create(java_obj). Also, have you verified that this works in DBR?

This works in DBR if the user runs from glow import * or from glow.conversions import * before running from pyspark.sql.functions import *.

[henrydavidge]

LGTM, thanks for the patience on this one!