Skip to content

Latest commit

 

History

History
19 lines (12 loc) · 837 Bytes

Utilizing Hive UDFs within Spark.md

File metadata and controls

19 lines (12 loc) · 837 Bytes

Utilizing Hive User Defined Functions (UDFs) within Spark

Built-in UDFs

You can call built-in Hive UDFs, UDAFs, and UDTFs from Spark SQL applications, as long as the functions are available in the standard Hive .jar file.

When using Hive UDFs, use HiveContext (not SQLContext).

Custom UDF

You can register custom functions in Python, Java, or Scala, and use them within SQL statements.

When using a custom UDF, make sure that the jar file for your UDF is included with your application, or use the --jars command-line option to specify the file.

Examples

define our own UDFs

Reference

Calling Hive User-Defined Functions