Skip to content

Unable to Create GlueContext via GlueContext Function in Local Python/awsglue Environment #69

@adamfortuno

Description

@adamfortuno

I'm having the issue described in issue #42.

I am attempting to run the following in my local PySpark console...

from awsglue.context import GlueContext
glueContext = GlueContext(sc)

We receive the following:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\XYZ\bin\aws-glue-libs\PyGlue.zip\awsglue\context.py", line 47, in __init__
  File "C:\Users\XYZ\bin\aws-glue-libs\PyGlue.zip\awsglue\context.py", line 68, in _get_glue_scala_context
TypeError: 'JavaPackage' object is not callable

The following is the complete picture:
image

The environment looks like the following:

  • OS: 10.0.17134.0
  • Python: 3.7.3
  • Hadoop (winutils.exe): 2.8.5
  • Spark: 2.4.3
  • PySpark: 2.4.6
  • awsglue: 1.0

My environment variables look like the following...

  • SPARK_HOME: \bin\spark-2.4.3-bin-hadoop2.8\
  • SPARK_CONF_DIR: \bin\aws-glue-libs\conf\
  • HADOOP_HOME: \bin\hadoop-2.8.5\
  • SPARK_CONF_DIR: \bin\spark-2.4.3-bin-hadoop2.8\
  • JAVA_HOME: C:\Progra~2\Java\jdk1.8.0\
  • CLASSPATH:
    • \bin\aws-glue-libs\jarsv1*
    • \bin\spark-2.4.3-bin-hadoop2.8\jars*
  • PYTHONPATH:
    • ${SPARK_HOME}\python\lib\py4j
    • \bin\aws-glue-libs\PyGlue.zip

Just to confirm which version awsglue repo I'm working with...

image

The following are the "netty" files in my ..\aws-glue-libs\jarsv1\:

image

I'm looking for a little guidance on how to tweak my configuration to resolve this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions