[FEATURE]: Detect usage of streaming checkpoint in Python code onto DBFS or other storage

[nfx]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

in

# Function to create and setup a new StreamingContext
def functionToCreateContext():
    sc = SparkContext(...)  # new context
    ssc = StreamingContext(...)
    lines = ssc.socketTextStream(...)  # create DStreams
    ...
    ssc.checkpoint(checkpointDirectory)  # set checkpoint directory
    return ssc

# Get StreamingContext from checkpoint data or create a new one
context = StreamingContext.getOrCreate(checkpointDirectory, functionToCreateContext)

the checkpoint directory can be on DBFS mountpoint or a storage account.

Proposed Solution

raise alert

Additional Context

Related issues:

• Migrate streaming checkpoints #492

[nfx]

Duplicate of #492