[FEATURE]: Detect notebook include graph by analysing dbutils.notebook.run(...) calls

[nfx]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

From https://docs.databricks.com/en/notebooks/notebook-workflows.html:

The dbutils.notebook API is a complement to %run because it lets you pass parameters to and return values from a notebook. This allows you to build complex workflows and pipelines with dependencies. For example, you can get a list of files in a directory and pass the names to another notebook, which is not possible with %run. You can also create if-then-else workflows based on return values or call other notebooks using relative paths. Unlike %run, the dbutils.notebook.run() method starts a new job to run the notebook. These methods, like all of the dbutils APIs, are available only in Python and Scala. However, you can use dbutils.notebook.run() to invoke an R notebook.

Proposed Solution

Scan Python code for dbutils.notebook.run(...) calls and treat them like a dependency

Additional Context

No response

[ericvergnaud]

What about Scala code ?

[nfx]

We don't have scala parser yet, for now let's create higher level epic and postpone until py and sql migrations are fully operational. Scala also has good almost builtin parser which is typesafer, so it is just a matter of priorities.

In this task, let's yield Advisory when non-py and non-sql notebook gets called

[nfx]

closed in:

• Detect dbutils.notebook.run #1284