align databricks-iris template to work with kedro-databricks

[JenspederM]

Motivation and Context

This PR is made to align the databricks-iris starter to the kedro-databricks plugin.

With these changes, creating a new Kedro project on Databricks should be as easy as:

• Create a project kedro new --starter="databricks-iris"
• Install dependencies python -m venv .venv && source ./.venv/bin/activate && pip install --upgrade pip && pip install -r requirements.txt
• Initialize databricks asset bundle kedro databricks init
• Bundle pipelines to asset bundle resources kedro databricks bundle
• Deploy project to Databricks kedro databricks deploy

How has this been tested?

The above has been tested against a private Databricks workspace.

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Assigned myself to the PR
• Added tests to cover my changes

[noklam]

Left some comment. Thanks for creating the plugin I think this is a huge improvement over the manual step deployment guide.

I have left some comment but I suggest separating necessary changes vs README.md, at this stage we are not ready to mention a third-party plugin in our official docs. You can actually create your own custom starter and register it in your plugin so that it can be created with kedro new -s. See the documentations

[JenspederM]

Left some comment. Thanks for creating the plugin I think this is a huge improvement over the manual step deployment guide.

... separating necessary changes vs README.md, at this stage we are not ready ...

Does that mean including the plugin

[JenspederM]

Hi @noklam,

Thank you for the thorough review! I have made changes according to your comments - I see I was a bit overenthusiastic in regards to kedro-databricks ;)

I have removed all mentions of it now, and simply ensure that the starter would work out of the box with the plugin instead.

Steps to deploy using the plugin is now:

# Create project
kedro new --starter="https://github.com/JenspederM/kedro-starters" --directory="databricks-iris" --checkout="feat/align-databricks-iris-with-kedro-databricks" --name "iris"

# Install dependencies
pip install -r requirements.txt
pip install kedro-databricks

# Initialization
kedro databricks init

# Bundling
kedro databricks bundle

# Deployment
kedro databricks deploy

This is much less invasive than before.

I also made some changes to the plugin, specifically regarding logging to make this more informative and closer aligned with other methods.

Please let me know what you think! :)

[noklam]

Thanks again, I think we are getting closed, added a few comments, I missed something in the last round.

[JenspederM]

FYI @noklam

I implemented the changes that you suggested. You're right, no need for the @pandas. :)

I have just tried the steps from my previous comment with this implementation and everything works as expected.

[noklam]

Left one minor styling comment, thanks for making this change! Exciting to see the datarbricks plugin, great work!

[datajoely]

Amazing work @JenspederM appreciate the effort. My comments are more for wider discussion not a firm recommendation!

[datajoely]

Is there an argument we should encourage the use of databricks.ManagedTableDataSet (even if it's commented out in the catalog) since it highlights Kedro's compatibility with Unity Catalog + Delta lake?

[JenspederM]

Agree 👆

But perhaps better saved for another PR? :)

[datajoely]

absolutely, this is a really important and frankly overdue piece of work and I'm just thinking about how to take it further, later!

[astrojuanlu]

For reference, +1 on this!

[JenspederM]

@datajoely, do you regard your comments as things that should be included in this PR or is it more general discussion for a future PR? 😊

I would like to merge this as I can then make an announcement for the plugin without having to specify a branch for the starter 😊

[noklam]

@JenspederM I see the DCO checking is failing, can you try to fix it? If you click into the "Details" button it shows you the instruction to fix it.

[JenspederM]

@noklam I just accepted your change here - apparently that doesn't sign..

Should be fixed now :)

[JenspederM]

@datajoely, I see you resolved your comments, does that mean you approve of the PR? :)

[JenspederM]

@noklam @datajoely any idea why tests aren't running?

[noklam]

I trigger it now, it's a setting thing. CI won't run for first time contributor.

[JenspederM]

Ah okay.

Thank you!

[JenspederM]

@noklam @datajoely Are we ready to merge? :)

[noklam]

@JenspederM congratulations for your first PR!