Add support for not nullable columns in delta lake

[homar]

Description

In delta lake add support for tables that have non nullable columns.

Is this change a fix, improvement, new feature, refactoring, or other?

new feature

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

a connector

How would you describe this change to a non-technical end user or system administrator?

During creation of table you can specify if column can store NULL values.

Related issues, pull requests, and links

#12635

Documentation

( ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
(x) Release notes entries required with the following suggested text:

# Delta Lake
* Add support for `NOT NULL` column constraint. ({issue}`13436`)

[homar]

@mosabua @colebow I know that it will be good to add this to the doc but I didn't figure out best way to do it. Could you suggest something?

[findepi]

Can you please uncomment the assertion and let the query succeed?
and add a TODO explaining what the expected result should be?

this would ensure this code gets updated when the bug is fixed

[alexjo2144]

Can you add a compatibility test that Databricks respects when Trino sets the non-null flag and vise versa. Besides that, LGTM

[ebyhr]

Question: What happens if the nullability is changed remotely and the connector writes rows within delta.metadata.cache-ttl?

[homar]

Question: What happens if the nullability is changed remotely and the connector writes rows within delta.metadata.cache-ttl?

So I expected it will be a problem but I added a test and it works - trino can't insert null into non nullable column.
In the test TTL is set to 30 mins.

[alexjo2144]

The cache auto-invalidates if a new transaction has been written to the log

[homar]

The cache auto-invalidates if a new transaction has been written to the log

Yes I know but in this case nothing new is written, or altering column is a transaction ? I guess that must be the case

[alexjo2144]

altering column is a transaction

Yep, it is in Delta. Still good to have the test through

[homar]

@ebyhr thanks for running these tests I forgot to remove test that checks if we can write into tables with non nullable columns. Now it should be fine

[homar]

@ebyhr tests passed can we merge it now ?

[ebyhr]

Rebased on upstream to resolve conflicts. Let me merge after CI completion.

[ebyhr]

I confirmed new MERGE statement respects NOT NULL constraints correctly in Delta Lake connector. Let's add the test in a follow-up PR.

[ebyhr]

Merged, thanks!

[colebow]

Are there other column constraints that would make sense to document alongside this? Or is it a unique/one-off thing?

[homar]

Are there other column constraints that would make sense to document alongside this? Or is it a unique/one-off thing?

I'd say this is a unique case, at least for now. Also keep in mind this is not delta specific. Non nullness should be supported for many connectors

[colebow]

Our CREATE TABLE and ALTER TABLE docs do reference NOT NULL as an option on a column. Based on that, I think it would be assumed to work everywhere? Seems very standard to me.

[homar]

Our CREATE TABLE and ALTER TABLE docs do reference NOT NULL as an option on a column. Based on that, I think it would be assumed to work everywhere? Seems very standard to me.

oh sorry, I must have missed that. BTW assumption is not correct. It is connector specific as connector must support nullability