Increase max writer version to 4 for delta lake

[homar]

Description

As we have checks that blocks writing to tables with specific features like CDF and generated columns we can safely update accepted writer version to 4.
I also refactored a little TestDeltaLakeDatabricksInsertCompatibility class

Non-technical explanation

Release notes

(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Delta Lake
* Add support for writing to tables using [version 4 of the writer](https://docs.delta.io/latest/versioning.html#features-by-protocol-version). 
  This does not yet include support for change data feeds and generated columns. ({issue}`14573 `)

[ebyhr]

Currently, the connector uses hard-coded reader & writer version when appending a protocol entry. If we changes metadata (e.g. COMMENT ON COLUMN) on tables having generated columns, it lowers the writer version and Databricks starts ignoring generated columns definition. I think we should copy the version in case of existing tables. #14611

[homar]

Currently, the connector uses hard-coded reader & writer version when appending a protocol entry. If we changes metadata (e.g. COMMENT ON COLUMN) on tables having generated columns, it lowers the writer version and Databricks starts ignoring generated columns definition. I think we should copy the version in case of existing tables. #14611

Yes i saw that, great you noticed that and made a fix. But does that affect this pr ? I haven't noticed a failure but there is so much flakiness that I may have missed something

[ebyhr]

#14612 is required as a preparatory commit, but there's no need to change this PR. I found the above issue when I was trying to add some tests based on this PR.

[ebyhr]

Rebased on upstream to resolve conflicts.

[ebyhr]

Merged, thanks!