Install JDBC drivers for Delta product tests at runtime and upgrade docker-images to version 81

[ebyhr]

Description

This upgrades Delta Lake version to 2.4.0.
https://github.com/delta-io/delta/releases/tag/v2.4.0

Release notes

(x) This is not user-visible or docs only and no release notes are required.

[ebyhr]

CI hit #17782 & #16416

[ebyhr]

Addressed comments.

[ebyhr]

testDatabricksVacuumRemoveChangeDataFeedFiles didn't pass because DeltaQueryExecutor opens a new connection every time.

[ebyhr]

CI hit #18266

[ebyhr]

CI hit #14239

[ebyhr]

@findepi Could you take another look?

[findepi]

I added some code sample how we can avoid regressions and shared state.

However, why not simply configure io.trino.tempto.query.JdbcConnectivityParamsState#jar?
Looks like Tempto has the support for what we need already?

[ebyhr]

I already tried JdbcConnectivityParamsState#jar, but it didn't work. I'm losing the root cause though.

[findepi]

I already tried JdbcConnectivityParamsState#jar, but it didn't work.

let's understand why. It looks like the "correct" approach to do what we're trying to do.
Of course requires a jar to be provided at runtime. Was getting the correct path to the jar a problem?

[ebyhr]

The path was correct. It looks classloader issue. Changing io.trino.tempto.internal.query.JdbcUtils#getDriverClassLoader to like below will resolve it:

    private static ClassLoader getDriverClassLoader(JdbcConnectivityParamsState jdbcParamsState)
    {
        try {
            if (!jdbcParamsState.jar.isPresent()) {
                return JdbcUtils.class.getClassLoader();
            }
            URL jarURL = new File(jdbcParamsState.jar.get()).toURL();
            return URLClassLoader.newInstance(new URL[] {jarURL}, ClassLoader.getPlatformClassLoader());
        }
        catch (MalformedURLException e) {
            throw new RuntimeException("could not create classloader for file" + jdbcParamsState.jar.get(), e);
        }
    }

[findepi]

if we use tempto driver jar feature, why is the state maintained explicitly?

please use tempto-configuration.yaml to configure drivers

[ebyhr]

Sent #18464