Feature: As a data engineer, I want Spark backed by the latest version of Hive so I have the latest fixes and features

[ewilkins-csi]

Description

Upgrade to a newer version of Hive. We are currently on Hive 3.0 and the latest version is 4.0.0, which is a good starting target. If there are too many breaking changes, or we decide we want to give 4.0 time to mature, 3.1.3 is the newest 3.x version. There are now public DockerHub images published by Apache that package hive and would be a great base to start from, however they are all still running Java 8 which is EOL. We could consider aligning our image with the paradigms of Apache's public images so that when they do update the Java version we are poised to switch more easily

DOD

• Hive metastore service Docker image is updated to run Hive 4.0.0 (or 3.1.3)
• Hive metastore service Helm chart is updated to maintain functionality with any image changes

--- If Helm chart changes are required, the following items must also be completed. ---

• Migration instructions are written for moving from the v1 Hive service chart to the v2 chart
• The v1 Helm chart Baton migration is updated to skip updating the image version for the v1 Hive chart
• The release notes are updated indicating the v2 Hive chart is available and v1 is deprecated and no longer compatible with the new docker image

Test Strategy/Script

• Create a project with Data Access and a pipeline that reads and writes data to both Hive and Delta Lake
• Run the pipeline to create and read test data
• Query Data Access to ensure the test data can be retrieved
• Create a v1.6 project with a simple pipeline
• Follow the v1 -> v2 migration instructions for Hive Metastore service
• Ensure the project can deploy and the pipeline can run