Add optional Glue Schema configuration to exclude Non-Current Fields

[duoxoud]

Closes #7584

This PR addresses a feature request for improving the Glue Schema generation process. It introduces a new configuration option that allows users to exclude non-current fields from the Glue Schema, providing clarity and reducing confusion for Athena users who primarily query current data.

In PR #3888, the Glue schema generation was modified to include all historical fields. This was intended to help users recognize previously used columns and avoid duplicating column names. However, in practice, this approach has led to confusion among users (for example, the same issue explained in #7584 ).

[duoxoud]

Hello team 👋 , is there a specific step I need to do before getting the workflow approval 😃 ?

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

[VitorNoro]

Bringing some attention to this PR, this feature would be useful

[VaultAges]

Bringing some interest as well, this feature is indeed needed.
Merge schema keeping old columns creates a lot of headscratching situation with old empty columns.