Fix flink versioning issues

[damccorm]

As I was doing some work to update Dataflow's notebooks product, I noticed that interactive beam on flink doesn't work with 2.56. When I dug deeper, I found a bunch of references that weren't updated in #30197

This tries to cover them all, and adds the known issue.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[damccorm]

R: @je-ik @Abacn

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[Abacn]

just a side note

this is stale implies this field is not used, actually currently this action is only used in update_python_dependencies and publish_github_release_notes workflow.

Currently there is hard coded python/java versions everywhere in our GHAs. As futurre clean up we could make use of this action to have single source of version setting for actions

[Abacn]

note - this would essentially prevent python user using flink 1.17 for Beam 2.56.0, and not prevent user choosing 1.12,1.13. Consider update the known issue.

[damccorm]

Yeah, I sorta mentioned this, but I updated to be clearer/more explicit. Thanks

[Abacn]

thank you!

[damccorm]

The test failures all seem to be preexisting, but I don't feel good about merging this until we get a greener signal since flakes are blocking our test coverage.

#31169 should fix it, at which point I'll merge in master and try this again

[damccorm]

Typescript tests are failing, but they're also permared; no meaningful signal there

[je-ik]

Looks like we need some post-commit check to cover this?

[damccorm]

Looks like we need some post-commit check to cover this?

Probably - I'm surprised nothing failed here

[je-ik]

Do we have a github issue for fixing the known issue? I'm struggling to find it. The mentioned #31168 links to this PR.

[damccorm]

No, I just went straight to PR since it was easier than enumerating the problems in an issue first

[je-ik]

Ah, sorry, I did not mean issue for this PR, I meant tracking issue for the mentioned problem https://github.com/apache/beam/pull/31168/files#diff-d975bf659606195d2165918f93e1cf680ef68ea3c9cab994f033705fea8238b2R131

[je-ik]

Context: there is already PR for 1.18, it would be good to avoid similar issue in the future, yet I'm not sure I understand the problem correctly.

[damccorm]

Basically, the problem is that we track Flink version support in a bunch of places which need updated when we add a version (and more importantly when we remove a version). The most important ones I noticed are:

1. beam/sdks/python/apache_beam/options/pipeline_options.py

   Line 1594 in 815639b

   PUBLISHED_FLINK_VERSIONS = ['1.14', '1.15', '1.16', '1.17']

   - controls which flink versions can be used from Python

2. beam/sdks/python/apache_beam/runners/interactive/interactive_beam.py

   Line 411 in 815639b

   DATAPROC_FLINK_VERSION = '1.15'

   - controls which flink version is used by interactive beam

But really all the files in this PR need updated anytime we update flink version availability

[je-ik]

Understood. I was confused by what else needs to be done, but the problem is that this PR was merged after the RC of 2.56.0 was built, so this PR fixes it for 2.57.0.

Got it, thanks!