Fix flink versioning issues (#31168)

* Fix flink versioning issues

* Add pr reference

* dataproc 2.2

* 1.15 for dataproc version

* More explicit changes

.github/actions/setup-default-test-properties/test-properties.json

@@ -14,7 +14,7 @@
   },
   "JavaTestProperties": {
     "SUPPORTED_VERSIONS": ["8", "11", "17"],
-    "FLINK_VERSIONS": ["1.13", "1.14", "1.15"],
+    "FLINK_VERSIONS": ["1.14", "1.15", "1.16", "1.17"],
     "SPARK_VERSIONS": ["2", "3"]
   },
   "GoTestProperties": {

CHANGES.md

@@ -127,6 +127,9 @@
 
 ## Known Issues
 
+* The beam interactive runner does not correctly run on flink ([#31168](https://github.com/apache/beam/issues/31168)).
+* When using the Flink runner from Python, 1.17 is not supported and 1.12/13 do not work correctly. Support for 1.17 will be added in 2.57.0, and the ability to choose 1.12/13 will be cleaned up and fully removed in 2.57.0 as well ([#31168](https://github.com/apache/beam/issues/31168)).
+
 # [2.55.1] - 2024-04-08
 
 ## Bugfixes

learning/tour-of-beam/learning-content/introduction/introduction-concepts/runner-concepts/description.md

@@ -191,7 +191,7 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
 {{if (eq .Sdk "java")}}
 
 ##### Portable
-1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.10`, `Flink 1.11`, `Flink 1.12`, `Flink 1.13`, `Flink 1.14`.
+1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.14`, `Flink 1.15`, `Flink 1.16`, `Flink 1.17`.
 2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
 3. Submit the pipeline to the above endpoint by using the PortableRunner, job_endpoint set to localhost:8099 (this is the default address of the JobService). Optionally set environment_type set to LOOPBACK. For example:
 

sdks/go/examples/wasm/README.md

@@ -68,7 +68,7 @@ cd $BEAM_HOME
 Expected output should include the following, from which you acquire the latest flink runner version.
 
 ```shell
-'flink_versions: 1.12,1.13,1.14,1.15,1.16'
+'flink_versions: 1.14,1.15,1.16,1.17'
 ```
 
 #### 2. Set to the latest flink runner version i.e. 1.16

sdks/python/apache_beam/options/pipeline_options.py

@@ -1591,7 +1591,7 @@ def _add_argparse_args(cls, parser):
 class FlinkRunnerOptions(PipelineOptions):
 
   # These should stay in sync with gradle.properties.
-  PUBLISHED_FLINK_VERSIONS = ['1.12', '1.13', '1.14', '1.15', '1.16']
+  PUBLISHED_FLINK_VERSIONS = ['1.14', '1.15', '1.16', '1.17']
 
   @classmethod
   def _add_argparse_args(cls, parser):

sdks/python/apache_beam/runners/interactive/interactive_beam.py

@@ -405,10 +405,10 @@ class Clusters:
   To configure a pipeline to run on a local FlinkRunner, explicitly set the
   default cluster metadata to None: ib.clusters.set_default_cluster(None).
   """
-  # Explicitly set the Flink version here to ensure compatibility with 2.0
+  # Explicitly set the Flink version here to ensure compatibility with 2.1
   # Dataproc images:
-  # https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.0
-  DATAPROC_FLINK_VERSION = '1.12'
+  # https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.1
+  DATAPROC_FLINK_VERSION = '1.15'
 
   # The minimum worker number to create a Dataproc cluster.
   DATAPROC_MINIMUM_WORKER_NUM = 2

sdks/typescript/src/apache_beam/runners/flink.ts

@@ -28,7 +28,7 @@ import { JavaJarService } from "../utils/service";
 const MAGIC_HOST_NAMES = ["[local]", "[auto]"];
 
 // These should stay in sync with gradle.properties.
-const PUBLISHED_FLINK_VERSIONS = ["1.12", "1.13", "1.14"];
+const PUBLISHED_FLINK_VERSIONS = ["1.14", "1.15", "1.16", "1.17"];
 
 const defaultOptions = {
   flinkMaster: "[local]",

website/www/site/content/en/documentation/runners/flink.md

@@ -196,13 +196,10 @@ The optional `flink_version` option may be required as well for older versions o
 
 {{< paragraph class="language-portable" >}}
 Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub:
-[Flink 1.10](https://hub.docker.com/r/apache/beam_flink1.10_job_server),
-[Flink 1.11](https://hub.docker.com/r/apache/beam_flink1.11_job_server),
-[Flink 1.12](https://hub.docker.com/r/apache/beam_flink1.12_job_server).
-[Flink 1.13](https://hub.docker.com/r/apache/beam_flink1.13_job_server).
 [Flink 1.14](https://hub.docker.com/r/apache/beam_flink1.14_job_server).
 [Flink 1.15](https://hub.docker.com/r/apache/beam_flink1.15_job_server).
 [Flink 1.16](https://hub.docker.com/r/apache/beam_flink1.16_job_server).
+[Flink 1.17](https://hub.docker.com/r/apache/beam_flink1.17_job_server).
 {{< /paragraph >}}
 
 <!-- TODO(BEAM-10214): Use actual lists here and below. -->
@@ -329,6 +326,11 @@ To find out which version of Flink is compatible with Beam please see the table
   <th>Artifact Id</th>
   <th>Supported Beam Versions</th>
 </tr>
+<tr>
+  <td>1.17.x</td>
+  <td>beam-runners-flink-1.16</td>
+  <td>&ge; 2.56.0</td>
+</tr>
 <tr>
   <td>1.16.x</td>
   <td>beam-runners-flink-1.16</td>
@@ -347,12 +349,12 @@ To find out which version of Flink is compatible with Beam please see the table
 <tr>
   <td>1.13.x</td>
   <td>beam-runners-flink-1.13</td>
-  <td>&ge; 2.31.0</td>
+  <td>&ge; 2.31.0 - 2.55.0</td>
 </tr>
 <tr>
   <td>1.12.x</td>
   <td>beam-runners-flink-1.12</td>
-  <td>&ge; 2.27.0</td>
+  <td>&ge; 2.27.0 - 2.55.0</td>
 </tr>
 <tr>
   <td>1.11.x</td>

website/www/site/content/en/documentation/sdks/java/testing/tpcds.md

@@ -138,7 +138,7 @@ When running via Gradle, the following two parameters control the execution:
 
     -P tpcds.runner
 	The Gradle project name of the runner, such as ":runners:spark:3" or
-	":runners:flink:1.13. The project names can be found in the root
+	":runners:flink:1.17. The project names can be found in the root
         `settings.gradle.kts`.
 
 Test data has to be generated before running a suite and stored to accessible file system. The query results will be written into output files.