Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix flink versioning issues #31168

Merged
merged 6 commits into from
May 3, 2024
Merged

Fix flink versioning issues #31168

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ad93a62
Fix flink versioning issues
damccorm May 3, 2024
55d2ef8
Add pr reference
damccorm May 3, 2024
2a9bf30
dataproc 2.2
damccorm May 3, 2024
d26d8d4
1.15 for dataproc version
damccorm May 3, 2024
1e98d6c
More explicit changes
damccorm May 3, 2024
7847077
Merge branch 'master' of https://github.com/apache/beam into users/da…
damccorm May 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actions/setup-default-test-properties/test-properties.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
},
"JavaTestProperties": {
"SUPPORTED_VERSIONS": ["8", "11", "17"],
"FLINK_VERSIONS": ["1.13", "1.14", "1.15"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a side note

this is stale implies this field is not used, actually currently this action is only used in update_python_dependencies and publish_github_release_notes workflow.

Currently there is hard coded python/java versions everywhere in our GHAs. As futurre clean up we could make use of this action to have single source of version setting for actions

"FLINK_VERSIONS": ["1.14", "1.15", "1.16", "1.17"],
"SPARK_VERSIONS": ["2", "3"]
},
"GoTestProperties": {
Expand Down
3 changes: 3 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,9 @@

## Known Issues

* The beam interactive runner does not correctly run on flink ([#31168](https://github.com/apache/beam/issues/31168)).
* When using the Flink runner from Python, 1.17 is not supported and 1.12/13 do not work correctly. Support for 1.17 will be added in 2.57.0, and the ability to choose 1.12/13 will be cleaned up and fully removed in 2.57.0 as well ([#31168](https://github.com/apache/beam/issues/31168)).

# [2.55.1] - 2024-04-08

## Bugfixes
Expand Down
2 changes: 1 addition & 1 deletion ...rning-content/introduction/introduction-concepts/runner-concepts/description.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
{{if (eq .Sdk "java")}}

##### Portable
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.10`, `Flink 1.11`, `Flink 1.12`, `Flink 1.13`, `Flink 1.14`.
1. Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub: `Flink 1.14`, `Flink 1.15`, `Flink 1.16`, `Flink 1.17`.
2. Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
3. Submit the pipeline to the above endpoint by using the PortableRunner, job_endpoint set to localhost:8099 (this is the default address of the JobService). Optionally set environment_type set to LOOPBACK. For example:

Expand Down
2 changes: 1 addition & 1 deletion sdks/go/examples/wasm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ cd $BEAM_HOME
Expected output should include the following, from which you acquire the latest flink runner version.

```shell
'flink_versions: 1.12,1.13,1.14,1.15,1.16'
'flink_versions: 1.14,1.15,1.16,1.17'
```

#### 2. Set to the latest flink runner version i.e. 1.16
Expand Down
2 changes: 1 addition & 1 deletion sdks/python/apache_beam/options/pipeline_options.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1591,7 +1591,7 @@ def _add_argparse_args(cls, parser):
class FlinkRunnerOptions(PipelineOptions):

# These should stay in sync with gradle.properties.
PUBLISHED_FLINK_VERSIONS = ['1.12', '1.13', '1.14', '1.15', '1.16']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note - this would essentially prevent python user using flink 1.17 for Beam 2.56.0, and not prevent user choosing 1.12,1.13. Consider update the known issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I sorta mentioned this, but I updated to be clearer/more explicit. Thanks

PUBLISHED_FLINK_VERSIONS = ['1.14', '1.15', '1.16', '1.17']

@classmethod
def _add_argparse_args(cls, parser):
Expand Down
6 changes: 3 additions & 3 deletions sdks/python/apache_beam/runners/interactive/interactive_beam.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,10 +405,10 @@ class Clusters:
To configure a pipeline to run on a local FlinkRunner, explicitly set the
default cluster metadata to None: ib.clusters.set_default_cluster(None).
"""
# Explicitly set the Flink version here to ensure compatibility with 2.0
# Explicitly set the Flink version here to ensure compatibility with 2.1
# Dataproc images:
# https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.0
DATAPROC_FLINK_VERSION = '1.12'
# https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.1
DATAPROC_FLINK_VERSION = '1.15'

# The minimum worker number to create a Dataproc cluster.
DATAPROC_MINIMUM_WORKER_NUM = 2
Expand Down
2 changes: 1 addition & 1 deletion sdks/typescript/src/apache_beam/runners/flink.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import { JavaJarService } from "../utils/service";
const MAGIC_HOST_NAMES = ["[local]", "[auto]"];

// These should stay in sync with gradle.properties.
const PUBLISHED_FLINK_VERSIONS = ["1.12", "1.13", "1.14"];
const PUBLISHED_FLINK_VERSIONS = ["1.14", "1.15", "1.16", "1.17"];

const defaultOptions = {
flinkMaster: "[local]",
Expand Down
14 changes: 8 additions & 6 deletions website/www/site/content/en/documentation/runners/flink.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,13 +196,10 @@ The optional `flink_version` option may be required as well for older versions o

{{< paragraph class="language-portable" >}}
Starting with Beam 2.18.0, pre-built Flink Job Service Docker images are available at Docker Hub:
[Flink 1.10](https://hub.docker.com/r/apache/beam_flink1.10_job_server),
[Flink 1.11](https://hub.docker.com/r/apache/beam_flink1.11_job_server),
[Flink 1.12](https://hub.docker.com/r/apache/beam_flink1.12_job_server).
[Flink 1.13](https://hub.docker.com/r/apache/beam_flink1.13_job_server).
[Flink 1.14](https://hub.docker.com/r/apache/beam_flink1.14_job_server).
[Flink 1.15](https://hub.docker.com/r/apache/beam_flink1.15_job_server).
[Flink 1.16](https://hub.docker.com/r/apache/beam_flink1.16_job_server).
[Flink 1.17](https://hub.docker.com/r/apache/beam_flink1.17_job_server).
{{< /paragraph >}}

<!-- TODO(BEAM-10214): Use actual lists here and below. -->
Expand Down Expand Up @@ -329,6 +326,11 @@ To find out which version of Flink is compatible with Beam please see the table
<th>Artifact Id</th>
<th>Supported Beam Versions</th>
</tr>
<tr>
<td>1.17.x</td>
<td>beam-runners-flink-1.16</td>
<td>&ge; 2.56.0</td>
</tr>
<tr>
<td>1.16.x</td>
<td>beam-runners-flink-1.16</td>
Expand All @@ -347,12 +349,12 @@ To find out which version of Flink is compatible with Beam please see the table
<tr>
<td>1.13.x</td>
<td>beam-runners-flink-1.13</td>
<td>&ge; 2.31.0</td>
<td>&ge; 2.31.0 - 2.55.0</td>
</tr>
<tr>
<td>1.12.x</td>
<td>beam-runners-flink-1.12</td>
<td>&ge; 2.27.0</td>
<td>&ge; 2.27.0 - 2.55.0</td>
</tr>
<tr>
<td>1.11.x</td>
Expand Down
2 changes: 1 addition & 1 deletion website/www/site/content/en/documentation/sdks/java/testing/tpcds.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ When running via Gradle, the following two parameters control the execution:

-P tpcds.runner
The Gradle project name of the runner, such as ":runners:spark:3" or
":runners:flink:1.13. The project names can be found in the root
":runners:flink:1.17. The project names can be found in the root
`settings.gradle.kts`.

Test data has to be generated before running a suite and stored to accessible file system. The query results will be written into output files.
Expand Down
Loading