added beam_PreCommit_Java_Examples_Dataflow job

added pull_request_target event added required steps improved if conditions Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (apache#27161) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.67 to 1.11.70. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.11.67...feature/s3/manager/v1.11.70) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Using capabilities instead of the container name to set use_single_core_per_container Add Exception metadata to the data sampling protocol Add apache#27000 as a known issue released in Beam 2.48.0 (apache#27235) Python data sampling optimization (apache#27157) * Python optimization work * Exception Sampling perf tests * add better element sampling microbenchmark * slowly move towards plumbing the samplers to the bundle processors * cleaned up * starting clean up and more testing * finish tests * fix unused data_sampler args and comments * yapf, comments, and simplifications * linter * lint and mypy * linter * run tests * address review comments * run tests --------- Co-authored-by: Sam Rohde <[email protected]> Improve autoscaler throughput estimates and account for heartbeats (apache#27056) The throughput estimates are improved by storing the estimates per partition (in StreamProgress) rather than per dofn instance. This required some refactoring of ThroughputEstimator/SizeEstimator. It now estimates throughput (per second) of the last ReadChangeStream request. fix(typescript): Publish types correctly, don't publish tests fix(typescript): added more exports Basic error handling for yaml. (apache#27145) Updates Python ExternalTransform to use the transform service when needed (apache#27228) * Updates Python ExternalTransform to use the transform service when needed * Addressing reviewer comments * Fix yapf * Fix lint * Fix yapf Add blog post for Managing Beam dependencies in Java Vertex AI Remote Model Handler (apache#27091) * Move WIP code to new branch at HEAD * Add client-side throttling * Various linting issues * More linting, add dependency to setup.py * Fix Docstring * new flags, types, framework for eventual file read * Route to correct tutorial * Better flag name * Align type hints, pylint cleanup * Move off of constant, preprocess image * Change to keyed version for output * Yapf * Pylint * Clean up ToDo * Add TODOs for file globs * Fix cast, remove unneccessary assert * whitespace * More whitespace * Import order * More import ordering * Amend comment [Playground] [Frontend] Brand colors in Playground flutter_code_editor (apache#27218) * brand colors in PG FCE * link to website * restart ci * restart ci --------- Co-authored-by: darkhan.nausharipov <[email protected]> Bump go.mongodb.org/mongo-driver from 1.11.7 to 1.12.0 in /sdks (apache#27215) Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.11.7 to 1.12.0. - [Release notes](https://github.com/mongodb/mongo-go-driver/releases) - [Commits](mongodb/mongo-go-driver@v1.11.7...v1.12.0) --- updated-dependencies: - dependency-name: go.mongodb.org/mongo-driver dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> update readme with Beam Playground, Tour of Beam (apache#27243) Co-authored-by: xqhu <[email protected]> runner image and arc changes (apache#27118) * runner image and arc changes * add state bucket comment in env file for reference * add readme for manual image push and sync beam environment --------- Co-authored-by: vdjerek <Vlado Djerek> Fix whitespace (apache#27247) [Tour of Beam] Learning content for "Final challenge" module (apache#26861) * add final challenge * correct examples * changge * correct golang * correct example hint * correct examples and add golang example * delete whitespace * delete whitespace * add file tag * correct final challenge * fixing incorrect tags and names * minor formatting * fixing example formatting * correct challenge * correct whitespace * change final-challenge-2 * correct imports * fix splittable unit id * formatting * format change * formatting * fixing template * temp fix of url * backend urls fix * restore backend urls * change * remove window --------- Co-authored-by: mende1esmende1es <[email protected]> Co-authored-by: Oleh Borysevych <[email protected]> Bump transformers (apache#27108) Bumps [transformers](https://github.com/huggingface/transformers) from 4.21.1 to 4.30.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](huggingface/transformers@v4.21.1...v4.30.0) --- updated-dependencies: - dependency-name: transformers dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Code completion plugin: add plugin tests workflow (apache#27212) * Code completion plugin: add plugin tests workflow * Testing changing directory in code_completion_plugin_tests.yml * Making gradlew executable * Fix gradlew intellij plugin import * change gradlew premissions * code-completion-plugin: fixing GA tests workflow * Revert "change gradlew premissions" This reverts commit ee396be. * Revert "code-completion-plugin: fixing GA tests workflow" This reverts commit 9baa351. * fix code_completion_plugin_tests.yml * Making gradlew executable * code-completion-plugin-fix intellij plugin version in build.gradle.kts * code-completion-plugin: change java version (17 -> 11) in code_completion_plugin_tests.yml * test code completion plugin CI * Add env vars to code_completion_plugin_test.yml * code-completion-plugin: test ci * code-completion-plugin: clone intellij-community to CI runner --------- Co-authored-by: Pablo <[email protected]> Co-authored-by: Pablo E <[email protected]> add job rerun action (apache#27210) * add job rerun action * Update .github/actions/rerun-job-action/action.yml Typo Co-authored-by: Danny McCormick <[email protected]> * header and a comment explaining the logic for going into re-run flow * deduplicate API calls * added some explanation within the action file * Update .github/actions/rerun-job-action/action.yml Typo Co-authored-by: Danny McCormick <[email protected]> --------- Co-authored-by: vdjerek <Vlado Djerek> Co-authored-by: Danny McCormick <[email protected]> Code completion plugin: Add Java SDK Transform Completions (Hard Coded) (apache#27168) * [Code Completion Plugin] Define Element Pattern for java sdk * Code Completion Plugin: Add java sdk transform completions * Verifying that tests are running as expected :) Sorry! * Update BeamCompletionContributorTestCase.java --------- Co-authored-by: Pablo <[email protected]> [Website] add hsbc case study (apache#27200) [Tour of Beam] add work example (apache#27080) * add work example * correct * correct tags for bigquery examples * correct read-query * correct read-query tag * correct imports * remove package * correct * fixed example name --------- Co-authored-by: mende1esmende1es <[email protected]> Co-authored-by: Oleh Borysevych <[email protected]> Remove py37 references (apache#27252) Handling issue where keystore file is empty for MongoDB SSL (apache#27250) Bump worker image dependency (apache#27253) [apache#22737] Add line about timer support for Go (apache#27263) Bump actions/checkout from 2 to 3 (apache#27259) Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Add Tour of Beam Page (apache#27244) * add tour of beam page to redirect. * add tour of beam page to redirect. * add tour of beam page to redirect.
akvelon · Jul 7, 2023 · fc7120e · fc7120e
1 parent f19c384
commit fc7120e
Show file tree

Hide file tree

Showing 24 changed files with 374 additions and 558 deletions.
diff --git a/.github/actions/rerun-job-action/action.yml b/.github/actions/rerun-job-action/action.yml
@@ -57,24 +57,17 @@ runs:
     shell: bash
     run: | 
       URL="${{github.api_url}}/repos/${{inputs.github_repository}}/commits/${{env.prsha}}/check-runs"
-      CHECK_RUN=$(curl  \
+      read -r STATUS CONCLUSION CHECK_SUITE_ID<<< $(curl  \
       -H 'Authorization: Bearer ${{inputs.github_token}}' \
       -H "Accept: application/vnd.github+json" \
       -H "X-GitHub-Api-Version: 2022-11-28" \
-      -s $URL | jq -r '.check_runs | .[] | select(.name=="${{inputs.github_job}}")')
-      if [ -z "$CHECK_RUN" ]; then
-        echo "No check runs found for this job"
-        echo skip=true >> $GITHUB_ENV
-        exit 0
-      fi
-      read -r STATUS CONCLUSION CHECK_SUITE_ID<<< $(echo $CHECK_RUN | jq -r '"\(.status) \(.conclusion) \(.check_suite.id)"')
+      -s $URL | jq -r '.check_runs | .[] | select(.name=="${{inputs.github_job}}") | "\(.status) \(.conclusion) \(.check_suite.id)"')
       echo status=$STATUS >> $GITHUB_ENV
       echo conclusion=$CONCLUSION >> $GITHUB_ENV
       echo check_suite_id=$CHECK_SUITE_ID >> $GITHUB_ENV
  
-
-  - name: Disable Rerun for Success or Skipped
-    if: ${{(env.status == 'completed' && (env.conclusion == 'success' || env.conclusion == 'skipped')) || env.skip == 'true'}}
+  - name: Enable Rerun
+    if: ${{env.status == 'completed' && (env.conclusion == 'success' || env.conclusion == 'skipped')}}
     shell: bash
     run: echo rerun=false >> $GITHUB_ENV
 
@@ -110,13 +103,6 @@ runs:
       -H "Accept: application/vnd.github+json" \
       -H "X-GitHub-Api-Version: 2022-11-28" \
       -s $URL 
-  - name: Install GH Cli
-    if: ${{env.rerun != 'false' }}
-    shell: bash
-    run: |
-      wget https://github.com/cli/cli/releases/download/v2.31.0/gh_2.31.0_linux_amd64.tar.gz
-      tar -xvf gh_2.31.0_linux_amd64.tar.gz
-      sudo mv gh_2.31.0_linux_amd64/bin/gh /usr/local/bin
   - name: Exit rest of the run
     if: ${{env.rerun != 'false' }}
     shell: bash

diff --git a/.github/gh-actions-self-hosted-runners/arc/config/arc_deployment.tpl b/.github/gh-actions-self-hosted-runners/arc/config/arc_deployment.tpl
@@ -43,6 +43,9 @@ spec:
       %{~ endfor ~}
       env: []
       resources:
+#        limits:
+#          cpu: "4.0"
+#          memory: "8Gi"
         requests:
           cpu: ${requests.cpu}
           memory: ${requests.memory}

diff --git a/.github/gh-actions-self-hosted-runners/arc/environments/beam.env b/.github/gh-actions-self-hosted-runners/arc/environments/beam.env
@@ -28,35 +28,9 @@ github_app_id_secret_name = "gh-app_id"
 github_app_install_id_secret_name = "gh-app_installation_id"
 github_private_key_secret_name = "gh-pem_key"
 deploy_webhook = "true"
+max_main_replicas = "50"
+min_main_replicas = "5"
+webhook_scaling = "true"
 runner_group = "beam"
-main_runner = {
-    name = "main-runner"
-    machine_type = "e2-standard-16"
-    min_node_count = "1"
-    max_node_count = "7"
-    min_replicas = "1"
-    max_replicas = "45"
-    webhook_scaling = true
-    disk_size_gb = 200
-    requests = {
-        cpu = "2"
-        memory = "3Gi"
-    }
-}
-additional_runner_pools = [{
-    name = "small-runner"
-    machine_type = "e2-standard-2"
-    min_node_count = "1"
-    max_node_count = "10"
-    min_replicas = "1"
-    max_replicas = "10"
-    webhook_scaling = "true"
-    requests = {
-        cpu = "1500m"
-        memory = "5Gi"
-    }
-    labels = ["self-hosted", "ubuntu-20.04", "small"]
-    enable_selector = true
-    enable_taint = true
-}]
-#state_bucket_name = "beam-arc-state"
+machine_type = "e2-standard-16"
+#state_bucket_name = "beam-arc-state"
diff --git a/.github/gh-actions-self-hosted-runners/arc/gke.tf b/.github/gh-actions-self-hosted-runners/arc/gke.tf
@@ -40,8 +40,7 @@ resource "google_container_node_pool" "main-actions-runner-pool" {
     auto_upgrade = "true"
    }
   node_config {
-    disk_size_gb = var.main_runner.disk_size_gb
-    machine_type = var.main_runner.machine_type
+    machine_type    = var.machine_type
     oauth_scopes = [
       "https://www.googleapis.com/auth/cloud-platform"
     ]

diff --git a/.github/gh-actions-self-hosted-runners/arc/images/Dockerfile b/.github/gh-actions-self-hosted-runners/arc/images/Dockerfile
@@ -21,14 +21,14 @@
 FROM ghcr.io/actions-runner-controller/actions-runner-controller/actions-runner:ubuntu-20.04
 
 USER root
-#Install Node
+#Install Node 
 RUN curl -OL https://nodejs.org/dist/v18.16.0/node-v18.16.0-linux-x64.tar.xz && \
     tar -C /usr/local -xf node-v18.16.0-linux-x64.tar.xz && \
     rm node-v18.16.0-linux-x64.tar.xz && \
     mv /usr/local/node-v18.16.0-linux-x64 /usr/local/node
 ENV PATH="${PATH}:/usr/local/node/bin"
 #Install Go
-ARG go_version=1.20.5
+ARG go_version=1.20.4
 RUN curl -OL https://go.dev/dl/go${go_version}.linux-amd64.tar.gz && \
     tar -C /usr/local -xzf go${go_version}.linux-amd64.tar.gz && \
     rm go${go_version}.linux-amd64.tar.gz
@@ -45,20 +45,8 @@ RUN curl -OL https://services.gradle.org/distributions/gradle-7.3.3-bin.zip && \
     rm gradle-7.3.3-bin.zip && \
     mv gradle-7.3.3 /usr/local/gradle
 ENV PATH="${PATH}:/usr/local/gradle/bin"
-#Install GitHub CLI
-RUN curl -OL https://github.com/cli/cli/releases/download/v2.31.0/gh_2.31.0_linux_amd64.tar.gz && \
-    tar -xvf gh_2.31.0_linux_amd64.tar.gz && \
-    rm gh_2.31.0_linux_amd64.tar.gz && \
-    mv gh_2.31.0_linux_amd64/bin/gh /usr/local/bin 
-#Install GCloud CLI and Kubectl
-RUN curl -OL https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-367.0.0-linux-x86_64.tar.gz && \
-    tar -xvf google-cloud-sdk-367.0.0-linux-x86_64.tar.gz && \
-    rm google-cloud-sdk-367.0.0-linux-x86_64.tar.gz && \
-    mv google-cloud-sdk /usr/local/google-cloud-sdk && \
-    /usr/local/google-cloud-sdk/install.sh --quiet && \
-    /usr/local/google-cloud-sdk/bin/gcloud components install kubectl 
-ENV PATH="${PATH}:/usr/local/google-cloud-sdk/bin"
+
 
 # Needed to transfer path addtitions to runner environment
 RUN echo PATH=$PATH >> /runnertmp/.env
-USER runner
+USER runner
diff --git a/.github/workflows/beam_PreCommit_Java_Examples_Dataflow.yml b/.github/workflows/beam_PreCommit_Java_Examples_Dataflow.yml
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: PreCommit Java Examples Dataflow
+
+on:
+  push:
+    tags: ['v*']
+    branches: ['master', 'release-*']
+    paths: ['model/**', 'sdks/java/**', 'runners/google-cloud-dataflow-java/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+  pull_request_target:
+    branches: ['master', 'release-*']
+    paths: ['model/**', 'sdks/java/**', 'runners/google-cloud-dataflow-java/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+  issue_comment:
+    types: [created]
+  schedule:
+    - cron: '* */6 * * *'
+
+permissions: read-all
+jobs:
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: check_gcp_variables
+    runs-on: [self-hosted, ubuntu-20.04]
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+      - name: Check are GCP variables set
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+          GCP_REGION: "not-needed-here"
+          GCP_PYTHON_WHEELS_BUCKET: "not-needed-here"
+
+  beam_PreCommit_Java_Examples_Dataflow:
+    name: beam_PreCommit_Java_Examples_Dataflow
+    needs:
+      - check_gcp_variables
+    runs-on: [self-hosted, ubuntu-20.04]
+    if: |
+      github.event_name == 'push' ||
+      github.event_name == 'pull_request_target' ||
+      github.event_name == 'schedule' ||
+      github.event.comment.body == 'Run Java_Examples_Dataflow PreCommit'
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+      - name: Setup self-hosted
+        uses: ./.github/actions/setup-self-hosted-action
+        with:
+          requires-py-38: false
+          requires-py-39: false
+          requires-go: false
+      - name: Setup Gradle
+        uses: gradle/gradle-build-action@v2
+        with:
+          cache-read-only: false
+      - name: Authenticate on GCP
+        uses: google-github-actions/setup-gcloud@v0
+        with:
+          service_account_email: ${{ secrets.GCP_SA_EMAIL }}
+          service_account_key: ${{ secrets.GCP_SA_KEY }}
+          project_id: ${{ secrets.GCP_PROJECT_ID }}
+          export_default_credentials: true
+      - name: run javaExamplesDataflowPrecommit script
+        run: ./gradlew :javaExamplesDataflowPrecommit
+          -PdisableSpotlessCheck=true
+          -PdisableCheckStyle=true
+      - name: Upload test report
+        uses: actions/upload-artifact@v3
+        with:
+          name: java-code-coverage-report
+          path: "**/build/test-results/**/*.xml"
diff --git a/.github/workflows/build_runner_image.yml b/.github/workflows/build_runner_image.yml
@@ -19,15 +19,10 @@
 name: Build and Version Runner Docker Image
 
 on:
-  push:
-    branches: ['master']
-    paths: ['.github/gh-actions-self-hosted-runners/arc/images/**','.github/workflows/build_runner_image.yml']
   pull_request_target:
     branches: ['master']
     paths: ['.github/gh-actions-self-hosted-runners/arc/images/**']
-env:
-  docker_registry: us-central1-docker.pkg.dev
-  docker_repo: apache-beam-testing/beam-github-actions/beam-arc-runner
+
 jobs:
   build-and-version-runner:
     env:
@@ -39,33 +34,16 @@ jobs:
       uses: actions/checkout@v3
       with:
         ref: ${{ github.event.pull_request.head.sha }}
-    - name: Authenticate on GCP
-      uses: google-github-actions/setup-gcloud@v0
-      with:
-        service_account_email: ${{ secrets.GCP_SA_EMAIL }}
-        service_account_key: ${{ secrets.GCP_SA_KEY }}
-        project_id: ${{ secrets.GCP_PROJECT_ID }}
-        export_default_credentials: true
-    - name: GCloud Docker credential helper
-      run: |
-        gcloud auth configure-docker ${{env.docker_registry}}
+
     - name: Set up Docker Buildx
       uses: docker/setup-buildx-action@v1
-    - name: Build and Load to docker
-      uses: docker/build-push-action@v4
-      with:
-        context: ${{ env.working-directory }}
-        load: true
-        tags: |
-          ${{env.docker_registry}}/${{env.docker_repo}}:latest
-          ${{env.docker_registry}}/${{env.docker_repo}}:${{ github.sha }}
-    - name: Push Docker image
-      if: github.event_name == 'push'
+
+    - name: Build and push Docker image
       id: docker_build
-      uses: docker/build-push-action@v4
+      uses: docker/build-push-action@v2
       with:
         context: ${{ env.working-directory }}
-        push: true
+        push: false
         tags: |
-          ${{env.docker_registry}}/${{env.docker_repo}}:latest
-          ${{env.docker_registry}}/${{env.docker_repo}}:${{ github.sha }}
+          us-central1-docker.pkg.dev/apache-beam-testing/beam-github-actions/beam-arc-runner:latest
+          us-central1-docker.pkg.dev/apache-beam-testing/beam-github-actions/beam-arc-runner:${{ github.sha }}
diff --git a/CHANGES.md b/CHANGES.md
@@ -65,7 +65,10 @@
 
 ## New Features / Improvements
 
-* X feature added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
+* Allow prebuilding large images when using `--prebuild_sdk_container_engine=cloud_build`, like images depending on `tensorflow` or `torch` ([#27023](https://github.com/apache/beam/pull/27023)).
+* Disabled `pip` cache when installing packages on the workers. This reduces the size of prebuilt Python container images ([#27035](https://github.com/apache/beam/pull/27035)).
+* Select dedicated avro datum reader and writer (Java) ([#18874](https://github.com/apache/beam/issues/18874)).
+* Timer API for the Go SDK (Go) ([#22737](https://github.com/apache/beam/issues/22737)).
 
 ## Breaking Changes
 

diff --git a/sdks/go.mod b/sdks/go.mod
@@ -33,8 +33,8 @@ require (
 	github.com/aws/aws-sdk-go-v2 v1.18.1
 	github.com/aws/aws-sdk-go-v2/config v1.18.27
 	github.com/aws/aws-sdk-go-v2/credentials v1.13.26
-	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.71
-	github.com/aws/aws-sdk-go-v2/service/s3 v1.36.0
+	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.70
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0
 	github.com/aws/smithy-go v1.13.5
 	github.com/docker/go-connections v0.4.0
 	github.com/dustin/go-humanize v1.0.1
@@ -52,8 +52,8 @@ require (
 	github.com/xitongsys/parquet-go v1.6.2
 	github.com/xitongsys/parquet-go-source v0.0.0-20220315005136-aec0fe3e777c
 	go.mongodb.org/mongo-driver v1.12.0
-	golang.org/x/net v0.12.0
-	golang.org/x/oauth2 v0.10.0
+	golang.org/x/net v0.11.0
+	golang.org/x/oauth2 v0.9.0
 	golang.org/x/sync v0.3.0
 	golang.org/x/sys v0.10.0
 	golang.org/x/text v0.11.0

diff --git a/sdks/go.sum b/sdks/go.sum
@@ -92,8 +92,8 @@ github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.3.0/go.mod h1:2LAuqPx1I6jNfaGDu
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4 h1:LxK/bitrAr4lnh9LnIS6i7zWbCOdMsfzKFBI6LUCS0I=
 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4/go.mod h1:E1hLXN/BL2e6YizK1zFlYd8vsfi2GTjbjBazinMmeaM=
 github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.3.2/go.mod h1:qaqQiHSrOUVOfKe6fhgQ6UzhxjwqVW8aHNegd6Ws4w4=
-github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.71 h1:SAB1UAVaf6nGCu3zyIrV+VWsendXrms1GqtW4zBotKA=
-github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.71/go.mod h1:ZNo5H4PR3/fwsXYqb+Ld5YAfvHcYCbltaTTtSay4l2o=
+github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.70 h1:4bh28MeeXoBFTjb0JjQ5sVatzlf5xA1DziV8mZed9v4=
+github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.70/go.mod h1:9yI5NXzqy2yOiMytv6QLZHvlyHLwYxO9iIq+bZIbrFg=
 github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34 h1:A5UqQEmPaCFpedKouS4v+dHCTUo2sKqhoKO9U5kxyWo=
 github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34/go.mod h1:wZpTEecJe0Btj3IYnDx/VlUzor9wm3fJHyvLpQF0VwY=
 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28 h1:srIVS45eQuewqz6fKKu6ZGXaq6FuFg5NzgQBAM6g8Y4=
@@ -115,8 +115,8 @@ github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.5.1/go.mod h1:6EQZIwNN
 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3 h1:dBL3StFxHtpBzJJ/mNEsjXVgfO+7jR0dAIEwLqMapEA=
 github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.14.3/go.mod h1:f1QyiAsvIv4B49DmCqrhlXqyaR+0IxMmyX+1P+AnzOM=
 github.com/aws/aws-sdk-go-v2/service/s3 v1.11.1/go.mod h1:XLAGFrEjbvMCLvAtWLLP32yTv8GpBquCApZEycDLunI=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.36.0 h1:lEmQ1XSD9qLk+NZXbgvLJI/IiTz7OIR2TYUTFH25EI4=
-github.com/aws/aws-sdk-go-v2/service/s3 v1.36.0/go.mod h1:aVbf0sko/TsLWHx30c/uVu7c62+0EAJ3vbxaJga0xCw=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0 h1:ya7fmrN2fE7s1P2gaPbNg5MTkERVWfsH8ToP1YC4Z9o=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.35.0/go.mod h1:aVbf0sko/TsLWHx30c/uVu7c62+0EAJ3vbxaJga0xCw=
 github.com/aws/aws-sdk-go-v2/service/sso v1.3.1/go.mod h1:J3A3RGUvuCZjvSuZEcOpHDnzZP/sKbhDWV2T1EOzFIM=
 github.com/aws/aws-sdk-go-v2/service/sso v1.12.12 h1:nneMBM2p79PGWBQovYO/6Xnc2ryRMw3InnDJq1FHkSY=
 github.com/aws/aws-sdk-go-v2/service/sso v1.12.12/go.mod h1:HuCOxYsF21eKrerARYO6HapNeh9GBNq7fius2AcwodY=
@@ -604,8 +604,8 @@ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.3.8/go.mod h1:E6s5w1FMmriuDzIBO73fBruAKo1PCIq6d2Q6DHfQ8WQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
-golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4=
-golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
+golang.org/x/text v0.10.0 h1:UpjohKhiEgNc0CSauXmwYftY1+LlaC75SJwh0SgCX58=
+golang.org/x/text v0.10.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=

diff --git a/...ain/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/ChangeStreamAction.java b/...ain/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/action/ChangeStreamAction.java
@@ -107,7 +107,8 @@ public Optional<DoFn.ProcessContinuation> run(
       RestrictionTracker<StreamProgress, StreamProgress> tracker,
       DoFn.OutputReceiver<KV<ByteString, ChangeStreamRecord>> receiver,
       ManualWatermarkEstimator<Instant> watermarkEstimator,
-      BytesThroughputEstimator<KV<ByteString, ChangeStreamRecord>> throughputEstimator) {
+      BytesThroughputEstimator<KV<ByteString, ChangeStreamRecord>> throughputEstimator,
+      boolean shouldDebug) {
     if (record instanceof Heartbeat) {
       Heartbeat heartbeat = (Heartbeat) record;
       final Instant watermark = toJodaTime(heartbeat.getEstimatedLowWatermark());