Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-12792] Install pipeline dependencies to temporary venv #16658

Merged
merged 9 commits into from
Nov 10, 2022
Merged

[BEAM-12792] Install pipeline dependencies to temporary venv #16658

merged 9 commits into from
Nov 10, 2022

Conversation

phoerious
Copy link
Contributor

@phoerious phoerious commented Jan 31, 2022 •
edited by tvalentyn
Loading

The change allows users to submit multiple pipelines to the same SDK container without dependency conflicts. Dependencies in --setup_only mode are still installed globally.

With this patch, the deployment cycle looks like this:

  1. Worker pool is spawned (PID 1 in SDK container => worker pool ID = 1).
  2. Flink starts a job and a worker gets spun up with ID 1-1.
  3. The worker creates a temporary venv in /opt/apache/beam-venv/beam-venv-worker-1-1 and deploys the artifact with dependencies.
  4. Processing happens ...
  5. Everything is cleaned up and a new job can be submitted.

The venvs are bound to individual workers (or sets of workers if they have siblings) and are cleaned up as soon as they finish. This makes multiProcessExactlyOnce() obsolete, so it was removed. Additional care was taken to ensure a clean signal flow between individual components, so anything that would cause a premature os.Exit() or a SIGKILL to be sent was adjusted to allow for clean worker shutdown.

fixes: #21123

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- Build Status Build Status Build Status Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status 		Build Status
Build Status 		Build Status
Build Status
Build Status 		Build Status
Python --- Build Status
Build Status
Build Status 		Build Status
Build Status 		Build Status Build Status ---
XLang Build Status Build Status
Build Status
Build Status
Build Status Build Status Build Status ---

Examples testing status on various runners

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- --- --- --- --- --- ---
Java --- Build Status
Build Status
Build Status
--- --- --- --- ---
Python --- --- --- --- --- --- ---
XLang --- --- --- --- --- --- ---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go Java Python
Build Status Build Status Build Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status
Build Status
Build Status
Build Status
Build Status 		Build Status Build Status Build Status Build Status
Portable --- Build Status Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

phoerious
phoerious commented Jan 31, 2022
View reviewed changes
sdks/python/container/boot.go Outdated Show resolved Hide resolved
sdks/python/container/boot.go Outdated Show resolved Hide resolved
@tvalentyn
Copy link
Contributor

cc: @ryanthompson591 for initial pass.

@phoerious
Copy link
Contributor Author

Any progress here?

ryanthompson591
ryanthompson591 approved these changes Feb 4, 2022
View reviewed changes
Copy link
Contributor

@ryanthompson591 ryanthompson591 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, just a few small questions. Please wait for valentyn's review before merging.

sdks/python/container/boot.go Outdated Show resolved Hide resolved
sdks/python/container/boot.go Outdated Show resolved Hide resolved
sdks/python/container/boot.go Outdated Show resolved Hide resolved
ryanthompson591
ryanthompson591 approved these changes Feb 7, 2022
View reviewed changes
sdks/python/container/boot.go Outdated Show resolved Hide resolved
sdks/python/container/boot.go Outdated Show resolved Hide resolved
@ryanthompson591
Copy link
Contributor

R: @tvalentyn

AnandInguva
AnandInguva reviewed Feb 11, 2022
View reviewed changes
CHANGES.md Outdated Show resolved Hide resolved
tvalentyn
tvalentyn reviewed Feb 19, 2022
View reviewed changes
sdks/python/container/boot.go Outdated Show resolved Hide resolved
sdks/python/container/boot.go Outdated Show resolved Hide resolved
@phoerious
Copy link
Contributor Author

phoerious commented Feb 21, 2022 •
edited
Loading

@ryanthompson591 @tvalentyn I updated the PR. The venvs are now using random names and are bound to the workers, which is the only way to make this safe.

I also fixed how workers are cleaned up. Previously, they were simply SIGKILL'ed by the worker pool Python executable, which prevented any kind of clean up and also caused zombie processes inside the containers. I think there are also still some cases where processes are not cleaned up properly and just keep running forever, but most of that should be fixed now. Processes that keep running forever happen particularly when I'm using a global CombineFn, which causes Flink to believe that the last remaining worker is still running even though it has long finished. When that happens, not even cancelling the job will send signals to the remaining workers. But that's another bug (I reported that before on the mailing list, but never got a response).

All of this needs some more testing, but it seems to be running fine on my Flink cluster at least.

@phoerious phoerious force-pushed the beam-12792/sdk-venv branch 2 times, most recently from 94a88d5 to fee9224 Compare February 21, 2022 18:41
@codecov
Copy link

codecov bot commented Feb 21, 2022 •
edited
Loading

Codecov Report

Merging #16658 (5a91e67) into master (4522f4c) will decrease coverage by 0.00%.
The diff coverage is 7.14%.

@@            Coverage Diff             @@
##           master   #16658      +/-   ##
==========================================
- Coverage   73.32%   73.32%   -0.01%     
==========================================
  Files         714      714              
  Lines       96403    96409       +6     
==========================================
+ Hits        70685    70688       +3     
- Misses      24398    24401       +3     
  Partials     1320     1320
Flag Coverage Δ
python 83.20% <7.14%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...hon/apache_beam/runners/worker/worker_pool_main.py 56.32% <7.14%> (-2.94%) ⬇️
.../python/apache_beam/transforms/periodicsequence.py 98.50% <0.00%> (+1.49%) ⬆️
sdks/python/apache_beam/utils/interactive_utils.py 97.56% <0.00%> (+2.43%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@phoerious
Copy link
Contributor Author

phoerious commented Feb 22, 2022 •
edited
Loading

@tvalentyn @ryanthompson591 All right. I think I have it a point where it's ready for a final review. It runs robustly on a 130 node Flink cluster, all processes are reliably and gracefully cleaned up, and I haven't seen any zombie processes.

(also pinging the guys from the OWNERs file, maybe someone has additional remarks: @herohde @aaltay @charlesccychen)

@phoerious phoerious force-pushed the beam-12792/sdk-venv branch 3 times, most recently from 1bdaf44 to 99aed21 Compare February 22, 2022 13:46
@phoerious phoerious force-pushed the beam-12792/sdk-venv branch 2 times, most recently from 8d429b8 to a41aed4 Compare February 22, 2022 15:09
@aaltay
Copy link
Member

aaltay commented Feb 22, 2022

Current set of reviewers looks great. I do not have much to add. Thanks for adding me.

worker_pool_main changes - R: @y1chi could review.

@phoerious
Copy link
Contributor Author

phoerious commented Feb 23, 2022 •
edited
Loading

I added one more change: If a worker exits with a non-zero exit code, the boot binary also exits with a non-zero code. That makes it easier to debug things like Python interpreter crashes, e.g., due to failures in a native extension. That case was very hard to debug previously.

@tvalentyn
Copy link
Contributor

tests passed: https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_PR/255/

@tvalentyn tvalentyn merged commit 08b6a52 into apache:master Nov 10, 2022
@tvalentyn
Copy link
Contributor

Merged. Thanks for your perseverance, @phoerious .

@phoerious phoerious changed the title [BEAM-12792] Install pipline dependencies to temporary venv [BEAM-12792] Install pipeline dependencies to temporary venv Nov 10, 2022
@phoerious phoerious deleted the beam-12792/sdk-venv branch November 10, 2022 17:40
nausharipov pushed a commit to akvelon/beam that referenced this pull request Nov 18, 2022
@nausharipov
auth, complete unit, user progress
5297d25 
AuthNotifier draft (#23692)

Comments (#23692)

Comments (#23692)(1)

sign in with google works (#23692)

new configs (#23692)

get user progress draft (#23692)

comment fixes (#23692)

sign in in IntroTextBody (#23692)

reverted config (#23692)

comment fixes (#23692)

WIP before rebase (merge) (#23692)

Squashed commit of the following:

commit bff4919ff00ec3b5d7186efde41c884dfc4c8344
Merge: 79ba69483a ce8d618c77
Author: Alexey Romanenko <[email protected]>
Date:   Thu Nov 17 10:34:02 2022 +0100

    Merge pull request #24186: Uses _all to follow alias/datastreams when estimating index size

commit 79ba69483a84ea0278d0b0ddb141200739607c77
Merge: 245fea9040 b7e860a762
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 20:47:40 2022 -0800

    Merge pull request #24218: Update Python wheel format for RC validation

commit 245fea904014cd58d4148807463dbaa40000774c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Nov 16 18:12:33 2022 -0800

    Bump loader-utils from 1.4.1 to 1.4.2 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel (#24191)

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit e1de8e78deeb5d17617fda6591429eaaf8abb8a2
Author: Yi Hu <[email protected]>
Date:   Wed Nov 16 20:48:06 2022 -0500

    Fix PythonLint (#24219)

commit b7e860a7621771c300dcec625655f87e62591323
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 17:28:31 2022 -0800

    updates

commit c2feb09ea49dd815b69c65e531ce34128756d988
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 17:06:08 2022 -0800

    updates

commit ce8d618c77d23e20a1ddb128bb8183048597d096
Author: egalpin <[email protected]>
Date:   Wed Nov 16 16:43:57 2022 -0800

    Adds test for following aliases when estimating index size

commit 959719d01c627328c0ca2849d2b7e2c9b322d4d1
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 15:16:06 2022 -0800

    Temporary update Python RC validation job

commit b952b41788acc20edbe5b75b2196f30dbf8fdeb0
Author: Yi Hu <[email protected]>
Date:   Wed Nov 16 14:18:12 2022 -0500

    Python TextIO Performance Test (#23951)

    * Python TextIO Performance Test

    * Add filebasedio_perf_test module for unified test framework for
      Python file-based IOs

    * Fix MetricsReader publishes metrics duplicately if more than one
      load test declared. This is because MetricsReader.publishers was
      static class variable

    * Fix pylint

    * Distribute Python performance tests random time at a day instead of all at 3PM

    * Add information about length conversion

commit 017f2cbde124af40a43be99ec88289fcf63c1c95
Merge: fef8acdbc0 88dba4f494
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 10:39:52 2022 -0800

    Merge pull request #24187: Add a reference to Java RunInference example

commit fef8acdbc0ecbcc85b49144adaf8830e3bc6b2de
Merge: 6e9187e67e ead245539d
Author: Ahmet Altay <[email protected]>
Date:   Wed Nov 16 10:24:53 2022 -0800

    Merge pull request #24199 from Laksh47/issue#24196

    refs: issue-24196, fix broken hyperlink

commit 6e9187e67e1bd8f73997f437f0ed4c29880ed73b
Author: Darkhan Nausharipov <[email protected]>
Date:   Wed Nov 16 22:33:50 2022 +0600

    [Tour of Beam] [Frontend] Content tree URLs (#23776)

    * Content tree navigation (#23593)

    Unit content navigation (#23593)

    Update URL on node click (#23593)

    Active unit color (#23593)

    removeListener in unit (#23593)

    First unit is opened on group title click (#23593)

    WIP by Alexey Inkin (#23593)

    selectedUnitColor (#23593)

    Unit borderRadius (#23593)

    RegExp todo (#23593)

    added referenced collection package to remove warning (#23593)

    small refinement (#23593)

    expand on group tap, padding, openNode (#23593)

    group expansion bug fix (#23593)

    selected & unselected progress indicators (#23593)

    * AnimatedBuilders instead of StatefulWidgets in unit & group (#23593)

    * fixed _getNodeAncestors (#23593)

    * get sdkId (#23593)

    * addressing comments (#23593)

    * sdkId getter & StatelessExpansionTile (#23593)

    * expand & collapse group (#23593)

    * StatelessExpansionTile (#23593)

    * license (#23593)

    * ValueChanged and ValueKey in StatelessExpansionTile (#23593)

    Co-authored-by: darkhan.nausharipov <[email protected]>
    Co-authored-by: Alexey Inkin <[email protected]>

commit b33fac2aa533d77cfa47f88466c8cd6bd3f3e864
Author: Bruno Volpato <[email protected]>
Date:   Wed Nov 16 10:51:11 2022 -0500

    Use only ValueProviders in SpannerConfig (#24156)

commit 5f013ab6567ec75b460b2081d7f89d332320caff
Author: Robert Burke <[email protected]>
Date:   Wed Nov 16 07:23:10 2022 -0800

    revert upgrade to go 1.19 for action unit tests (#24189)

commit 9337f4dbecc929886f8559949a082a649fd9d1bb
Author: Yi Hu <[email protected]>
Date:   Wed Nov 16 10:18:42 2022 -0500

    Fix Python PostCommit Example CustomPTransformIT on portable (#24159)

    * Fix Python PostCommit Examples on portable

    * Fix custom_ptransform pipeline options gets modified

    * Specify flinkConfDir

commit ead245539d01dec0f3e08699c1e1cc6777a5ef0e
Author: Laksh <[email protected]>
Date:   Wed Nov 16 09:32:46 2022 -0500

    refs: issue-24196, fix broken hyperlink

commit e83a996d4374d467d95bcfad7166905622ec615c
Merge: 2fc56ec663 ffdee0b6ed
Author: Jan Lukavský <[email protected]>
Date:   Wed Nov 16 15:15:31 2022 +0100

    Merge pull request #24192: Re-use serializable pipeline options when already available.

commit ffdee0b6edb8638c78a65ec85c727ea5dde1cb2f
Author: Jozef Vilcek <[email protected]>
Date:   Mon Nov 14 16:48:18 2022 +0100

    Re-use serializable pipeline options when already available (#24192)

commit 88dba4f494829b2b3530b767fb8c5252e0d2ba44
Author: Chamikara Jayalath <[email protected]>
Date:   Tue Nov 15 16:21:22 2022 -0800

    Add a reference to Java RunInference example

commit 2fc56ec663e335cfcf37dc57d471f79b601414f4
Merge: f763186987 83f1bc19b9
Author: Kenn Knowles <[email protected]>
Date:   Tue Nov 15 16:16:47 2022 -0800

    Merge pull request #24142: Fix arguments to checkState in BatchViewOverrides

commit f763186987c00ba1d26efdc35406436a1fa69a9a
Merge: c2bc2135e9 0d7ca04182
Author: Ning Kang <[email protected]>
Date:   Tue Nov 15 15:25:20 2022 -0800

    Addresses #24161 Updated README of Interactive Beam

commit c2bc2135e9bce715990a5d5551e2bc2dc0311da4
Author: Doug Judd <[email protected]>
Date:   Tue Nov 15 14:48:26 2022 -0800

    Strip FGAC database role from changestreams metadata requests (#24177)

    Co-authored-by: Doug Judd <[email protected]>

commit af637974f96ad1b5110d7dea3f9a26c68e19a51b
Author: Jack McCluskey <[email protected]>
Date:   Tue Nov 15 17:16:43 2022 -0500

    Add custom inference function support to the PyTorch model handler (#24062)

    * Initial type def and function signature

    * [Draft] Add custom inference fn support to Pytorch Model Handler

    * Formatting

    * Split out default

    * Remove Keyed version for testing

    * Move device optimization

    * Make default available for import, add to test classes

    * Remove incorrect default from keyed test

    * Keyed impl

    * Fix device arg

    * custom inference test

    * formatting

    * Add helpers to define custom inference functions using model methods

    * Trailing whitespace

    * Unit tests

    * Fix incorrect getattr syntax

    * Type typo

    * Fix docstring

    * Fix keyed helper, add basic generate route

    * Modify generate() to be different than forward()

    * formatting

    * Remove extra generate() def

commit a014637106970a0a0e9eb7944aa5caf79fa5fd37
Author: egalpin <[email protected]>
Date:   Tue Nov 15 13:57:54 2022 -0800

    Uses _all to follow alias/datastreams when estimating index size

    Fixes #24117

commit 0d7ca041823bc2b09f76f86fdfd1d0b9508c9c88
Author: Ning Kang <[email protected]>
Date:   Tue Nov 15 13:57:27 2022 -0800

    Minor update

commit e8fc759d756f4a987e41d2b9da56b906a6cd7736
Author: Ning Kang <[email protected]>
Date:   Tue Nov 15 13:52:18 2022 -0800

    Updated README of Interactive Beam

    Removed deprecated cache_dir runner param in favor of the cache_root global option.

commit 08d5f72e5f35d41f3e9fa9fe799caea6bed1b7a7
Author: Anand Inguva <[email protected]>
Date:   Tue Nov 15 16:34:21 2022 -0500

    [Python]Support pipe operator as Union (PEP -604) (#24106)

    Fixes https://github.com/apache/beam/issues/21972

commit 526e7a58b62682582c27173ab21ed8667ddab766
Author: Scott Strong <[email protected]>
Date:   Tue Nov 15 16:26:45 2022 -0500

    Using Teardown context instead of deprecated finalize (#24180)

    * Using Teardown context instead of deprecated finalize

    * making function public

    Co-authored-by: Scott Strong <[email protected]>

commit fb4d1d4dea7b26ed538a9f6aca0ed41e8c300e37
Author: Danny McCormick <[email protected]>
Date:   Tue Nov 15 16:25:22 2022 -0500

    Fix broken json for notebook (#24183)

commit f98db2008a97f4546d036ddf0dddfee8c87eb58a
Author: Robert Burke <[email protected]>
Date:   Tue Nov 15 12:49:23 2022 -0800

    Update automation to use Go 1.19 (#24175)

    Co-authored-by: lostluck <[email protected]>

commit e5f58504eef1fdeebe0402cda8a2df259169c704
Author: Brian Hulette <[email protected]>
Date:   Tue Nov 15 12:25:13 2022 -0800

    Add error reporting for BatchConverter match failure (#24022)

    * add error reporting for BatchConverters

    * Test pytorch

    * Finish up torch tests

    * yapf

    * yapf

    * Remove else

commit 3037747f66f0d71d65b6c65745b4f8942c22f05a
Author: Danny McCormick <[email protected]>
Date:   Tue Nov 15 14:13:04 2022 -0500

    Fix broken notebook (#24179)

commit b2b1c739ce37690923891934ee317f799db937a2
Author: MakarkinSAkvelon <[email protected]>
Date:   Tue Nov 15 21:53:06 2022 +0500

    [Playground] Move Playground in GKE and Infrastructure change (#23928)

    * changes to updated master branch

    * Change workflow

    * ingress changes

    * Certificate was added

    * Updates for cloud build backend

    * Update main.tf

    * Create main.tf

    * Create variables.tf

    * Update variables.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Create output.tf

    * Update output.tf

    * Update output.tf

    * Update main.tf

    * Update build.gradle.kts

    * Update output.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update output.tf

    * Update main.tf

    * Update main.tf

    * Update output.tf

    * Create variables.tf

    * Update main.tf

    * Update main.tf

    * Delete playground/terraform/infrastructure/cluddns directory

    * Update main.tf

    * Update output.tf

    * Update output.tf

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update README.md

    * Update README.md

    * helm folder name was changed

    * Update README.md

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Updates to readme

    * Fix DNS name

    * HelmChart was changed

    * Some workflows were changed

    * Remove unused file

    * playground-examples return

    * add license information

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * remove "stg" folder

    * Update README.md

    * Remove unused files

    * DNS Removed

    * var name changed

    * remove DNSName from var file

    * 1

    * Clear terraform

    * remove unused records

    * gradle check

    * grade last change

    * issue fix

    * fix

    * 1

    * run

    * test

    * Index creation for Gradle

    * Add IndexCreation in gradle

    * Update README.md

    * Update README.md

    * Fix names for Frontend

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Redis fix

    * services fix

    * Update variables.tf

    * change order in gradle

    * Fix Config.g.dart file issue

    * Update README.md

    * Playground workflow update

    Co-authored-by: Sergey Makarkin <[email protected]>
    Co-authored-by: Sergey Makarkin <[email protected]>
    Co-authored-by: ruslan-ikhsan <[email protected]>
    Co-authored-by: Alex Kosolapov <[email protected]>

commit 85df5f2eb2f299e28b36be0cce7b9c19d62124da
Author: Yi Hu <[email protected]>
Date:   Tue Nov 15 11:38:13 2022 -0500

    Eliminate CalciteUtil.CharType logical type (#24013)

    * Eliminate CalciteUtils.CharType logical type

    * Replace CalciteUtils.CharType to String
      Note that CalciteUtils still omits the precision of BINARY/VARBINARY/CHAR/VARCHAR
      as what it originally did. Support of the precision of these calcite types involves
      make use of making use of the overload method RelDataTypeFactory.createSqlType(var1, var2).

    * Replace every reference of CalciteUtil.CharType to generic
      PassThroughLogicalType check

    * Add TODO to Support sql types with arguments

    * Use VariableString in LogicalTypeTestCase

commit f349f41010c5b238ff6020f7de718f938eef3c5e
Author: alexeyinkin <[email protected]>
Date:   Tue Nov 15 20:04:01 2022 +0400

    Configure flutter_code_editor options with Hugo shortcode (#23926) (#24031)

    * Configure flutter_code_editor options with Hugo shortcode (#23926)

    * Minor fixes (#23926)

    * Refactor after review (#23926)

commit 0f4ca6363b3ce0e5de3ad36517bb406aa6391a18
Author: Rebecca Szper <[email protected]>
Date:   Tue Nov 15 06:10:13 2022 -0800

    Editorial review of the ML notebooks. (#24125)

    * Editorial review of the ML notebooks.

    * Editorial review of the ML notebooks.

    * Editorial review of the ML notebooks.

    * Update examples/notebooks/beam-ml/custom_remote_inference.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Updating based on feedback

    * Update examples/notebooks/beam-ml/run_inference_sklearn.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Updating based on feedback

    Co-authored-by: Danny McCormick <[email protected]>

commit 5bd34ede026253326ebff1a7e4f9edb5f71b4a2c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Nov 15 07:17:28 2022 -0500

    Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#24131)

    Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.3.2 to 1.11.39.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.3.2...feature/s3/manager/v1.11.39)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 2ee809fa0ca7689dd0279e186ebc02d9569a8429
Merge: e3b9bdb2e6 563c66d6fd
Author: Alexey Romanenko <[email protected]>
Date:   Tue Nov 15 11:01:14 2022 +0100

    Merge pull request #23065: [Website] Update copy icon styles

commit e3b9bdb2e607d85a4017ba7839000e92a0ad83c4
Author: Moritz Mack <[email protected]>
Date:   Tue Nov 15 10:40:50 2022 +0100

    [Dockerized Jenkins] Fix build of dockerized jenkins (fixes #24053) (#24054)

commit faaac2ab6e010374cb2be0e95a5dd345836a2a2c
Author: Moritz Mack <[email protected]>
Date:   Tue Nov 15 10:38:59 2022 +0100

    [Dockerized Jenkins] Update README how to use local repo (#24055)

commit 689e70b5131620540faf52e2f1e2dca7a36f269d
Author: Damon <[email protected]>
Date:   Mon Nov 14 17:34:29 2022 -0800

    Implement embedded WebAssembly example (#24081)

commit e1bf6c42950e8013f35e35fb9fee8017e01e5010
Merge: eddac84126 10337d2868
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 14 15:22:14 2022 -0800

    Merge pull request #24160 Rename the test_splits flag to direct_test_splits.

commit eddac841261228a2c63fa9b225c520ae0f853806
Author: Pablo <[email protected]>
Date:   Mon Nov 14 15:05:05 2022 -0800

    More dataset templates to clean up (#24162)

commit 2adb68bd12743566cc89b596bf204d7c807eb62d
Author: Pablo <[email protected]>
Date:   Mon Nov 14 13:28:13 2022 -0800

    Adding a quickstart to README for the TS SDK (#23509)

    * More of a quickstart for the TS SDK

    * Update sdks/typescript/README.md

    Co-authored-by: Danny McCormick <[email protected]>

    * Update sdks/typescript/README.md

    Co-authored-by: Danny McCormick <[email protected]>

    Co-authored-by: Danny McCormick <[email protected]>

commit 10337d28685ad5712e2ad8608977ec5c5e0e6b6b
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 14 12:46:32 2022 -0800

    Rename the test_splits flag to direct_test_splits.

    This avoids possible flag conflicts.

commit 48c70cc30742b45b17a1d18ece2f0d079bee3915
Author: arne-alex <[email protected]>
Date:   Mon Nov 14 21:33:02 2022 +0100

    Merge pull request #23333: Track time on Cloud Dataflow streaming data reads and export via heartbeats

commit 9c83de646ab52bd0b05e3346190dd55cd68b2a8b
Author: Johanna Öjeling <[email protected]>
Date:   Mon Nov 14 21:19:44 2022 +0100

    Add more tests for S3 filesystem (#24138)

commit 9e9c6d797ba52b460f83131431c8e53aebbbc9ac
Merge: d5d76b9745 c600444e1d
Author: Ning Kang <[email protected]>
Date:   Mon Nov 14 12:06:15 2022 -0800

    Merge pull request #24029 from apache/dependabot/npm_and_yarn/sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/loader-utils-1.4.1

    Bump loader-utils from 1.4.0 to 1.4.1 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel

commit d5d76b974592d45de368ab641647ca5cc4ec12ec
Author: Yi Hu <[email protected]>
Date:   Mon Nov 14 15:03:28 2022 -0500

    Support SqlTypes Date and Timestamp (MicrosInstant) in AvroUtils (#23969)

    * Support SqlTypes Date and Timestamp (MicrosInstant) in AvroUtils

    * Add TODO about java.time migration

commit 330cc2010c9f4a2d4e30318bf50a4109ec1cd392
Author: Pablo <[email protected]>
Date:   Mon Nov 14 12:02:10 2022 -0800

    Cleanup stale BQ datasets (#24158)

    * Cleanup stale BQ datasets

    * addressing comments

commit 4a044999b8ed4bcd41f816f3a23ccb5da00c4c38
Merge: e563b9dd2f 5bd75c25de
Author: Heejong Lee <[email protected]>
Date:   Mon Nov 14 11:16:00 2022 -0800

    Merge pull request #24076 from chamikaramj/multilang_java_updates

    Updates Multi-lang Java quickstart

commit e563b9dd2f3aa0484e6cdc08869991b5e438023e
Author: Evgeny Antyshev <[email protected]>
Date:   Mon Nov 14 20:56:35 2022 +0300

    [Tour Of Beam] verify that unit exists when saving progress (#24118)

    * AIO

    * Update learning/tour-of-beam/backend/integration_tests/auth_test.go

    Co-authored-by: Danny McCormick <[email protected]>

    * nit

    Co-authored-by: Danny McCormick <[email protected]>

commit 774923e0dd089de870bfa5c77063ae2b28f79347
Merge: 71785de528 1ad0cbc445
Author: Kenn Knowles <[email protected]>
Date:   Mon Nov 14 09:52:26 2022 -0800

    Merge pull request #24141: Fix checkArgument format in GcsPath

commit 71785de52864313c2e3b14fe72a2a63281343617
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Nov 14 11:54:37 2022 -0500

    Bump github.com/aws/aws-sdk-go-v2/config from 1.17.10 to 1.18.0 in /sdks (#24151)

    Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.17.10 to 1.18.0.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.17.10...config/v1.18.0)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/config
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 50d591d6cb3e799bee4e29dfc593c693a86e6276
Author: Bruno Volpato <[email protected]>
Date:   Mon Nov 14 11:50:01 2022 -0500

    Change DataflowBatchWorkerHarness doWork error level to INFO (#24135)

commit 5a72696bfda09fdb905ba8e58b636f8494ef955f
Merge: ee0a5836d6 0633fe9634
Author: Kenn Knowles <[email protected]>
Date:   Mon Nov 14 08:12:12 2022 -0800

    Merge pull request #24149: Remove extraneous jetbrains annotation

commit ee0a5836d69b776834eb3bd9b2bd02eb5252c333
Merge: d001a69e1a 137799672e
Author: Kenn Knowles <[email protected]>
Date:   Mon Nov 14 08:11:00 2022 -0800

    Merge pull request #24132: Fix checkArgument format string in AvroIO

commit d001a69e1a58701d6ed4fcb5e3fb7a0921301dad
Author: Yi Hu <[email protected]>
Date:   Mon Nov 14 10:56:54 2022 -0500

    Test Dataproc 2.1 with Flink load tests (#24129)

    * Test Dataproc 2.1 with Flink load tests

    * Minor fix flink_cluster script

commit caabd9be52887ad70c8a4269395c893811ac6a84
Author: Israel Herraiz <[email protected]>
Date:   Mon Nov 14 16:03:39 2022 +0100

    Make MonotonicWatermarkEstimator work like its Java SDK equivalent (#24146)

    * Make MonotonicWatermarkEstimator work like its Java SDK equivalent

    The current implementation of MonotonicWatermarkEstimator raises an exception
    with late messages, which makes the watermark estimator barely usable in real
    world scenarios.

    This PR fixes #20041 by making this watermark estimator work like its Java SDK
    equivalent (`WatermarkEstimators.MonotonicallyIncreasing`).

    * Update unit tests too

    * Make linter happy

commit 451f6b3e7f58d0a3782ad942c6a1fd9f63932024
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Nov 14 09:48:23 2022 -0500

    Bump golang.org/x/net from 0.1.0 to 0.2.0 in /sdks (#24153)

    Bumps [golang.org/x/net](https://github.com/golang/net) from 0.1.0 to 0.2.0.
    - [Release notes](https://github.com/golang/net/releases)
    - [Commits](https://github.com/golang/net/compare/v0.1.0...v0.2.0)

    ---
    updated-dependencies:
    - dependency-name: golang.org/x/net
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 2bb03d62e2d7dc2d8e39040fc9adebccbde74fde
Merge: 4e39ef2041 623083cd0a
Author: Alexey Romanenko <[email protected]>
Date:   Mon Nov 14 15:01:13 2022 +0100

    Merge pull request #24000: [Website] Change headers size from h4,h3 to h2

commit 563c66d6fd32165da14a07747f2764c17a5d24ea
Author: bulat safiullin <[email protected]>
Date:   Wed Sep 7 18:28:42 2022 +0600

    [Website] update pre tag copy link styles #23064

commit 4e39ef20410ee51c6040317bcd60171e64c5171f
Merge: 223768f782 105ed6fedc
Author: Alexey Romanenko <[email protected]>
Date:   Mon Nov 14 10:55:33 2022 +0100

    Merge pull request #24115: [Website] update go-dependencies.md java-dependencies.md links

commit 223768f782f771f0033b8d0686d86cf4c71fad75
Merge: aa0a35dabf a9da2abee6
Author: Kenn Knowles <[email protected]>
Date:   Sun Nov 13 18:53:13 2022 -0800

    Merge pull request #24136: Fix checkArgument format string in ExecutionStateTracker

commit 0633fe9634fe61df7cbc0ecac205d81124fd504a
Author: Kenneth Knowles <[email protected]>
Date:   Sat Nov 12 15:15:16 2022 -0800

    Remove extraneous jetbrains annotation

commit 83f1bc19b95935e60ca1f4027d4b60c7e738a84a
Author: Kenneth Knowles <[email protected]>
Date:   Sat Nov 12 14:16:09 2022 -0800

    Fix arguments to checkState in BatchViewOverrides

commit 1ad0cbc44594d8405bf4b07a126265238013a02a
Author: Kenneth Knowles <[email protected]>
Date:   Sat Nov 12 13:41:02 2022 -0800

    Fix checkArgument format in GcsPath

commit aa0a35dabf9c2a0d9822faff06d939d9a77a3ab6
Author: Kenn Knowles <[email protected]>
Date:   Fri Nov 11 20:26:30 2022 -0800

    Fix checkArgument format string in TestStream (#24134)

commit a9da2abee6455bc2cf0f18ba5f6cd7bbaeae669f
Author: Kenneth Knowles <[email protected]>
Date:   Fri Nov 11 16:54:27 2022 -0800

    Fix checkArgument format string in ExecutionStateTracker

commit 369e2ba8622d3474c14c39b941b2c618842d1e47
Author: Ryan Thompson <[email protected]>
Date:   Fri Nov 11 19:46:07 2022 -0500

    Add a ValidatesContainer integration test for use_sibling_sdk_workers (#24099)

commit 137799672eb559a7586262e6a8a73d1ab3580e44
Author: Kenneth Knowles <[email protected]>
Date:   Fri Nov 11 15:30:01 2022 -0800

    Fix checkArgument format string in AvroIO

commit 5d2dbf957e4e82fb3980726940df02ac67e563cd
Author: Anand Inguva <[email protected]>
Date:   Fri Nov 11 15:57:28 2022 -0500

    Update staging of Python wheels (#24114)

    Fixes https://github.com/apache/beam/issues/24110

commit c2021bee1eba0322b43c90841397859048296b21
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Nov 11 15:33:14 2022 -0500

    Bump google.golang.org/api from 0.102.0 to 0.103.0 in /sdks (#24049)

    Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.102.0 to 0.103.0.
    - [Release notes](https://github.com/googleapis/google-api-go-client/releases)
    - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.102.0...v0.103.0)

    ---
    updated-dependencies:
    - dependency-name: google.golang.org/api
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 6557c91c79480b9d90573d52d257a11c2b160196
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Nov 11 11:47:12 2022 -0800

    Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#24112)

    Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.29.1 to 1.29.2.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.29.1...service/s3/v1.29.2)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 96f9da1ab652156cd143d57e3aa3d94836338f2b
Author: Yi Hu <[email protected]>
Date:   Fri Nov 11 14:34:22 2022 -0500

    More cleanup containers (#24105)

    * More cleanup containers

    * prebuilt_beam_sdk/beam_python_prebuilt_sdk no longer exists in gcr.io
      Add correct paths to cleanup

    * Bump grace time to 15 days

    * Run job daily as in code comment

    * Set grace period to 30 days

commit 836766ddb83d37463e4b036f156b8d7e15e3864b
Author: Ritesh Ghorse <[email protected]>
Date:   Fri Nov 11 14:30:30 2022 -0500

    upgrade testcontainer dependency (#24123)

commit 9fcd20c3712536f2d4580beead678cdbb6fd4746
Author: Damon <[email protected]>
Date:   Fri Nov 11 11:12:11 2022 -0800

    Implement PubsubRowToMessage transform (#23897)

    * Begin PubsubRowToMessage Impl

    * Complete working draft

    * Unit tests validate user and non-user fields

    * Finish tests on supporting methods

    * Pass checks before finalizing tests

    * WIP

    * fix timestamp

    * finalize tests

    * Finalize code comments

    * Clean up check findings

    * Add InputSchemaFactory

    * Patch code comment typo

commit 3a6fcc1ca4d07a467464ed1214a94b5c9c147295
Author: Evgeny Antyshev <[email protected]>
Date:   Fri Nov 11 19:13:45 2022 +0300

    disable (#24121)

commit 027fb142038b45c443d87af96cac082264c43188
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Nov 11 10:26:23 2022 -0500

    Bump cloud.google.com/go/bigtable from 1.17.0 to 1.18.0 in /sdks (#24113)

    Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.17.0 to 1.18.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.17.0...pubsub/v1.18.0)

    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/bigtable
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit b2d28a64bb0837dd97e8be7f97f31f88d380f110
Author: Danny McCormick <[email protected]>
Date:   Fri Nov 11 08:45:14 2022 -0500

    Add TFX support in pydoc (#23960)

    * Add TFX support in pydoc

    * Wording

commit 105ed6fedcb6ff33d4a43940c342a653e67cb25e
Author: bulat safiullin <[email protected]>
Date:   Thu Nov 10 17:47:54 2022 +0600

    [Website] update go-dependencies.md java-dependencies.md programming-guide.md links #24084

commit 4b96193250a63b27721a8c5e1a3bd6ecff983093
Author: Brian Hulette <[email protected]>
Date:   Thu Nov 10 16:51:06 2022 -0800

    Remove TheNeuralBit from the pool of Python reviewers (#24108)

commit 8f8f089a5f565f179905984cef3522a0577d9219
Author: Sanil Jain <[email protected]>
Date:   Thu Nov 10 16:47:31 2022 -0800

    Wire SamzaPipelineOptions to Exeption listener interface (#24109)

commit b3186ba91f9bb22764d47b78eb2b7ff017080f75
Author: Ahmed Abualsaud <[email protected]>
Date:   Thu Nov 10 18:04:50 2022 -0500

    Support using BigQueryIO Storage Read API with SchemaTransforms (#23827)

    * support schema transform for bq direct read method

    * use vendor Strings import

    * add BigQueryServices argument to config object

    * suppress nullability errors

    * add package-info.java file to providers subdirectory

    * removing accidentally created file

    * added documentation and moved configuration filclass into provider class

    * validate config params before expansion. config has a validaate me method

    * updated URN identifier to follow standards

commit a4a94da1936d8cd2f8d1145fd2e8329fe06a2990
Author: Oleh Borysevych <[email protected]>
Date:   Fri Nov 11 00:28:43 2022 +0200

    fixing linter error (#24104)

commit 156a6099d71a155c3379340a0b98256306e1755e
Author: Danny McCormick <[email protected]>
Date:   Thu Nov 10 17:26:15 2022 -0500

    Add blog post on new ML resources (#24071)

    * Create ml-resources.md

    * Add ensemble notebook image

    * Add image link

    * Fix image link

    * Add ml-landing page image

    * Add image + move around

    * Add paragraph on upcoming changes

    * Remove bad whitespace

commit 38742d40c895f5d38ee6ee5eb328d11b3262307e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Nov 10 16:12:40 2022 -0500

    Bump github.com/aws/aws-sdk-go-v2/config from 1.5.0 to 1.17.10 in /sdks (#24080)

    Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.5.0 to 1.17.10.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.5.0...config/v1.17.10)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/config
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 92cef32915c40d353b51529a87d8f0131b0a3538
Author: BjornPrime <[email protected]>
Date:   Thu Nov 10 15:30:37 2022 -0500

    Num failed inferences (#23830)

    * created test_increment_num_failed_inferences and test_num_failed_inferences_no_failures

    * added assertRaises to test_increment_num_failed_inferences

    * added num_failed_inferences to _MetricsCollector

    * changed error handling and update() implementation

    * updated metric name in tests

    * removed unnecessary else blocking

    * removed unnecessary inference_args from test_increment_failed_batches_counter()

    * changed final test_increment_failed_batches_counter assertion

    * clarified error handling and updated failed_batches_counter initialization

    * decreased examples array length to 1 to ensure repeatability

    * troubleshooting tests

    * trying to get test_increment_failed_batches_counter to fail as expected

    * corrected assertion details

    * simplified assertRaises and added reminder comment to assertEqual counter

    * lint test

    * lint test passed, resetting pre-commit-config.yaml

    * fixed lingering linting issues

    * shortened comment line to comply with linting

    * formatter worked its magic

commit 2341f61d48d838ddaf2e4fb990a49987244c3513
Author: Evgeny Antyshev <[email protected]>
Date:   Thu Nov 10 21:24:55 2022 +0300

    [Tour Of Beam] handle CORS pre-flight requests (#24083)

    * cors

    * README

    * nit

    * nit

    * -headers, -cache

    * allow-headers

    * +1h cache,comment

commit 36b0c8f1a76865e4dc3b1eaec1566c3ef9fd4345
Author: Evgeny Antyshev <[email protected]>
Date:   Thu Nov 10 21:21:21 2022 +0300

    [Playground] update snippet by persistence_key (#24056)

    * proto

    * AIO

    * -frontend

    * testify

    * +license

    * -build

    * index

    * skipKey

    * Update playground/backend/internal/db/datastore/datastore_db.go

    Co-authored-by: Danny McCormick <[email protected]>

    Co-authored-by: Danny McCormick <[email protected]>

commit df553d1a4c6c6cc9b37ada2a6c84c88d8aef38b9
Author: Ahmed Abualsaud <[email protected]>
Date:   Thu Nov 10 13:17:36 2022 -0500

    Add random string at the end of BigQuery query job name to make it resilient to retries (#24041)

    * add random string at the end of query job name

    * use deterministic temp table name

commit e439f4120ef4c25aa36e5b03756dc7391bdbd211
Author: Pablo <[email protected]>
Date:   Thu Nov 10 10:06:26 2022 -0800

    Improving stale container cleanup script (#24040)

    * Improving stale container cleanup script

    * Avoid also latest image

    * Update .test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh

    Co-authored-by: Yi Hu <[email protected]>

    * Update .test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh

    Co-authored-by: Yi Hu <[email protected]>

    Co-authored-by: Yi Hu <[email protected]>

commit 8d585242e158babd8dd2ca5a6d5a49d353be2935
Author: Anand Inguva <[email protected]>
Date:   Thu Nov 10 12:58:41 2022 -0500

    [Python]Set pickle library at the Pipeline creation stage (#24069)

    Fixes https://github.com/apache/beam/issues/21615

commit 08b6a524fecc333a12ce42971733ef64ae7d02f1
Author: Janek Bevendorff <[email protected]>
Date:   Thu Nov 10 18:37:09 2022 +0100

    [BEAM-12792] Install pipline dependencies to temporary venv (#16658)

commit 74f87b0282a23aa01ecb63288013311d207960aa
Merge: 38a85b288a 383ea77bc9
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 10 09:10:37 2022 -0800

    Merge pull request #23985 Support dynamic sharding in the worker.

commit 38a85b288ad10171c444d4092eb969ca5de58af3
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Nov 10 10:39:15 2022 -0500

    Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#24077)

    Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.11.1 to 1.29.1.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.11.1...service/s3/v1.29.1)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c5110bae06fb9db6cace1f7657745fa97e3f58af
Author: Yi Hu <[email protected]>
Date:   Thu Nov 10 10:33:55 2022 -0500

    Fix FhirIO javadoc format broken (#24072)

commit 623083cd0a72d86c7ecba7653e9a8564861abc4d
Author: bulat safiullin <[email protected]>
Date:   Mon Nov 7 16:08:39 2022 +0600

    [Website] change headers size from h4,h3 to h2 #24082

commit 5bd75c25de291e517cc5c5799ae4adaaaaceacb7
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 9 18:04:28 2022 -0800

    Updates Multi-lang Java quickstart

commit 4522f4ce916223afb3c4edd02a7be6018d0ce5a4
Author: Trevor Gevers <[email protected]>
Date:   Wed Nov 9 18:56:18 2022 -0600

    Update datastore_wordcount.py (#23724)

commit 96cc252348ce7229731be3c74bf5848d1b004c08
Author: tvalentyn <[email protected]>
Date:   Wed Nov 9 16:08:00 2022 -0800

    Remove a duplicate label (#24043)

commit 62a6bd1e9317a7658810e9a0eebfe925460e8982
Author: Thiago Nunes <[email protected]>
Date:   Thu Nov 10 08:45:27 2022 +1100

    test: add more tests to throughput estimator (#23915)

    Verifies that getting the throughput when no updates have occurred for
    the size of the window should return 0.

commit e6647c34bb367b3058c42d0201150395c534e7ce
Author: Ritesh Ghorse <[email protected]>
Date:   Wed Nov 9 15:12:59 2022 -0500

    skip output coder field in exp request (#24066)

commit 383ea77bc982416afcbbbe11a4dc1f1424670589
Author: Robert Bradshaw <[email protected]>
Date:   Wed Nov 9 10:49:37 2022 -0800

    Reduce flakiness of time-based split manager test.

    Increase wait time from a tenth to half a second.

commit 539fa9159ffc116b2e79e6de2804dfdd1c1e4722
Author: Danny McCormick <[email protected]>
Date:   Wed Nov 9 11:45:20 2022 -0500

    Convert initialisms to all caps (#24061)

    * Convert initialisms to all caps

    * Fix test refs

commit 4941b9633c6ccf9eb8aad554186aac24ca7b7492
Author: Ryan Thompson <[email protected]>
Date:   Wed Nov 9 11:25:48 2022 -0500

    added comments for tensorflow notebook (#23726)

    * added comments for tensorflow notebook

    * added note to string type

commit 858f5048f578548a63e5e4319a7ab5363311c269
Merge: 95c121b254 1929968891
Author: Alexey Romanenko <[email protected]>
Date:   Wed Nov 9 16:35:19 2022 +0100

    Merge pull request #24057: [Spark Dataset runner] Enable projection pushdown for Spark dataset runner

commit 95c121b2549d3f8fda7668e3759b8991b55d9865
Author: Oleh Borysevych <[email protected]>
Date:   Wed Nov 9 16:03:33 2022 +0200

    Fix dependency mismatch in Playground Java runner  (#24059)

    * fixing existing and potential dependency mismatch

    * extract grpc version

commit 1929968891d629c83249c574c28db6c103ba9271
Author: Moritz Mack <[email protected]>
Date:   Wed Nov 9 13:35:12 2022 +0100

    [Spark Dataset runner] Enable projection pushdown for Spark dataset runner.

commit 63362f5ba60a22e77e54c2df47a844f031036309
Author: Johanna Öjeling <[email protected]>
Date:   Wed Nov 9 07:49:54 2022 +0100

    [Go SDK] S3 implementation of the Beam filesystem (#23992)

    * Implement filesystem for S3

    * Update CHANGES.md

commit 73142bad2472e4516831c24a8dfb05c9acebf791
Author: Kanishk Karanawat <[email protected]>
Date:   Tue Nov 8 20:46:04 2022 -0500

    Handle Avro schema generation for logical data types in BigQueryAvroUtils (#23620): handle avro logical types for TableFieldSchema to Avro schema conversion

    Co-authored-by: Kanishk Karanawat <[email protected]>

commit 8db8fae173056c0d89a36a8a5ce0443b6b9e54fa
Author: Robert Burke <[email protected]>
Date:   Tue Nov 8 15:56:53 2022 -0800

    [Go] Add pipeline resource hints to CHANGES.md (#24036)

commit a84b7dc179fed5dfc1d48dddfb04266de3716e31
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 16:35:25 2022 -0800

    Make mypy happy.

commit b9655e7de1a682d8ec4efcafb4d610f794e1b40e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Nov 8 12:27:38 2022 -0800

    Bump cloud.google.com/go/storage from 1.27.0 to 1.28.0 in /sdks (#24028)

    Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.27.0 to 1.28.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.27.0...spanner/v1.28.0)

    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/storage
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 708e05f0dc6c0a2e62674cd0ecb8a6744ae57d3a
Author: Brian Hulette <[email protected]>
Date:   Tue Nov 8 12:18:23 2022 -0800

    Retroactively announce Batched DoFn support in 2.42.0 Blog (#24011)

    * Retroactively announce Batched DoFn support in 2.42.0

    * Add to blog as well

commit b9ff75a18b40141a8d2340e397fd70382b3d4e8f
Author: Israel Herraiz <[email protected]>
Date:   Tue Nov 8 20:03:43 2022 +0100

    Update my Twitter handle (#23653)

commit bf621ccb88023e0bc09752978c099ac4629086e2
Author: Rebecca Szper <[email protected]>
Date:   Tue Nov 8 10:50:02 2022 -0800

    Editorial review of the ML base API descriptions (#24026)

commit f0a6ff46619ee1fd0639fa3f06e04708f69dee74
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Nov 8 13:04:03 2022 -0500

    Bump cloud.google.com/go/bigtable from 1.16.0 to 1.17.0 in /sdks (#24027)

    Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.16.0 to 1.17.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.16.0...pubsub/v1.17.0)

    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/bigtable
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 7dac3f5ef40b5d24b24d9ce5bec4717284260b85
Author: Lukasz Cwik <[email protected]>
Date:   Tue Nov 8 09:42:29 2022 -0800

    [#21250] Trivial removal of loop over something that always has one element (#24014)

    Multiplexing was put into the PCollectionConsumerRegistry a long time ago and this seems to have been missed during that migration.

commit a6a9b23fd5b74142c35948f10d1840b882817246
Author: Robert Burke <[email protected]>
Date:   Tue Nov 8 09:19:49 2022 -0800

    [Go] Pipeline Resource Hints (#23990)

commit 8cf2a63cc25399109f64b84ddfcd7c74d21e61ce
Author: tvalentyn <[email protected]>
Date:   Tue Nov 8 09:19:16 2022 -0800

    Update release notes. (#23986)

commit c600444e1dfa2f4afaad056bc65fa009a60a32bb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Nov 8 06:22:14 2022 +0000

    Bump loader-utils

    Bumps [loader-utils](https://github.com/webpack/loader-utils) from 1.4.0 to 1.4.1.
    - [Release notes](https://github.com/webpack/loader-utils/releases)
    - [Changelog](https://github.com/webpack/loader-utils/blob/v1.4.1/CHANGELOG.md)
    - [Commits](https://github.com/webpack/loader-utils/compare/v1.4.0...v1.4.1)

    ---
    updated-dependencies:
    - dependency-name: loader-utils
      dependency-type: indirect
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

commit 62d8bdc604dfeccae08d00225bce8c9b9c3ffb28
Merge: d4142f7eab 930c4d2d87
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 15:31:37 2022 -0800

    Merge pull request #23789 Better surfacing of Scala support via Scio.

commit 7f7713de09b3b0e400ed02ffdfcb34a6a8b6e1e4
Merge: 5f587b9339 d4142f7eab
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 15:29:27 2022 -0800

    Merge branch 'master' into javascript-liquid-sharding

commit 5f587b9339b3b19ae5c5b1cf15e3ce218f75d21a
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 15:25:33 2022 -0800

    Clarifying comments.

commit 60fe3cdddf9d3c27d9493b345d1f5f2a9284a76f
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 15:06:26 2022 -0800

    Update style sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py

    Co-authored-by: Danny McCormick <[email protected]>

commit d4142f7eab7d5a3d21a3d3930c448fe403880f93
Author: Robert Burke <[email protected]>
Date:   Mon Nov 7 12:11:21 2022 -0800

    Switch && for || to fix bug in #23889 resolution (#24017)

commit 45100d708d954b25b5a884dc3c7a965f008228c6
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 14:53:24 2022 -0500

    Add files then check cached diff to get untracked files

commit 380d4730d3c52b5c2165bdb84aac11d0feccfe4f
Author: scwhittle <[email protected]>
Date:   Mon Nov 7 19:33:37 2022 +0100

    Enforce splitting invariants by ensuring split state is reset in the same synchronized block as window index increments.  (#23882)

    * Enforce splitting invariants by ensuring split state is reset in the same synchronized block as window index increments. Fixes #23881.

    * Add missing currentElement = null;

    Co-authored-by: Lukasz Cwik <[email protected]>

commit 8b3fd2e0d652045bf94b83638129bb12562ce738
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Nov 7 10:33:26 2022 -0800

    Bump google.golang.org/api from 0.101.0 to 0.102.0 in /sdks (#23957)

    Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.101.0 to 0.102.0.
    - [Release notes](https://github.com/googleapis/google-api-go-client/releases)
    - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.101.0...v0.102.0)

    ---
    updated-dependencies:
    - dependency-name: google.golang.org/api
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 9533fc3757d3041135cbe2f5078dff0855adc1f1
Author: capthiron <[email protected]>
Date:   Mon Nov 7 19:32:18 2022 +0100

    feat: implement bigtable io connector with write capabilities (#23411)

commit c4218e572aa8a8f46b433b60073a59fdd7fdf148
Merge: 23676a9ec1 a5ee669a43
Author: Alexey Romanenko <[email protected]>
Date:   Mon Nov 7 18:24:45 2022 +0100

    Merge pull request #23934: [Website] change case studies page mobile layout

commit 23676a9ec17069b309e8803d1fbfeb8009035828
Author: Oleh Borysevych <[email protected]>
Date:   Mon Nov 7 17:24:59 2022 +0200

    [Tour of Beam] Learning content for "Introduction" module (#23085)

    * learning content for introduction module

    * removing white spaces from md files

    * delete whitespaces

    * delete whitespaces in python

    * delete whitespace #2

    * divide pipeline concepts

    * add pipeline example concepts

    * adding category tag to python examples

    * adding category to java examples

    * adding category to go examples

    * fixed go example

    * fixed go example compilation

    * fixing python duplicate example names

    * add runner concepts

    * fixing java examples

    * add licence for runner unit

    * some minor fixes for unit names

    * fixed unit name

    * resolving CR comments

    * adding complexity to examples

    * adding tags

    * fixed go example compilation

    * fixed python example with duplicate transform

    * change indent python

    * fixing missing pipeline options

    * change arrow symbol

    * delete example prefix

    * minor formatting and readability fixes

    * add example description

    * minor fix

    * minor code review comment

    Co-authored-by: Abzal Tuganbay <[email protected]>

commit 98d2ffd69ab9d5c49196a78f7c25daacb57b6478
Author: Brian Hulette <[email protected]>
Date:   Mon Nov 7 07:03:12 2022 -0800

    Make `documentation/io/connectors/` canonical (#23877)

    * Drop documentation/io/built-in/

    * Update links in Custom I/O pattern

commit c0b41fa045418beb782c0a59e6f5c511da898c53
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 09:29:08 2022 -0500

    Use git diff instead of git diff-index to avoid file timestamp changes being picked up

commit 0476d6498217b7b0e833643c47fef2c4bdaec529
Merge: 1d5fc14031 6ae37b6f82
Author: Alexey Romanenko <[email protected]>
Date:   Mon Nov 7 15:28:59 2022 +0100

    Merge pull request #23962: [CdapIO] Add sparkreceiver:2 module.

commit 1d5fc14031d583e92018ebfbeaff9b4e3b978fdb
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 09:14:24 2022 -0500

    Remove quiet flag on debug

commit c08273b9355566181d2d5c0792130f8d5cc1428c
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 09:12:26 2022 -0500

    Correctly print diff and swallow empty commits for the moment

commit 66fb431deb77db5f7f0fac31e6bf3b2c094c8ac5
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 09:08:53 2022 -0500

    Print diff and scope to state path

commit e0e10b9e5432643b884c381d145e5924cc4ef193
Author: Danny McCormick <[email protected]>
Date:   Sun Nov 6 12:12:22 2022 -0500

    PR Bot - Dont throw error on return code 1

commit bbcb790461c47d22399c681cd42c434b206eb550
Author: Danny McCormick <[email protected]>
Date:   Sun Nov 6 11:59:32 2022 -0500

    Fix pr bot - exec doesn't allow command chaining

commit 88f9a66a9f4cc2fe3259d5ebd83ebb472a29dd5e
Author: Danny McCormick <[email protected]>
Date:   Sun Nov 6 11:46:34 2022 -0500

    Fix diff to stop repeated bot runs

commit ddb4f2d53538fea68383fe43221d4074d05d11b9
Author: bullet03 <[email protected]>
Date:   Fri Nov 4 16:11:10 2022 -0700

    Website add and update logos (#23899)

    * [Website] add and update logos

    * Delete talend.svg:Zone.Identifier

commit 6d42219ff547720996489aa29de0d64246db2541
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 14:30:18 2022 -0700

    Enable more portable-runner requiring tests. (#23970)

    * Enable more portable-runner requiring tests.

    * Run all basic tests on portable runner as well.

    * Mark ULR tests.

commit 1a643d16112a5f3f1418f1be393de51af65d752b
Author: David Cavazos <[email protected]>
Date:   Fri Nov 4 13:45:12 2022 -0700

    Beam starter projects blog post (#23964)

    Co-authored-by: Ahmet Altay <[email protected]>

commit d10b4a28fff667289c146152535275035f2dedf7
Author: Ryan Thompson <[email protected]>
Date:   Fri Nov 4 16:06:49 2022 -0400

    removed trailing whitespace (#23987)

commit 3bd697561aab4a5d2aa49d29143695d6d57a834a
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 11:43:47 2022 -0700

    Add dynamic splitting support to the worker.

commit ed520243f472c7224a1f8c74a0c2e8f81965cbfc
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 11:07:40 2022 -0700

    Add the a Reshuffle operation and use it in Create.

    This allows created sources to be staticly and dynamically split.

commit 9283512544b8de169be9cd7795b42284fdf45b87
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 10:28:51 2022 -0700

    Add the ability to schedule splits on the ULR via a pipeline option.

    This can be used to test arbitrary SDKs for behavioral correctness
    for dynamic splitting.

    These splits are wall-time based, which isn't ideal, but easier than
    setting up and managing a cross-process synchronization channel.

commit ed7b560d7ce7dda0b55a067b9d9a1a9d692a7d7a
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 09:46:19 2022 -0700

    Compute element counts for all PCollections.

commit 1cfdb127983a55c77b3259c4fdd248497b28e250
Author: Jack McCluskey <[email protected]>
Date:   Fri Nov 4 15:43:16 2022 -0400

    Add custom inference fn suport to the sklearn model handlers (#23642)

    * Add custom inference fn suport to the sklearn model handlers

    * Clean up import order

    * Update typing, add numpy unit test

    * Add Pandas unit test

    * Formatting, linting

    * yapf run

    * Remove trailing whitespace

    * import order

    * Change inference_fn to keyword-only arg

commit 8617b86fd792ac555233dd79c8ede34c2ce67ecf
Merge: aa178f8252 2496a0d1e0
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 11:42:21 2022 -0700

    Merge pull request #23978 Add basic counter support to the typescript SDK.

commit 2496a0d1e0fda74e8fc0f24eee011362426369a2
Merge: 64abcbe4a3 aa178f8252
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 11:15:47 2022 -0700

    Merge branch 'master' into javascript-metrics

commit aa178f825223cbb0a6208fad0e02f98292eeded1
Merge: 97628802fe 1807c307aa
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 09:34:11 2022 -0700

    Merge pull request #23976 Correctly capture log levels and attach stage information.

commit 64abcbe4a3a15160aba3f3163ff8cbf5186bbfdb
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 09:27:22 2022 -0700

    Only report counters that were actually used.

commit 1807c307aa1e493acb2d8ba542baffefcb96e7e2
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 08:52:21 2022 -0700

    Remove obsolete TODO.

commit 97628802fe4cf6640a3b6854e27b410dd5731379
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Nov 4 11:47:07 2022 -0400

    Bump cloud.google.com/go/datastore from 1.8.0 to 1.9.0 in /sdks (#23916)

    Bumps [cloud.google.com/go/datastore](https://github.com/googleapis/google-cloud-go) from 1.8.0 to 1.9.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-cloud-go/compare/asset/v1.8.0...asset/v1.9.0)

    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/datastore
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 058c56af735e10184d1501cc08cd16af4d21e64a
Author: Ayush Sharma <[email protected]>
Date:   Fri Nov 4 20:58:11 2022 +0530

    [Task]: PR Bot will push commits only if they are non-empty (#23937)

    * fix pr-bot

    * fix the git pr bot

commit 7dba78d611089a1826c5655ed7ffcdf3f6007eae
Author: Danny McCormick <[email protected]>
Date:   Fri Nov 4 09:35:16 2022 -0400

    Immediately truncate full restriction on drain of periodic impulse (#23765)

    * Immediately truncate full restriction on drain of periodic impulse

    * Whitespace format

commit 16852518861514a56ad1b57113e3815cfcc8533a
Author: Anand Inguva <[email protected]>
Date:   Fri Nov 4 08:39:34 2022 -0400

    TFX image classification example (#23456)

    * TFX image classification example

    * TFX image classification with serving model

    * Add TF Model Wrapper

    * Clean up code

    * Refactoring

    * Add a test for tfx_bsl RunInference

    * fixup lint

    * Refactoring

    * Fixup lint

    * Add brief summary for the example

    * Apply suggestions from code review

    Co-authored-by: Andy Ye <[email protected]>

    * Refactoring code and add comments

    * Update help description

    * reorder imports

    * Reorder imports again

    * Add docstring

    * Refactoring

    * Add pillow to tfx pipeline requirements

    * Move inferencePostCommitIT to Python 3.9 suite

    * Uncomment other postcommit suites

    Co-authored-by: Andy Ye <[email protected]>

commit 30b26171b4038c1c976adbc3be386e118b35153b
Author: Danny McCormick <[email protected]>
Date:   Fri Nov 4 08:22:17 2022 -0400

    Update watermark during periodic sequence/impulse (#23507)

    * Update watermark during periodic sequence/impulse

    * Remove extraneous import

    * Formatting

    * Linting

    * Only run on dataflow for guaranteed watermark support

    * More permissive test to avoid timing issues

    * Test pipeline options

    * Fix test

    * Formatting

    * Formatting

    * Apply feedback - cleanup/naming/flink

    * Format

    * Unused import

commit 1563b9721b4bee5a22d384b282762726247e0703
Author: Brian Hulette <[email protected]>
Date:   Fri Nov 4 05:09:39 2022 -0700

    Improve Iterator error message (#23972)

commit e3987b5435d39c5fb058d4085eb5e210f8756224
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 17:39:49 2022 -0700

    old prettier change

commit 89c8dd5ca23591be6c2778045df32c7d08d9f99e
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 17:12:26 2022 -0700

    Add distribution metric type.

commit e4e606a8f2652c2fa761a95462f230385b6bcb47
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 17:09:03 2022 -0700

    Support metrics over the portability API.

commit e04f3d6fdb39528264cb2a14e60e565991121f1c
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 16:52:51 2022 -0700

    Add basic counter setting and getting to the typescript SDK.

commit 1daa9c373ac7fbdb696120eeaa2ca9e85a679a1f
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 16:01:59 2022 -0700

    Cleanup worker logging.

    Attempt to correctly capture and propagate log levels
    and attaches stage information to them.

    Also remove some excessively verbose logs.

commit d3a173cf16355bfe23a9d918afa12cd61d290b59
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 15:11:40 2022 -0700

    Move logging to its own module.

commit 6ae37b6f82ae3cd77cc82aff4566d8218a8acc57
Author: vitaly.terentyev <[email protected]>
Date:   Thu Nov 3 19:44:54 2022 +0400

    Fix sparkreceiver dependencies

commit 8223375d2f9606a29d6498a73c1914a61f3277c4
Author: vitaly.terentyev <[email protected]>
Date:   Thu Nov 3 19:34:04 2022 +0400

    Add sparkreceiver:2 module.

commit a5ee669a439dc3d13dfc4926c1c5ae2351bb9135
Author: bulat safiullin <[email protected]>
Date:   Wed Nov 2 13:53:08 2022 +0600

    [Website] change case-study-card width on mobile

commit 930c4d2d87d462b2457369fd06066203a6628a46
Author: Robert Bradshaw <[email protected]>
Date:   Fri Oct 21 12:22:13 2022 -0700

    Better surfacing of Scala support via Scio.

draft working code

test commit

Revert "Squashed commit of the following:"

This reverts commit 8fc833a5e7abf4cbfb63e56d450631bbfc1985ff.

auth files
damccorm pushed a commit that referenced this pull request Jan 13, 2023
@nausharipov
[ToB] [Frontend] Authentication, unit progress, complete unit, SDK se…
a5efba6 
…lection (#24457)

* Content tree navigation (#23593)

Unit content navigation (#23593)

Update URL on node click (#23593)

Active unit color (#23593)

removeListener in unit (#23593)

First unit is opened on group title click (#23593)

WIP by Alexey Inkin (#23593)

selectedUnitColor (#23593)

Unit borderRadius (#23593)

RegExp todo (#23593)

added referenced collection package to remove warning (#23593)

small refinement (#23593)

expand on group tap, padding, openNode (#23593)

group expansion bug fix (#23593)

selected & unselected progress indicators (#23593)

* AnimatedBuilders instead of StatefulWidgets in unit & group (#23593)

* fixed _getNodeAncestors (#23593)

* get sdkId (#23593)

* addressing comments (#23593)

* sdkId getter & StatelessExpansionTile (#23593)

* expand & collapse group (#23593)

* StatelessExpansionTile (#23593)

* license (#23593)

* ValueChanged and ValueKey in StatelessExpansionTile (#23593)

* optional SDK selector in tour scaffold

* AppNotifier

* moved AppNotifier into _initializeState

* StorageKeys

* remove listener

* auth, complete unit, user progress

AuthNotifier draft (#23692)

Comments (#23692)

Comments (#23692)(1)

sign in with google works (#23692)

new configs (#23692)

get user progress draft (#23692)

comment fixes (#23692)

sign in in IntroTextBody (#23692)

reverted config (#23692)

comment fixes (#23692)

WIP before rebase (merge) (#23692)

Squashed commit of the following:

commit bff4919ff00ec3b5d7186efde41c884dfc4c8344
Merge: 79ba69483a ce8d618c77
Author: Alexey Romanenko <[email protected]>
Date:   Thu Nov 17 10:34:02 2022 +0100

    Merge pull request #24186: Uses _all to follow alias/datastreams when estimating index size

commit 79ba69483a84ea0278d0b0ddb141200739607c77
Merge: 245fea9040 b7e860a762
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 20:47:40 2022 -0800

    Merge pull request #24218: Update Python wheel format for RC validation

commit 245fea904014cd58d4148807463dbaa40000774c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Nov 16 18:12:33 2022 -0800

    Bump loader-utils from 1.4.1 to 1.4.2 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel (#24191)

    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit e1de8e78deeb5d17617fda6591429eaaf8abb8a2
Author: Yi Hu <[email protected]>
Date:   Wed Nov 16 20:48:06 2022 -0500

    Fix PythonLint (#24219)

commit b7e860a7621771c300dcec625655f87e62591323
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 17:28:31 2022 -0800

    updates

commit c2feb09ea49dd815b69c65e531ce34128756d988
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 17:06:08 2022 -0800

    updates

commit ce8d618c77d23e20a1ddb128bb8183048597d096
Author: egalpin <[email protected]>
Date:   Wed Nov 16 16:43:57 2022 -0800

    Adds test for following aliases when estimating index size

commit 959719d01c627328c0ca2849d2b7e2c9b322d4d1
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 15:16:06 2022 -0800

    Temporary update Python RC validation job

commit b952b41788acc20edbe5b75b2196f30dbf8fdeb0
Author: Yi Hu <[email protected]>
Date:   Wed Nov 16 14:18:12 2022 -0500

    Python TextIO Performance Test (#23951)

    * Python TextIO Performance Test

    * Add filebasedio_perf_test module for unified test framework for
      Python file-based IOs

    * Fix MetricsReader publishes metrics duplicately if more than one
      load test declared. This is because MetricsReader.publishers was
      static class variable

    * Fix pylint

    * Distribute Python performance tests random time at a day instead of all at 3PM

    * Add information about length conversion

commit 017f2cbde124af40a43be99ec88289fcf63c1c95
Merge: fef8acdbc0 88dba4f494
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 16 10:39:52 2022 -0800

    Merge pull request #24187: Add a reference to Java RunInference example

commit fef8acdbc0ecbcc85b49144adaf8830e3bc6b2de
Merge: 6e9187e67e ead245539d
Author: Ahmet Altay <[email protected]>
Date:   Wed Nov 16 10:24:53 2022 -0800

    Merge pull request #24199 from Laksh47/issue#24196

    refs: issue-24196, fix broken hyperlink

commit 6e9187e67e1bd8f73997f437f0ed4c29880ed73b
Author: Darkhan Nausharipov <[email protected]>
Date:   Wed Nov 16 22:33:50 2022 +0600

    [Tour of Beam] [Frontend] Content tree URLs (#23776)

    * Content tree navigation (#23593)

    Unit content navigation (#23593)

    Update URL on node click (#23593)

    Active unit color (#23593)

    removeListener in unit (#23593)

    First unit is opened on group title click (#23593)

    WIP by Alexey Inkin (#23593)

    selectedUnitColor (#23593)

    Unit borderRadius (#23593)

    RegExp todo (#23593)

    added referenced collection package to remove warning (#23593)

    small refinement (#23593)

    expand on group tap, padding, openNode (#23593)

    group expansion bug fix (#23593)

    selected & unselected progress indicators (#23593)

    * AnimatedBuilders instead of StatefulWidgets in unit & group (#23593)

    * fixed _getNodeAncestors (#23593)

    * get sdkId (#23593)

    * addressing comments (#23593)

    * sdkId getter & StatelessExpansionTile (#23593)

    * expand & collapse group (#23593)

    * StatelessExpansionTile (#23593)

    * license (#23593)

    * ValueChanged and ValueKey in StatelessExpansionTile (#23593)

    Co-authored-by: darkhan.nausharipov <[email protected]>
    Co-authored-by: Alexey Inkin <[email protected]>

commit b33fac2aa533d77cfa47f88466c8cd6bd3f3e864
Author: Bruno Volpato <[email protected]>
Date:   Wed Nov 16 10:51:11 2022 -0500

    Use only ValueProviders in SpannerConfig (#24156)

commit 5f013ab6567ec75b460b2081d7f89d332320caff
Author: Robert Burke <[email protected]>
Date:   Wed Nov 16 07:23:10 2022 -0800

    revert upgrade to go 1.19 for action unit tests (#24189)

commit 9337f4dbecc929886f8559949a082a649fd9d1bb
Author: Yi Hu <[email protected]>
Date:   Wed Nov 16 10:18:42 2022 -0500

    Fix Python PostCommit Example CustomPTransformIT on portable (#24159)

    * Fix Python PostCommit Examples on portable

    * Fix custom_ptransform pipeline options gets modified

    * Specify flinkConfDir

commit ead245539d01dec0f3e08699c1e1cc6777a5ef0e
Author: Laksh <[email protected]>
Date:   Wed Nov 16 09:32:46 2022 -0500

    refs: issue-24196, fix broken hyperlink

commit e83a996d4374d467d95bcfad7166905622ec615c
Merge: 2fc56ec663 ffdee0b6ed
Author: Jan Lukavský <[email protected]>
Date:   Wed Nov 16 15:15:31 2022 +0100

    Merge pull request #24192: Re-use serializable pipeline options when already available.

commit ffdee0b6edb8638c78a65ec85c727ea5dde1cb2f
Author: Jozef Vilcek <[email protected]>
Date:   Mon Nov 14 16:48:18 2022 +0100

    Re-use serializable pipeline options when already available (#24192)

commit 88dba4f494829b2b3530b767fb8c5252e0d2ba44
Author: Chamikara Jayalath <[email protected]>
Date:   Tue Nov 15 16:21:22 2022 -0800

    Add a reference to Java RunInference example

commit 2fc56ec663e335cfcf37dc57d471f79b601414f4
Merge: f763186987 83f1bc19b9
Author: Kenn Knowles <[email protected]>
Date:   Tue Nov 15 16:16:47 2022 -0800

    Merge pull request #24142: Fix arguments to checkState in BatchViewOverrides

commit f763186987c00ba1d26efdc35406436a1fa69a9a
Merge: c2bc2135e9 0d7ca04182
Author: Ning Kang <[email protected]>
Date:   Tue Nov 15 15:25:20 2022 -0800

    Addresses #24161 Updated README of Interactive Beam

commit c2bc2135e9bce715990a5d5551e2bc2dc0311da4
Author: Doug Judd <[email protected]>
Date:   Tue Nov 15 14:48:26 2022 -0800

    Strip FGAC database role from changestreams metadata requests (#24177)

    Co-authored-by: Doug Judd <[email protected]>

commit af637974f96ad1b5110d7dea3f9a26c68e19a51b
Author: Jack McCluskey <[email protected]>
Date:   Tue Nov 15 17:16:43 2022 -0500

    Add custom inference function support to the PyTorch model handler (#24062)

    * Initial type def and function signature

    * [Draft] Add custom inference fn support to Pytorch Model Handler

    * Formatting

    * Split out default

    * Remove Keyed version for testing

    * Move device optimization

    * Make default available for import, add to test classes

    * Remove incorrect default from keyed test

    * Keyed impl

    * Fix device arg

    * custom inference test

    * formatting

    * Add helpers to define custom inference functions using model methods

    * Trailing whitespace

    * Unit tests

    * Fix incorrect getattr syntax

    * Type typo

    * Fix docstring

    * Fix keyed helper, add basic generate route

    * Modify generate() to be different than forward()

    * formatting

    * Remove extra generate() def

commit a014637106970a0a0e9eb7944aa5caf79fa5fd37
Author: egalpin <[email protected]>
Date:   Tue Nov 15 13:57:54 2022 -0800

    Uses _all to follow alias/datastreams when estimating index size

    Fixes #24117

commit 0d7ca041823bc2b09f76f86fdfd1d0b9508c9c88
Author: Ning Kang <[email protected]>
Date:   Tue Nov 15 13:57:27 2022 -0800

    Minor update

commit e8fc759d756f4a987e41d2b9da56b906a6cd7736
Author: Ning Kang <[email protected]>
Date:   Tue Nov 15 13:52:18 2022 -0800

    Updated README of Interactive Beam

    Removed deprecated cache_dir runner param in favor of the cache_root global option.

commit 08d5f72e5f35d41f3e9fa9fe799caea6bed1b7a7
Author: Anand Inguva <[email protected]>
Date:   Tue Nov 15 16:34:21 2022 -0500

    [Python]Support pipe operator as Union (PEP -604) (#24106)

    Fixes https://github.com/apache/beam/issues/21972

commit 526e7a58b62682582c27173ab21ed8667ddab766
Author: Scott Strong <[email protected]>
Date:   Tue Nov 15 16:26:45 2022 -0500

    Using Teardown context instead of deprecated finalize (#24180)

    * Using Teardown context instead of deprecated finalize

    * making function public

    Co-authored-by: Scott Strong <[email protected]>

commit fb4d1d4dea7b26ed538a9f6aca0ed41e8c300e37
Author: Danny McCormick <[email protected]>
Date:   Tue Nov 15 16:25:22 2022 -0500

    Fix broken json for notebook (#24183)

commit f98db2008a97f4546d036ddf0dddfee8c87eb58a
Author: Robert Burke <[email protected]>
Date:   Tue Nov 15 12:49:23 2022 -0800

    Update automation to use Go 1.19 (#24175)

    Co-authored-by: lostluck <[email protected]>

commit e5f58504eef1fdeebe0402cda8a2df259169c704
Author: Brian Hulette <[email protected]>
Date:   Tue Nov 15 12:25:13 2022 -0800

    Add error reporting for BatchConverter match failure (#24022)

    * add error reporting for BatchConverters

    * Test pytorch

    * Finish up torch tests

    * yapf

    * yapf

    * Remove else

commit 3037747f66f0d71d65b6c65745b4f8942c22f05a
Author: Danny McCormick <[email protected]>
Date:   Tue Nov 15 14:13:04 2022 -0500

    Fix broken notebook (#24179)

commit b2b1c739ce37690923891934ee317f799db937a2
Author: MakarkinSAkvelon <[email protected]>
Date:   Tue Nov 15 21:53:06 2022 +0500

    [Playground] Move Playground in GKE and Infrastructure change (#23928)

    * changes to updated master branch

    * Change workflow

    * ingress changes

    * Certificate was added

    * Updates for cloud build backend

    * Update main.tf

    * Create main.tf

    * Create variables.tf

    * Update variables.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Create output.tf

    * Update output.tf

    * Update output.tf

    * Update main.tf

    * Update build.gradle.kts

    * Update output.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update variables.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update main.tf

    * Update output.tf

    * Update main.tf

    * Update main.tf

    * Update output.tf

    * Create variables.tf

    * Update main.tf

    * Update main.tf

    * Delete playground/terraform/infrastructure/cluddns directory

    * Update main.tf

    * Update output.tf

    * Update output.tf

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update README.md

    * Update README.md

    * helm folder name was changed

    * Update README.md

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Update build.gradle.kts

    * Updates to readme

    * Fix DNS name

    * HelmChart was changed

    * Some workflows were changed

    * Remove unused file

    * playground-examples return

    * add license information

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * remove "stg" folder

    * Update README.md

    * Remove unused files

    * DNS Removed

    * var name changed

    * remove DNSName from var file

    * 1

    * Clear terraform

    * remove unused records

    * gradle check

    * grade last change

    * issue fix

    * fix

    * 1

    * run

    * test

    * Index creation for Gradle

    * Add IndexCreation in gradle

    * Update README.md

    * Update README.md

    * Fix names for Frontend

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Update README.md

    * Redis fix

    * services fix

    * Update variables.tf

    * change order in gradle

    * Fix Config.g.dart file issue

    * Update README.md

    * Playground workflow update

    Co-authored-by: Sergey Makarkin <[email protected]>
    Co-authored-by: Sergey Makarkin <[email protected]>
    Co-authored-by: ruslan-ikhsan <[email protected]>
    Co-authored-by: Alex Kosolapov <[email protected]>

commit 85df5f2eb2f299e28b36be0cce7b9c19d62124da
Author: Yi Hu <[email protected]>
Date:   Tue Nov 15 11:38:13 2022 -0500

    Eliminate CalciteUtil.CharType logical type (#24013)

    * Eliminate CalciteUtils.CharType logical type

    * Replace CalciteUtils.CharType to String
      Note that CalciteUtils still omits the precision of BINARY/VARBINARY/CHAR/VARCHAR
      as what it originally did. Support of the precision of these calcite types involves
      make use of making use of the overload method RelDataTypeFactory.createSqlType(var1, var2).

    * Replace every reference of CalciteUtil.CharType to generic
      PassThroughLogicalType check

    * Add TODO to Support sql types with arguments

    * Use VariableString in LogicalTypeTestCase

commit f349f41010c5b238ff6020f7de718f938eef3c5e
Author: alexeyinkin <[email protected]>
Date:   Tue Nov 15 20:04:01 2022 +0400

    Configure flutter_code_editor options with Hugo shortcode (#23926) (#24031)

    * Configure flutter_code_editor options with Hugo shortcode (#23926)

    * Minor fixes (#23926)

    * Refactor after review (#23926)

commit 0f4ca6363b3ce0e5de3ad36517bb406aa6391a18
Author: Rebecca Szper <[email protected]>
Date:   Tue Nov 15 06:10:13 2022 -0800

    Editorial review of the ML notebooks. (#24125)

    * Editorial review of the ML notebooks.

    * Editorial review of the ML notebooks.

    * Editorial review of the ML notebooks.

    * Update examples/notebooks/beam-ml/custom_remote_inference.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Updating based on feedback

    * Update examples/notebooks/beam-ml/run_inference_sklearn.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

    Co-authored-by: Danny McCormick <[email protected]>

    * Updating based on feedback

    Co-authored-by: Danny McCormick <[email protected]>

commit 5bd34ede026253326ebff1a7e4f9edb5f71b4a2c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Nov 15 07:17:28 2022 -0500

    Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#24131)

    Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.3.2 to 1.11.39.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.3.2...feature/s3/manager/v1.11.39)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 2ee809fa0ca7689dd0279e186ebc02d9569a8429
Merge: e3b9bdb2e6 563c66d6fd
Author: Alexey Romanenko <[email protected]>
Date:   Tue Nov 15 11:01:14 2022 +0100

    Merge pull request #23065: [Website] Update copy icon styles

commit e3b9bdb2e607d85a4017ba7839000e92a0ad83c4
Author: Moritz Mack <[email protected]>
Date:   Tue Nov 15 10:40:50 2022 +0100

    [Dockerized Jenkins] Fix build of dockerized jenkins (fixes #24053) (#24054)

commit faaac2ab6e010374cb2be0e95a5dd345836a2a2c
Author: Moritz Mack <[email protected]>
Date:   Tue Nov 15 10:38:59 2022 +0100

    [Dockerized Jenkins] Update README how to use local repo (#24055)

commit 689e70b5131620540faf52e2f1e2dca7a36f269d
Author: Damon <[email protected]>
Date:   Mon Nov 14 17:34:29 2022 -0800

    Implement embedded WebAssembly example (#24081)

commit e1bf6c42950e8013f35e35fb9fee8017e01e5010
Merge: eddac84126 10337d2868
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 14 15:22:14 2022 -0800

    Merge pull request #24160 Rename the test_splits flag to direct_test_splits.

commit eddac841261228a2c63fa9b225c520ae0f853806
Author: Pablo <[email protected]>
Date:   Mon Nov 14 15:05:05 2022 -0800

    More dataset templates to clean up (#24162)

commit 2adb68bd12743566cc89b596bf204d7c807eb62d
Author: Pablo <[email protected]>
Date:   Mon Nov 14 13:28:13 2022 -0800

    Adding a quickstart to README for the TS SDK (#23509)

    * More of a quickstart for the TS SDK

    * Update sdks/typescript/README.md

    Co-authored-by: Danny McCormick <[email protected]>

    * Update sdks/typescript/README.md

    Co-authored-by: Danny McCormick <[email protected]>

    Co-authored-by: Danny McCormick <[email protected]>

commit 10337d28685ad5712e2ad8608977ec5c5e0e6b6b
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 14 12:46:32 2022 -0800

    Rename the test_splits flag to direct_test_splits.

    This avoids possible flag conflicts.

commit 48c70cc30742b45b17a1d18ece2f0d079bee3915
Author: arne-alex <[email protected]>
Date:   Mon Nov 14 21:33:02 2022 +0100

    Merge pull request #23333: Track time on Cloud Dataflow streaming data reads and export via heartbeats

commit 9c83de646ab52bd0b05e3346190dd55cd68b2a8b
Author: Johanna Öjeling <[email protected]>
Date:   Mon Nov 14 21:19:44 2022 +0100

    Add more tests for S3 filesystem (#24138)

commit 9e9c6d797ba52b460f83131431c8e53aebbbc9ac
Merge: d5d76b9745 c600444e1d
Author: Ning Kang <[email protected]>
Date:   Mon Nov 14 12:06:15 2022 -0800

    Merge pull request #24029 from apache/dependabot/npm_and_yarn/sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/loader-utils-1.4.1

    Bump loader-utils from 1.4.0 to 1.4.1 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel

commit d5d76b974592d45de368ab641647ca5cc4ec12ec
Author: Yi Hu <[email protected]>
Date:   Mon Nov 14 15:03:28 2022 -0500

    Support SqlTypes Date and Timestamp (MicrosInstant) in AvroUtils (#23969)

    * Support SqlTypes Date and Timestamp (MicrosInstant) in AvroUtils

    * Add TODO about java.time migration

commit 330cc2010c9f4a2d4e30318bf50a4109ec1cd392
Author: Pablo <[email protected]>
Date:   Mon Nov 14 12:02:10 2022 -0800

    Cleanup stale BQ datasets (#24158)

    * Cleanup stale BQ datasets

    * addressing comments

commit 4a044999b8ed4bcd41f816f3a23ccb5da00c4c38
Merge: e563b9dd2f 5bd75c25de
Author: Heejong Lee <[email protected]>
Date:   Mon Nov 14 11:16:00 2022 -0800

    Merge pull request #24076 from chamikaramj/multilang_java_updates

    Updates Multi-lang Java quickstart

commit e563b9dd2f3aa0484e6cdc08869991b5e438023e
Author: Evgeny Antyshev <[email protected]>
Date:   Mon Nov 14 20:56:35 2022 +0300

    [Tour Of Beam] verify that unit exists when saving progress (#24118)

    * AIO

    * Update learning/tour-of-beam/backend/integration_tests/auth_test.go

    Co-authored-by: Danny McCormick <[email protected]>

    * nit

    Co-authored-by: Danny McCormick <[email protected]>

commit 774923e0dd089de870bfa5c77063ae2b28f79347
Merge: 71785de528 1ad0cbc445
Author: Kenn Knowles <[email protected]>
Date:   Mon Nov 14 09:52:26 2022 -0800

    Merge pull request #24141: Fix checkArgument format in GcsPath

commit 71785de52864313c2e3b14fe72a2a63281343617
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Nov 14 11:54:37 2022 -0500

    Bump github.com/aws/aws-sdk-go-v2/config from 1.17.10 to 1.18.0 in /sdks (#24151)

    Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.17.10 to 1.18.0.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.17.10...config/v1.18.0)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/config
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 50d591d6cb3e799bee4e29dfc593c693a86e6276
Author: Bruno Volpato <[email protected]>
Date:   Mon Nov 14 11:50:01 2022 -0500

    Change DataflowBatchWorkerHarness doWork error level to INFO (#24135)

commit 5a72696bfda09fdb905ba8e58b636f8494ef955f
Merge: ee0a5836d6 0633fe9634
Author: Kenn Knowles <[email protected]>
Date:   Mon Nov 14 08:12:12 2022 -0800

    Merge pull request #24149: Remove extraneous jetbrains annotation

commit ee0a5836d69b776834eb3bd9b2bd02eb5252c333
Merge: d001a69e1a 137799672e
Author: Kenn Knowles <[email protected]>
Date:   Mon Nov 14 08:11:00 2022 -0800

    Merge pull request #24132: Fix checkArgument format string in AvroIO

commit d001a69e1a58701d6ed4fcb5e3fb7a0921301dad
Author: Yi Hu <[email protected]>
Date:   Mon Nov 14 10:56:54 2022 -0500

    Test Dataproc 2.1 with Flink load tests (#24129)

    * Test Dataproc 2.1 with Flink load tests

    * Minor fix flink_cluster script

commit caabd9be52887ad70c8a4269395c893811ac6a84
Author: Israel Herraiz <[email protected]>
Date:   Mon Nov 14 16:03:39 2022 +0100

    Make MonotonicWatermarkEstimator work like its Java SDK equivalent (#24146)

    * Make MonotonicWatermarkEstimator work like its Java SDK equivalent

    The current implementation of MonotonicWatermarkEstimator raises an exception
    with late messages, which makes the watermark estimator barely usable in real
    world scenarios.

    This PR fixes #20041 by making this watermark estimator work like its Java SDK
    equivalent (`WatermarkEstimators.MonotonicallyIncreasing`).

    * Update unit tests too

    * Make linter happy

commit 451f6b3e7f58d0a3782ad942c6a1fd9f63932024
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Nov 14 09:48:23 2022 -0500

    Bump golang.org/x/net from 0.1.0 to 0.2.0 in /sdks (#24153)

    Bumps [golang.org/x/net](https://github.com/golang/net) from 0.1.0 to 0.2.0.
    - [Release notes](https://github.com/golang/net/releases)
    - [Commits](https://github.com/golang/net/compare/v0.1.0...v0.2.0)

    ---
    updated-dependencies:
    - dependency-name: golang.org/x/net
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 2bb03d62e2d7dc2d8e39040fc9adebccbde74fde
Merge: 4e39ef2041 623083cd0a
Author: Alexey Romanenko <[email protected]>
Date:   Mon Nov 14 15:01:13 2022 +0100

    Merge pull request #24000: [Website] Change headers size from h4,h3 to h2

commit 563c66d6fd32165da14a07747f2764c17a5d24ea
Author: bulat safiullin <[email protected]>
Date:   Wed Sep 7 18:28:42 2022 +0600

    [Website] update pre tag copy link styles #23064

commit 4e39ef20410ee51c6040317bcd60171e64c5171f
Merge: 223768f782 105ed6fedc
Author: Alexey Romanenko <[email protected]>
Date:   Mon Nov 14 10:55:33 2022 +0100

    Merge pull request #24115: [Website] update go-dependencies.md java-dependencies.md links

commit 223768f782f771f0033b8d0686d86cf4c71fad75
Merge: aa0a35dabf a9da2abee6
Author: Kenn Knowles <[email protected]>
Date:   Sun Nov 13 18:53:13 2022 -0800

    Merge pull request #24136: Fix checkArgument format string in ExecutionStateTracker

commit 0633fe9634fe61df7cbc0ecac205d81124fd504a
Author: Kenneth Knowles <[email protected]>
Date:   Sat Nov 12 15:15:16 2022 -0800

    Remove extraneous jetbrains annotation

commit 83f1bc19b95935e60ca1f4027d4b60c7e738a84a
Author: Kenneth Knowles <[email protected]>
Date:   Sat Nov 12 14:16:09 2022 -0800

    Fix arguments to checkState in BatchViewOverrides

commit 1ad0cbc44594d8405bf4b07a126265238013a02a
Author: Kenneth Knowles <[email protected]>
Date:   Sat Nov 12 13:41:02 2022 -0800

    Fix checkArgument format in GcsPath

commit aa0a35dabf9c2a0d9822faff06d939d9a77a3ab6
Author: Kenn Knowles <[email protected]>
Date:   Fri Nov 11 20:26:30 2022 -0800

    Fix checkArgument format string in TestStream (#24134)

commit a9da2abee6455bc2cf0f18ba5f6cd7bbaeae669f
Author: Kenneth Knowles <[email protected]>
Date:   Fri Nov 11 16:54:27 2022 -0800

    Fix checkArgument format string in ExecutionStateTracker

commit 369e2ba8622d3474c14c39b941b2c618842d1e47
Author: Ryan Thompson <[email protected]>
Date:   Fri Nov 11 19:46:07 2022 -0500

    Add a ValidatesContainer integration test for use_sibling_sdk_workers (#24099)

commit 137799672eb559a7586262e6a8a73d1ab3580e44
Author: Kenneth Knowles <[email protected]>
Date:   Fri Nov 11 15:30:01 2022 -0800

    Fix checkArgument format string in AvroIO

commit 5d2dbf957e4e82fb3980726940df02ac67e563cd
Author: Anand Inguva <[email protected]>
Date:   Fri Nov 11 15:57:28 2022 -0500

    Update staging of Python wheels (#24114)

    Fixes https://github.com/apache/beam/issues/24110

commit c2021bee1eba0322b43c90841397859048296b21
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Nov 11 15:33:14 2022 -0500

    Bump google.golang.org/api from 0.102.0 to 0.103.0 in /sdks (#24049)

    Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.102.0 to 0.103.0.
    - [Release notes](https://github.com/googleapis/google-api-go-client/releases)
    - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.102.0...v0.103.0)

    ---
    updated-dependencies:
    - dependency-name: google.golang.org/api
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 6557c91c79480b9d90573d52d257a11c2b160196
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Nov 11 11:47:12 2022 -0800

    Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#24112)

    Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.29.1 to 1.29.2.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.29.1...service/s3/v1.29.2)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 96f9da1ab652156cd143d57e3aa3d94836338f2b
Author: Yi Hu <[email protected]>
Date:   Fri Nov 11 14:34:22 2022 -0500

    More cleanup containers (#24105)

    * More cleanup containers

    * prebuilt_beam_sdk/beam_python_prebuilt_sdk no longer exists in gcr.io
      Add correct paths to cleanup

    * Bump grace time to 15 days

    * Run job daily as in code comment

    * Set grace period to 30 days

commit 836766ddb83d37463e4b036f156b8d7e15e3864b
Author: Ritesh Ghorse <[email protected]>
Date:   Fri Nov 11 14:30:30 2022 -0500

    upgrade testcontainer dependency (#24123)

commit 9fcd20c3712536f2d4580beead678cdbb6fd4746
Author: Damon <[email protected]>
Date:   Fri Nov 11 11:12:11 2022 -0800

    Implement PubsubRowToMessage transform (#23897)

    * Begin PubsubRowToMessage Impl

    * Complete working draft

    * Unit tests validate user and non-user fields

    * Finish tests on supporting methods

    * Pass checks before finalizing tests

    * WIP

    * fix timestamp

    * finalize tests

    * Finalize code comments

    * Clean up check findings

    * Add InputSchemaFactory

    * Patch code comment typo

commit 3a6fcc1ca4d07a467464ed1214a94b5c9c147295
Author: Evgeny Antyshev <[email protected]>
Date:   Fri Nov 11 19:13:45 2022 +0300

    disable (#24121)

commit 027fb142038b45c443d87af96cac082264c43188
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Nov 11 10:26:23 2022 -0500

    Bump cloud.google.com/go/bigtable from 1.17.0 to 1.18.0 in /sdks (#24113)

    Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.17.0 to 1.18.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.17.0...pubsub/v1.18.0)

    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/bigtable
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit b2d28a64bb0837dd97e8be7f97f31f88d380f110
Author: Danny McCormick <[email protected]>
Date:   Fri Nov 11 08:45:14 2022 -0500

    Add TFX support in pydoc (#23960)

    * Add TFX support in pydoc

    * Wording

commit 105ed6fedcb6ff33d4a43940c342a653e67cb25e
Author: bulat safiullin <[email protected]>
Date:   Thu Nov 10 17:47:54 2022 +0600

    [Website] update go-dependencies.md java-dependencies.md programming-guide.md links #24084

commit 4b96193250a63b27721a8c5e1a3bd6ecff983093
Author: Brian Hulette <[email protected]>
Date:   Thu Nov 10 16:51:06 2022 -0800

    Remove TheNeuralBit from the pool of Python reviewers (#24108)

commit 8f8f089a5f565f179905984cef3522a0577d9219
Author: Sanil Jain <[email protected]>
Date:   Thu Nov 10 16:47:31 2022 -0800

    Wire SamzaPipelineOptions to Exeption listener interface (#24109)

commit b3186ba91f9bb22764d47b78eb2b7ff017080f75
Author: Ahmed Abualsaud <[email protected]>
Date:   Thu Nov 10 18:04:50 2022 -0500

    Support using BigQueryIO Storage Read API with SchemaTransforms (#23827)

    * support schema transform for bq direct read method

    * use vendor Strings import

    * add BigQueryServices argument to config object

    * suppress nullability errors

    * add package-info.java file to providers subdirectory

    * removing accidentally created file

    * added documentation and moved configuration filclass into provider class

    * validate config params before expansion. config has a validaate me method

    * updated URN identifier to follow standards

commit a4a94da1936d8cd2f8d1145fd2e8329fe06a2990
Author: Oleh Borysevych <[email protected]>
Date:   Fri Nov 11 00:28:43 2022 +0200

    fixing linter error (#24104)

commit 156a6099d71a155c3379340a0b98256306e1755e
Author: Danny McCormick <[email protected]>
Date:   Thu Nov 10 17:26:15 2022 -0500

    Add blog post on new ML resources (#24071)

    * Create ml-resources.md

    * Add ensemble notebook image

    * Add image link

    * Fix image link

    * Add ml-landing page image

    * Add image + move around

    * Add paragraph on upcoming changes

    * Remove bad whitespace

commit 38742d40c895f5d38ee6ee5eb328d11b3262307e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Nov 10 16:12:40 2022 -0500

    Bump github.com/aws/aws-sdk-go-v2/config from 1.5.0 to 1.17.10 in /sdks (#24080)

    Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.5.0 to 1.17.10.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.5.0...config/v1.17.10)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/config
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 92cef32915c40d353b51529a87d8f0131b0a3538
Author: BjornPrime <[email protected]>
Date:   Thu Nov 10 15:30:37 2022 -0500

    Num failed inferences (#23830)

    * created test_increment_num_failed_inferences and test_num_failed_inferences_no_failures

    * added assertRaises to test_increment_num_failed_inferences

    * added num_failed_inferences to _MetricsCollector

    * changed error handling and update() implementation

    * updated metric name in tests

    * removed unnecessary else blocking

    * removed unnecessary inference_args from test_increment_failed_batches_counter()

    * changed final test_increment_failed_batches_counter assertion

    * clarified error handling and updated failed_batches_counter initialization

    * decreased examples array length to 1 to ensure repeatability

    * troubleshooting tests

    * trying to get test_increment_failed_batches_counter to fail as expected

    * corrected assertion details

    * simplified assertRaises and added reminder comment to assertEqual counter

    * lint test

    * lint test passed, resetting pre-commit-config.yaml

    * fixed lingering linting issues

    * shortened comment line to comply with linting

    * formatter worked its magic

commit 2341f61d48d838ddaf2e4fb990a49987244c3513
Author: Evgeny Antyshev <[email protected]>
Date:   Thu Nov 10 21:24:55 2022 +0300

    [Tour Of Beam] handle CORS pre-flight requests (#24083)

    * cors

    * README

    * nit

    * nit

    * -headers, -cache

    * allow-headers

    * +1h cache,comment

commit 36b0c8f1a76865e4dc3b1eaec1566c3ef9fd4345
Author: Evgeny Antyshev <[email protected]>
Date:   Thu Nov 10 21:21:21 2022 +0300

    [Playground] update snippet by persistence_key (#24056)

    * proto

    * AIO

    * -frontend

    * testify

    * +license

    * -build

    * index

    * skipKey

    * Update playground/backend/internal/db/datastore/datastore_db.go

    Co-authored-by: Danny McCormick <[email protected]>

    Co-authored-by: Danny McCormick <[email protected]>

commit df553d1a4c6c6cc9b37ada2a6c84c88d8aef38b9
Author: Ahmed Abualsaud <[email protected]>
Date:   Thu Nov 10 13:17:36 2022 -0500

    Add random string at the end of BigQuery query job name to make it resilient to retries (#24041)

    * add random string at the end of query job name

    * use deterministic temp table name

commit e439f4120ef4c25aa36e5b03756dc7391bdbd211
Author: Pablo <[email protected]>
Date:   Thu Nov 10 10:06:26 2022 -0800

    Improving stale container cleanup script (#24040)

    * Improving stale container cleanup script

    * Avoid also latest image

    * Update .test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh

    Co-authored-by: Yi Hu <[email protected]>

    * Update .test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh

    Co-authored-by: Yi Hu <[email protected]>

    Co-authored-by: Yi Hu <[email protected]>

commit 8d585242e158babd8dd2ca5a6d5a49d353be2935
Author: Anand Inguva <[email protected]>
Date:   Thu Nov 10 12:58:41 2022 -0500

    [Python]Set pickle library at the Pipeline creation stage (#24069)

    Fixes https://github.com/apache/beam/issues/21615

commit 08b6a524fecc333a12ce42971733ef64ae7d02f1
Author: Janek Bevendorff <[email protected]>
Date:   Thu Nov 10 18:37:09 2022 +0100

    [BEAM-12792] Install pipline dependencies to temporary venv (#16658)

commit 74f87b0282a23aa01ecb63288013311d207960aa
Merge: 38a85b288a 383ea77bc9
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 10 09:10:37 2022 -0800

    Merge pull request #23985 Support dynamic sharding in the worker.

commit 38a85b288ad10171c444d4092eb969ca5de58af3
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Nov 10 10:39:15 2022 -0500

    Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#24077)

    Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.11.1 to 1.29.1.
    - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
    - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.11.1...service/s3/v1.29.1)

    ---
    updated-dependencies:
    - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c5110bae06fb9db6cace1f7657745fa97e3f58af
Author: Yi Hu <[email protected]>
Date:   Thu Nov 10 10:33:55 2022 -0500

    Fix FhirIO javadoc format broken (#24072)

commit 623083cd0a72d86c7ecba7653e9a8564861abc4d
Author: bulat safiullin <[email protected]>
Date:   Mon Nov 7 16:08:39 2022 +0600

    [Website] change headers size from h4,h3 to h2 #24082

commit 5bd75c25de291e517cc5c5799ae4adaaaaceacb7
Author: Chamikara Jayalath <[email protected]>
Date:   Wed Nov 9 18:04:28 2022 -0800

    Updates Multi-lang Java quickstart

commit 4522f4ce916223afb3c4edd02a7be6018d0ce5a4
Author: Trevor Gevers <[email protected]>
Date:   Wed Nov 9 18:56:18 2022 -0600

    Update datastore_wordcount.py (#23724)

commit 96cc252348ce7229731be3c74bf5848d1b004c08
Author: tvalentyn <[email protected]>
Date:   Wed Nov 9 16:08:00 2022 -0800

    Remove a duplicate label (#24043)

commit 62a6bd1e9317a7658810e9a0eebfe925460e8982
Author: Thiago Nunes <[email protected]>
Date:   Thu Nov 10 08:45:27 2022 +1100

    test: add more tests to throughput estimator (#23915)

    Verifies that getting the throughput when no updates have occurred for
    the size of the window should return 0.

commit e6647c34bb367b3058c42d0201150395c534e7ce
Author: Ritesh Ghorse <[email protected]>
Date:   Wed Nov 9 15:12:59 2022 -0500

    skip output coder field in exp request (#24066)

commit 383ea77bc982416afcbbbe11a4dc1f1424670589
Author: Robert Bradshaw <[email protected]>
Date:   Wed Nov 9 10:49:37 2022 -0800

    Reduce flakiness of time-based split manager test.

    Increase wait time from a tenth to half a second.

commit 539fa9159ffc116b2e79e6de2804dfdd1c1e4722
Author: Danny McCormick <[email protected]>
Date:   Wed Nov 9 11:45:20 2022 -0500

    Convert initialisms to all caps (#24061)

    * Convert initialisms to all caps

    * Fix test refs

commit 4941b9633c6ccf9eb8aad554186aac24ca7b7492
Author: Ryan Thompson <[email protected]>
Date:   Wed Nov 9 11:25:48 2022 -0500

    added comments for tensorflow notebook (#23726)

    * added comments for tensorflow notebook

    * added note to string type

commit 858f5048f578548a63e5e4319a7ab5363311c269
Merge: 95c121b254 1929968891
Author: Alexey Romanenko <[email protected]>
Date:   Wed Nov 9 16:35:19 2022 +0100

    Merge pull request #24057: [Spark Dataset runner] Enable projection pushdown for Spark dataset runner

commit 95c121b2549d3f8fda7668e3759b8991b55d9865
Author: Oleh Borysevych <[email protected]>
Date:   Wed Nov 9 16:03:33 2022 +0200

    Fix dependency mismatch in Playground Java runner  (#24059)

    * fixing existing and potential dependency mismatch

    * extract grpc version

commit 1929968891d629c83249c574c28db6c103ba9271
Author: Moritz Mack <[email protected]>
Date:   Wed Nov 9 13:35:12 2022 +0100

    [Spark Dataset runner] Enable projection pushdown for Spark dataset runner.

commit 63362f5ba60a22e77e54c2df47a844f031036309
Author: Johanna Öjeling <[email protected]>
Date:   Wed Nov 9 07:49:54 2022 +0100

    [Go SDK] S3 implementation of the Beam filesystem (#23992)

    * Implement filesystem for S3

    * Update CHANGES.md

commit 73142bad2472e4516831c24a8dfb05c9acebf791
Author: Kanishk Karanawat <[email protected]>
Date:   Tue Nov 8 20:46:04 2022 -0500

    Handle Avro schema generation for logical data types in BigQueryAvroUtils (#23620): handle avro logical types for TableFieldSchema to Avro schema conversion

    Co-authored-by: Kanishk Karanawat <[email protected]>

commit 8db8fae173056c0d89a36a8a5ce0443b6b9e54fa
Author: Robert Burke <[email protected]>
Date:   Tue Nov 8 15:56:53 2022 -0800

    [Go] Add pipeline resource hints to CHANGES.md (#24036)

commit a84b7dc179fed5dfc1d48dddfb04266de3716e31
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 16:35:25 2022 -0800

    Make mypy happy.

commit b9655e7de1a682d8ec4efcafb4d610f794e1b40e
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Nov 8 12:27:38 2022 -0800

    Bump cloud.google.com/go/storage from 1.27.0 to 1.28.0 in /sdks (#24028)

    Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.27.0 to 1.28.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.27.0...spanner/v1.28.0)

    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/storage
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 708e05f0dc6c0a2e62674cd0ecb8a6744ae57d3a
Author: Brian Hulette <[email protected]>
Date:   Tue Nov 8 12:18:23 2022 -0800

    Retroactively announce Batched DoFn support in 2.42.0 Blog (#24011)

    * Retroactively announce Batched DoFn support in 2.42.0

    * Add to blog as well

commit b9ff75a18b40141a8d2340e397fd70382b3d4e8f
Author: Israel Herraiz <[email protected]>
Date:   Tue Nov 8 20:03:43 2022 +0100

    Update my Twitter handle (#23653)

commit bf621ccb88023e0bc09752978c099ac4629086e2
Author: Rebecca Szper <[email protected]>
Date:   Tue Nov 8 10:50:02 2022 -0800

    Editorial review of the ML base API descriptions (#24026)

commit f0a6ff46619ee1fd0639fa3f06e04708f69dee74
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Nov 8 13:04:03 2022 -0500

    Bump cloud.google.com/go/bigtable from 1.16.0 to 1.17.0 in /sdks (#24027)

    Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.16.0 to 1.17.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.16.0...pubsub/v1.17.0)

    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/bigtable
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 7dac3f5ef40b5d24b24d9ce5bec4717284260b85
Author: Lukasz Cwik <[email protected]>
Date:   Tue Nov 8 09:42:29 2022 -0800

    [#21250] Trivial removal of loop over something that always has one element (#24014)

    Multiplexing was put into the PCollectionConsumerRegistry a long time ago and this seems to have been missed during that migration.

commit a6a9b23fd5b74142c35948f10d1840b882817246
Author: Robert Burke <[email protected]>
Date:   Tue Nov 8 09:19:49 2022 -0800

    [Go] Pipeline Resource Hints (#23990)

commit 8cf2a63cc25399109f64b84ddfcd7c74d21e61ce
Author: tvalentyn <[email protected]>
Date:   Tue Nov 8 09:19:16 2022 -0800

    Update release notes. (#23986)

commit c600444e1dfa2f4afaad056bc65fa009a60a32bb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Nov 8 06:22:14 2022 +0000

    Bump loader-utils

    Bumps [loader-utils](https://github.com/webpack/loader-utils) from 1.4.0 to 1.4.1.
    - [Release notes](https://github.com/webpack/loader-utils/releases)
    - [Changelog](https://github.com/webpack/loader-utils/blob/v1.4.1/CHANGELOG.md)
    - [Commits](https://github.com/webpack/loader-utils/compare/v1.4.0...v1.4.1)

    ---
    updated-dependencies:
    - dependency-name: loader-utils
      dependency-type: indirect
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

commit 62d8bdc604dfeccae08d00225bce8c9b9c3ffb28
Merge: d4142f7eab 930c4d2d87
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 15:31:37 2022 -0800

    Merge pull request #23789 Better surfacing of Scala support via Scio.

commit 7f7713de09b3b0e400ed02ffdfcb34a6a8b6e1e4
Merge: 5f587b9339 d4142f7eab
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 15:29:27 2022 -0800

    Merge branch 'master' into javascript-liquid-sharding

commit 5f587b9339b3b19ae5c5b1cf15e3ce218f75d21a
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 15:25:33 2022 -0800

    Clarifying comments.

commit 60fe3cdddf9d3c27d9493b345d1f5f2a9284a76f
Author: Robert Bradshaw <[email protected]>
Date:   Mon Nov 7 15:06:26 2022 -0800

    Update style sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner.py

    Co-authored-by: Danny McCormick <[email protected]>

commit d4142f7eab7d5a3d21a3d3930c448fe403880f93
Author: Robert Burke <[email protected]>
Date:   Mon Nov 7 12:11:21 2022 -0800

    Switch && for || to fix bug in #23889 resolution (#24017)

commit 45100d708d954b25b5a884dc3c7a965f008228c6
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 14:53:24 2022 -0500

    Add files then check cached diff to get untracked files

commit 380d4730d3c52b5c2165bdb84aac11d0feccfe4f
Author: scwhittle <[email protected]>
Date:   Mon Nov 7 19:33:37 2022 +0100

    Enforce splitting invariants by ensuring split state is reset in the same synchronized block as window index increments.  (#23882)

    * Enforce splitting invariants by ensuring split state is reset in the same synchronized block as window index increments. Fixes #23881.

    * Add missing currentElement = null;

    Co-authored-by: Lukasz Cwik <[email protected]>

commit 8b3fd2e0d652045bf94b83638129bb12562ce738
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Nov 7 10:33:26 2022 -0800

    Bump google.golang.org/api from 0.101.0 to 0.102.0 in /sdks (#23957)

    Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.101.0 to 0.102.0.
    - [Release notes](https://github.com/googleapis/google-api-go-client/releases)
    - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.101.0...v0.102.0)

    ---
    updated-dependencies:
    - dependency-name: google.golang.org/api
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 9533fc3757d3041135cbe2f5078dff0855adc1f1
Author: capthiron <[email protected]>
Date:   Mon Nov 7 19:32:18 2022 +0100

    feat: implement bigtable io connector with write capabilities (#23411)

commit c4218e572aa8a8f46b433b60073a59fdd7fdf148
Merge: 23676a9ec1 a5ee669a43
Author: Alexey Romanenko <[email protected]>
Date:   Mon Nov 7 18:24:45 2022 +0100

    Merge pull request #23934: [Website] change case studies page mobile layout

commit 23676a9ec17069b309e8803d1fbfeb8009035828
Author: Oleh Borysevych <[email protected]>
Date:   Mon Nov 7 17:24:59 2022 +0200

    [Tour of Beam] Learning content for "Introduction" module (#23085)

    * learning content for introduction module

    * removing white spaces from md files

    * delete whitespaces

    * delete whitespaces in python

    * delete whitespace #2

    * divide pipeline concepts

    * add pipeline example concepts

    * adding category tag to python examples

    * adding category to java examples

    * adding category to go examples

    * fixed go example

    * fixed go example compilation

    * fixing python duplicate example names

    * add runner concepts

    * fixing java examples

    * add licence for runner unit

    * some minor fixes for unit names

    * fixed unit name

    * resolving CR comments

    * adding complexity to examples

    * adding tags

    * fixed go example compilation

    * fixed python example with duplicate transform

    * change indent python

    * fixing missing pipeline options

    * change arrow symbol

    * delete example prefix

    * minor formatting and readability fixes

    * add example description

    * minor fix

    * minor code review comment

    Co-authored-by: Abzal Tuganbay <[email protected]>

commit 98d2ffd69ab9d5c49196a78f7c25daacb57b6478
Author: Brian Hulette <[email protected]>
Date:   Mon Nov 7 07:03:12 2022 -0800

    Make `documentation/io/connectors/` canonical (#23877)

    * Drop documentation/io/built-in/

    * Update links in Custom I/O pattern

commit c0b41fa045418beb782c0a59e6f5c511da898c53
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 09:29:08 2022 -0500

    Use git diff instead of git diff-index to avoid file timestamp changes being picked up

commit 0476d6498217b7b0e833643c47fef2c4bdaec529
Merge: 1d5fc14031 6ae37b6f82
Author: Alexey Romanenko <[email protected]>
Date:   Mon Nov 7 15:28:59 2022 +0100

    Merge pull request #23962: [CdapIO] Add sparkreceiver:2 module.

commit 1d5fc14031d583e92018ebfbeaff9b4e3b978fdb
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 09:14:24 2022 -0500

    Remove quiet flag on debug

commit c08273b9355566181d2d5c0792130f8d5cc1428c
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 09:12:26 2022 -0500

    Correctly print diff and swallow empty commits for the moment

commit 66fb431deb77db5f7f0fac31e6bf3b2c094c8ac5
Author: Danny McCormick <[email protected]>
Date:   Mon Nov 7 09:08:53 2022 -0500

    Print diff and scope to state path

commit e0e10b9e5432643b884c381d145e5924cc4ef193
Author: Danny McCormick <[email protected]>
Date:   Sun Nov 6 12:12:22 2022 -0500

    PR Bot - Dont throw error on return code 1

commit bbcb790461c47d22399c681cd42c434b206eb550
Author: Danny McCormick <[email protected]>
Date:   Sun Nov 6 11:59:32 2022 -0500

    Fix pr bot - exec doesn't allow command chaining

commit 88f9a66a9f4cc2fe3259d5ebd83ebb472a29dd5e
Author: Danny McCormick <[email protected]>
Date:   Sun Nov 6 11:46:34 2022 -0500

    Fix diff to stop repeated bot runs

commit ddb4f2d53538fea68383fe43221d4074d05d11b9
Author: bullet03 <[email protected]>
Date:   Fri Nov 4 16:11:10 2022 -0700

    Website add and update logos (#23899)

    * [Website] add and update logos

    * Delete talend.svg:Zone.Identifier

commit 6d42219ff547720996489aa29de0d64246db2541
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 14:30:18 2022 -0700

    Enable more portable-runner requiring tests. (#23970)

    * Enable more portable-runner requiring tests.

    * Run all basic tests on portable runner as well.

    * Mark ULR tests.

commit 1a643d16112a5f3f1418f1be393de51af65d752b
Author: David Cavazos <[email protected]>
Date:   Fri Nov 4 13:45:12 2022 -0700

    Beam starter projects blog post (#23964)

    Co-authored-by: Ahmet Altay <[email protected]>

commit d10b4a28fff667289c146152535275035f2dedf7
Author: Ryan Thompson <[email protected]>
Date:   Fri Nov 4 16:06:49 2022 -0400

    removed trailing whitespace (#23987)

commit 3bd697561aab4a5d2aa49d29143695d6d57a834a
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 11:43:47 2022 -0700

    Add dynamic splitting support to the worker.

commit ed520243f472c7224a1f8c74a0c2e8f81965cbfc
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 11:07:40 2022 -0700

    Add the a Reshuffle operation and use it in Create.

    This allows created sources to be staticly and dynamically split.

commit 9283512544b8de169be9cd7795b42284fdf45b87
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 10:28:51 2022 -0700

    Add the ability to schedule splits on the ULR via a pipeline option.

    This can be used to test arbitrary SDKs for behavioral correctness
    for dynamic splitting.

    These splits are wall-time based, which isn't ideal, but easier than
    setting up and managing a cross-process synchronization channel.

commit ed7b560d7ce7dda0b55a067b9d9a1a9d692a7d7a
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 09:46:19 2022 -0700

    Compute element counts for all PCollections.

commit 1cfdb127983a55c77b3259c4fdd248497b28e250
Author: Jack McCluskey <[email protected]>
Date:   Fri Nov 4 15:43:16 2022 -0400

    Add custom inference fn suport to the sklearn model handlers (#23642)

    * Add custom inference fn suport to the sklearn model handlers

    * Clean up import order

    * Update typing, add numpy unit test

    * Add Pandas unit test

    * Formatting, linting

    * yapf run

    * Remove trailing whitespace

    * import order

    * Change inference_fn to keyword-only arg

commit 8617b86fd792ac555233dd79c8ede34c2ce67ecf
Merge: aa178f8252 2496a0d1e0
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 11:42:21 2022 -0700

    Merge pull request #23978 Add basic counter support to the typescript SDK.

commit 2496a0d1e0fda74e8fc0f24eee011362426369a2
Merge: 64abcbe4a3 aa178f8252
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 11:15:47 2022 -0700

    Merge branch 'master' into javascript-metrics

commit aa178f825223cbb0a6208fad0e02f98292eeded1
Merge: 97628802fe 1807c307aa
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 09:34:11 2022 -0700

    Merge pull request #23976 Correctly capture log levels and attach stage information.

commit 64abcbe4a3a15160aba3f3163ff8cbf5186bbfdb
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 09:27:22 2022 -0700

    Only report counters that were actually used.

commit 1807c307aa1e493acb2d8ba542baffefcb96e7e2
Author: Robert Bradshaw <[email protected]>
Date:   Fri Nov 4 08:52:21 2022 -0700

    Remove obsolete TODO.

commit 97628802fe4cf6640a3b6854e27b410dd5731379
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Nov 4 11:47:07 2022 -0400

    Bump cloud.google.com/go/datastore from 1.8.0 to 1.9.0 in /sdks (#23916)

    Bumps [cloud.google.com/go/datastore](https://github.com/googleapis/google-cloud-go) from 1.8.0 to 1.9.0.
    - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
    - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md)
    - [Commits](https://github.com/googleapis/google-cloud-go/compare/asset/v1.8.0...asset/v1.9.0)

    ---
    updated-dependencies:
    - dependency-name: cloud.google.com/go/datastore
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <[email protected]>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 058c56af735e10184d1501cc08cd16af4d21e64a
Author: Ayush Sharma <[email protected]>
Date:   Fri Nov 4 20:58:11 2022 +0530

    [Task]: PR Bot will push commits only if they are non-empty (#23937)

    * fix pr-bot

    * fix the git pr bot

commit 7dba78d611089a1826c5655ed7ffcdf3f6007eae
Author: Danny McCormick <[email protected]>
Date:   Fri Nov 4 09:35:16 2022 -0400

    Immediately truncate full restriction on drain of periodic impulse (#23765)

    * Immediately truncate full restriction on drain of periodic impulse

    * Whitespace format

commit 16852518861514a56ad1b57113e3815cfcc8533a
Author: Anand Inguva <[email protected]>
Date:   Fri Nov 4 08:39:34 2022 -0400

    TFX image classification example (#23456)

    * TFX image classification example

    * TFX image classification with serving model

    * Add TF Model Wrapper

    * Clean up code

    * Refactoring

    * Add a test for tfx_bsl RunInference

    * fixup lint

    * Refactoring

    * Fixup lint

    * Add brief summary for the example

    * Apply suggestions from code review

    Co-authored-by: Andy Ye <[email protected]>

    * Refactoring code and add comments

    * Update help description

    * reorder imports

    * Reorder imports again

    * Add docstring

    * Refactoring

    * Add pillow to tfx pipeline requirements

    * Move inferencePostCommitIT to Python 3.9 suite

    * Uncomment other postcommit suites

    Co-authored-by: Andy Ye <[email protected]>

commit 30b26171b4038c1c976adbc3be386e118b35153b
Author: Danny McCormick <[email protected]>
Date:   Fri Nov 4 08:22:17 2022 -0400

    Update watermark during periodic sequence/impulse (#23507)

    * Update watermark during periodic sequence/impulse

    * Remove extraneous import

    * Formatting

    * Linting

    * Only run on dataflow for guaranteed watermark support

    * More permissive test to avoid timing issues

    * Test pipeline options

    * Fix test

    * Formatting

    * Formatting

    * Apply feedback - cleanup/naming/flink

    * Format

    * Unused import

commit 1563b9721b4bee5a22d384b282762726247e0703
Author: Brian Hulette <[email protected]>
Date:   Fri Nov 4 05:09:39 2022 -0700

    Improve Iterator error message (#23972)

commit e3987b5435d39c5fb058d4085eb5e210f8756224
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 17:39:49 2022 -0700

    old prettier change

commit 89c8dd5ca23591be6c2778045df32c7d08d9f99e
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 17:12:26 2022 -0700

    Add distribution metric type.

commit e4e606a8f2652c2fa761a95462f230385b6bcb47
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 17:09:03 2022 -0700

    Support metrics over the portability API.

commit e04f3d6fdb39528264cb2a14e60e565991121f1c
Author: Robert Bradshaw <[email protected]>
Date:   Thu Nov 3 16:52:51 2022 -0700

    Add basic counter setting and getting to the typescript SDK.

commit 1daa9c373ac7fbdb696120eeaa2ca9e85a679a1f
Author: Robert Bradshaw <robertwb@gmail.…
@tvalentyn tvalentyn mentioned this pull request Feb 11, 2023
Closed
15 tasks
@tvalentyn
Copy link
Contributor

tvalentyn commented Feb 11, 2023 •
edited
Loading

Looks like we have users that were affected by these changes. Currently saw couple of cases, both with users who provide custom containers, patched with an entrypoint from Beam.

  1. If base image doesn't have python3-venv, environment creation fails. We could fall-back on single env in this case.
  2. (Likely but not confirmed) see: [Bug]: Dependencies from private repositories unable to be seen #25085.

Still thinking how to respond and keeping an eye on other issues. An opt-in / opt-out capability via environment variable is another option.

nausharipov added a commit to akvelon/beam that referenced this pull request Mar 28, 2023
@nausharipov
Squashed commit of the following:
fcf7f46 
commit 95c82ec5be90e64f8e62072d4b7742db77b0558f
Merge: 7066061152 c796d856f7
Author: Darkhan Nausharipov <[email protected]>
Date:   Tue Mar 28 16:07:43 2023 +0600

    Merge pull request #441 from akvelon/issue25283-tobf-sucfuu

    [ToB] [Frontend] Save code for unauthenticated users

commit c796d856f74dfb3d3371c9421e2c09eb4a5196a8
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Mar 28 16:05:08 2023 +0600

    _isSnippetTypeSavable (#25283)

commit 7b487a0b4c2fd83512bc2da53bf403d60520ae3d
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Mar 28 15:49:29 2023 +0600

    disabled saving solution code (#25283)

commit be3e3460527263146eb13c440056d553ae867938
Author: Alexey Inkin <[email protected]>
Date:   Tue Mar 28 13:22:05 2023 +0400

    Improve comments, remove unused code (#25283)

commit 3434381ed159e646949e5e736a491b95e0cad66d
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Mar 27 20:28:39 2023 +0600

    getSavedDescriptor (#25283)

commit 7dbb6b44dd7dcb5e0835f62d4d9454b4c5cb7c9a
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Mar 27 20:28:07 2023 +0600

    _getStandardOrEmptyDescriptor (#25283)

commit b9d76e61660b4180efed79cfb19470aad84e7e00
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Mar 27 18:19:49 2023 +0600

    comments (#25283)

commit cc7ff54d6c231285dc38eb048da753cb7ddf59a8
Merge: 8410493f38 7066061152
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Mar 24 21:23:19 2023 +0600

    Merge branch 'issue24538-save-user-code' into issue25283-tobf-sucfuu

commit 8410493f388ab91f42bb04c26a087331dc25f2e5
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Mar 24 20:37:49 2023 +0600

    comments (#25283)

commit 7066061152a1415297be5b1fb4729a3322b9676f
Merge: 879795b632 ddae966f33
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Mar 24 20:03:34 2023 +0600

    Merge branch 'master' into issue24538-save-user-code

commit 8575c6e8f1f4a837c62403e941cf31e1511cf290
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Mar 24 12:33:28 2023 +0600

    addressing comments (#25283)

commit 172f18f33dc5648e0d5a6105430a10e2baaca8fb
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Mar 20 19:26:20 2023 +0600

    comment fixes (#25283)

commit 099b0f19955dc119e0de53a098d424d4cf2d6ad3
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Mar 17 13:28:42 2023 +0600

    changes for demo (#25283)

commit 426d43ee066b02816ceef4deab62f96ee855fdc8
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Mar 15 21:42:15 2023 +0600

    untested multi-file saving (#25283)

commit 879795b632f5863ae9f477c30f40cb18164cb205
Merge: 1521ab9d93 04c2de61e5
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Mar 14 20:39:29 2023 +0600

    Merge branch 'master' into issue24538-save-user-code

commit 8c7807bbb47906cfc4d68c6a30dc740325688bf5
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Mar 2 20:24:01 2023 +0600

    setUnit & HiveLocalStorageCache (#25283)

commit 6b54331285e912d3e989ffcc6d7a5c446616ea2f
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Feb 28 18:32:03 2023 +0600

    unfinished ContentExampleLoadingDescriptor (#25283)

commit 20e097ef00ab04cc25c4fca0598de4d4cd6a7b9a
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Feb 27 12:03:59 2023 +0600

    deleted google_sign_in from pubspec (#25283)

commit bdf6f4948f9a9a8624f2c4417dd9645b6e188f7e
Merge: f755775829 1521ab9d93
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Feb 27 12:01:19 2023 +0600

    Merge branch 'issue24538-save-user-code' into issue25283-tobf-sucfuu

commit 1521ab9d93f1ea1d964a58ac0f7b87213702cd01
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Feb 23 19:36:50 2023 +0600

    CodeBlock widget (#25529)

commit b638355e33cfbc1cd73e8ccaf7b8b65dfa6c7184
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Feb 23 19:34:00 2023 +0600

    added missing notifyListeners (#24538)

commit 5d22b4cda7a944cbbd650e6673ed55d63f4b5dc1
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Feb 23 19:33:01 2023 +0600

    scrollbar in markdown code block (#25529)

commit f7557758290b48fda83b8abaf6487bce36ea6c5e
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Feb 22 11:37:58 2023 +0600

    resolve deprecation warnings todo (#25283)

commit 56264acd2da1a565f3dbb98dab1a41dff54ef54b
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Feb 22 11:30:22 2023 +0600

    flutter v3.7 support (#24538)

commit 08989ab7e0a8bf7623aff19de10e22b38ecb20dd
Merge: 9e3666c706 40838f7644
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Feb 22 11:00:47 2023 +0600

    Merge branch 'master' into issue24538-save-user-code

commit 6860d1b367e739abc73e35e48774869ae016872f
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Feb 20 11:39:36 2023 +0600

    Flutter v3.7.3 support for ToB (#25283)

commit 9e3666c706ff3bc6a3a02ccaf33738a116820502
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Feb 8 17:29:53 2023 +0600

    set sdkId to fix a bug (#24538)

commit 5381d9e0c10d29e2e605c941d0b74d0db16a7b27
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Feb 6 13:37:00 2023 +0600

    _Buttons padding (#24538)

commit 4a419a611c8e7e28e178fb0488bd308d6c52add8
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Feb 6 11:26:47 2023 +0600

    moved hints to the left (#24538)

commit 5637c4171a65ccb4a87019b492245b8cdaec38a5
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Feb 6 11:22:07 2023 +0600

    scrollable hints (#24538)

commit c7c51b1234b87fe374bce6f9b7dc27d299605b00
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Feb 6 11:00:15 2023 +0600

    commit to relaunch RAT check (#24538)

commit 8eed5d3174fdf193b9a4c62631e3687c2c505d6f
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Feb 3 14:53:06 2023 +0600

    comment fixes (#24538)

commit 7f943d51dcc975f39dc666b19e107c8956fb2048
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Feb 3 12:48:46 2023 +0600

    organized TourNotifier (#24538)

commit 52227d538363a74fb96eb7410eb1d606ef34429f
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Feb 3 12:37:55 2023 +0600

     deleted unitProgressByUnitId getter (#24538)

commit 4e3d3cfe134688a8e14813092d1819e97a552b01
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Feb 3 12:31:54 2023 +0600

    alphabetic order (#24538)

commit bfc759666b71f4c265285633f1b57bb3e549eabd
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Feb 3 12:26:10 2023 +0600

    getUnitSavedSnippetId (#24538)

commit a836d0180e328d0bcecf614eb7178734422446ff
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Feb 2 18:57:05 2023 +0600

    comment fixes(#24538)

commit 01611b02f9947d4401ed056528b92e707c619a27
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Feb 1 20:26:36 2023 +0600

    comment fixes (#24538)

    comment fixes (#24538)

commit fff0bfa7e4f3e68ae5ac6e7191f7600b22236dca
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Jan 27 01:29:59 2023 +0600

    save code status (#24538)

commit d4401c9156a8599cb44b2f92e189906615eff5cf
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Jan 26 14:14:18 2023 +0600

    comment fixes (#24538)

commit f3d1c0f66fad5072008434516ec93bb23bf1f138
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Jan 26 10:48:50 2023 +0600

    snippet type enum switcher (#24538)

commit 25dec321066e89e2ce9f1f1574b42d002c61fa80
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Jan 25 00:48:14 2023 +0600

    comment fixes (#24538)

commit f2d2f8fa304b19c6ebf7411745365e5cb59c3557
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Jan 23 21:32:19 2023 +0600

    reverted to unitProgress (#24538)

commit a5310e0eb21d00dc8e3256562e5530bc21fa496c
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Jan 23 21:28:40 2023 +0600

    missing notifyListeners (#24538)

commit 18ecd23b17cc9397311c2a0a51bbef25c62e2e0b
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Jan 23 21:26:52 2023 +0600

     reset is shown immediately (#24538)

commit ea8acc0d68528fd435a2134feb1ad35076114fa4
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Jan 23 18:58:41 2023 +0600

    removed comment after discussion (#24538)

commit 92b4ec47c0526523da672f4472f2776a46e40de5
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Jan 23 09:06:57 2023 +0600

    tested with show solution (#24538)

commit ebce111cc242e6f2323717c6fda0761d1e69bdc9
Author: darkhan.nausharipov <[email protected]>
Date:   Sun Jan 22 22:28:47 2023 +0600

    set show solution false on reset (#24538)

commit ece905cfb6d8abe5fc64f3f8d023699a17047fd5
Author: darkhan.nausharipov <[email protected]>
Date:   Sun Jan 22 22:08:53 2023 +0600

    pre-review fixes (#24538)

commit 725e8d108678c0b9da1718a478468207c1e4c446
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Jan 20 19:06:51 2023 +0600

    post-merge commit (#24538)

commit b6cb6b93e92df4571df84bdbc524a815d7ca2308
Merge: eb23673d01 428ec97e30
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Jan 20 19:06:03 2023 +0600

    Merge branch 'master' into issue24538-save-user-code

commit eb23673d0178f90b313fb18c591505f8952eb24b
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Jan 20 18:39:35 2023 +0600

    setSaveCodeListener to activeFileController.codeController (#24538)

commit d3b0085253c985e52351bf541587d447db08c94d
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Jan 17 15:28:17 2023 +0600

    merge fixes (#24538)

commit 8cd106d00f6369b8dec1f92ca44190ded3378d30
Merge: a8502624ca a5e6d90081
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Jan 17 15:27:42 2023 +0600

    Merge branch 'master' into issue24538-save-user-code

commit a8502624ca74909e27e5f90508c0bea22619b382
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Dec 27 19:06:23 2022 +0600

    getUnitSnippets in unit progress (#24538)

commit 8dd70984cdb3aebd676c2d92f4508b3b0d6a7ae5
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 8 22:33:32 2022 +0600

    user menu without padding

commit 90b9e3d49adf6c075fbdeef881ee161b109fd1df
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 8 22:32:37 2022 +0600

    save user code barely works

commit 44172c57c2d806b6521fc966febabb949e3e77a7
Merge: d442e63668 59d0fd3129
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 8 14:15:37 2022 +0600

    Merge branch 'issue24394-tobf-hint' into issue24538-tobf-save-user-code

commit d442e63668b15330bd8a68fb18312f5cc06e5640
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Dec 7 15:22:57 2022 +0600

    no duplicate slash with reg exp (#24538)

commit e76860497a6983cd28e29f3d4065c9331ac32660
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Dec 6 10:11:12 2022 +0600

    updated import (#24538)

commit 63745d564ace082d34be5360cfd264cfaef6e86a
Merge: d15913b289 af936d0b11
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Dec 6 10:02:57 2022 +0600

    Merge branch 'issue23692_auth' into issue24538-tobf-save-user-code

commit 59d0fd3129bcab3b55a66f671898ce6c5bcc84df
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Dec 2 20:50:12 2022 +0600

    solution managed inside of tour notifier (#24394)

commit 5ca484661d330a0d7bea371a767f8d37402db520
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Dec 2 15:57:30 2022 +0600

    show solution snippets (#24394)

commit 5e3ac5857e6fbe585e8f6b43cca75265b59e25c2
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Dec 2 14:20:43 2022 +0600

    renamed to unit content widget (#24394)

commit 372b97fb5cd457125137c3ceb71332acecbc9117
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Dec 2 12:01:39 2022 +0600

    group progress indicator size adjustment (#24394)

commit 48ea57d7c97a817458741d7062955add3b758c76
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Dec 2 11:50:08 2022 +0600

    title in unit content (#24394)

commit 338c67317cdc9eec5a4046af9e5599737408baf2
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Dec 2 11:01:57 2022 +0600

    extracted widgets, hints emptiness check (#24394)

commit 36d3105f52fad0aa37ddaeaac340ee31a8a4e86b
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 1 21:12:05 2022 +0600

    hint in dialog (#24394)

commit af936d0b115bf42f21310719dc1b18e580010421
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 1 14:59:52 2022 +0600

    added firebase_options.dart into gradle rat exclusions & added a missing license (#23692)

commit 9e1bbb081c575358445750694d27e393ebd72406
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 1 14:02:46 2022 +0600

    specific imports (#23692)

commit 9720c134b5733fb439ae2b4df12bff07c359a95b
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 1 13:06:57 2022 +0600

    moved dismissible overlay (#23692)

commit 869b40cefa9222b5a61db0125f6c54340dfc918c
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 1 13:06:40 2022 +0600

    extracted overlay body (#23692)

commit c8db1ef778f401a9029c113a784faeb94919d660
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 1 13:05:12 2022 +0600

    refined open overlay  (#23692)

commit 8b639d08b3d6101ba9b13a6f6fe32b7cc7261559
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 1 12:50:35 2022 +0600

    all caches extend cache (#23692)

commit a10795205507a66b8cd90ed8caaf112663a60c8c
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Dec 1 12:47:19 2022 +0600

    renamed user progress cache to unit progress (#23692)

commit e215ec4c676e88d85fad3ba08e222ac19d8953b8
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Nov 30 10:37:05 2022 +0600

    generated files

commit 96e41f7d01c50e2cfa96fbc120fca69b3d29cb39
Merge: 1e38328e56 37fb90c98d
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Nov 30 10:30:05 2022 +0600

    Merge from master

commit 1e38328e564f72b4e18439c60610902423223dbd
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 29 15:40:15 2022 +0600

    removed "fix exception" todo after filing an issue (#23692)

commit b6c4e3e66c30fa4cbdfbbefd1a41e649c32b9ecb
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 29 13:52:29 2022 +0600

    added async (#23692)

commit 27b26e41a81f07faac060c3d2e45f5b7bea19d5a
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 29 13:31:24 2022 +0600

    clearUpdatingUnitId (#23692)

commit 4b2d855920d6d63dfff4412fc9bdfb9ddbbdd695
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 29 13:30:12 2022 +0600

    renamed user progress model to unit progress (#23692)

commit ef637403aa114f43f82ac005254632f2d35e0bad
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 29 13:09:58 2022 +0600

    rearranged completeUnit (#23692)

commit fb1fea1af8069ae02ba261781ca923cd40796c97
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 29 11:43:06 2022 +0600

    comment fixes (4)

commit 749d8b19bea072be84cace76bd1fb998a3a76531
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 29 11:03:34 2022 +0600

    pubspec.lock

commit 0a76b4019647ebcbec76739b078b01fddd37789b
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 29 11:03:17 2022 +0600

    pubspec.lock ignored only in PGC

commit c6004b0f220be0959b1fadc1fcffb04e4004cd97
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 28 21:17:16 2022 +0600

    no final else

commit d55622e30df308eb61a42ecd1f03930012ec0cb8
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 28 21:15:15 2022 +0600

    missing await

commit 65dc5219943a948e7922da44ce34b54474bffaf1
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 28 21:12:22 2022 +0600

    updating & blocking complete unit button (#23692)

commit 1bbc09279604dd2025ca6c93cfc85d2df9bece09
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 28 14:36:25 2022 +0600

    comment fixes (3)

commit 860780836bdf8cdbbcbe5952232abc3544effc9e
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 28 12:44:52 2022 +0600

    pubspec.lock in gitignore

commit a575717869b239f59374caf74f81db74191c3781
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 28 12:41:44 2022 +0600

    deleted pubspec.lock

commit c2f74a59ed6a364d04a9cb0d11ea0e1f0ce5ffbf
Author: darkhan.nausharipov <[email protected]>
Date:   Sun Nov 27 21:05:49 2022 +0600

    review comments (2)

commit adbdeb6eec823178d487a3a3b3bba07ceda6efa9
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Nov 24 15:15:57 2022 +0600

    deleted file (#23692)

commit 6ed38acb2beb4b960eb7a77da1282683e2805bc0
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Nov 24 15:14:07 2022 +0600

    unused import (#23692)

commit 2c90ced90376df81d0989c233995320a73c7feb1
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Nov 24 14:55:05 2022 +0600

    login_overlay (#23692)

commit 6d21e1ea3f0837616a185e4b1f0d48398799791d
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Nov 24 14:52:08 2022 +0600

    kOpenLoginOverlay (#23692)

commit 64965a1e1f49f0240b4e27a387aad099caade1b2
Author: darkhan.nausharipov <[email protected]>
Date:   Thu Nov 24 14:44:19 2022 +0600

    canCompleteCurrentUnit (#23692)

commit 7fbaf3843394530602a87805fa274ace7620ae29
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Nov 23 15:21:56 2022 +0600

    new configs & todos

commit 0b6fc8f805d3417f819ebafadc40f2571d73df99
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 22 21:07:35 2022 +0600

    candidate repository

commit 3df27bc7fc71f12d48b3aa0bf5d58e57ba1327b9
Author: darkhan.nausharipov <[email protected]>
Date:   Tue Nov 22 15:12:05 2022 +0600

    untested refinement (1)

commit 52838e500fc106414dbb47d0c05fe4042c812509
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 21 21:54:53 2022 +0600

    notifier objects naming

commit 2068ce9e8da8b6caa41e1be09dfa04e26e2ca567
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 21 21:50:25 2022 +0600

    deleted show sdk selector

commit a6597d434a3aed640fc230976afd3637217d8792
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 21 21:44:55 2022 +0600

    untested refinement

commit 10b287b89456368b146fd9bc1edc38bf0c28fc73
Merge: 4aedd5a607 5ad067f7cc
Author: Darkhan Nausharipov <[email protected]>
Date:   Mon Nov 21 19:06:30 2022 +0600

    Merge pull request #324 from akvelon/tour-header-sdk-selector

    SDK selector in tour scaffold

commit 5ad067f7ccdb976fcdb26102d983bea936a9ba83
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 21 17:55:38 2022 +0600

    non-nullable onChanged

commit 4aedd5a6072e92dbfd7487652619e7a14e291375
Merge: 6f53e30d39 70c0d61d3a
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 21 16:06:38 2022 +0600

    SDK selector with fixed comments

commit 6f53e30d39ba793850153a2a6077fbe6bbab2b51
Author: darkhan.nausharipov <[email protected]>
Date:   Mon Nov 21 16:04:31 2022 +0600

    cache license

commit 70c0d61d3a96e120d312f1aecc6a38e73e551697
Author: darkhan.nausharipov <[email protected]>
Date:   Sat Nov 19 20:46:18 2022 +0600

    comments (0)

commit de273baf06cc76e4da1f94064131f0093e265f0c
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Nov 18 20:51:31 2022 +0600

    hide back button after navigating from welcome

commit 3a10e8d7633bb4d6d9bd39ba5515aacae6208554
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Nov 18 20:48:13 2022 +0600

    show sdk selector in welcome screen

commit de0f7c34f2a686f22137c464d8815a5f6c2cb9c4
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Nov 18 20:00:17 2022 +0600

    sdk selection works with auth

commit 3c534d864b5a36cf4f63b64af0332753fd420b9a
Merge: 12a685d1f0 b78bbd33dc
Author: Darkhan Nausharipov <[email protected]>
Date:   Fri Nov 18 19:41:06 2022 +0600

    Merge remote-tracking branch 'origin/tour-header-sdk-selector' into auth_sdk_demo

commit 12a685d1f00140981c8241a2477b5ee0707e47be
Author: darkhan.nausharipov <[email protected]>
Date:   Fri Nov 18 19:36:34 2022 +0600

    unit file fix

commit d25eaf52df834b655421319b14e43e0afc7c63f1
Merge: 5297d2548c cf56af2799
Author: Darkhan Nausharipov <[email protected]>
Date:   Fri Nov 18 19:22:22 2022 +0600

    Merge branch 'master' into issue23692_auth

commit 5297d2548ced6027c58b75b9d917fbfeb7052369
Author: darkhan.nausharipov <[email protected]>
Date:   Wed Oct 19 19:50:43 2022 +0600

    auth, complete unit, user progress

    AuthNotifier draft (#23692)

    Comments (#23692)

    Comments (#23692)(1)

    sign in with google works (#23692)

    new configs (#23692)

    get user progress draft (#23692)

    comment fixes (#23692)

    sign in in IntroTextBody (#23692)

    reverted config (#23692)

    comment fixes (#23692)

    WIP before rebase (merge) (#23692)

    Squashed commit of the following:

    commit bff4919ff00ec3b5d7186efde41c884dfc4c8344
    Merge: 79ba69483a ce8d618c77
    Author: Alexey Romanenko <[email protected]>
    Date:   Thu Nov 17 10:34:02 2022 +0100

        Merge pull request #24186: Uses _all to follow alias/datastreams when estimating index size

    commit 79ba69483a84ea0278d0b0ddb141200739607c77
    Merge: 245fea9040 b7e860a762
    Author: Chamikara Jayalath <[email protected]>
    Date:   Wed Nov 16 20:47:40 2022 -0800

        Merge pull request #24218: Update Python wheel format for RC validation

    commit 245fea904014cd58d4148807463dbaa40000774c
    Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Date:   Wed Nov 16 18:12:33 2022 -0800

        Bump loader-utils from 1.4.1 to 1.4.2 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel (#24191)

        Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

    commit e1de8e78deeb5d17617fda6591429eaaf8abb8a2
    Author: Yi Hu <[email protected]>
    Date:   Wed Nov 16 20:48:06 2022 -0500

        Fix PythonLint (#24219)

    commit b7e860a7621771c300dcec625655f87e62591323
    Author: Chamikara Jayalath <[email protected]>
    Date:   Wed Nov 16 17:28:31 2022 -0800

        updates

    commit c2feb09ea49dd815b69c65e531ce34128756d988
    Author: Chamikara Jayalath <[email protected]>
    Date:   Wed Nov 16 17:06:08 2022 -0800

        updates

    commit ce8d618c77d23e20a1ddb128bb8183048597d096
    Author: egalpin <[email protected]>
    Date:   Wed Nov 16 16:43:57 2022 -0800

        Adds test for following aliases when estimating index size

    commit 959719d01c627328c0ca2849d2b7e2c9b322d4d1
    Author: Chamikara Jayalath <[email protected]>
    Date:   Wed Nov 16 15:16:06 2022 -0800

        Temporary update Python RC validation job

    commit b952b41788acc20edbe5b75b2196f30dbf8fdeb0
    Author: Yi Hu <[email protected]>
    Date:   Wed Nov 16 14:18:12 2022 -0500

        Python TextIO Performance Test (#23951)

        * Python TextIO Performance Test

        * Add filebasedio_perf_test module for unified test framework for
          Python file-based IOs

        * Fix MetricsReader publishes metrics duplicately if more than one
          load test declared. This is because MetricsReader.publishers was
          static class variable

        * Fix pylint

        * Distribute Python performance tests random time at a day instead of all at 3PM

        * Add information about length conversion

    commit 017f2cbde124af40a43be99ec88289fcf63c1c95
    Merge: fef8acdbc0 88dba4f494
    Author: Chamikara Jayalath <[email protected]>
    Date:   Wed Nov 16 10:39:52 2022 -0800

        Merge pull request #24187: Add a reference to Java RunInference example

    commit fef8acdbc0ecbcc85b49144adaf8830e3bc6b2de
    Merge: 6e9187e67e ead245539d
    Author: Ahmet Altay <[email protected]>
    Date:   Wed Nov 16 10:24:53 2022 -0800

        Merge pull request #24199 from Laksh47/issue#24196

        refs: issue-24196, fix broken hyperlink

    commit 6e9187e67e1bd8f73997f437f0ed4c29880ed73b
    Author: Darkhan Nausharipov <[email protected]>
    Date:   Wed Nov 16 22:33:50 2022 +0600

        [Tour of Beam] [Frontend] Content tree URLs (#23776)

        * Content tree navigation (#23593)

        Unit content navigation (#23593)

        Update URL on node click (#23593)

        Active unit color (#23593)

        removeListener in unit (#23593)

        First unit is opened on group title click (#23593)

        WIP by Alexey Inkin (#23593)

        selectedUnitColor (#23593)

        Unit borderRadius (#23593)

        RegExp todo (#23593)

        added referenced collection package to remove warning (#23593)

        small refinement (#23593)

        expand on group tap, padding, openNode (#23593)

        group expansion bug fix (#23593)

        selected & unselected progress indicators (#23593)

        * AnimatedBuilders instead of StatefulWidgets in unit & group (#23593)

        * fixed _getNodeAncestors (#23593)

        * get sdkId (#23593)

        * addressing comments (#23593)

        * sdkId getter & StatelessExpansionTile (#23593)

        * expand & collapse group (#23593)

        * StatelessExpansionTile (#23593)

        * license (#23593)

        * ValueChanged and ValueKey in StatelessExpansionTile (#23593)

        Co-authored-by: darkhan.nausharipov <[email protected]>
        Co-authored-by: Alexey Inkin <[email protected]>

    commit b33fac2aa533d77cfa47f88466c8cd6bd3f3e864
    Author: Bruno Volpato <[email protected]>
    Date:   Wed Nov 16 10:51:11 2022 -0500

        Use only ValueProviders in SpannerConfig (#24156)

    commit 5f013ab6567ec75b460b2081d7f89d332320caff
    Author: Robert Burke <[email protected]>
    Date:   Wed Nov 16 07:23:10 2022 -0800

        revert upgrade to go 1.19 for action unit tests (#24189)

    commit 9337f4dbecc929886f8559949a082a649fd9d1bb
    Author: Yi Hu <[email protected]>
    Date:   Wed Nov 16 10:18:42 2022 -0500

        Fix Python PostCommit Example CustomPTransformIT on portable (#24159)

        * Fix Python PostCommit Examples on portable

        * Fix custom_ptransform pipeline options gets modified

        * Specify flinkConfDir

    commit ead245539d01dec0f3e08699c1e1cc6777a5ef0e
    Author: Laksh <[email protected]>
    Date:   Wed Nov 16 09:32:46 2022 -0500

        refs: issue-24196, fix broken hyperlink

    commit e83a996d4374d467d95bcfad7166905622ec615c
    Merge: 2fc56ec663 ffdee0b6ed
    Author: Jan Lukavský <[email protected]>
    Date:   Wed Nov 16 15:15:31 2022 +0100

        Merge pull request #24192: Re-use serializable pipeline options when already available.

    commit ffdee0b6edb8638c78a65ec85c727ea5dde1cb2f
    Author: Jozef Vilcek <[email protected]>
    Date:   Mon Nov 14 16:48:18 2022 +0100

        Re-use serializable pipeline options when already available (#24192)

    commit 88dba4f494829b2b3530b767fb8c5252e0d2ba44
    Author: Chamikara Jayalath <[email protected]>
    Date:   Tue Nov 15 16:21:22 2022 -0800

        Add a reference to Java RunInference example

    commit 2fc56ec663e335cfcf37dc57d471f79b601414f4
    Merge: f763186987 83f1bc19b9
    Author: Kenn Knowles <[email protected]>
    Date:   Tue Nov 15 16:16:47 2022 -0800

        Merge pull request #24142: Fix arguments to checkState in BatchViewOverrides

    commit f763186987c00ba1d26efdc35406436a1fa69a9a
    Merge: c2bc2135e9 0d7ca04182
    Author: Ning Kang <[email protected]>
    Date:   Tue Nov 15 15:25:20 2022 -0800

        Addresses #24161 Updated README of Interactive Beam

    commit c2bc2135e9bce715990a5d5551e2bc2dc0311da4
    Author: Doug Judd <[email protected]>
    Date:   Tue Nov 15 14:48:26 2022 -0800

        Strip FGAC database role from changestreams metadata requests (#24177)

        Co-authored-by: Doug Judd <[email protected]>

    commit af637974f96ad1b5110d7dea3f9a26c68e19a51b
    Author: Jack McCluskey <[email protected]>
    Date:   Tue Nov 15 17:16:43 2022 -0500

        Add custom inference function support to the PyTorch model handler (#24062)

        * Initial type def and function signature

        * [Draft] Add custom inference fn support to Pytorch Model Handler

        * Formatting

        * Split out default

        * Remove Keyed version for testing

        * Move device optimization

        * Make default available for import, add to test classes

        * Remove incorrect default from keyed test

        * Keyed impl

        * Fix device arg

        * custom inference test

        * formatting

        * Add helpers to define custom inference functions using model methods

        * Trailing whitespace

        * Unit tests

        * Fix incorrect getattr syntax

        * Type typo

        * Fix docstring

        * Fix keyed helper, add basic generate route

        * Modify generate() to be different than forward()

        * formatting

        * Remove extra generate() def

    commit a014637106970a0a0e9eb7944aa5caf79fa5fd37
    Author: egalpin <[email protected]>
    Date:   Tue Nov 15 13:57:54 2022 -0800

        Uses _all to follow alias/datastreams when estimating index size

        Fixes #24117

    commit 0d7ca041823bc2b09f76f86fdfd1d0b9508c9c88
    Author: Ning Kang <[email protected]>
    Date:   Tue Nov 15 13:57:27 2022 -0800

        Minor update

    commit e8fc759d756f4a987e41d2b9da56b906a6cd7736
    Author: Ning Kang <[email protected]>
    Date:   Tue Nov 15 13:52:18 2022 -0800

        Updated README of Interactive Beam

        Removed deprecated cache_dir runner param in favor of the cache_root global option.

    commit 08d5f72e5f35d41f3e9fa9fe799caea6bed1b7a7
    Author: Anand Inguva <[email protected]>
    Date:   Tue Nov 15 16:34:21 2022 -0500

        [Python]Support pipe operator as Union (PEP -604) (#24106)

        Fixes https://github.com/apache/beam/issues/21972

    commit 526e7a58b62682582c27173ab21ed8667ddab766
    Author: Scott Strong <[email protected]>
    Date:   Tue Nov 15 16:26:45 2022 -0500

        Using Teardown context instead of deprecated finalize (#24180)

        * Using Teardown context instead of deprecated finalize

        * making function public

        Co-authored-by: Scott Strong <[email protected]>

    commit fb4d1d4dea7b26ed538a9f6aca0ed41e8c300e37
    Author: Danny McCormick <[email protected]>
    Date:   Tue Nov 15 16:25:22 2022 -0500

        Fix broken json for notebook (#24183)

    commit f98db2008a97f4546d036ddf0dddfee8c87eb58a
    Author: Robert Burke <[email protected]>
    Date:   Tue Nov 15 12:49:23 2022 -0800

        Update automation to use Go 1.19 (#24175)

        Co-authored-by: lostluck <[email protected]>

    commit e5f58504eef1fdeebe0402cda8a2df259169c704
    Author: Brian Hulette <[email protected]>
    Date:   Tue Nov 15 12:25:13 2022 -0800

        Add error reporting for BatchConverter match failure (#24022)

        * add error reporting for BatchConverters

        * Test pytorch

        * Finish up torch tests

        * yapf

        * yapf

        * Remove else

    commit 3037747f66f0d71d65b6c65745b4f8942c22f05a
    Author: Danny McCormick <[email protected]>
    Date:   Tue Nov 15 14:13:04 2022 -0500

        Fix broken notebook (#24179)

    commit b2b1c739ce37690923891934ee317f799db937a2
    Author: MakarkinSAkvelon <[email protected]>
    Date:   Tue Nov 15 21:53:06 2022 +0500

        [Playground] Move Playground in GKE and Infrastructure change (#23928)

        * changes to updated master branch

        * Change workflow

        * ingress changes

        * Certificate was added

        * Updates for cloud build backend

        * Update main.tf

        * Create main.tf

        * Create variables.tf

        * Update variables.tf

        * Update main.tf

        * Update variables.tf

        * Update main.tf

        * Create output.tf

        * Update output.tf

        * Update output.tf

        * Update main.tf

        * Update build.gradle.kts

        * Update output.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update variables.tf

        * Update main.tf

        * Update variables.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update variables.tf

        * Update main.tf

        * Update variables.tf

        * Update main.tf

        * Update variables.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update main.tf

        * Update output.tf

        * Update main.tf

        * Update main.tf

        * Update output.tf

        * Create variables.tf

        * Update main.tf

        * Update main.tf

        * Delete playground/terraform/infrastructure/cluddns directory

        * Update main.tf

        * Update output.tf

        * Update output.tf

        * Update build.gradle.kts

        * Update build.gradle.kts

        * Update build.gradle.kts

        * Update build.gradle.kts

        * Update build.gradle.kts

        * Update build.gradle.kts

        * Update README.md

        * Update README.md

        * helm folder name was changed

        * Update README.md

        * Update build.gradle.kts

        * Update build.gradle.kts

        * Update build.gradle.kts

        * Updates to readme

        * Fix DNS name

        * HelmChart was changed

        * Some workflows were changed

        * Remove unused file

        * playground-examples return

        * add license information

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * remove "stg" folder

        * Update README.md

        * Remove unused files

        * DNS Removed

        * var name changed

        * remove DNSName from var file

        * 1

        * Clear terraform

        * remove unused records

        * gradle check

        * grade last change

        * issue fix

        * fix

        * 1

        * run

        * test

        * Index creation for Gradle

        * Add IndexCreation in gradle

        * Update README.md

        * Update README.md

        * Fix names for Frontend

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Update README.md

        * Redis fix

        * services fix

        * Update variables.tf

        * change order in gradle

        * Fix Config.g.dart file issue

        * Update README.md

        * Playground workflow update

        Co-authored-by: Sergey Makarkin <[email protected]>
        Co-authored-by: Sergey Makarkin <[email protected]>
        Co-authored-by: ruslan-ikhsan <[email protected]>
        Co-authored-by: Alex Kosolapov <[email protected]>

    commit 85df5f2eb2f299e28b36be0cce7b9c19d62124da
    Author: Yi Hu <[email protected]>
    Date:   Tue Nov 15 11:38:13 2022 -0500

        Eliminate CalciteUtil.CharType logical type (#24013)

        * Eliminate CalciteUtils.CharType logical type

        * Replace CalciteUtils.CharType to String
          Note that CalciteUtils still omits the precision of BINARY/VARBINARY/CHAR/VARCHAR
          as what it originally did. Support of the precision of these calcite types involves
          make use of making use of the overload method RelDataTypeFactory.createSqlType(var1, var2).

        * Replace every reference of CalciteUtil.CharType to generic
          PassThroughLogicalType check

        * Add TODO to Support sql types with arguments

        * Use VariableString in LogicalTypeTestCase

    commit f349f41010c5b238ff6020f7de718f938eef3c5e
    Author: alexeyinkin <[email protected]>
    Date:   Tue Nov 15 20:04:01 2022 +0400

        Configure flutter_code_editor options with Hugo shortcode (#23926) (#24031)

        * Configure flutter_code_editor options with Hugo shortcode (#23926)

        * Minor fixes (#23926)

        * Refactor after review (#23926)

    commit 0f4ca6363b3ce0e5de3ad36517bb406aa6391a18
    Author: Rebecca Szper <[email protected]>
    Date:   Tue Nov 15 06:10:13 2022 -0800

        Editorial review of the ML notebooks. (#24125)

        * Editorial review of the ML notebooks.

        * Editorial review of the ML notebooks.

        * Editorial review of the ML notebooks.

        * Update examples/notebooks/beam-ml/custom_remote_inference.ipynb

        Co-authored-by: Danny McCormick <[email protected]>

        * Updating based on feedback

        * Update examples/notebooks/beam-ml/run_inference_sklearn.ipynb

        Co-authored-by: Danny McCormick <[email protected]>

        * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

        Co-authored-by: Danny McCormick <[email protected]>

        * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

        Co-authored-by: Danny McCormick <[email protected]>

        * Update examples/notebooks/beam-ml/run_inference_tensorflow.ipynb

        Co-authored-by: Danny McCormick <[email protected]>

        * Updating based on feedback

        Co-authored-by: Danny McCormick <[email protected]>

    commit 5bd34ede026253326ebff1a7e4f9edb5f71b4a2c
    Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Date:   Tue Nov 15 07:17:28 2022 -0500

        Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#24131)

        Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.3.2 to 1.11.39.
        - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
        - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
        - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.3.2...feature/s3/manager/v1.11.39)

        ---
        updated-dependencies:
        - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
          dependency-type: direct:production
          update-type: version-update:semver-minor
        ...

        Signed-off-by: dependabot[bot] <[email protected]>

        Signed-off-by: dependabot[bot] <[email protected]>
        Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

    commit 2ee809fa0ca7689dd0279e186ebc02d9569a8429
    Merge: e3b9bdb2e6 563c66d6fd
    Author: Alexey Romanenko <[email protected]>
    Date:   Tue Nov 15 11:01:14 2022 +0100

        Merge pull request #23065: [Website] Update copy icon styles

    commit e3b9bdb2e607d85a4017ba7839000e92a0ad83c4
    Author: Moritz Mack <[email protected]>
    Date:   Tue Nov 15 10:40:50 2022 +0100

        [Dockerized Jenkins] Fix build of dockerized jenkins (fixes #24053) (#24054)

    commit faaac2ab6e010374cb2be0e95a5dd345836a2a2c
    Author: Moritz Mack <[email protected]>
    Date:   Tue Nov 15 10:38:59 2022 +0100

        [Dockerized Jenkins] Update README how to use local repo (#24055)

    commit 689e70b5131620540faf52e2f1e2dca7a36f269d
    Author: Damon <[email protected]>
    Date:   Mon Nov 14 17:34:29 2022 -0800

        Implement embedded WebAssembly example (#24081)

    commit e1bf6c42950e8013f35e35fb9fee8017e01e5010
    Merge: eddac84126 10337d2868
    Author: Robert Bradshaw <[email protected]>
    Date:   Mon Nov 14 15:22:14 2022 -0800

        Merge pull request #24160 Rename the test_splits flag to direct_test_splits.

    commit eddac841261228a2c63fa9b225c520ae0f853806
    Author: Pablo <[email protected]>
    Date:   Mon Nov 14 15:05:05 2022 -0800

        More dataset templates to clean up (#24162)

    commit 2adb68bd12743566cc89b596bf204d7c807eb62d
    Author: Pablo <[email protected]>
    Date:   Mon Nov 14 13:28:13 2022 -0800

        Adding a quickstart to README for the TS SDK (#23509)

        * More of a quickstart for the TS SDK

        * Update sdks/typescript/README.md

        Co-authored-by: Danny McCormick <[email protected]>

        * Update sdks/typescript/README.md

        Co-authored-by: Danny McCormick <[email protected]>

        Co-authored-by: Danny McCormick <[email protected]>

    commit 10337d28685ad5712e2ad8608977ec5c5e0e6b6b
    Author: Robert Bradshaw <[email protected]>
    Date:   Mon Nov 14 12:46:32 2022 -0800

        Rename the test_splits flag to direct_test_splits.

        This avoids possible flag conflicts.

    commit 48c70cc30742b45b17a1d18ece2f0d079bee3915
    Author: arne-alex <[email protected]>
    Date:   Mon Nov 14 21:33:02 2022 +0100

        Merge pull request #23333: Track time on Cloud Dataflow streaming data reads and export via heartbeats

    commit 9c83de646ab52bd0b05e3346190dd55cd68b2a8b
    Author: Johanna Öjeling <[email protected]>
    Date:   Mon Nov 14 21:19:44 2022 +0100

        Add more tests for S3 filesystem (#24138)

    commit 9e9c6d797ba52b460f83131431c8e53aebbbc9ac
    Merge: d5d76b9745 c600444e1d
    Author: Ning Kang <[email protected]>
    Date:   Mon Nov 14 12:06:15 2022 -0800

        Merge pull request #24029 from apache/dependabot/npm_and_yarn/sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/loader-utils-1.4.1

        Bump loader-utils from 1.4.0 to 1.4.1 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel

    commit d5d76b974592d45de368ab641647ca5cc4ec12ec
    Author: Yi Hu <[email protected]>
    Date:   Mon Nov 14 15:03:28 2022 -0500

        Support SqlTypes Date and Timestamp (MicrosInstant) in AvroUtils (#23969)

        * Support SqlTypes Date and Timestamp (MicrosInstant) in AvroUtils

        * Add TODO about java.time migration

    commit 330cc2010c9f4a2d4e30318bf50a4109ec1cd392
    Author: Pablo <[email protected]>
    Date:   Mon Nov 14 12:02:10 2022 -0800

        Cleanup stale BQ datasets (#24158)

        * Cleanup stale BQ datasets

        * addressing comments

    commit 4a044999b8ed4bcd41f816f3a23ccb5da00c4c38
    Merge: e563b9dd2f 5bd75c25de
    Author: Heejong Lee <[email protected]>
    Date:   Mon Nov 14 11:16:00 2022 -0800

        Merge pull request #24076 from chamikaramj/multilang_java_updates

        Updates Multi-lang Java quickstart

    commit e563b9dd2f3aa0484e6cdc08869991b5e438023e
    Author: Evgeny Antyshev <[email protected]>
    Date:   Mon Nov 14 20:56:35 2022 +0300

        [Tour Of Beam] verify that unit exists when saving progress (#24118)

        * AIO

        * Update learning/tour-of-beam/backend/integration_tests/auth_test.go

        Co-authored-by: Danny McCormick <[email protected]>

        * nit

        Co-authored-by: Danny McCormick <[email protected]>

    commit 774923e0dd089de870bfa5c77063ae2b28f79347
    Merge: 71785de528 1ad0cbc445
    Author: Kenn Knowles <[email protected]>
    Date:   Mon Nov 14 09:52:26 2022 -0800

        Merge pull request #24141: Fix checkArgument format in GcsPath

    commit 71785de52864313c2e3b14fe72a2a63281343617
    Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Date:   Mon Nov 14 11:54:37 2022 -0500

        Bump github.com/aws/aws-sdk-go-v2/config from 1.17.10 to 1.18.0 in /sdks (#24151)

        Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.17.10 to 1.18.0.
        - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
        - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
        - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.17.10...config/v1.18.0)

        ---
        updated-dependencies:
        - dependency-name: github.com/aws/aws-sdk-go-v2/config
          dependency-type: direct:production
          update-type: version-update:semver-minor
        ...

        Signed-off-by: dependabot[bot] <[email protected]>

        Signed-off-by: dependabot[bot] <[email protected]>
        Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

    commit 50d591d6cb3e799bee4e29dfc593c693a86e6276
    Author: Bruno Volpato <[email protected]>
    Date:   Mon Nov 14 11:50:01 2022 -0500

        Change DataflowBatchWorkerHarness doWork error level to INFO (#24135)

    commit 5a72696bfda09fdb905ba8e58b636f8494ef955f
    Merge: ee0a5836d6 0633fe9634
    Author: Kenn Knowles <[email protected]>
    Date:   Mon Nov 14 08:12:12 2022 -0800

        Merge pull request #24149: Remove extraneous jetbrains annotation

    commit ee0a5836d69b776834eb3bd9b2bd02eb5252c333
    Merge: d001a69e1a 137799672e
    Author: Kenn Knowles <[email protected]>
    Date:   Mon Nov 14 08:11:00 2022 -0800

        Merge pull request #24132: Fix checkArgument format string in AvroIO

    commit d001a69e1a58701d6ed4fcb5e3fb7a0921301dad
    Author: Yi Hu <[email protected]>
    Date:   Mon Nov 14 10:56:54 2022 -0500

        Test Dataproc 2.1 with Flink load tests (#24129)

        * Test Dataproc 2.1 with Flink load tests

        * Minor fix flink_cluster script

    commit caabd9be52887ad70c8a4269395c893811ac6a84
    Author: Israel Herraiz <[email protected]>
    Date:   Mon Nov 14 16:03:39 2022 +0100

        Make MonotonicWatermarkEstimator work like its Java SDK equivalent (#24146)

        * Make MonotonicWatermarkEstimator work like its Java SDK equivalent

        The current implementation of MonotonicWatermarkEstimator raises an exception
        with late messages, which makes the watermark estimator barely usable in real
        world scenarios.

        This PR fixes #20041 by making this watermark estimator work like its Java SDK
        equivalent (`WatermarkEstimators.MonotonicallyIncreasing`).

        * Update unit tests too

        * Make linter happy

    commit 451f6b3e7f58d0a3782ad942c6a1fd9f63932024
    Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Date:   Mon Nov 14 09:48:23 2022 -0500

        Bump golang.org/x/net from 0.1.0 to 0.2.0 in /sdks (#24153)

        Bumps [golang.org/x/net](https://github.com/golang/net) from 0.1.0 to 0.2.0.
        - [Release notes](https://github.com/golang/net/releases)
        - [Commits](https://github.com/golang/net/compare/v0.1.0...v0.2.0)

        ---
        updated-dependencies:
        - dependency-name: golang.org/x/net
          dependency-type: direct:production
          update-type: version-update:semver-minor
        ...

        Signed-off-by: dependabot[bot] <[email protected]>

        Signed-off-by: dependabot[bot] <[email protected]>
        Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

    commit 2bb03d62e2d7dc2d8e39040fc9adebccbde74fde
    Merge: 4e39ef2041 623083cd0a
    Author: Alexey Romanenko <[email protected]>
    Date:   Mon Nov 14 15:01:13 2022 +0100

        Merge pull request #24000: [Website] Change headers size from h4,h3 to h2

    commit 563c66d6fd32165da14a07747f2764c17a5d24ea
    Author: bulat safiullin <[email protected]>
    Date:   Wed Sep 7 18:28:42 2022 +0600

        [Website] update pre tag copy link styles #23064

    commit 4e39ef20410ee51c6040317bcd60171e64c5171f
    Merge: 223768f782 105ed6fedc
    Author: Alexey Romanenko <[email protected]>
    Date:   Mon Nov 14 10:55:33 2022 +0100

        Merge pull request #24115: [Website] update go-dependencies.md java-dependencies.md links

    commit 223768f782f771f0033b8d0686d86cf4c71fad75
    Merge: aa0a35dabf a9da2abee6
    Author: Kenn Knowles <[email protected]>
    Date:   Sun Nov 13 18:53:13 2022 -0800

        Merge pull request #24136: Fix checkArgument format string in ExecutionStateTracker

    commit 0633fe9634fe61df7cbc0ecac205d81124fd504a
    Author: Kenneth Knowles <[email protected]>
    Date:   Sat Nov 12 15:15:16 2022 -0800

        Remove extraneous jetbrains annotation

    commit 83f1bc19b95935e60ca1f4027d4b60c7e738a84a
    Author: Kenneth Knowles <[email protected]>
    Date:   Sat Nov 12 14:16:09 2022 -0800

        Fix arguments to checkState in BatchViewOverrides

    commit 1ad0cbc44594d8405bf4b07a126265238013a02a
    Author: Kenneth Knowles <[email protected]>
    Date:   Sat Nov 12 13:41:02 2022 -0800

        Fix checkArgument format in GcsPath

    commit aa0a35dabf9c2a0d9822faff06d939d9a77a3ab6
    Author: Kenn Knowles <[email protected]>
    Date:   Fri Nov 11 20:26:30 2022 -0800

        Fix checkArgument format string in TestStream (#24134)

    commit a9da2abee6455bc2cf0f18ba5f6cd7bbaeae669f
    Author: Kenneth Knowles <[email protected]>
    Date:   Fri Nov 11 16:54:27 2022 -0800

        Fix checkArgument format string in ExecutionStateTracker

    commit 369e2ba8622d3474c14c39b941b2c618842d1e47
    Author: Ryan Thompson <[email protected]>
    Date:   Fri Nov 11 19:46:07 2022 -0500

        Add a ValidatesContainer integration test for use_sibling_sdk_workers (#24099)

    commit 137799672eb559a7586262e6a8a73d1ab3580e44
    Author: Kenneth Knowles <[email protected]>
    Date:   Fri Nov 11 15:30:01 2022 -0800

        Fix checkArgument format string in AvroIO

    commit 5d2dbf957e4e82fb3980726940df02ac67e563cd
    Author: Anand Inguva <[email protected]>
    Date:   Fri Nov 11 15:57:28 2022 -0500

        Update staging of Python wheels (#24114)

        Fixes https://github.com/apache/beam/issues/24110

    commit c2021bee1eba0322b43c90841397859048296b21
    Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Date:   Fri Nov 11 15:33:14 2022 -0500

        Bump google.golang.org/api from 0.102.0 to 0.103.0 in /sdks (#24049)

        Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.102.0 to 0.103.0.
        - [Release notes](https://github.com/googleapis/google-api-go-client/releases)
        - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
        - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.102.0...v0.103.0)

        ---
        updated-dependencies:
        - dependency-name: google.golang.org/api
          dependency-type: direct:production
          update-type: version-update:semver-minor
        ...

        Signed-off-by: dependabot[bot] <[email protected]>

        Signed-off-by: dependabot[bot] <[email protected]>
        Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

    commit 6557c91c79480b9d90573d52d257a11c2b160196
    Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Date:   Fri Nov 11 11:47:12 2022 -0800

        Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#24112)

        Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.29.1 to 1.29.2.
        - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
        - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
        - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.29.1...service/s3/v1.29.2)

        ---
        updated-dependencies:
        - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
          dependency-type: direct:production
          update-type: version-update:semver-patch
        ...

        Signed-off-by: dependabot[bot] <[email protected]>

        Signed-off-by: dependabot[bot] <[email protected]>
        Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

    commit 96f9da1ab652156cd143d57e3aa3d94836338f2b
    Author: Yi Hu <[email protected]>
    Date:   Fri Nov 11 14:34:22 2022 -0500

        More cleanup containers (#24105)

        * More cleanup containers

        * prebuilt_beam_sdk/beam_python_prebuilt_sdk no longer exists in gcr.io
          Add correct paths to cleanup

        * Bump grace time to 15 days

        * Run job daily as in code comment

        * Set grace period to 30 days

    commit 836766ddb83d37463e4b036f156b8d7e15e3864b
    Author: Ritesh Ghorse <[email protected]>
    Date:   Fri Nov 11 14:30:30 2022 -0500

        upgrade testcontainer dependency (#24123)

    commit 9fcd20c3712536f2d4580beead678cdbb6fd4746
    Author: Damon <[email protected]>
    Date:   Fri Nov 11 11:12:11 2022 -0800

        Implement PubsubRowToMessage transform (#23897)

        * Begin PubsubRowToMessage Impl

        * Complete working draft

        * Unit tests validate user and non-user fields

        * Finish tests on supporting methods

        * Pass checks before finalizing tests

        * WIP

        * fix timestamp

        * finalize tests

        * Finalize code comments

        * Clean up check findings

        * Add InputSchemaFactory

        * Patch code comment typo

    commit 3a6fcc1ca4d07a467464ed1214a94b5c9c147295
    Author: Evgeny Antyshev <[email protected]>
    Date:   Fri Nov 11 19:13:45 2022 +0300

        disable (#24121)

    commit 027fb142038b45c443d87af96cac082264c43188
    Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Date:   Fri Nov 11 10:26:23 2022 -0500

        Bump cloud.google.com/go/bigtable from 1.17.0 to 1.18.0 in /sdks (#24113)

        Bumps [cloud.google.com/go/bigtable](https://github.com/googleapis/google-cloud-go) from 1.17.0 to 1.18.0.
        - [Release notes](https://github.com/googleapis/google-cloud-go/releases)
        - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
        - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.17.0...pubsub/v1.18.0)

        ---
        updated-dependencies:
        - dependency-name: cloud.google.com/go/bigtable
          dependency-type: direct:production
          update-type: version-update:semver-minor
        ...

        Signed-off-by: dependabot[bot] <[email protected]>

        Signed-off-by: dependabot[bot] <[email protected]>
        Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

    commit b2d28a64bb0837dd97e8be7f97f31f88d380f110
    Author: Danny McCormick <[email protected]>
    Date:   Fri Nov 11 08:45:14 2022 -0500

        Add TFX support in pydoc (#23960)

        * Add TFX support in pydoc

        * Wording

    commit 105ed6fedcb6ff33d4a43940c342a653e67cb25e
    Author: bulat safiullin <[email protected]>
    Date:   Thu Nov 10 17:47:54 2022 +0600

        [Website] update go-dependencies.md java-dependencies.md programming-guide.md links #24084

    commit 4b96193250a63b27721a8c5e1a3bd6ecff983093
    Author: Brian Hulette <[email protected]>
    Date:   Thu Nov 10 16:51:06 2022 -0800

        Remove TheNeuralBit from the pool of Python reviewers (#24108)

    commit 8f8f089a5f565f179905984cef3522a0577d9219
    Author: Sanil Jain <[email protected]>
    Date:   Thu Nov 10 16:47:31 2022 -0800

        Wire SamzaPipelineOptions to Exeption listener interface (#24109)

    commit b3186ba91f9bb22764d47b78eb2b7ff017080f75
    Author: Ahmed Abualsaud <[email protected]>
    Date:   Thu Nov 10 18:04:50 2022 -0500

        Support using BigQueryIO Storage Read API with SchemaTransforms (#23827)

        * support schema transform for bq direct read method

        * use vendor Strings import

        * add BigQueryServices argument to config object

        * suppress nullability errors

        * add package-info.java file to providers subdirectory

        * removing accidentally created file

        * added documentation and moved configuration filclass into provider class

        * validate config params before expansion. config has a validaate me method

        * updated URN identifier to follow standards

    commit a4a94da1936d8cd2f8d1145fd2e8329fe06a2990
    Author: Oleh Borysevych <[email protected]>
    Date:   Fri Nov 11 00:28:43 2022 +0200

        fixing linter error (#24104)

    commit 156a6099d71a155c3379340a0b98256306e1755e
    Author: Danny McCormick <[email protected]>
    Date:   Thu Nov 10 17:26:15 2022 -0500

        Add blog post on new ML resources (#24071)

        * Create ml-resources.md

        * Add ensemble notebook image

        * Add image link

        * Fix image link

        * Add ml-landing page image

        * Add image + move around

        * Add paragraph on upcoming changes

        * Remove bad whitespace

    commit 38742d40c895f5d38ee6ee5eb328d11b3262307e
    Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Date:   Thu Nov 10 16:12:40 2022 -0500

        Bump github.com/aws/aws-sdk-go-v2/config from 1.5.0 to 1.17.10 in /sdks (#24080)

        Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.5.0 to 1.17.10.
        - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
        - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
        - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/v1.5.0...config/v1.17.10)

        ---
        updated-dependencies:
        - dependency-name: github.com/aws/aws-sdk-go-v2/config
          dependency-type: direct:production
          update-type: version-update:semver-minor
        ...

        Signed-off-by: dependabot[bot] <[email protected]>

        Signed-off-by: dependabot[bot] <[email protected]>
        Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

    commit 92cef32915c40d353b51529a87d8f0131b0a3538
    Author: BjornPrime <[email protected]>
    Date:   Thu Nov 10 15:30:37 2022 -0500

        Num failed inferences (#23830)

        * created test_increment_num_failed_inferences and test_num_failed_inferences_no_failures

        * added assertRaises to test_increment_num_failed_inferences

        * added num_failed_inferences to _MetricsCollector

        * changed error handling and update() implementation

        * updated metric name in tests

        * removed unnecessary else blocking

        * removed unnecessary inference_args from test_increment_failed_batches_counter()

        * changed final test_increment_failed_batches_counter assertion

        * clarified error handling and updated failed_batches_counter initialization

        * decreased examples array length to 1 to ensure repeatability

        * troubleshooting tests

        * trying to get test_increment_failed_batches_counter to fail as expected

        * corrected assertion details

        * simplified assertRaises and added reminder comment to assertEqual counter

        * lint test

        * lint test passed, resetting pre-commit-config.yaml

        * fixed lingering linting issues

        * shortened comment line to comply with linting

        * formatter worked its magic

    commit 2341f61d48d838ddaf2e4fb990a49987244c3513
    Author: Evgeny Antyshev <[email protected]>
    Date:   Thu Nov 10 21:24:55 2022 +0300

        [Tour Of Beam] handle CORS pre-flight requests (#24083)

        * cors

        * README

        * nit

        * nit

        * -headers, -cache

        * allow-headers

        * +1h cache,comment

    commit 36b0c8f1a76865e4dc3b1eaec1566c3ef9fd4345
    Author: Evgeny Antyshev <[email protected]>
    Date:   Thu Nov 10 21:21:21 2022 +0300

        [Playground] update snippet by persistence_key (#24056)

        * proto

        * AIO

        * -frontend

        * testify

        * +license

        * -build

        * index

        * skipKey

        * Update playground/backend/internal/db/datastore/datastore_db.go

        Co-authored-by: Danny McCormick <[email protected]>

        Co-authored-by: Danny McCormick <[email protected]>

    commit df553d1a4c6c6cc9b37ada2a6c84c88d8aef38b9
    Author: Ahmed Abualsaud <[email protected]>
    Date:   Thu Nov 10 13:17:36 2022 -0500

        Add random string at the end of BigQuery query job name to make it resilient to retries (#24041)

        * add random string at the end of query job name

        * use deterministic temp table name

    commit e439f4120ef4c25aa36e5b03756dc7391bdbd211
    Author: Pablo <[email protected]>
    Date:   Thu Nov 10 10:06:26 2022 -0800

        Improving stale container cleanup script (#24040)

        * Improving stale container cleanup script

        * Avoid also latest image

        * Update .test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh

        Co-authored-by: Yi Hu <[email protected]>

        * Update .test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh

        Co-authored-by: Yi Hu <[email protected]>

        Co-authored-by: Yi Hu <[email protected]>

    commit 8d585242e158babd8dd2ca5a6d5a49d353be2935
    Author: Anand Inguva <[email protected]>
    Date:   Thu Nov 10 12:58:41 2022 -0500

        [Python]Set pickle library at the Pipeline creation stage (#24069)

        Fixes https://github.com/apache/beam/issues/21615

    commit 08b6a524fecc333a12ce42971733ef64ae7d02f1
    Author: Janek Bevendorff <[email protected]>
    Date:   Thu Nov 10 18:37:09 2022 +0100

        [BEAM-12792] Install pipline dependencies to temporary venv (#16658)

    commit 74f87b0282a23aa01ecb63288013311d207960aa
    Merge: 38a85b288a 383ea77bc9
    Author: Robert Bradshaw <[email protected]>
    Date:   Thu Nov 10 09:10:37 2022 -0800

        Merge pull request #23985 Support dynamic sharding in the worker.

    commit 38a85b288ad10171c444d4092eb969ca5de58af3
    Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    Date:   Thu Nov 10 10:39:15 2022 -0500

        Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#24077)

        Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.11.1 to 1.29.1.
        - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
        - [Changelog](https://github.com/aws/aws-sdk-g…
This was referenced May 18, 2023
Merged
Closed
@carlthome
Copy link

carlthome commented May 31, 2023 •
edited
Loading

We're also affected by this on Dataflow when upgrading our Python container image from copying in Apache Beam 2.43 to now 2.46. Weirdly, python3-venv is installed but still seeing an infinite loop with

Failed to initialize Python venv.

until the Dataflow job times out.

This is the image contents as per syft:

NAME                             VERSION                                  TYPE      
Markdown                         3.4.3                                    python     
MarkupSafe                       2.1.2                                    python     
Pillow                           9.5.0                                    python     
PyYAML                           6.0                                      python     
Werkzeug                         2.3.4                                    python     
absl-py                          1.4.0                                    python     
adduser                          3.118ubuntu5                             deb        
apache-beam                      2.46.0                                   python     
apt                              2.4.9                                    deb        
array-record                     0.2.0                                    python     
astunparse                       1.6.3                                    python     
audioread                        3.0.0                                    python     
base-files                       12ubuntu4.3                              deb        
base-passwd                      3.5.52build1                             deb        
bash                             5.1-6ubuntu1                             deb        
bsdutils                         1:2.37.2-4ubuntu3                        deb        
ca-certificates                  20230311ubuntu0.22.04.1                  deb        
cachetools                       4.2.4                                    python     
certifi                          2023.5.7                                 python     
cffi                             1.15.1                                   python     
charset-normalizer               3.1.0                                    python     
click                            8.1.3                                    python     
cloudpickle                      2.2.1                                    python     
coreutils                        8.32-4.1ubuntu1                          deb        
crcmod                           1.7                                      python     
dash                             0.5.11+git20210903+057cd650a4ed-3build1  deb        
db-dtypes                        1.1.1                                    python     
debconf                          1.5.79ubuntu1                            deb        
debianutils                      5.5-1ubuntu2                             deb        
decorator                        5.1.1                                    python     
diffutils                        1:3.8-0ubuntu2                           deb        
dill                             0.3.1.1                                  python     
dm-tree                          0.1.8                                    python     
docopt                           0.6.2                                    python     
dpkg                             1.21.1ubuntu2.1                          deb        
e2fsprogs                        1.46.5-2ubuntu1.1                        deb        
etils                            1.3.0                                    python     
fastavro                         1.7.4                                    python     
fasteners                        0.18                                     python     
ffmpeg                           7:4.4.2-0ubuntu0.22.04.1                 deb        
filelock                         3.12.0                                   python     
findutils                        4.8.0-1ubuntu3                           deb        
flatbuffers                      23.5.26                                  python     
fontconfig                       2.13.1-4.2ubuntu5                        deb        
fontconfig-config                2.13.1-4.2ubuntu5                        deb        
fonts-dejavu-core                2.37-2build1                             deb        
fsspec                           2023.5.0                                 python     
gast                             0.4.0                                    python     
gcc-12-base                      12.1.0-2ubuntu1~22.04                    deb        
github.com/apache/beam/sdks/v2   (devel)                                  go-module  
github.com/golang/protobuf       v1.5.2                                   go-module  
golang.org/x/net                 v0.5.0                                   go-module  
golang.org/x/sys                 v0.5.0                                   go-module  
golang.org/x/text                v0.7.0                                   go-module  
google-api-core                  2.11.0                                   python     
google-apitools                  0.5.31                                   python     
google-auth                      2.19.0                                   python     
google-auth-httplib2             0.1.0                                    python     
google-auth-oauthlib             1.0.0                                    python     
google-cloud-bigquery            3.10.0                                   python     
google-cloud-bigquery-storage    2.16.2                                   python     
google-cloud-bigtable            1.7.3                                    python     
google-cloud-core                2.3.2                                    python     
google-cloud-datastore           1.15.5                                   python     
google-cloud-dlp                 3.12.1                                   python     
google-cloud-language            1.3.2                                    python     
google-cloud-pubsub              2.17.1                                   python     
google-cloud-pubsublite          1.8.2                                    python     
google-cloud-recommendations-ai  0.7.1                                    python     
google-cloud-spanner             3.35.0                                   python     
google-cloud-storage             2.8.0                                    python     
google-cloud-videointelligence   1.16.3                                   python     
google-cloud-vision              3.4.1                                    python     
google-crc32c                    1.5.0                                    python     
google-pasta                     0.2.0                                    python     
google-resumable-media           2.5.0                                    python     
google.golang.org/genproto       v0.0.0-20230110181048-76db0878b65f       go-module  
google.golang.org/grpc           v1.52.3                                  go-module  
google.golang.org/protobuf       v1.28.1                                  go-module  
googleapis-common-protos         1.59.0                                   python     
gpgv                             2.2.27-3ubuntu2.1                        deb        
grep                             3.7-1build1                              deb        
grpc-google-iam-v1               0.12.6                                   python     
grpcio                           1.54.2                                   python     
grpcio-status                    1.48.2                                   python     
gzip                             1.10-4ubuntu4.1                          deb        
h5py                             3.8.0                                    python     
hdfs                             2.7.0                                    python     
hostname                         3.23ubuntu2                              deb        
httplib2                         0.21.0                                   python     
huggingface-hub                  0.14.1                                   python     
idna                             3.4                                      python     
importlib-resources              5.12.0                                   python     
init-system-helpers              1.62                                     deb        
jax                              0.4.10                                   python     
joblib                           1.2.0                                    python     
keras                            2.12.0                                   python     
libacl1                          2.3.1-1                                  deb        
libaom3                          3.3.0-1                                  deb        
libapparmor1                     3.0.4-2ubuntu2.2                         deb        
libapt-pkg6.0                    2.4.9                                    deb        
libasound2                       1.2.6.1-1ubuntu1                         deb        
libasound2-data                  1.2.6.1-1ubuntu1                         deb        
libass9                          1:0.15.2-1                               deb        
libasyncns0                      0.8-6build2                              deb        
libattr1                         1:2.5.1-1build1                          deb        
libaudit-common                  1:3.0.7-1build1                          deb        
libaudit1                        1:3.0.7-1build1                          deb        
libavc1394-0                     0.5.4-5build2                            deb        
libavcodec58                     7:4.4.2-0ubuntu0.22.04.1                 deb        
libavdevice58                    7:4.4.2-0ubuntu0.22.04.1                 deb        
libavfilter7                     7:4.4.2-0ubuntu0.22.04.1                 deb        
libavformat58                    7:4.4.2-0ubuntu0.22.04.1                 deb        
libavutil56                      7:4.4.2-0ubuntu0.22.04.1                 deb        
libblas3                         3.10.0-2ubuntu1                          deb        
libblkid1                        2.37.2-4ubuntu3                          deb        
libbluray2                       1:1.3.1-1                                deb        
libbrotli1                       1.0.9-2build6                            deb        
libbs2b0                         3.1.0+dfsg-2.2build1                     deb        
libbsd0                          0.11.5-1                                 deb        
libbz2-1.0                       1.0.8-5build1                            deb        
libc-bin                         2.35-0ubuntu3.1                          deb        
libc6                            2.35-0ubuntu3.1                          deb        
libcaca0                         0.99.beta19-2.2ubuntu4                   deb        
libcairo-gobject2                1.16.0-5ubuntu2                          deb        
libcairo2                        1.16.0-5ubuntu2                          deb        
libcap-ng0                       0.7.9-2.2build3                          deb        
libcap2                          1:2.44-1build3                           deb        
libcdio-cdda2                    10.2+2.0.0-1build3                       deb        
libcdio-paranoia2                10.2+2.0.0-1build3                       deb        
libcdio19                        2.1.0-3build1                            deb        
libchromaprint1                  1.5.1-2                                  deb        
libclang                         16.0.0                                   python     
libcodec2-1.0                    1.0.1-3                                  deb        
libcom-err2                      1.46.5-2ubuntu1.1                        deb        
libcrypt1                        1:4.4.27-1                               deb        
libdatrie1                       0.2.13-2                                 deb        
libdav1d5                        0.9.2-1                                  deb        
libdb5.3                         5.3.28+dfsg1-0.8ubuntu3                  deb        
libdbus-1-3                      1.12.20-2ubuntu4.1                       deb        
libdc1394-25                     2.2.6-4                                  deb        
libdebconfclient0                0.261ubuntu1                             deb        
libdecor-0-0                     0.1.0-3build1                            deb        
libdeflate0                      1.10-2                                   deb        
libdrm-amdgpu1                   2.4.113-2~ubuntu0.22.04.1                deb        
libdrm-common                    2.4.113-2~ubuntu0.22.04.1                deb        
libdrm-nouveau2                  2.4.113-2~ubuntu0.22.04.1                deb        
libdrm-radeon1                   2.4.113-2~ubuntu0.22.04.1                deb        
libdrm2                          2.4.113-2~ubuntu0.22.04.1                deb        
libedit2                         3.1-20210910-1build1                     deb        
libelf1                          0.186-1build1                            deb        
libexpat1                        2.4.7-1ubuntu0.2                         deb        
libext2fs2                       1.46.5-2ubuntu1.1                        deb        
libffi8                          3.4.2-4                                  deb        
libflac8                         1.3.3-2ubuntu0.1                         deb        
libflite1                        2.2-3                                    deb        
libfontconfig1                   2.13.1-4.2ubuntu5                        deb        
libfreetype6                     2.11.1+dfsg-1ubuntu0.2                   deb        
libfribidi0                      1.0.8-2ubuntu3.1                         deb        
libgbm1                          22.2.5-0ubuntu0.1~22.04.1                deb        
libgcc-s1                        12.1.0-2ubuntu1~22.04                    deb        
libgcrypt20                      1.9.4-3ubuntu3                           deb        
libgdk-pixbuf-2.0-0              2.42.8+dfsg-1ubuntu0.2                   deb        
libgdk-pixbuf2.0-common          2.42.8+dfsg-1ubuntu0.2                   deb        
libgfortran5                     12.1.0-2ubuntu1~22.04                    deb        
libgl1                           1.4.0-1                                  deb        
libgl1-mesa-dri                  22.2.5-0ubuntu0.1~22.04.1                deb        
libglapi-mesa                    22.2.5-0ubuntu0.1~22.04.1                deb        
libglib2.0-0                     2.72.4-0ubuntu2                          deb        
libglvnd0                        1.4.0-1                                  deb        
libglx-mesa0                     22.2.5-0ubuntu0.1~22.04.1                deb        
libglx0                          1.4.0-1                                  deb        
libgme0                          0.6.3-2                                  deb        
libgmp10                         2:6.2.1+dfsg-3ubuntu1                    deb        
libgnutls30                      3.7.3-4ubuntu1.2                         deb        
libgomp1                         12.1.0-2ubuntu1~22.04                    deb        
libgpg-error0                    1.43-3                                   deb        
libgraphite2-3                   1.3.14-1build2                           deb        
libgsm1                          1.0.19-1                                 deb        
libgssapi-krb5-2                 1.19.2-2ubuntu0.1                        deb        
libharfbuzz0b                    2.7.4-1ubuntu3.1                         deb        
libhogweed6                      3.7.3-1build2                            deb        
libicu70                         70.1-2                                   deb        
libidn2-0                        2.3.2-2build1                            deb        
libiec61883-0                    1.2.0-4build3                            deb        
libjack-jackd2-0                 1.9.20~dfsg-1                            deb        
libjbig0                         2.1-3.1ubuntu0.22.04.1                   deb        
libjpeg-turbo8                   2.1.2-0ubuntu1                           deb        
libjpeg8                         8c-2ubuntu10                             deb        
libk5crypto3                     1.19.2-2ubuntu0.1                        deb        
libkeyutils1                     1.6.1-2ubuntu3                           deb        
libkrb5-3                        1.19.2-2ubuntu0.1                        deb        
libkrb5support0                  1.19.2-2ubuntu0.1                        deb        
liblapack3                       3.10.0-2ubuntu1                          deb        
liblilv-0-0                      0.24.12-2                                deb        
libllvm15                        1:15.0.7-0ubuntu0.22.04.1                deb        
liblz4-1                         1.9.3-2build2                            deb        
liblzma5                         5.2.5-2ubuntu1                           deb        
libmd0                           1.0.4-1build1                            deb        
libmfx1                          22.3.0-1                                 deb        
libmount1                        2.37.2-4ubuntu3                          deb        
libmp3lame0                      3.100-3build2                            deb        
libmpdec3                        2.5.1-2build2                            deb        
libmpg123-0                      1.29.3-1build1                           deb        
libmysofa1                       1.2.1~dfsg0-1                            deb        
libncurses6                      6.3-2                                    deb        
libncursesw6                     6.3-2                                    deb        
libnettle8                       3.7.3-1build2                            deb        
libnorm1                         1.5.9+dfsg-2                             deb        
libnsl2                          1.3.0-2build2                            deb        
libnuma1                         2.0.14-3ubuntu2                          deb        
libogg0                          1.3.5-0ubuntu3                           deb        
libopenal-data                   1:1.19.1-2build3                         deb        
libopenal1                       1:1.19.1-2build3                         deb        
libopenjp2-7                     2.4.0-6                                  deb        
libopenmpt0                      0.6.1-1                                  deb        
libopus0                         1.3.1-0.1build2                          deb        
libp11-kit0                      0.24.0-6build1                           deb        
libpam-modules                   1.4.0-11ubuntu2.3                        deb        
libpam-modules-bin               1.4.0-11ubuntu2.3                        deb        
libpam-runtime                   1.4.0-11ubuntu2.3                        deb        
libpam0g                         1.4.0-11ubuntu2.3                        deb        
libpango-1.0-0                   1.50.6+ds-2ubuntu1                       deb        
libpangocairo-1.0-0              1.50.6+ds-2ubuntu1                       deb        
libpangoft2-1.0-0                1.50.6+ds-2ubuntu1                       deb        
libpcre2-8-0                     10.39-3ubuntu0.1                         deb        
libpcre3                         2:8.39-13ubuntu0.22.04.1                 deb        
libpgm-5.3-0                     5.3.128~dfsg-2                           deb        
libpixman-1-0                    0.40.0-1ubuntu0.22.04.1                  deb        
libpng16-16                      1.6.37-3build5                           deb        
libpocketsphinx3                 0.8.0+real5prealpha+1-14ubuntu1          deb        
libpostproc55                    7:4.4.2-0ubuntu0.22.04.1                 deb        
libprocps8                       2:3.3.17-6ubuntu2                        deb        
libpulse0                        1:15.99.1+dfsg1-1ubuntu2.1               deb        
libpython3-stdlib                3.10.6-1~22.04                           deb        
libpython3.10-minimal            3.10.6-1~22.04.2ubuntu1                  deb        
libpython3.10-stdlib             3.10.6-1~22.04.2ubuntu1                  deb        
libquadmath0                     12.1.0-2ubuntu1~22.04                    deb        
librabbitmq4                     0.10.0-1ubuntu2                          deb        
libraw1394-11                    2.1.2-2build2                            deb        
libreadline8                     8.1.2-1                                  deb        
librosa                          0.8.0                                    python     
librsvg2-2                       2.52.5+dfsg-3                            deb        
librubberband2                   2.0.0-2                                  deb        
libsamplerate0                   0.2.2-1build1                            deb        
libsdl2-2.0-0                    2.0.20+dfsg-2ubuntu1.22.04.1             deb        
libseccomp2                      2.5.3-2ubuntu2                           deb        
libselinux1                      3.3-1build2                              deb        
libsemanage-common               3.3-1build2                              deb        
libsemanage2                     3.3-1build2                              deb        
libsensors-config                1:3.6.0-7ubuntu1                         deb        
libsensors5                      1:3.6.0-7ubuntu1                         deb        
libsepol2                        3.3-1build1                              deb        
libserd-0-0                      0.30.10-2                                deb        
libshine3                        3.1.1-2                                  deb        
libslang2                        2.3.2-5build4                            deb        
libsmartcols1                    2.37.2-4ubuntu3                          deb        
libsnappy1v5                     1.1.8-1build3                            deb        
libsndfile1                      1.0.31-2build1                           deb        
libsndio7.0                      1.8.1-1.1                                deb        
libsodium23                      1.0.18-1build2                           deb        
libsord-0-0                      0.16.8-2                                 deb        
libsoxr0                         0.1.3-4build2                            deb        
libspeex1                        1.2~rc1.2-1.1ubuntu3                     deb        
libsphinxbase3                   0.8+5prealpha+1-13build1                 deb        
libsqlite3-0                     3.37.2-2ubuntu0.1                        deb        
libsratom-0-0                    0.6.8-1                                  deb        
libsrt1.4-gnutls                 1.4.4-4                                  deb        
libss2                           1.46.5-2ubuntu1.1                        deb        
libssh-gcrypt-4                  0.9.6-2build1                            deb        
libssl3                          3.0.2-0ubuntu1.9                         deb        
libstdc++6                       12.1.0-2ubuntu1~22.04                    deb        
libswresample3                   7:4.4.2-0ubuntu0.22.04.1                 deb        
libswscale5                      7:4.4.2-0ubuntu0.22.04.1                 deb        
libsystemd0                      249.11-0ubuntu3.9                        deb        
libtasn1-6                       4.18.0-4build1                           deb        
libthai-data                     0.1.29-1build1                           deb        
libthai0                         0.1.29-1build1                           deb        
libtheora0                       1.1.1+dfsg.1-15ubuntu4                   deb        
libtiff5                         4.3.0-6ubuntu0.4                         deb        
libtinfo6                        6.3-2                                    deb        
libtirpc-common                  1.3.2-2ubuntu0.1                         deb        
libtirpc3                        1.3.2-2ubuntu0.1                         deb        
libtwolame0                      0.4.0-2build2                            deb        
libudev1                         249.11-0ubuntu3.9                        deb        
libudfread0                      1.1.2-1                                  deb        
libunistring2                    1.0-1                                    deb        
libusb-1.0-0                     2:1.0.25-1ubuntu2                        deb        
libuuid1                         2.37.2-4ubuntu3                          deb        
libva-drm2                       2.14.0-1                                 deb        
libva-x11-2                      2.14.0-1                                 deb        
libva2                           2.14.0-1                                 deb        
libvdpau1                        1.4-3build2                              deb        
libvidstab1.1                    1.1.0-2                                  deb        
libvorbis0a                      1.3.7-1build2                            deb        
libvorbisenc2                    1.3.7-1build2                            deb        
libvorbisfile3                   1.3.7-1build2                            deb        
libvpx7                          1.11.0-2ubuntu2                          deb        
libwayland-client0               1.20.0-1ubuntu0.1                        deb        
libwayland-cursor0               1.20.0-1ubuntu0.1                        deb        
libwayland-egl1                  1.20.0-1ubuntu0.1                        deb        
libwayland-server0               1.20.0-1ubuntu0.1                        deb        
libwebp7                         1.2.2-2ubuntu0.22.04.1                   deb        
libwebpmux3                      1.2.2-2ubuntu0.22.04.1                   deb        
libx11-6                         2:1.7.5-1                                deb        
libx11-data                      2:1.7.5-1                                deb        
libx11-xcb1                      2:1.7.5-1                                deb        
libx264-163                      2:0.163.3060+git5db6aa6-2build1          deb        
libx265-199                      3.5-2                                    deb        
libxau6                          1:1.0.9-1build5                          deb        
libxcb-dri2-0                    1.14-3ubuntu3                            deb        
libxcb-dri3-0                    1.14-3ubuntu3                            deb        
libxcb-glx0                      1.14-3ubuntu3                            deb        
libxcb-present0                  1.14-3ubuntu3                            deb        
libxcb-render0                   1.14-3ubuntu3                            deb        
libxcb-shape0                    1.14-3ubuntu3                            deb        
libxcb-shm0                      1.14-3ubuntu3                            deb        
libxcb-sync1                     1.14-3ubuntu3                            deb        
libxcb-xfixes0                   1.14-3ubuntu3                            deb        
libxcb1                          1.14-3ubuntu3                            deb        
libxcursor1                      1:1.2.0-2build4                          deb        
libxdmcp6                        1:1.1.3-0ubuntu5                         deb        
libxext6                         2:1.3.4-1build1                          deb        
libxfixes3                       1:6.0.0-1                                deb        
libxi6                           2:1.8-1build1                            deb        
libxinerama1                     2:1.1.4-3                                deb        
libxkbcommon0                    1.4.0-1                                  deb        
libxml2                          2.9.13+dfsg-1ubuntu0.3                   deb        
libxrandr2                       2:1.5.2-1build1                          deb        
libxrender1                      1:0.9.10-1build4                         deb        
libxshmfence1                    1.3-1build4                              deb        
libxss1                          1:1.2.3-1build2                          deb        
libxv1                           2:1.0.11-1build2                         deb        
libxvidcore4                     2:1.3.7-1                                deb        
libxxf86vm1                      1:1.1.4-1build3                          deb        
libxxhash0                       0.8.1-1                                  deb        
libzimg2                         3.0.3+ds1-1                              deb        
libzmq5                          4.3.4-2                                  deb        
libzstd1                         1.4.8+dfsg-3build1                       deb        
libzvbi-common                   0.2.35-19                                deb        
libzvbi0                         0.2.35-19                                deb        
llvmlite                         0.40.0                                   python     
login                            1:4.8.1-2ubuntu2.1                       deb        
logsave                          1.46.5-2ubuntu1.1                        deb        
lsb-base                         11.1.0ubuntu4                            deb        
mawk                             1.3.4.20200120-3                         deb        
media-types                      7.0.0                                    deb        
ml-dtypes                        0.1.0                                    python     
mount                            2.37.2-4ubuntu3                          deb        
ncurses-base                     6.3-2                                    deb        
ncurses-bin                      6.3-2                                    deb        
numba                            0.57.0                                   python     
numpy                            1.23.5                                   python     
oauth2client                     4.1.3                                    python     
oauthlib                         3.2.2                                    python     
objsize                          0.6.1                                    python     
ocl-icd-libopencl1               2.2.14-3                                 deb        
openssl                          3.0.2-0ubuntu1.9                         deb        
opt-einsum                       3.3.0                                    python     
orjson                           3.8.14                                   python     
overrides                        6.5.0                                    python     
packaging                        23.1                                     python     
pandas                           2.0.1                                    python     
pandas-gbq                       0.19.1                                   python     
passwd                           1:4.8.1-2ubuntu2.1                       deb        
perl-base                        5.34.0-3ubuntu1.1                        deb        
pip                              22.0.2                                   python     
platformdirs                     3.5.1                                    python     
pooch                            1.7.0                                    python     
procps                           2:3.3.17-6ubuntu2                        deb        
promise                          2.3                                      python     
proto-plus                       1.22.2                                   python     
protobuf                         3.20.3                                   python     
psutil                           5.9.5                                    python     
pyarrow                          9.0.0                                    python     
pyasn1                           0.5.0                                    python     
pyasn1-modules                   0.3.0                                    python     
pycparser                        2.21                                     python     
pydata-google-auth               1.8.0                                    python     
pydot                            1.4.2                                    python     
pymongo                          3.13.0                                   python     
pyparsing                        3.0.9                                    python     
python-dateutil                  2.8.2                                    python     
python3                          3.10.6-1~22.04                           deb        
python3-distutils                3.10.6-1~22.04                           deb        
python3-lib2to3                  3.10.6-1~22.04                           deb        
python3-minimal                  3.10.6-1~22.04                           deb        
python3-pip                      22.0.2+dfsg-1ubuntu0.2                   deb        
python3-pip-whl                  22.0.2+dfsg-1ubuntu0.2                   deb        
python3-pkg-resources            59.6.0-1.2ubuntu0.22.04.1                deb        
python3-setuptools               59.6.0-1.2ubuntu0.22.04.1                deb        
python3-setuptools-whl           59.6.0-1.2ubuntu0.22.04.1                deb        
python3-venv                     3.10.6-1~22.04                           deb        
python3-wheel                    0.37.1-2ubuntu0.22.04.1                  deb        
python3.10                       3.10.6-1~22.04.2ubuntu1                  deb        
python3.10-minimal               3.10.6-1~22.04.2ubuntu1                  deb        
python3.10-venv                  3.10.6-1~22.04.2ubuntu1                  deb        
pytz                             2023.3                                   python     
readline-common                  8.1.2-1                                  deb        
regex                            2023.5.5                                 python     
requests                         2.31.0                                   python     
requests-oauthlib                1.3.1                                    python     
resampy                          0.4.2                                    python     
rsa                              4.9                                      python     
scikit-learn                     1.2.2                                    python     
scipy                            1.10.1                                   python     
sed                              4.8-1ubuntu2                             deb        
sensible-utils                   0.0.17                                   deb        
setuptools                       59.6.0                                   python     
shared-mime-info                 2.1-2                                    deb        
six                              1.16.0                                   python     
soundfile                        0.12.1                                   python     
sqlparse                         0.4.4                                    python     
sysvinit-utils                   3.01-1ubuntu1                            deb        
tar                              1.34+dfsg-1ubuntu0.1.22.04.1             deb        
tensorboard                      2.12.3                                   python     
tensorboard-data-server          0.7.0                                    python     
tensorflow                       2.12.0                                   python     
tensorflow-datasets              4.9.2                                    python     
tensorflow-estimator             2.12.0                                   python     
tensorflow-hub                   0.13.0                                   python     
tensorflow-io-gcs-filesystem     0.32.0                                   python     
tensorflow-metadata              1.13.1                                   python     
termcolor                        2.3.0                                    python     
threadpoolctl                    3.1.0                                    python     
tokenizers                       0.13.3                                   python     
toml                             0.10.2                                   python     
tqdm                             4.65.0                                   python     
transformers                     4.28.1                                   python     
typing_extensions                4.6.2                                    python     
tzdata                           2023.3                                   python     
ubuntu-keyring                   2021.03.26                               deb        
ucf                              3.0043                                   deb        
urllib3                          1.26.16                                  python     
usrmerge                         25ubuntu2                                deb        
util-linux                       2.37.2-4ubuntu3                          deb        
wheel                            0.37.1                                   python     
wrapt                            1.14.1                                   python     
x11-common                       1:7.7+23ubuntu2                          deb        
xkb-data                         2.33-1                                   deb        
zipp                             3.15.0                                   python     
zlib1g                           1:1.2.11.dfsg-2ubuntu9.2                 deb        
zstandard                        0.21.0                                   python

and the corresponding Dockerfile

# syntax=docker/dockerfile:1
FROM ubuntu:22.04

# Install system dependencies.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get --no-install-recommends install --yes \
    libsndfile1 \
    ffmpeg \
    python3.10 \
    python3-pip \
    python3-venv \
    python3-wheel

# Install Python dependencies.
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Install Apache Beam SDK. Should match the version installed by requirements.txt above.
COPY --from=apache/beam_python3.10_sdk:2.46.0 /opt/apache/beam /opt/apache/beam

# Set the entrypoint to Apache Beam SDK launcher.
ENTRYPOINT ["/opt/apache/beam/boot"]

# Run the SDK process in default Python environment.
ENV RUN_PYTHON_SDK_IN_DEFAULT_ENVIRONMENT=1

@tvalentyn
Copy link
Contributor

tvalentyn commented May 31, 2023 •
edited
Loading

@carlthome can you try with Beam 2.48.0 or higher ? it should have a better error message, and there is now an option to disable this behavior by setting ENV RUN_PYTHON_SDK_IN_DEFAULT_ENVIRONMENT=1 in your Dockerfile.

@tvalentyn
Copy link
Contributor

tvalentyn commented May 31, 2023 •
edited
Loading

Given you use Ubuntu, I wonder if this may be due to python command missing, since I believe at least some Ubuntu versions come with python3, not python, and Beam assumes python. Filed #26958

@carlthome
Copy link

carlthome commented Jun 1, 2023 •
edited
Loading

Thanks @tvalentyn! Did both now (setting RUN_PYTHON_SDK_IN_DEFAULT_ENVIRONMENT in 2.48, and added python-is-python3 just to be safe) and got working Dataflow jobs again.

@phoerious phoerious mentioned this pull request Dec 7, 2023
Open
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AnandInguva AnandInguva AnandInguva left review comments

@lostluck lostluck lostluck approved these changes

@y1chi y1chi y1chi left review comments

@tvalentyn tvalentyn tvalentyn approved these changes

@ryanthompson591 ryanthompson591 ryanthompson591 approved these changes

Assignees
No one assigned
Labels
docker python
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Multiple jobs running on Flink session cluster reuse the persistent Python environment.
9 participants
@phoerious @tvalentyn @ryanthompson591 @aaltay @y1chi @robertwb @carlthome @lostluck @AnandInguva