Use existing pickle_library flag in expansion service. #23111
Conversation
|
R: @tvalentyn |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
Codecov Report
@@ Coverage Diff @@
## master #23111 +/- ##
=======================================
Coverage 73.58% 73.58%
=======================================
Files 716 716
Lines 95301 95300 -1
=======================================
+ Hits 70125 70127 +2
+ Misses 23880 23877 -3
Partials 1296 1296
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
| pickler.set_library(known_args.default_pickler) | ||
| # Set this before any pipeline construction occurs. | ||
| # See https://github.com/apache/beam/issues/21615 | ||
| pickler.set_library(pipeline_options.view_as(SetupOptions).pickle_library) |
There was a problem hiding this comment.
How early in construction does this code get called? If its early enough it should resolve the issue of needing to set the pickle library as well as the pipeline options.
There was a problem hiding this comment.
looks like it is called right at the entrypoint of the expansion service (the process that will be responsible for expanding an external python transform to incorporate into a Java pipeline).
| @@ -449,8 +449,7 @@ public OutputT expand(InputT input) { | |||
| } else { | |||
| int port = PythonService.findAvailablePort(); | |||
| ImmutableList.Builder<String> args = ImmutableList.builder(); | |||
| args.add( | |||
| "--port=" + port, "--fully_qualified_name_glob=*", "--default_pickler=cloudpickle"); | |||
| args.add("--port=" + port, "--fully_qualified_name_glob=*", "--pickle_library=cloudpickle"); | |||
There was a problem hiding this comment.
I'm also curious if this will break a bunch of tests. Maybe in the portable version of beam it is fine.
There was a problem hiding this comment.
this only applies to external transforms where Python transform is called from Java. this is new functionality and there are not a lot of tests to break.
|
@robertwb do we need to run some postcommit tests that exercise the relevant XLang functionality? |
|
Run XVR_Direct PostCommit |
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.