Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sideinputs to the RunInference Transform #25200

Merged
merged 54 commits into from
Feb 2, 2023
Merged
Changes from 1 commit
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
e54a2de
Add model pcoll param to the RunInference Ptransform
AnandInguva Jan 6, 2023
443120a
Add sklearn side input example
AnandInguva Jan 12, 2023
50b58b1
Add ModeMetadata and some refactoring
AnandInguva Jan 13, 2023
3cb7074
refactor _convert_to_result and add it to the utils.py
AnandInguva Jan 19, 2023
f614a3d
Add tag to the RunInference DoFn
AnandInguva Jan 19, 2023
43b5ca6
Add enable_side_input_loading flag
AnandInguva Jan 20, 2023
c42b903
Add helper functions
AnandInguva Jan 20, 2023
e2c2833
Add doc string, refactor utils code
AnandInguva Jan 21, 2023
f4a6c2b
Fix pytorch inference tests
AnandInguva Jan 21, 2023
530f61f
Fix up sklearn inference
AnandInguva Jan 23, 2023
9bd8d8f
Remove logging
AnandInguva Jan 24, 2023
676600a
Add thread Lock when there is an update to side input
AnandInguva Jan 25, 2023
0a2a56e
Check if side input is EmptySideInput
AnandInguva Jan 25, 2023
7a1ed15
Add unit test for side input loading
AnandInguva Jan 25, 2023
eae1837
Remove examples
AnandInguva Jan 25, 2023
af19536
Add log when side input path is updated
AnandInguva Jan 25, 2023
3d0821e
Add test to Dataflow
AnandInguva Jan 26, 2023
fe434df
Refactor side input loading code
AnandInguva Jan 26, 2023
2a3e6a4
Add documentation, changelog
AnandInguva Jan 27, 2023
dd6d494
Add Singleton view doc
AnandInguva Jan 27, 2023
037c80a
Fix whitespace, tests
AnandInguva Jan 27, 2023
f9a61a1
fix weird spacing
AnandInguva Jan 27, 2023
d2edaa9
Remove beam website udpate
AnandInguva Jan 27, 2023
73ed494
Revert "fix weird spacing"
AnandInguva Jan 27, 2023
ae91ffe
Add WatchFilePattern transform
AnandInguva Jan 27, 2023
b796560
undo changes to beam wesbite
AnandInguva Jan 27, 2023
ca178a2
Pass side inputs only in streaming mode
AnandInguva Jan 27, 2023
0d96c77
Revert "Add WatchFilePattern transform"
AnandInguva Jan 27, 2023
2d97fdb
Add lines to website page
AnandInguva Jan 27, 2023
cf3893f
Add test
AnandInguva Jan 28, 2023
9295dce
Add unit test to catch --streaming flag and Singleton SideInput
AnandInguva Jan 30, 2023
3ce25f8
Addressing PR comments
AnandInguva Jan 30, 2023
912e36f
Add logic to detect windows on side inputs
AnandInguva Jan 30, 2023
0191cdd
Add more tests
AnandInguva Jan 30, 2023
a860b98
Remove redundant code
AnandInguva Jan 30, 2023
fba66b3
Update test
AnandInguva Jan 31, 2023
babcd24
Fix lint
AnandInguva Jan 31, 2023
016bbc5
Add postcommit markers.
AnandInguva Jan 31, 2023
88cb09d
Remove `and`
AnandInguva Jan 31, 2023
53f5a7c
fixup lint
AnandInguva Jan 31, 2023
d1abcc0
Modify message
AnandInguva Jan 31, 2023
b2c40db
Add check for default model
AnandInguva Jan 31, 2023
60b33fa
Update message
AnandInguva Jan 31, 2023
036f321
Add validates runner
AnandInguva Jan 31, 2023
5b40d1f
Fix test
AnandInguva Jan 31, 2023
0b58f10
Add PipelineVisitor for RunInference during construction time
AnandInguva Feb 1, 2023
bcdb871
Address comments based on PR
AnandInguva Feb 1, 2023
e6f8eaa
Remove restriction on the side inputs
AnandInguva Feb 1, 2023
ec3ca9b
Remove/add tests
AnandInguva Feb 1, 2023
6d07b6f
Modify logging
AnandInguva Feb 1, 2023
25c662c
Add tests
AnandInguva Feb 1, 2023
f7ab2d7
Merge branch 'master' into model-updates-api
damccorm Feb 2, 2023
0d325fe
Add 2.46.0 change log
AnandInguva Feb 2, 2023
637d3c3
fix typo
AnandInguva Feb 2, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,37 @@

* ([#X](https://github.com/apache/beam/issues/X)).
-->

# [2.46.0] - Unreleased

## Highlights

* * RunInference PTransform will accept model paths as SideInputs in Python SDK. ([#24042](https://github.com/apache/beam/issues/24042))
damccorm marked this conversation as resolved.
Show resolved Hide resolved

## I/Os

* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).

## New Features / Improvements

* X feature added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).

## Breaking Changes

* X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).

## Deprecations

* X behavior is deprecated and will be removed in X versions ([#X](https://github.com/apache/beam/issues/X)).

## Bugfixes

* Fixed X (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).

## Known Issues

* ([#X](https://github.com/apache/beam/issues/X)).

# [2.45.0] - Unreleased

## Highlights
Expand All @@ -69,7 +100,6 @@
present in 2.43.0 (up to 1.8.0_342, 11.0.16, 17.0.2 for respective Java versions). This is accompanied
by an explicit re-enabling of TLSv1 and TLSv1.1 for Java 8 and Java 11.
* Add UDF metrics support for Samza portable mode.
* RunInference PTransform will accept Singleton SideInputs in Python SDK. ([#24042](https://github.com/apache/beam/issues/24042))
* Option for SparkRunner to avoid the need of SDF output to fit in memory ([#23852](https://github.com/apache/beam/issues/23852)).
This helps e.g. with ParquetIO reads. Turn the feature on by adding experiment `use_bounded_concurrent_output_for_sdf`.

Expand Down