-
Notifications
You must be signed in to change notification settings - Fork 254
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add remote client that directly utilizes the flyteidl-rust
Python bindings
#2536
base: flyrs-v2
Are you sure you want to change the base?
Add remote client that directly utilizes the flyteidl-rust
Python bindings
#2536
Conversation
Signed-off-by: Austin Liu <[email protected]>
Signed-off-by: Austin Liu <[email protected]>
Signed-off-by: Austin Liu <[email protected]>
Signed-off-by: Austin Liu <[email protected]>
Signed-off-by: Austin Liu <[email protected]> refactor remote entities into `flyteidl-rust` Signed-off-by: Austin Liu <[email protected]>
Signed-off-by: Austin Liu <[email protected]>
Signed-off-by: Austin Liu <[email protected]> refactor `pyflyte` interaction into `flyteidl-rust` Signed-off-by: Austin Liu <[email protected]>
Signed-off-by: Austin Liu <[email protected]>
Signed-off-by: Austin Liu <[email protected]>
Thanks, Ketan, for reminding me not to compromise on the earlier serialization approach, and thanks to Kevin and Troy for sharing helpful experiences as they are developing the |
There are several implementation details need to be discussed (in regards to the developer experience):
|
flyteidl-rust
Python bindingsflyteidl-rust
Python bindings
Tracking issue
As a result of the initiative, issues, discussions and serial trials and errors in
flyrs
branch 1 2 this PR aims to create another FlyteRemote inremote/remote_rs.py
which wrappingflyteidl-rust
1 as its core. Python package,flyteidl-rust
, was built directly from Rust. With the helps of works in flyte/flyteidl/src/lib.rs, this PR eliminates the protobuf objects serialization overhead2 while calling gRPC services. Unlike the original approach inflyrs
, this method optimizes performance and reduces the maintenance cost.Why are the changes needed?
With
flyteidl-rust
Python bindings, Flytekit can get rid of not onlygrpc
but alsoprotobuf
Python dependencies, ultimately removing themodel
layer betweenprotobuf
andremote entities
. This approach leverages Rust structures directly.For Rust side, developers still can wrap
flyte/flytidl
as their remote gRPC clients or use its API directly.For Python, developers can build another remote client by wrapping
flyteidl-rust
as its core.What changes were proposed in this pull request?
To this end, we have to replace all types import from
model
files to useflyteidl-rust
package instead. Including,Modify remote entities to inherit from
flyteidl-rust
classes instead ofmodel
files.Refactor
type transoformers
incore/type_engine.py
andget_serializable()
intranslator.py
to adapt withflyteidl-rust
classes.get_serializable()
intools/translator.py
ReferenceEntity
PythonTask
WorkflowBase
Node
LaunchPlan
BranchNode
FlyteTask
SimpleTransformer
incore/type_engine.py
int
float
bool
str
datetime
timedelta
date
none
Transformer
incore/type_engine.py
How was this patch tested?
Warning
For now, we havn't migrate the test cases to use
flyteidl-rust
and other type transformation still works in progress, so the CI must fail, don't run.FlyteRemote only works in
Task
register / fetch / execute strictly with primitive python types,Workflow
andLaunchPlan
still works in progress.Setup process
Install
![Screenshot 2024-06-27 at 7 11 54 PM](https://private-user-images.githubusercontent.com/35886692/343748132-986d1d2d-ca5c-40be-8790-4d7ee95a19b6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ0Njk5NTMsIm5iZiI6MTcyNDQ2OTY1MywicGF0aCI6Ii8zNTg4NjY5Mi8zNDM3NDgxMzItOTg2ZDFkMmQtY2E1Yy00MGJlLTg3OTAtNGQ3ZWU5NWExOWI2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI0VDAzMjA1M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTVkNjZlNGFmY2I3ODFiMDNjM2Q5YWZjOWVjODdlMWM1NzEyOWY3ZTQ1NWYwYWE3NTRkNDUyYjNmNjJmZDczY2UmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.kJUu_r2HR-pytuhpUPeCp7fNFDALf1IxAvhiFYlhv24)
flyteidl-rust
pip install -i https://test.pypi.org/simple/ flyteidl-rust
Run the tasks which only using string, integer, float as input or output. For example,
pyflyte register ./echo_string_task.py
and run on web console, orpyflyte run --remote ./echo_string_task.py test_echo_string_task --s "hey"
Screenshots
pyflyte register
echo_int
andecho_float
task.echo_string
task from console.pyflyte run --remote
Check all the applicable boxes
Related PRs
Docs link
Cross References
Footnotes
Temporarily registered in PyPI test index as test package that will not affect the real PyPI registry in production. ↩
In previous PRs, it serialize the python types into bytes string and pass into PyO3-exposed Rust functions. ↩