Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request][Go SDK]: Rewrite the dot runner in terms of a Portable pipeline. #27508

Open
1 of 15 tasks
lostluck opened this issue Jul 14, 2023 · 4 comments · May be fixed by #31787
Open
1 of 15 tasks

[Feature Request][Go SDK]: Rewrite the dot runner in terms of a Portable pipeline. #27508

lostluck opened this issue Jul 14, 2023 · 4 comments · May be fixed by #31787
Assignees
@mls3odp
Labels
go good first issue new feature P3

Comments

@lostluck
Copy link
Contributor

What would you like to happen?

The Go SDK's dot runner current generates a pipeline graph based on the Go SDK internal structures.

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/util/dot/dot.go#L89

If it were rewritten in terms of the portable pipeline proto (https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L91) then it could be used to render Cross Language pipelines and be used as a basis for the stand alone Prism Runner's web UI.

That would sufficiently allow #25896 to be closed.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@github-actions github-actions bot added the go label Jul 14, 2023
@mls3odp
Copy link
Contributor

mls3odp commented Jul 4, 2024

.take-issue

@mls3odp
Copy link
Contributor

mls3odp commented Jul 5, 2024 •
edited
Loading

Does this mean to use Pipeline proto as the input of Execute function in dot.go?

Or modify the Render function to transforming edges and nodes into the Pipeline proto and writing the corresponding information (e.g. Pipeline::Components containing transforms and pcolllections) as texts? CMIIW thanks!

@lostluck
Copy link
Contributor Author

lostluck commented Jul 5, 2024

In this case it means rendering the pipeline structure from the Pipeline Proto instead of from the raw Go structures.

This would let it take any pipeline proto graph and produce a dot graph for it.

So this means building the pipeline proto with the existing code to do so (see how the Universal runner is doing it) and then building the visual graph based on the information only from the proto.

@mls3odp
Copy link
Contributor

mls3odp commented Jul 5, 2024

Gotcha! So we need to convert beam.Pipeline to Pipeline proto like https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/universal/universal.go#L64-L91 in https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/dot/dot.go#L40.

@mls3odp mls3odp linked a pull request Jul 5, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mls3odp mls3odp

Labels
go good first issue new feature P3
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support portable pipeline dot graph rendering mls3odp/apache-beam
2 participants
@lostluck @mls3odp