Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request]: Pass joinSubsetType to BeamSqlSeekableTable in JoinAsLookup #28283

Closed
1 of 15 tasks
gabrywu opened this issue Sep 1, 2023 · 3 comments
Closed
1 of 15 tasks

Comments

@gabrywu
Copy link
Member

gabrywu commented Sep 1, 2023

What would you like to happen?

As of now, BeamSqlSeekableTable has 3 methods, setUp, seekRow and tearDown,
seekRow is the most important method to BeamSqlSeekableTable in JoinAsLookup.
However BeamSqlSeekableTable doesn't know any details of the join condition before it startup. In many cases, we'd better to know the join keys for a lookup table.
With join keys, we can build a local cache based on them when startup. So I recommend that we push the join condition to BeamSqlSeekableTable

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@gabrywu
Copy link
Member Author

gabrywu commented Sep 5, 2023

@Abacn how about this feature? If you think it is reasonable, I can create a PR

@Abacn
Copy link
Contributor

Abacn commented Sep 6, 2023

Hi @gabrywu thanks, this sounds good to me.

I see recently there were other BeamSqlSeekableTable improvements (e.g. #28253). If these are part of larger effort, consider sending a design doc to devlist so more people could see and help it

@gabrywu
Copy link
Member Author

gabrywu commented Sep 7, 2023

Hi @gabrywu thanks, this sounds good to me.

I see recently there were other BeamSqlSeekableTable improvements (e.g. #28253). If these are part of larger effort, consider sending a design doc to devlist so more people could see and help it

Sure. I think there is not too much effort

@gabrywu gabrywu changed the title [Feature Request]: pass joinSubsetType to BeamSqlSeekableTable in JoinAsLookup [Feature Request]: Pass joinSubsetType to BeamSqlSeekableTable in JoinAsLookup Sep 14, 2023
@gabrywu gabrywu closed this as completed Sep 21, 2023
@github-actions GitHub Actions bot added this to the 2.52.0 Release milestone Sep 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants