Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Each connector support just one database #189

Open
VicoWu opened this issue Apr 27, 2017 · 2 comments
Open

Each connector support just one database #189

VicoWu opened this issue Apr 27, 2017 · 2 comments

Comments

@VicoWu
Copy link

VicoWu commented Apr 27, 2017

If hive is enabled,all topic in a connector can just share a same database.I think it has strongly restricted my practical usage of it.
Yes , I can create multiple connectors to solve this problem, but I think in most practical cases there exists only one connector.So , I want to add the multiple-database-support feature to current connector.I think it is reasonable and necessary.
Any suggestions?

@ewencp
Copy link
Contributor

ewencp commented May 2, 2017

@VicoWu Today the tables are created by topic, so it should not be limited to one table per topic. I think this is what many people want because they have one schema (or set of compatible schemas) per topic.

Beyond that, I'd expect sending multiple topics to the same table to be the more common need (in contrast to taking the data from one topic and splitting it among multiple tables). I think you can do this now with single message transforms -- the RegexpRouter could, for example, allow you to subscribe to logs-* and then transform the topic in each record to simply be logs, which will cause all the data to be "routed" together into the same partitioning & same table.

Barring doing these transformations, we'd need some sort of routing within the connector itself, i.e. a mapping from input topics to output tables. This is something we could consider, though I think it probably gets pretty complex/messy pretty fast.

@VicoWu
Copy link
Author

VicoWu commented May 22, 2017

From the doc of RegexRouter, it is used to
Update the record topic using the configured regular expression and replacement string
That means a many-to-one topic transformation , so , what you want , is a more arbitrary transformation?Or do I have misunderstand your meaning ?What do you mean by we'd need some sort of routing within the connector itself, i.e. a mapping from input topics to output tables

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants