You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If hive is enabled,all topic in a connector can just share a same database.I think it has strongly restricted my practical usage of it.
Yes , I can create multiple connectors to solve this problem, but I think in most practical cases there exists only one connector.So , I want to add the multiple-database-support feature to current connector.I think it is reasonable and necessary.
Any suggestions?
The text was updated successfully, but these errors were encountered:
@VicoWu Today the tables are created by topic, so it should not be limited to one table per topic. I think this is what many people want because they have one schema (or set of compatible schemas) per topic.
Beyond that, I'd expect sending multiple topics to the same table to be the more common need (in contrast to taking the data from one topic and splitting it among multiple tables). I think you can do this now with single message transforms -- the RegexpRouter could, for example, allow you to subscribe to logs-* and then transform the topic in each record to simply be logs, which will cause all the data to be "routed" together into the same partitioning & same table.
Barring doing these transformations, we'd need some sort of routing within the connector itself, i.e. a mapping from input topics to output tables. This is something we could consider, though I think it probably gets pretty complex/messy pretty fast.
From the doc of RegexRouter, it is used to Update the record topic using the configured regular expression and replacement string
That means a many-to-one topic transformation , so , what you want , is a more arbitrary transformation?Or do I have misunderstand your meaning ?What do you mean by we'd need some sort of routing within the connector itself, i.e. a mapping from input topics to output tables
If hive is enabled,all topic in a connector can just share a same database.I think it has strongly restricted my practical usage of it.
Yes , I can create multiple connectors to solve this problem, but I think in most practical cases there exists only one connector.So , I want to add the multiple-database-support feature to current connector.I think it is reasonable and necessary.
Any suggestions?
The text was updated successfully, but these errors were encountered: