how to manage manticore.conf when sharding database and table

[jacentsao]

for example, 16 database and each database has 16 user table. should we create 16 * 16 source and table definition in conf file , Can it be configured like Flink? or any other simple conf way.

-- Flink SQL
Flink SQL> CREATE TABLE user_source (
    database_name STRING METADATA VIRTUAL,
    table_name STRING METADATA VIRTUAL,
    `id` DECIMAL(20, 0) NOT NULL,
    name STRING,
    address STRING,
    phone_number STRING,
    email STRING,
    PRIMARY KEY (`id`) NOT ENFORCED
  ) WITH (
    'connector' = 'mysql-cdc',
    'hostname' = 'mysql',
    'port' = '3306',
    'username' = 'root',
    'password' = '123456',
    'database-name' = 'db_[0-9]+',
    'table-name' = 'user_[0-9]+'
  );

[barryhunter]

If you want to use 'Plain Indexes' and built by indexer command (eg run on cron), would need all the indexes 'defined' in the manticore.conf file.

But a key point is the config file can actually be a script. ie a script that 'outputs' a config file. Eg could be coded in bash, perl, PHP or python. So in effect it could contain a loops that outputs lots of 'source/index pairs' following a template.

It could using a database connection to get the list of indexes to build.

---

The other way is to use 'Real Time Mode', where indexes can be created dynamically without then needed to be 'hardcoded' in config file.

This way the application can create the indexes on the fly, following whatever structure it likes. It could do on a loop.

(Similar how with mysql, can just execute a 'CREATE TABLE' to create a table, can use a similar command with manticore to create the index, then just 'INSERT' documents into the index)

[jacentsao]

Thanks for your reply, already create conf file by template. I'm wondering if there's a better way. After search through manticore search document, maybe it's best for now.