Kafka queues do not support special characters in source field names

[djklim87]

Bug Description:

When creating a source to read from Kafka queues, we need to iterate through the fields of the incoming message (JSON). In JSON, it’s acceptable for keys to contain $ or other special characters, but such characters are completely prohibited in Manticore fields. This needs to be handled appropriately

CREATE SOURCE wiki_source (id bigint, $schema json) 
type='kafka' 
broker_list='kafka:9092' 
topic_list='wikimedia' 
consumer_group='ms_wikimedia' 
num_consumers='4' 
batch=200

Manticore Search Version:

6.3.8

Operating System Version:

Any

Have you tried the latest development version?

None

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• Changelog updated

[djklim87]

Blocked by manticoresoftware/manticoresearch-buddy#421

[djklim87]

• On review Feat/kafka mapping support manticoresearch-buddy#420
• Don't forget to update tests in ManticoreSearch repo

[djklim87]

We decided to use the syntax:

new_name 'original_name' type

• new_name: A Manticore-compatible field name.
• original_name: The original key from the JSON syntax, which may include special characters. If you need to use an apostrophe within the name, it should be escaped using \'

[djklim87]

@sanikolaev The Windows tests failed here. Please assign this issue to someone from the core team (my changes can't affect this)

https://github.com/manticoresoftware/manticoresearch/actions/runs/12745846202

[djklim87]

implemented in #2926

[djklim87]

tests and docs updated #2927

[sanikolaev]

Suggested docs improvements here #2940