hack out a connector to index index metadata

[seanstory]

Relates to https://github.com/elastic/search-team/issues/10828

Just naively searching on this with a single semantic query got us to:

Exact match accuracy: 80.00%

Accuracy by difficulty:
  easy: 95.83% (24 queries)
  medium: 70.00% (30 queries)
  hard: 75.00% (16 queries)
==================================================

which is slightly better than just using index names, but not as good as handing index descriptions directly to the LLM. May try a hybrid approach next.

My mappings were:

PUT connector-mappings
{
    "mappings": {
        "properties": {
            "semantic_content": {
                "type": "semantic_text"
            },
            "index_name": {
                "type": "text",
                "copy_to": "semantic_content"
            },
            "mapping": {
                "type": "keyword",
                "copy_to": "semantic_content"
            },
            "source_cluster": {
                "type": "keyword"
            },
            "meta_description": {
                "type": "text",
                "copy_to": "semantic_content"
            },
            "field_descriptions": {
                "type": "text",
                "copy_to": "semantic_content"
            }
        }
    }
}

Checklists

Pre-Review Checklist

• this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
• this PR has a meaningful title
• this PR links to all relevant github issues that it fixes or partially addresses
• if there is no GH issue, please create it. Each PR should have a link to an issue
• this PR has a thorough description
• Covered the changes with automated tests
• Tested the changes locally
• Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)
• For bugfixes: backport safely to all minor branches still receiving patch releases
• Considered corresponding documentation changes
• Contributed any configuration settings changes to the configuration reference
• if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in Kibana

Changes Requiring Extra Attention

• Security-related changes (encryption, TLS, SSRF, etc)
• New external service dependencies added.

Related Pull Requests

Release Note