LogsDB - Quality Assurance Testing

[salvatore-campagna]

Description

As part of the effort to test integration of LogsDB we would like to run quality assurance testing. LogsDB should work seamlessly when used in place of standard indices. Ideally LogsDB should transparently replace standard indices except for non-functional characteristics when it comes to indexing throughput, query latency and storage footprint.

• Create a new Elasticsearch datastream using standard index and another datastream using LogsDB. Index the same set of documents in both datastreams. Make sure no indexing error takes place while indexing in both datastreams.
• Create a new Elasticsearch datastream using standard index and another datastream using LogsDB. Index the same set of documents in both clusters’ datastreams. Run the same query in both clusters and check results are the same, taking into account that the cluster using LogsDB uses synthetic source. Make sure to generate queries both using ESQL and Elasticsearch query DSL.
• Create a new Elasticsearch datastream using standard index and another datastream using LogsDB. Retrieve all field capabilities using both datastreams as targets. Compare field caps api responses making sure the result is the same.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[lkts]

#110952 adds initial implementation of data generation logic.

Test data generation roadmap beyond it:

• nested
• object mapper configuration parameters like enabled
• object arrays
• more leaf fields, more leaf field mapping parameters
• dynamic mapping
• runtime fields

[lkts]

Currently pending items:

• tests using ESQL queries
• tests using fieldcaps api

Data generation:

• duse ignore_dynamic_beyond_limit
• generate malformed values, support ignore_malformed and ignore_above
• generate more field types including some complex ones (flattened, geo_*, *_range, text with keyword multifield)