Support distinct count aggregation

[chloe-zh]

Signed-off-by: Chloe Zhang [email protected]

Description

• Grammar: enabled distinct count
  SQL: count(DISTINCT field)
  PPL: distinct_count(field)/dc(field) in stats command
  Note that distinct all count(distinct *) is not supported, the grammar of MySQL to get the distinct count of all fields is to put all fields in the distinct count field: https://dev.mysql.com/doc/refman/8.0/en/aggregate-functions.html#function_count-distinct Not supported in this PR since the distinct count of multi-field is not implemented here. Will create another issue for this case.

• Core engine
  Added distinct option in aggregators, and the distinct option is off by default, distinct count turns on this option.

• Push down
  Distinct count of single field in OpenSearch is achieved by building the cardinality aggregation, for example:

select count(distinct Dest) from opensearch_dashboards_sample_data_flights

DSL:
{
  "from":0,
  "size":0,
  "timeout":"1m",
  "aggregations":{
    "count(distinct Dest)":{
      "cardinality":{
        "field":"Dest"
      }
    }
  }
}

Full explain:
{
  "root": {
    "name": "ProjectOperator",
    "description": {
      "fields": "[count(distinct Dest)]"
    },
    "children": [
      {
        "name": "OpenSearchIndexScan",
        "description": {
          "request": """OpenSearchQueryRequest(indexName=opensearch_dashboards_sample_data_flights, sourceBuilder={"from":0,"size":0,"timeout":"1m","aggregations":{"count(distinct Dest)":{"cardinality":{"field":"Dest"}}}}, searchDone=false)"""
        },
        "children": []
      }
    ]
  }
}

Explain of distinct count with filter:

SELECT COUNT(distinct OriginWeather) filter(where AvgTicketPrice < 500) FROM opensearch_dashboards_sample_data_flights

{
  "root": {
    "name": "ProjectOperator",
    "description": {
      "fields": """[COUNT(distinct OriginWeather) filter(where AvgTicketPrice < 500)]"""
    },
    "children": [
      {
        "name": "OpenSearchIndexScan",
        "description": {
          "request": """OpenSearchQueryRequest(indexName=opensearch_dashboards_sample_data_flights, sourceBuilder={"from":0,"size":0,"timeout":"1m","aggregations":{"COUNT(distinct OriginWeather) filter(where AvgTicketPrice < 500)":{"filter":{"range":{"AvgTicketPrice":{"from":null,"to":500,"include_lower":true,"include_upper":false,"boost":1.0}}},"aggregations":{"COUNT(distinct OriginWeather) filter(where AvgTicketPrice < 500)":{"cardinality":{"field":"OriginWeather"}}}}}}, searchDone=false)"""
        },
        "children": []
      }
    ]
  }
}

SQL example:

SELECT COUNT(distinct OriginWeather) filter(where AvgTicketPrice < 500) FROM opensearch_dashboards_sample_data_flights

{
  "schema": [
    {
      "name": """COUNT(distinct OriginWeather) filter(where AvgTicketPrice < 500)""",
      "type": "integer"
    }
  ],
  "datarows": [
    [
      8
    ]
  ],
  "total": 1,
  "size": 1,
  "status": 200
}

PPL example:

source=opensearch_dashboards_sample_data_flights | stats distinct_count(Dest) by Origin | head 3

{
  "schema": [
    {
      "name": "distinct_count(Dest)",
      "type": "integer"
    },
    {
      "name": "Origin",
      "type": "string"
    }
  ],
  "datarows": [
    [
      72,
      "Abu Dhabi International Airport"
    ],
    [
      78,
      "Adelaide International Airport"
    ],
    [
      72,
      "Adolfo Suarez Madrid— Barajas Airport"
    ]
  ],
  "total": 3,
  "size": 3
}