Rollup indices break on missing timezone in global timefilter

[flash1293]

The change introduced by #31807 breaks visualizing rollup indices because the time filter is sent as a strict_date_optional_time without timezone instead of epoch_millis, which makes the specification of a timezone necessary.

The date format was switched in the first place to support dates before the unix epoch which isn't the case for epoch_millis in 7.0. If we find a date respresentation which supports dates before the epoch and doesn't require a timezone in rollup indices, we could use that one and get around trying to get the timezone into all of these places.

Edit: Unfortunately it also doesn't work with the epoch_millis format (see comment below) which means this isn't solvable by reverting #31807 - the timezone has to be propagated to all the filters when dealing with rollup indices.

Request generated:

[{
  "index": "logstash_rollup",
  "query": {
    "size": 0,
    "aggregations": {
      "2": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "2h",
          "time_zone": "Europe/Berlin",
          "min_doc_count": 1
        },
        "aggs": {
          "1": {
            "max": {
              "field": "bytes"
            }
          }
        }
      }
    },
    "query": {
      "bool": {
        "must": [{
          "range": {
            "@timestamp": {
              // At this point "time_zone": "Europe/Berlin", has to be specificed
              "format": "strict_date_optional_time",
              "gte": "2019-03-27T17:39:41.248Z",
              "lte": "2019-03-27T17:54:41.248Z"
            }
          }
        }],
        "filter": [{
          "match_all": {}
        }, {
          "match_all": {}
        }],
        "should": [],
        "must_not": []
      }
    }
  }
}]

Exception shown:

{"statusCode":400,"error":"Bad Request","message":"[illegal_argument_exception] Field [@timestamp] in [range] query was found in rollup indices, but requested timezone is not compatible. Options include: [Europe/Berlin]","cause":["Field [@timestamp] in [range] query was found in rollup indices, but requested timezone is not compatible. Options include: [Europe/Berlin]"]}

cc @timroes @jen-huang

[elasticmachine]

Pinging @elastic/kibana-app

[polyfractal]

Had a quick look, I think there's something else going on here. E.g with this setup:

POST /test_index/_doc/
{
  "date": "2019-03-27T17:39:41.248Z"
}
  
PUT /_rollup/job/test
{
    "index_pattern": "test_index",
    "rollup_index": "rollup",
    "cron": "*/30 * * * * ?",
    "page_size" :1000,
    "groups" : {
      "date_histogram": {
        "field": "date",
        "interval": "1h",
        "time_zone":"Europe/Berlin"
      }
    }
}

POST /_rollup/job/test/_start

This query works just fine. Note the dates in the range query are explicitly UTC (Z timezone). That conflicts with the Rollup config of Berlin which is why the error is thrown. You can override the timezone with time_zone parameter and it works as expected:

GET /rollup/_rollup_search
{
  "size": 0,
  "aggregations": {
    "2": {
      "date_histogram": {
        "field": "date",
        "interval": "1h",
        "time_zone": "Europe/Berlin",
        "min_doc_count": 1
      }
    }
  },
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "date": {
              "format": "strict_date_optional_time",
              "gte": "2019-03-27T17:39:41.248Z",
              "lte": "2019-03-27T17:54:41.248Z",
              "time_zone": "Europe/Berlin"
            }
          }
        }
      ]
    }
  }
}

And regarding epoch_millis, I'm not sure how or why it worked before... I think something else changed. If you just specify epoch_millis without a timezone it's the same error:

GET /rollup/_rollup_search
{
  "size": 0,
  "aggregations": {
    "2": {
      "date_histogram": {
        "field": "date",
        "interval": "1h",
        "time_zone": "Europe/Berlin",
        "min_doc_count": 1
      }
    }
  },
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "date": {
              "format": "epoch_millis",
              "gte": 1553735319910,
              "lte": 1553735319910
            }
          }
        }
      ]
    }
  }
}

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Field [date] in [range] query was found in rollup indices, but requested timezone is not compatible. Options include: [Europe/Berlin]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Field [date] in [range] query was found in rollup indices, but requested timezone is not compatible. Options include: [Europe/Berlin]"
  },
  "status": 400
}

The Rollup search endpoint does two things with regards to timezones:

• It verifies that any date_histogram aggregations use the same timezone as the configured job. If no timezone is configured, UTC is assumed
• Similarly, if there is a range aggregation on a date field in the config, it verifies that the timezone matches the confg (and assumes UTC if not specified).

There is a known issue where Rollup is doing string timezone comparisons instead of rule-based comparisons, which means "Canada/Mountain" != "America/Edmonton" even though they are the same timezone... but I don't think that's the issue that's happening here.

While poking around the code and testing, I noticed that there is another bug where inline timezones are not validated correctly. E.g. if the format supports inline timezones and it is supplied, we throw an error even though the timezone is fine:

GET /rollup/_rollup_search
{
  "size": 0,
  "aggregations": {
    "2": {
      "date_histogram": {
        "field": "date",
        "interval": "1h",
        "time_zone": "Europe/Berlin",
        "min_doc_count": 1
      }
    }
  },
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "date": {
              "gte": "2001-07-04T12:08:56.235+0100",
              "lte": "2001-07-04T12:08:56.235+0100"
            }
          }
        }
      ]
    }
  }
}

But I also don't think this is the issue, since timezones aren't being set here.

[flash1293]

Thanks a lot for your analysis, @polyfractal - It seems like the switch from epoch_millis to strict_date_optional_time just coincidentally happened around the same time when date ranges without timezone stopped working from the Elasticsearch side.

@timroes This probably means we have to find a way to propagate the correct timezone to all of the filters.

[polyfractal]

It could also be some kind of interaction between Kibana, Rollups and the java-time changes. E.g. we just recently found multiple date formats were broken under new java-time. Something like that could have knock-on effect like throwing errors during indexing which, if not caught, might cause issues later.

Or just a bug in Rollup that I'm not seeing, which is very possible :) Timezone support is dicey in a few places. If you have the original rollup job config and the index mapping that'd help too :)

[timroes]

This has been fixed on the Elasticsearch side via elastic/elasticsearch#40647