Retention policy didn't kick in for nativelog when disk usage reaches 90%

[chenziliang]

Describe what's wrong

k8s_metrics has the following defintion, but seem the retention policy didn't kick in for nativelog when disk usage reaches 90%.

proton :) show create k8s_metrics;

SHOW CREATE STREAM k8s_metrics

Query id: 4f74b1cb-50a9-4195-8d58-bb593ea2d645

┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE STREAM default.k8s_metrics
(
  `name` low_cardinality(string),
  `type` low_cardinality(string),
  `value` float64,
  `tags` map(string, string),
  `_tp_time` datetime64(3, 'UTC') DEFAULT now64(3, 'UTC') CODEC(DoubleDelta, LZ4),
  INDEX _tp_time_index _tp_time TYPE minmax GRANULARITY 2
)
ENGINE = Stream(1, 1, rand())
PARTITION BY to_YYYYMM(_tp_time)
PRIMARY KEY to_YYYYMMDD(_tp_time)
ORDER BY to_YYYYMMDD(_tp_time)
TTL to_datetime(_tp_time) + INTERVAL 14 DAY
SETTINGS subtype = 'tabular', mode = 'append', logstore_retention_bytes = '10737418240', logstore_retention_ms = '86400000', index_granularity = 8192 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.001 sec. 

How to reproduce

Error message and/or stacktrace

Additional context

[chenziliang]

We need validate

1. Create a stream without retention policy for the logstore
2. Using the REST API to reconfig the retention policy for the logstore. Without reboot, the retention policy shall kick in in 5 mins (10 mins maximum). If not, bug.
3. When max disk usage reached, validate if the background retention scheduler still works. It appeared not working in this case.