Deletion request with invalid query causes subsequent queries to fail

[erichiller]

Describe the bug
Deletion request with invalid query causes subsequent queries to fail with:

parse error : queries require at least one regexp or equality matcher that does not have an empty-compatible value. For instance, app=~".*" does not meet this requirement, but app=~".+" will

When creating a new deletion request such as {host=""} (which I understand is invalid - that's not the issue) The query will be processed (as seen from the list endpoint), but upon making any queries that include data between when the deletion request started and when it was processed an error about the query used in the deletion request will be received.

I ended up fixing it using a modified version of the procedure noted here:

1. shutting down loki
2. deleting chunks/index/delete_requests/delete_requests.gz
3. deleting compactor/deletion/delete_requests/delete_requests

Expected behavior
Ideally the deletion request with an invalid query would be rejected and never created.

Environment:

• Loki v3.3.1
• Infrastructure: Docker
• Deployment tool: docker compose