Ruler API returns 202 but fails to store rules #2901
Description
Using this rules yaml:
name: test
rules:
- record: test
expr: sum(rate(container_cpu_usage_seconds_total{job="kube-system/cadvisor", image!="", container_name!=""}[5m])) by (namespace)
This curl command:
curl -v --data-binary @./rules.yaml -X POST -H "X-Scope-OrgID: cortex_enterprise_dev" http://localhost:8080/api/v1/rules/ns
Receives a 202 response:
* upload completely sent off: 167 out of 167 bytes
< HTTP/1.1 202 Accepted
< Content-Type: application/json
< Date: Mon, 20 Jul 2020 14:15:25 GMT
< Content-Length: 58
<
* Connection #0 to host localhost left intact
{"status":"success","data":null,"errorType":"","error":""}
The "ns" namespace folder is created in GCS, but it's empty.
This causes Cortex to log the following:
level=info ts=2020-07-20T14:16:24.886031631Z caller=mapper.go:116 msg="updating rule file" file=/rules/cortex_enterprise_dev/ns
level=error ts=2020-07-20T14:16:24.886715617Z caller=manager.go:910 user=cortex_enterprise_dev msg="loading groups failed" err="/rules/cortex_enterprise_dev/ns: 11:3: Groupname must not be empty"
level=error ts=2020-07-20T14:16:24.886759343Z caller=ruler.go:523 msg="unable to update rule manager" user=cortex_enterprise_dev err="error loading rules, previous rule set restored"
There are no negative logs during the save and it appears to succeed, but there's no corresponding object in GCS.