[CCR] Stats show follower index even after it has been deleted

[sebelga]

After deleting index (that was a follower), the _ccr/stats API still show the index in the stats.

Expected behavior: When an index does not exist anymore, it should not be listed as a follower index in the ccr stats.

Steps to reproduces

1. Create a follower index

PUT /my-follower-index/_ccr/follow
{
  "remote_cluster" : "new-york",
  "leader_index" : "kibana_sample"
}

2. Delete the index

DELETE /my-follower-index

3. Fetch the CCR stats

GET /_ccr/stats

Response (200 status)
{
    "auto_follow_stats": {
        "number_of_failed_follow_indices": 0,
        ...
    },
    "follow_stats": {
        "indices": [
            {
                "index": "my-follower-index",
                "shards": [
                    {
                        "remote_cluster": "new-york",
                        "leader_index": "kibana_sample",
                        "follower_index": "my-follower-index",
                        ...
                    }
                ]
            }
        ]
    }
}

[elasticmachine]

Pinging @elastic/es-distributed

[martijnvg]

@sebelga The follow stats api returns the stats of the shard follow tasks for a particular follower index. When a follower shard is removed, the shard follow tasks are not removed automatically. The shard following process itself fails and the follow stats api include a fatal error for the shard follow tasks that have a follower index that no longer exists. The reason that the shard follow tasks don't stop automatically is because there can be other errors that can cause index following to fail and keeping the shard follow tasks around with a fatal error set, will help debugging (instead of going though logs).

If indexing following is paused then the shard follow tasks are removed and then the follow stats api will no longer include stats for this follower index.

[sebelga]

@martijnvg, I understand that there can be multiple causes for a following to fail. But doesn't it make sense, if we delete the follower index, that the following tasks associated with it get deleted too?

[martijnvg]

if we delete the follower index, that the following tasks associated with it get deleted too?

The delete index api doesn't know about ccr and that an index is a follower index, so that isn't something that can be done automatically as part of the delete index itself. However a follower shard task could remove itself when it detects that the follower index no longer exists. Today we don't do this and a shard follow tasks stop following and sets the fatal error field.

[sebelga]

Let's keep this as an improvement for the future 😊 cheers

[tlrx]

However a follower shard task could remove itself when it detects that the follower index no longer exists. Today we don't do this and a shard follow tasks stop following and sets the fatal error field.

This has been implemented in #44702: shard follow tasks are now removed if the task concerns a deleted index. As a consequence, the Follow Stats API does not report stats for deleted indices anymore.