libbeat/idxmgmt: alias creation fails when `setup.ilm.overwrite: true` and alias exists

[axw]

When ILM is enabled, libbeat will attempt to create a write alias. It does this by creating an index, and attempting to pointing the write alias to it:

beats/libbeat/idxmgmt/ilm/client_handler.go

Lines 178 to 208 in dc57628

	// CreateAlias sends request to Elasticsearch for creating alias.
	func (h *ESClientHandler) CreateAlias(alias Alias) error {
	// Escaping because of date pattern
	// This always assume it's a date pattern by sourrounding it by <...>
	firstIndex := fmt.Sprintf("<%s-%s>", alias.Name, alias.Pattern)
	firstIndex = url.PathEscape(firstIndex)
	body := common.MapStr{
	"aliases": common.MapStr{
	alias.Name: common.MapStr{
	"is_write_index": true,
	},
	},
	}
	// Note: actual aliases are accessible via the index
	status, res, err := h.client.Request("PUT", "/"+firstIndex, "", nil, body)
	if status == 400 {
	// HasAlias fails if there is an index with the same name, that is
	// what we want to check here.
	_, err := h.HasAlias(alias.Name)
	if err != nil {
	return err
	}
	return errOf(ErrAliasAlreadyExists)
	} else if err != nil {
	return wrapErrf(err, ErrAliasCreateFailed, "failed to create alias: %s", res)
	}
	return nil
	}

If the index already exists, Elasticsearch will return 400; libbeat then checks if the alias exists, and carries on without error if it does.

If the index does not exist (e.g. because it rolled over), but the write alias exists and points to some other index, then Elasticsearch returns 500. In this case, libbeat fails index management setup with an error like (taken from apm-server):

Index Alias apm-7.13.0-span setup failed: failed to create alias: {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"alias [apm-7.13.0-span] has more than one write index [apm-7.13.0-span-000001,apm-7.13.0-span-000008]"}],"type":"illegal_state_exception","reason":"alias [apm-7.13.0-span] has more than one write index [apm-7.13.0-span-000001,apm-7.13.0-span-000008]"},"status":500}: 500 Internal Server Error: {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"alias [apm-7.13.0-span] has more than one write index [apm-7.13.0-span-000001,apm-7.13.0-span-000008]"}],"type":"illegal_state_exception","reason":"alias [apm-7.13.0-span] has more than one write index [apm-7.13.0-span-000001,apm-7.13.0-span-000008]"},"status":500}.

[botelastic]

This issue doesn't have a Team:<team> label.

[mfinelli]

I'm having the same issue when the initial index gets created on one day and then a beat agent restarts on a different day and tries to write to an index now named with the current day. As a temporary workaround I need to restart all of the beats so that they all try to write to the new day index and then I update the old index alias no no longer be a write index:

{"error":{"root_cause":[{"type":"illegal_state_ex
ception","reason":"alias [metricbeat-7.13.1] has more than one write index [metricbeat-7.13.1-2021.06.03-000001,metricbeat-7.13.1-2021.06.04-000001]"}],"type":"illegal_state_exception","reason":"a
lias [metricbeat-7.13.1] has more than one write index [metricbeat-7.13.1-2021.06.03-000001,metricbeat-7.13.1-2021.06.04-000001]"},"status":500}: 500 Internal Server Error: {"error":{"root_cause":
[{"type":"illegal_state_exception","reason":"alias [metricbeat-7.13.1] has more than one write index [metricbeat-7.13.1-2021.06.03-000001,metricbeat-7.13.1-2021.06.04-000001]"}],"type":"illegal_state_exception","reason":"alias [metricbeat-7.13.1] has more than one write index [metricbeat-7.13.1-2021.06.03-000001,metricbeat-7.13.1-2021.06.04-000001]"},"status":500}

PUT metricbeat-7.13.1-2021.06.03-000001/_alias/metricbeat-7.13.1
{
  "is_write_index": false
}