fix: memberlist delete operation

[NickAnge]

What this PR does:

After the rollout of delete operation in memberlist we observed 2 things:

• Error failed to decode value: snappy: corrupt input. (logs)
• Delete was happening but the actual entry was not deleted completed. So the entry is deleted it and then comes back.

In this PR we tackle these two issues by:

• returning an actual change when the only thing that gets updated is the Delete status, so we do not have to handlePossibleNilEncode which was returning an []byte{} that was causing the Decode issue.
• returning no change when the incoming entry has Delete true. Thist was an edge case in which the current HA tracker had deleted the entry and another one was coming with delete True. This can happen if we have regular push/pull interval.

Verified it locally.

• Need to test it in a dev-cell. mimir-dev-11, would be the cell. https://github.com/grafana/deployment_tools/pull/220742/files

Tests

• Introduces checkMemberlistEntry, that will constantly checks the memberlist entries up to specific amount of time before considers the test failure. It utilizes test.poll
• extend TestDelete to check if there are any encoding errors during the Delete flow. Covers failed to decode value: snappy: corrupt input

Which issue(s) this PR fixes:

Fixes #

Checklist

• Tests updated
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[dimitarvdimitrov]

the change makes sense, but it definitely needs unit tests. If you can even cover cases which haven't been covered so far, that'd be great

[NickAnge]

I decided that the handle* funcions are not needed since we handle the change=nil at the mergeValueForKey level. That was the main reason these were needed at the first place. I validated that the []byte{}was the one causing the Error failed to decode value: snappy: corrupt input at the first place. Please let me know what you think

[NickAnge]

Update after the test in mimir-dev-11

• The entry is deleted as expected ✅

Bad news

• We still see failed to decode value: snappy: corrupt input (logs) after the delete operation , which I am planning to investigate. Its not causing any impact apart from some "bad" messages that are not ingested