Backport of Handle errors properly when services are de-registered from the catalog into release/1.0.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #2571 to be assessed for backporting due to the inclusion of the label backport/1.0.x.

The below text is copied from the body of the original PR.

---

Possible fix for:

#2085
#2065
hashicorp/consul#15908
#1935
#1817

which are all the same issue of ghost services left in the catalog when nodes are recycled.

Associated issue: #2491

Changes proposed in this PR:

• In the past, kubernetes nodes were used as the source of truth to determine the list of services that should exist in Consul.
• In most cases this was ok but becomes a problem when nodes are quickly deleted from kubernetes such as the case when using spot instances.
• Instead, use consul synthetic-nodes to get the list of services and deregister the services that do not have endpoint addresses.

How I've tested this PR:

• Ran and updated the unit tests so that they pass. This required adding synthetic node meta data to many of the tests.
• Acceptance tests
• Manually tested using my demo repo.
  • Created a multi-node kind cluster, deployed several static apps to it with a nodeSelector and deleted nodes.
  • When the node is deleted, endpoints controller detects an endpoint change and deregisters
  • Adding the node back re-registers the service
  • Note: You can simulate node deletions using a combination of kubectl delete node + docker stop <image>. docker start <image> will re-register the node.

How I expect reviewers to test this PR:

👀

Checklist:

• Tests added
• CHANGELOG entry added

---

Overview of commits

• 35ff586 - 6dd53d5