Internal Error failed to validate: role:: when using backup & restore. 4.2 -> 4.3

[benarent]

Description

What happened:

A customer wasn't able to connect to the UI after upgrading, via a backup & restore procedure.

Internal Error
failed to validate: role:: Additional property role: is not allowed

If I completely remove the /var/lib/teleport dir from the proxy system I can get the UI to work but I loose my trusted cluster info and also get the WARN below. I restored the original teleport contents and restarted the proxy and I connected 1 time then logged out then back in and got the error again....

WARN [REVERSE:I] Re-init the cache on error: failed to validate: role:: Additional property role: is not allowed. cache/cache.go:333
WARN [PROXY:1:C] Re-init the cache on error: failed to validate: role:: Additional property role: is not allowed. cache/cache.go:333

What you expected to happen:

Customers should be able to access upgraded clusters when using tctl get all as defined by https://gravitational.com/teleport/docs/admin-guide/#backing-up-teleport

How to reproduce it (as minimally and precisely as possible):

cd /var/lib/teleport
mkdir backup
tctl get all > state.yaml
stop teleport
unload old rpm
load new rpm
cd /var/lib
mv teleport teleport.(old_version)
mkdir teleport
chmod 700 teleport
start with bootstrap to read old config
  cd teleport.(old)/backup
   teleport start --insecure --config /etc/teleport.yaml --bootstrap state.yaml
  from another login check config.
   tctl get all |more
   tctl status
   tctl users ls
   tctl nodes ls

kill current teleport process.
systemctl start teleport; tail -f /var/lib/teleport/teleport.log 

Version: Teleport 4.3 OSS

[russjones]

Best: 1
Realistic: 3

[fspmarshall]

Upon further investigation, this issue seems to be composed of two separate (and essentially unrelated) component problems:

1. The --bootstrap flag doesn't correctly setup trusted clusters (see Bootstrap flag doesn't correctly setup trusted clusters. #3642).
2. Under certain conditions, teleport nodes with invalid labels can successfully start and register. In the case of this issue, a teleport node with the label role: (not role) was able to register.