Improve performance of resource filtering

[rosstimothy]

There are three main ways that performance was increased

1. Switch the order of operations for ListUnifiedResources such that RBAC happens after filtering. This reduces the amount of time processing roles for each resource to just the ones that would be returned.
2. Prevent creating a logrus entry during RBAC evaluation if debug logging is disabled
3. Reduce the number of allocations performed in searching and filtering code

Benchmarks

The output below is comparing the benchmark run on the first commit with the last commit. The changes result in a 60% reduction in test time and 91% reduction of allocations.

$ benchstat 309068ff441efbc6cf6c9638b0548562e376c66a.txt b6fc568d95ade92b61737b55b19cfceeb8d7f099.txt
goos: darwin
goarch: arm64
pkg: github.com/gravitational/teleport/lib/auth
                                        │ 309068ff441efbc6cf6c9638b0548562e376c66a.txt │ b6fc568d95ade92b61737b55b19cfceeb8d7f099.txt │
                                        │                    sec/op                    │        sec/op         vs base                │
ListUnifiedResourcesFilter/labels-12                                            267.26m ± 1%            67.75m ± 6%  -74.65% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_path-12                                     500.1m ± 1%            281.3m ± 2%  -43.75% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_index-12                                    533.8m ± 8%            290.0m ± 1%  -45.68% (p=0.000 n=10)
ListUnifiedResourcesFilter/search_lower-12                                       365.0m ± 4%            122.8m ± 6%  -66.35% (p=0.000 n=10)
ListUnifiedResourcesFilter/search_upper-12                                       337.5m ± 1%            137.2m ± 1%  -59.35% (p=0.000 n=10)
geomean                                                                    388.0m                 156.3m       -59.73%

                                        │ 309068ff441efbc6cf6c9638b0548562e376c66a.txt │ b6fc568d95ade92b61737b55b19cfceeb8d7f099.txt │
                                        │                     B/op                     │         B/op          vs base                │
ListUnifiedResourcesFilter/labels-12                                         162555.6Ki ± 0%           728.3Ki ± 1%  -99.55% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_path-12                                    396.9Mi ± 0%           238.9Mi ± 0%  -39.81% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_index-12                                   396.9Mi ± 0%           238.9Mi ± 0%  -39.82% (p=0.000 n=10)
ListUnifiedResourcesFilter/search_lower-12                                     158.62Mi ± 0%           16.76Mi ± 0%  -89.44% (p=0.000 n=10)
ListUnifiedResourcesFilter/search_upper-12                                     159.59Mi ± 0%           16.76Mi ± 0%  -89.50% (p=0.000 n=10)
geomean                                                                   229.2Mi                25.79Mi       -88.75%

                                        │ 309068ff441efbc6cf6c9638b0548562e376c66a.txt │ b6fc568d95ade92b61737b55b19cfceeb8d7f099.txt │
                                        │                  allocs/op                   │      allocs/op        vs base                │
ListUnifiedResourcesFilter/labels-12                                           2411.98k ± 0%            11.42k ± 0%  -99.53% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_path-12                                     7.362M ± 0%            4.962M ± 0%  -32.60% (p=0.000 n=10)
ListUnifiedResourcesFilter/predicate_index-12                                    7.213M ± 0%            4.812M ± 0%  -33.28% (p=0.000 n=10)
ListUnifiedResourcesFilter/search_lower-12                                      3162.2k ± 0%            161.6k ± 0%  -94.89% (p=0.000 n=10)
ListUnifiedResourcesFilter/search_upper-12                                      3312.2k ± 0%            161.6k ± 0%  -95.12% (p=0.000 n=10)
geomean                                                                    4.222M                 371.9k       -91.19%

Real world tests

The changes were tested in a cluster with 150,002 static nodes. 150,000 had varying and overlapping labels and the final two nodes had unique labels to allow finding a needle in the haystack. The overall time of listing the entire set was not improved, though searching for one item out of the whole lot was improved by several hundred milliseconds.

#### 15.1.9

time ./releases/15.1.9/tsh ls --format=json | jq .[].metadata.name | wc -l
  150002

________________________________________________________
Executed in   20.40 secs    fish           external
   usr time    3.95 secs    0.21 millis    3.95 secs
   sys time    0.60 secs    2.22 millis    0.60 secs


time ./releases/15.1.9/tsh ls --search="10.20.30.40" --format=json | jq .[].metadata.name | wc -l
      1

________________________________________________________
Executed in    1.08 secs      fish           external
  usr time   88.14 millis    0.23 millis   87.92 millis
  sys time   38.26 millis    2.54 millis   35.72 millis



#### tross/listing_performance


time ./releases/15.1.9/tsh ls --format=json | jq .[].metadata.name | wc -l
  150002

________________________________________________________
Executed in   20.84 secs    fish           external
   usr time    4.05 secs    0.19 millis    4.05 secs
   sys time    0.59 secs    2.57 millis    0.59 secs

time ./releases/15.1.9/tsh ls --search="10.20.30.40" --format=json | jq .[].metadata.name | wc -l
       1

________________________________________________________
Executed in  704.20 millis    fish           external
   usr time   75.38 millis    0.24 millis   75.14 millis
   sys time   49.23 millis    2.77 millis   46.46 millis

Changelog: Improve resource filtering performance for tsh ls and the web ui.

[fspmarshall]

lgtm, with a few minor nits. exciting stuff!

[rosstimothy]

PTAL @atburke

[public-teleport-github-review-bot]

@rosstimothy See the table below for backport results.

Branch	Result
branch/v13	Failed
branch/v14	Failed
branch/v15	Failed