[Stack Monitoring] health api support for data ingested from package

[klacabane]

Summary

Closes #138864

Add support for data ingested from packages in the metrics-{product}.* patterns.

This change only fetches the data for the successfully monitored products, but we can also retrieve the errors like we do for metricbeat since an error.message may be populated on failure. This will be handled in elastic/integrations#4011

Testing

• setup a stack with an agent - you can follow the testing steps for the logstash package in [logstash] align mappings with metricbeat integrations#3993 to start an elastic-package stack with elasticsearch and logstash data
• connect your local kibana to the elastic-package elasticsearch - howto
• hit localhost:5602/api/monitoring/v1/_health - a monitored cluster should show up with cluster, elasticsearch and logstash data collected with the package method

[klacabane]

Took the opportunity to reuse the functions used to build the patterns throughout the server code

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[matschaffer]

@elasticmachine merge upstream

[matschaffer]

I tried using your profile from elastic/integrations#4018 then running this branch against it. Since I don't think this branch supports metrics-* for the regular UI I then enabled internal collection, but so far I'm only seeing internal-collection data in the health api. Nothing agent based yet.

GET http://localhost:5602/ftw/api/monitoring/v1/_health

{
  "monitoredClusters": {
    "clusters": {
      "0pMSN574ShOT3UQkRDRHxg": {
        "cluster": {
          "0pMSN574ShOT3UQkRDRHxg": {
            "index_summary": {
              "internal-monitoring": {
                "index": ".monitoring-es-7-2022.09.06",
                "lastSeen": "2022-09-06T09:13:15.108Z"
              }
            },
            "index_recovery": {
              "internal-monitoring": {
                "index": ".monitoring-es-7-2022.09.06",
                "lastSeen": "2022-09-06T09:13:15.111Z"
              }
            },
            "index": {
              "internal-monitoring": {
                "index": ".monitoring-es-7-2022.09.06",
                "lastSeen": "2022-09-06T09:13:15.108Z"
              }
            },
            "cluster_stats": {
              "internal-monitoring": {
                "index": ".monitoring-es-7-2022.09.06",
                "lastSeen": "2022-09-06T09:13:15.099Z"
              }
            }
          }
        },
        "elasticsearch": {
          "6TyLurl-R3ChzhNEa2u8zw": {
            "node_stats": {
              "internal-monitoring": {
                "index": ".monitoring-es-7-2022.09.06",
                "lastSeen": "2022-09-06T09:13:15.061Z"
              }
            },
            "shard": {
              "internal-monitoring": {
                "index": ".monitoring-es-7-2022.09.06",
                "lastSeen": "2022-09-06T09:13:15.108Z"
              }
            }
          }
        },
        "kibana": {
          "71916d11-34bc-451d-b87e-f9ff2fb1137c": {
            "stats": {
              "internal-monitoring": {
                "index": ".monitoring-kibana-7-2022.09.06",
                "lastSeen": "2022-09-06T09:13:22.268Z"
              }
            }
          },
          "5b2de169-2785-441b-ae8c-186a1936b17d": {
            "stats": {
              "internal-monitoring": {
                "index": ".monitoring-kibana-7-2022.09.06",
                "lastSeen": "2022-09-06T09:13:23.866Z"
              }
            }
          }
        }
      },
      "standalone": {
        
      }
    },
    "execution": {
      "timedOut": false,
      "errors": [
        
      ]
    }
  },
  "metricbeatErrors": {
    "products": {
      
    },
    "execution": {
      "timedOut": false,
      "errors": [
        
      ]
    }
  },
  "settings": {
    "ccs": true,
    "logsIndex": "filebeat-*",
    "metricbeatIndex": "metricbeat-*",
    "hasRemoteClusterConfigured": false
  }
}

Maybe we can toss it around in our 1:1

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 296cb0d

Metrics [docs]

✅ unchanged

History

• 💛 Build #69563 was flaky f8d1a09
• 💚 Build #68041 succeeded 6b99818
• 💚 Build #67967 succeeded 7e3a1c1
• 💚 Build #67782 succeeded aec57b9
• 💚 Build #67446 succeeded cc8f319
• 💚 Build #65606 succeeded 8c39be9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @klacabane

[matschaffer]

We paired up on this and that last commit should have things working nicely!