Validate new prometheus metrics are fixed

[MasterPtato]

• Reduce metrics sent by node exporter (cpu + memory + network + disk)

• Fix Vector high memory

  • Check allocation tracing

• 2 Vector instances

• Check buffering from vector ↔ vector

• Vector origin OOM killed

• Disable excess Traefik metrics

  • Does open_conn still show entrypoint label?

  curl localhost:9980/metrics | grep -v '^#'  | wc -l
  
  

• Expose API on 0.0.0.0 for vector

• Reduce metrics (erase memory)?

  # Total
  cat /tmp/metrics |  grep -v '^#'  | wc -l
  # Without Traefik
  cat /tmp/metrics |  grep -v '^#' | grep -v traefik | wc -l
  
  

• Remove extra node exporter meterics

  curl localhost:9100/metrics | grep -v '^#'  | wc -l
  
  

• Culptrits

  • Edge vector (high memory, sus) (TBD)

  • Traefik tunnel (seems to work because api-internal works)

    May 31 08:21:02 localhost traefik[2896]: 2024-05-31T08:21:02Z ERR Error while dialing backend error=EOF
    
    

  • Vector HTTP source (can we make this HA?) (TBD, why are some working fine)

  • Prometheus itself (might have helpful logs?)

• 2 fold

  • A: Vector doesn’t recover well when it can’t send events
  • B: A lot of requests are getting dropped in the tunnel (either tunnel or Vector itself)

• Disable high cardinality metrics

  

[linear]

RVT-3771 Validate new prometheus metrics are fixed