Validate Kubernetes Integration with TSDB

[gizas]

TSDB feature has been introduced since 8.5. Goal is to enable TSDB with latest Kubernetes Integration package and validate functionality of collection, ingestion and visualisation of Kubernetes resources.

Relevant information can be found here.

Goal*

Validate Kubernetes Integration functionality and UX with TSDB feature enabled

TSDB very simple example

Steps to reproduce the example

1. Create TSDB index:

PUT /testing-timeseries-mode
{
  "settings": {
    "index": {
        "routing_path": [
          "dimension-1",
          "dimension-2"
        ],
        "mode": "time_series"
    }
  },
  "mappings": {
    "properties": {
      "counter-field": { 
        "type": "integer",
        "time_series_metric": "counter"
      },
      "dimension-1": { 
        "type": "keyword",
        "time_series_dimension": true
      },
      "dimension-2": { 
        "type": "keyword",
        "time_series_dimension": true
      }
    }
  }
}

2. Try to add a document without any dimension (this is, no fields in the routing_path:

POST testing-timeseries-mode/_doc/
{
  "@timestamp": "2099-11-15T13:12:00",
  "counter-field": 123
}

It should fail:

3. Adding a document with a unique combination between timestamp and dimension should work:

POST testing-timeseries-mode/_doc/
{
  "@timestamp": "2099-11-15T13:12:00",
  "counter-field": 123,
  "dimension-1": "AA"
}

4. If we try to POST a document with the same dimension and timestamp, but with a different counter-field value, then it will fail because the combination already exists:

POST testing-timeseries-mode/_doc/
{
  "@timestamp": "2099-11-15T13:12:00",
  "counter-field": 321,
  "dimension-1": "AA"
}

6. If we still use the same dimension and timestamp, but use a new dimension, the new combination will be (timestamp, dimension1, dimension2) and that means it is still unique. It should work without any problems.

POST testing-timeseries-mode/_doc/
{
  "@timestamp": "2099-11-15T13:12:00",
  "counter-field": 321,
  "dimension-1": "AA",
  "dimension-2": "BB"
}

Outcomes:

• Configure a TSDB Elastic Stack with our Kubernetes Integration. We will need to update
• Verify functionality of Kubernetes Integration
  • Indexing Functionality
  • Kubernetes Dashboard functionality
• Validate our current Dimension fields, if they are accurate for TSDb configuration
• Validate any performance (storage savings, number of documents, performance of Dashboards/ queries). (This goal can be done in combination with the K8s Scaling Performance story - Link to TBD
• For evaluation of performance we can use rally tool:
  https://github.com/elastic/rally-tracks/tree/master/tsdb : Specific Track for TSDB
  https://github.com/elastic/k8s-integration-infra/tree/cloudnative/cloudnative : Track with data focused on K8s integration

Breaking the steps of the Story:

• Understanding TSDB Design
• TSDB design
• Time series datastream
• TSDB - Kibana behavior
• TSDB migration
• Checking relevant Prototyping work: 1, 2, 3, 4
• All metrics datastreams ([Kubernetes] Add support for TSDB - all metrics datastreams except the state ones #5464 and [Kubernetes] Add support for TSDB - set dimension fields for state data streams #5621):
  • Set dimension fields in all Kubernetes metrics datastreams
  • Toggle Option to enable TSDB in Integration UI (relevant link)
  • Validate UX when Kubernetes data already have been ingested inside Elastic Search and user enables TSDB feature. Searching in Discovery should be with no problems
  • Retrieve basic metrics for storage, indexing time etc. with TSDB enabled
• Check that each pre defined dashboard works as expected
  • Proxy: [Kubernetes][Dashboard] Fix proxy, controller manager and scheduler dashboards to support TSDB #5992
  • Scheduler: [Kubernetes][Dashboard] Fix proxy, controller manager and scheduler dashboards to support TSDB #5992
  • Controller manager: [Kubernetes][Dashboard] Fix proxy, controller manager and scheduler dashboards to support TSDB #5992
  • API Server
  • Cronjob
  • StatefulSet
  • Volume
  • Pods
  • Deployments
  • Daemonset
  • Job
  • Nodes
  • PV/PVC
  • Cluster Overview
  • Services

For the future:

• If TSDB mode is the default, then update all Kubernetes Datastreams with index_mode: "time_series"

[lalit-satapathy]

@constanca-m,

Only few data streams have the sum aggregation issue which causes problem using the dashboard.

Our understanding is that, the remaining data streams of k8 package, work fine (both in dashboard and regular usage) with TSDB enabled, Please confirm.

[constanca-m]

I updated the description to include which data streams have the dashboards working @lalit-satapathy. I already opened a PR to fix 3 of the 5 that were broken. I will start working on trying to fix the other 2.