Failing test: API Integration Tests.test/api_integration/apis/data_views/fields_for_wildcard_route/response·ts - apis index_patterns index_patterns/_fields_for_wildcard route fields_for_wildcard_route response returns 200 when index is closed

[kibanamachine]

A test failed on a tracked branch

Error: expected 200 "OK", got 404 "Not Found"
    at Context.<anonymous> (response.ts:252:10)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:74:16)
----
    at Test._assertStatus (node_modules/supertest/lib/test.js:252:14)
    at /opt/buildkite-agent/builds/bk-agent-prod-gcp-1731037086955086518/elastic/kibana-elasticsearch-snapshot-verify/kibana/node_modules/supertest/lib/test.js:308:13
    at Test._assertFunction (node_modules/supertest/lib/test.js:285:13)
    at Test.assert (node_modules/supertest/lib/test.js:164:23)
    at localAssert (node_modules/supertest/lib/test.js:120:14)
    at fn (node_modules/supertest/lib/test.js:125:7)
    at Test.callback (node_modules/superagent/src/node/index.js:899:3)
    at fn (node_modules/superagent/src/node/index.js:1165:18)
    at IncomingMessage.<anonymous> (node_modules/superagent/src/node/parsers/json.js:19:7)
    at IncomingMessage.emit (node:events:531:35)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21)

First failure: kibana-elasticsearch-snapshot-verify - 8.x

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 8.x

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - main

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - main

[delanni]

@elastic/kibana-data-discovery - This is not flakiness, this is due to a new ES snapshot. Can you help us triage if this is something we need to adjust to on the Kibana side?

[jughosta]

It does not look like the ES response for a closed index changed. I tested with the following and it's the same on older version.

PUT test
PUT test/_mapping
{
  "properties": {
    "actual": {
      "type": "keyword"
    }
  }
}
POST test/_field_caps?fields=*
POST test/_close
POST test/_field_caps?fields=*

Maybe it takes longer in some cases to create an index.

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 8.x

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 8.x

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - main

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - main

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - main

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 8.x

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - main

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 8.x

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - main

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 8.x

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - main

[kibanamachine]

New failure: kibana-elasticsearch-snapshot-verify - 8.x

[kertal]

@jughosta did you test with the latest snapshot? Looking at the test, I don't see a reason why there should be a 404

  it('returns 200 when index is closed', async () => {
      const es = getService('es');

      await es.indices.close({ index: 'basic_index' });

      await supertest
        .get(FIELDS_FOR_WILDCARD_PATH)
        .set(ELASTIC_HTTP_VERSION_HEADER, INITIAL_REST_VERSION_INTERNAL)
        .set(X_ELASTIC_INTERNAL_ORIGIN_REQUEST, 'kibana')
        .query({ pattern: 'basic_index' })
        .expect(200, {
          fields: [],
          indices: [],
        });
    });

according to the logs the closing was successful, but also it seems the cluster health state was RED before going to YELLOW? I don't think this is expected?

”-> "before each" hook: global before each for "returns 200 when index is closed"
[00:00:16]             │ info [o.e.c.m.MetadataIndexStateService] [ftr] closing indices [basic_index/UCol_hTtQJK87jIxTgQg5A]
[00:00:16]             │ info [o.e.c.m.MetadataIndexStateService] [ftr] completed closing of indices [basic_index]
[00:00:17]             │ info [o.e.c.r.a.AllocationService] [ftr] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[basic_index][0]]])." previous.health="RED" reason="shards started [[basic_index][0]]"
[00:00:17]             └- ✖ fail: apis index_patterns index_patterns/_fields_for_wildcard route fields_for_wildcard_route response returns 200 when index is closed

FYI @jbudz

[kertal]

Not sure if this is expected from ES side, that when an index is closed it's switching from RED to YELLOW? if this is, maybe @jughosta is right about "taking longer", it might take longer from ES side to get from the RED to the YELLOW state, and in between it causes a 404?

info [o.e.c.r.a.AllocationService] [ftr] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[basic_index][0]]])." previous.health="RED" reason="shards started [[basic_index][0]]"
[00:00:32]             └- ✖ fail: apis index_patterns index_patterns/_fields_for_wildcard route fields_for_wildcard_route response returns 200 when index is closed

[jughosta]

There is a change in ES response, please see #199654 (comment)

[jbudz]

We're going to skip, promote the new snapshot, and apply a fix in

kibana/src/plugins/data_views/server/fetcher/lib/es_api.ts

Line 97 in b628770

if (error.message.startsWith('cluster_block_exception')) {

[jbudz]

/skip

[kibanamachine]

Skipped

8.x: faf0496
main: e6265f2