Skip to content

[Usage counters] Relocate counters to a dedicated index. Unify / simplify 'ui' and 'server' counters logic. #187064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gsoldevila merged 14 commits into from
Jul 5, 2024
Merged

[Usage counters] Relocate counters to a dedicated index. Unify / simplify 'ui' and 'server' counters logic. #187064

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
6f94e8a
Relocate usage-counters to dedicated index and split server Vs ui
gsoldevila Jun 27, 2024
f1eca22
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine Jun 27, 2024
b9a828f
Use a single SO type, introduce 'source' attribute to distinguish 'ui…
gsoldevila Jun 28, 2024
c656fb3
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine Jun 28, 2024
1c226c6
[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…
kibanamachine Jul 1, 2024
6c91c98
Update tests to take into account new SO type `counters` and index `.…
gsoldevila Jul 1, 2024
fff9e1c
Fix typo in test
gsoldevila Jul 2, 2024
a273eb1
Misc enhancements
gsoldevila Jul 2, 2024
b7a2d54
Address PR feedback
gsoldevila Jul 3, 2024
b28d019
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
kibanamachine Jul 3, 2024
c61df3b
[CI] Auto-commit changed files from 'node scripts/jest_integration -u…
kibanamachine Jul 3, 2024
aed12b4
Refactor / fix UTs
gsoldevila Jul 4, 2024
6c7a6cf
Small tweaks
gsoldevila Jul 5, 2024
f0fe90c
Merge branch 'main' into kbn-dashboards-plus-plus
gsoldevila Jul 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/core/saved-objects/core-saved-objects-server/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ export {
ALERTING_CASES_SAVED_OBJECT_INDEX,
SECURITY_SOLUTION_SAVED_OBJECT_INDEX,
ANALYTICS_SAVED_OBJECT_INDEX,
USAGE_COUNTERS_SAVED_OBJECT_INDEX,
ALL_SAVED_OBJECT_INDICES,
} from './src/saved_objects_index_pattern';
export type {
Expand Down
3 changes: 3 additions & 0 deletions packages/core/saved-objects/core-saved-objects-server/src/saved_objects_index_pattern.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,14 @@ export const INGEST_SAVED_OBJECT_INDEX = `${MAIN_SAVED_OBJECT_INDEX}_ingest`;
export const ALERTING_CASES_SAVED_OBJECT_INDEX = `${MAIN_SAVED_OBJECT_INDEX}_alerting_cases`;
export const SECURITY_SOLUTION_SAVED_OBJECT_INDEX = `${MAIN_SAVED_OBJECT_INDEX}_security_solution`;
export const ANALYTICS_SAVED_OBJECT_INDEX = `${MAIN_SAVED_OBJECT_INDEX}_analytics`;
export const USAGE_COUNTERS_SAVED_OBJECT_INDEX = `${MAIN_SAVED_OBJECT_INDEX}_usage_counters`;
Copy link
Copy Markdown
Member

@afharo afharo Jun 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧡


export const ALL_SAVED_OBJECT_INDICES = [
MAIN_SAVED_OBJECT_INDEX,
TASK_MANAGER_SAVED_OBJECT_INDEX,
ALERTING_CASES_SAVED_OBJECT_INDEX,
INGEST_SAVED_OBJECT_INDEX,
SECURITY_SOLUTION_SAVED_OBJECT_INDEX,
ANALYTICS_SAVED_OBJECT_INDEX,
USAGE_COUNTERS_SAVED_OBJECT_INDEX,
];
6 changes: 6 additions & 0 deletions packages/kbn-check-mappings-update-cli/current_fields.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1029,6 +1029,12 @@
"createDate",
"slug"
],
"usage-counter": [
"counterName",
"counterType",
"domainId",
"source"
],
"usage-counters": [
"domainId"
],
Expand Down
17 changes: 17 additions & 0 deletions packages/kbn-check-mappings-update-cli/current_mappings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3407,6 +3407,23 @@
}
}
},
"usage-counter": {
"dynamic": false,
"properties": {
"counterName": {
"type": "keyword"
},
"counterType": {
"type": "keyword"
},
"domainId": {
"type": "keyword"
},
"source": {
"type": "keyword"
}
}
},
"usage-counters": {
"dynamic": false,
"properties": {
Expand Down
1 change: 1 addition & 0 deletions src/core/server/integration_tests/ci_checks/saved_objects/check_registered_types.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ describe('checking migration metadata changes on all registered SO types', () =>
"uptime-dynamic-settings": "b6756ff71d6b5258971b1c8fd433d167affbde52",
"uptime-synthetics-api-key": "7ae976a461248f9dbd8442af14a179bdbc229eca",
"url": "c923a4a5002a09c0080c9095e958f07d518e6704",
"usage-counter": "3a104db0c9867da2d0436e20604a11dc5d0bb59e",
"usage-counters": "48782b3bcb6b5a23ba6f2bfe3a380d835e68890a",
"visualization": "93a3e73994ad836fe2b1dccbe208238f41f63da0",
"workplace_search_telemetry": "52b32b47ee576f554ac77cb1d5896dfbcfe9a1fb",
Expand Down
2 changes: 1 addition & 1 deletion ...re/server/integration_tests/saved_objects/migrations/group2/check_target_mappings.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ describe('migration v2 - CHECK_TARGET_MAPPINGS', () => {

// Check for migration steps present in the logs
logs = await fs.readFile(logFilePath, 'utf-8');
expect(logs).not.toMatch('CREATE_NEW_TARGET');
expect(logs).not.toMatch('[.kibana] CREATE_NEW_TARGET');
Copy link
Copy Markdown
Member Author

@gsoldevila gsoldevila Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test now creates a new index for the new type.
I made the constraint more specific and ensure that a new .kibana is not being created.

expect(logs).toMatch('CHECK_TARGET_MAPPINGS -> UPDATE_TARGET_MAPPINGS_PROPERTIES');
expect(logs).toMatch(
'UPDATE_TARGET_MAPPINGS_PROPERTIES -> UPDATE_TARGET_MAPPINGS_PROPERTIES_WAIT_FOR_TASK'
Expand Down
3 changes: 2 additions & 1 deletion src/core/server/integration_tests/saved_objects/migrations/group3/type_registrations.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,8 @@ const previouslyRegisteredTypes = [
'synthetics-dynamic-settings',
'uptime-synthetics-api-key',
'url',
'usage-counters',
'usage-counter', // added in 8.16.0: richer mappings, located in .kibana_usage_counters
'usage-counters', // deprecated in favor of 'usage-counter'
'visualization',
'workplace_search_telemetry',
].sort();
Expand Down
139 changes: 10 additions & 129 deletions src/core/server/integration_tests/saved_objects/migrations/group5/dot_kibana_split.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,135 +173,15 @@ describe('split .kibana index into multiple system indices', () => {
})
);

expect(indicesInfo[`.kibana_${currentVersion}_001`].mappings?._meta?.indexTypesMap)
.toMatchInlineSnapshot(`
Object {
".kibana": Array [
"action",
"action_task_params",
"ad_hoc_run_params",
"alert",
"api_key_pending_invalidation",
"apm-custom-dashboards",
"apm-indices",
"apm-server-schema",
"apm-service-group",
"apm-telemetry",
"app_search_telemetry",
"application_usage_daily",
"application_usage_totals",
"canvas-element",
"canvas-workpad-template",
"cases",
"cases-comments",
"cases-configure",
"cases-connector-mappings",
"cases-rules",
"cases-telemetry",
"cases-user-actions",
"cloud-security-posture-settings",
"config",
"config-global",
"connector_token",
"core-usage-stats",
"csp-rule-template",
"endpoint:unified-user-artifact-manifest",
"endpoint:user-artifact-manifest",
"enterprise_search_telemetry",
"entity-definition",
"entity-discovery-api-key",
"epm-packages",
"epm-packages-assets",
"event-annotation-group",
"event_loop_delays_daily",
"exception-list",
"exception-list-agnostic",
"file",
"file-upload-usage-collection-telemetry",
"fileShare",
"fleet-fleet-server-host",
"fleet-message-signing-keys",
"fleet-preconfiguration-deletion-record",
"fleet-proxy",
"fleet-setup-lock",
"fleet-uninstall-tokens",
"graph-workspace",
"guided-onboarding-guide-state",
"guided-onboarding-plugin-state",
"index-pattern",
"infra-custom-dashboards",
"infrastructure-monitoring-log-view",
"infrastructure-ui-source",
"ingest-agent-policies",
"ingest-download-sources",
"ingest-outputs",
"ingest-package-policies",
"ingest_manager_settings",
"inventory-view",
"kql-telemetry",
"legacy-url-alias",
"lens",
"lens-ui-telemetry",
"links",
"maintenance-window",
"map",
"metrics-data-source",
"metrics-explorer-view",
"ml-job",
"ml-module",
"ml-trained-model",
"monitoring-telemetry",
"observability-onboarding-state",
"osquery-manager-usage-metric",
"osquery-pack",
"osquery-pack-asset",
"osquery-saved-query",
"policy-settings-protection-updates-note",
"query",
"risk-engine-configuration",
"rules-settings",
"sample-data-telemetry",
"search-session",
"search-telemetry",
"security-rule",
"security-solution-signals-migration",
"siem-detection-engine-rule-actions",
"siem-ui-timeline",
"siem-ui-timeline-note",
"siem-ui-timeline-pinned-event",
"slo",
"slo-settings",
"space",
"spaces-usage-stats",
"synthetics-dynamic-settings",
"synthetics-monitor",
"synthetics-param",
"synthetics-privates-locations",
"tag",
"telemetry",
"threshold-explorer-view",
"ui-metric",
"upgrade-assistant-ml-upgrade-operation",
"upgrade-assistant-reindex-operation",
"uptime-dynamic-settings",
"uptime-synthetics-api-key",
"url",
"usage-counters",
"workplace_search_telemetry",
],
".kibana_so_search": Array [
"search",
],
".kibana_so_ui": Array [
"canvas-workpad",
"dashboard",
"visualization",
],
".kibana_task_manager": Array [
"task",
],
}
`);
const typesMap = indicesInfo[`.kibana_${currentVersion}_001`].mappings?._meta?.indexTypesMap;

expect(Array.isArray(typesMap['.kibana'])).toEqual(true);
expect(typesMap['.kibana'].length > 50).toEqual(true);
expect(typesMap['.kibana'].includes('action')).toEqual(true);
Copy link
Copy Markdown
Member Author

@gsoldevila gsoldevila Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking that the map contains an exhaustive list of mappings is not necessary.
This test is broken everytime anyone adds a new SO type.
I've relaxed the constraint a bit.

expect(typesMap['.kibana'].includes('cases')).toEqual(true);
expect(typesMap['.kibana_so_search']).toEqual(['search']);
expect(typesMap['.kibana_so_ui']).toEqual(['canvas-workpad', 'dashboard', 'visualization']);
expect(typesMap['.kibana_task_manager']).toEqual(['task']);

const logs = await parseLogFile(logFilePathFirstRun);

Expand Down Expand Up @@ -506,6 +386,7 @@ describe('split .kibana index into multiple system indices', () => {
'.kibana_task_manager': {
task: 5,
},
'.kibana_usage_counters': {},
});
});

Expand Down
60 changes: 29 additions & 31 deletions src/core/server/integration_tests/saved_objects/service/lib/repository_with_proxy.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import Hapi from '@hapi/hapi';
import h2o2 from '@hapi/h2o2';
import { URL } from 'url';
import { SavedObject, ALL_SAVED_OBJECT_INDICES } from '@kbn/core-saved-objects-server';
import type { SavedObject } from '@kbn/core-saved-objects-server';
import type { ISavedObjectsRepository } from '@kbn/core-saved-objects-api-server';
import type { InternalCoreSetup, InternalCoreStart } from '@kbn/core-lifecycle-server-internal';
import { Root } from '@kbn/core-root-server-internal';
Expand All @@ -18,7 +18,6 @@ import {
createTestServers,
type TestElasticsearchUtils,
} from '@kbn/core-test-helpers-kbn-server';
import { kibanaPackageJson as pkg } from '@kbn/repo-info';
import {
declareGetRoute,
declareDeleteRoute,
Expand All @@ -32,7 +31,8 @@ import {
declarePassthroughRoute,
declareIndexRoute,
setProxyInterrupt,
allCombinationsPermutations,
getVersionedKibanaIndex,
getIndicesWithNamespaceAwareTypes,
} from './repository_with_proxy_utils';

let esServer: TestElasticsearchUtils;
Expand Down Expand Up @@ -93,29 +93,40 @@ describe('404s from proxies', () => {
? parseInt(process.env.TEST_PROXY_SERVER_PORT, 10)
: 5698;

// Setup kibana configured to use proxy as ES backend
root = createRootWithCorePlugins({
elasticsearch: {
hosts: [`http://${esHostname}:${proxyPort}`],
},
migrations: {
skip: false,
},
});
await root.preboot();
const setup = await root.setup();
registerSOTypes(setup);

// Setup custom hapi hapiServer with h2o2 plugin for proxying
hapiServer = Hapi.server({
port: proxyPort,
});

const mainIndex = getVersionedKibanaIndex();
await hapiServer.register(h2o2);
// register specific routes to modify the response and a catch-all to relay the request/response as-is
declareGetRoute(hapiServer, esHostname, esPort, mainIndex);
declareDeleteRoute(hapiServer, esHostname, esPort, mainIndex);
declarePostUpdateRoute(hapiServer, esHostname, esPort, mainIndex);

allCombinationsPermutations(
ALL_SAVED_OBJECT_INDICES.map((indexPattern) => `${indexPattern}_${pkg.version}`)
)
.map((indices) => indices.join(','))
.forEach((kbnIndexPath) => {
declareGetRoute(hapiServer, esHostname, esPort, kbnIndexPath);
declareDeleteRoute(hapiServer, esHostname, esPort, kbnIndexPath);
declarePostUpdateRoute(hapiServer, esHostname, esPort, kbnIndexPath);

declareGetSearchRoute(hapiServer, esHostname, esPort, kbnIndexPath);
declarePostSearchRoute(hapiServer, esHostname, esPort, kbnIndexPath);
declarePostPitRoute(hapiServer, esHostname, esPort, kbnIndexPath);
declarePostUpdateByQueryRoute(hapiServer, esHostname, esPort, kbnIndexPath);
declareIndexRoute(hapiServer, esHostname, esPort, kbnIndexPath);
});
declareGetSearchRoute(hapiServer, esHostname, esPort, mainIndex);
declarePostSearchRoute(hapiServer, esHostname, esPort, mainIndex);
declarePostPitRoute(hapiServer, esHostname, esPort, mainIndex);
declareIndexRoute(hapiServer, esHostname, esPort, mainIndex);

// the deleteByNamespace performs an updateByQuery under the hood.
// It targets all SO indices that have namespace-aware types
const nsIndices = getIndicesWithNamespaceAwareTypes(setup.savedObjects.getTypeRegistry());
Copy link
Copy Markdown
Member Author

@gsoldevila gsoldevila Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previously we were running a very costly operation, registering thousands of routes, when 99% of requests target .kibana solely.
The only case where multiple indices were specified in the HTTP request path is with namespace-aware indices.
The test was timing out due to the new SO index, which added a lot more permutations.
Hence, I used the opportunity to simplify it.
The test should now be noticeably ligher/faster.

declarePostUpdateByQueryRoute(hapiServer, esHostname, esPort, nsIndices);

// register index-agnostic routes
declarePostBulkRoute(hapiServer, esHostname, esPort);
Expand All @@ -124,19 +135,6 @@ describe('404s from proxies', () => {

await hapiServer.start();

// Setup kibana configured to use proxy as ES backend
root = createRootWithCorePlugins({
elasticsearch: {
hosts: [`http://${esHostname}:${proxyPort}`],
},
migrations: {
skip: false,
},
});
await root.preboot();
const setup = await root.setup();
registerSOTypes(setup);

start = await root.start();
});

Expand Down
Loading