[Task Manager] Adding list of explicitly de-registered task types

[ymao1]

Resolves #122389

Summary

Adds a list of explicitly de-registered task types and updates claim script to only mark a task as unrecognized if the task type exists in that list.

In order to explicitly stop registering a task, plugin authors will need to remove the task registration code as well as add the task type to the REMOVED_TYPES list in x-pack/plugins/task_manager/server/task_type_dictionary.ts

• If the task is not registered but not added to REMOVED_TYPES, it will not be marked as unrecognized. It will just not be claimed.
• If a task is added to REMOVED_TYPES but still registered, an error will be thrown. These scenarios should be caught in development.

Todo

• Add old siem rule type to list.

To Verify

1. Create a rule that runs frequently and make sure it runs (I used the People in Space rule). Query for the rule in the task manager index and see that status is either idle or running.
2. Update the code so that this rule type is no longer registered with alerting

--- a/x-pack/examples/alerting_example/server/plugin.ts
+++ b/x-pack/examples/alerting_example/server/plugin.ts
@@ -28,7 +28,7 @@ export interface AlertingExampleDeps {
 export class AlertingExamplePlugin implements Plugin<void, void, AlertingExampleDeps> {
   public setup(core: CoreSetup, { alerting, features }: AlertingExampleDeps) {
     alerting.registerType(alwaysFiringAlert);
-    alerting.registerType(peopleInSpaceAlert);
+    // alerting.registerType(peopleInSpaceAlert);

3. Verify that the rule no longer runs and the status stays idle
4. Revert the update made in step 2. Verify that the rule starts running again.
5. Update the code to add this rule type to the list of removed types

--- a/x-pack/plugins/task_manager/server/task_type_dictionary.ts
+++ b/x-pack/plugins/task_manager/server/task_type_dictionary.ts
@@ -14,6 +14,7 @@ import { Logger } from '../../../../src/core/server';
 export const REMOVED_TYPES: string[] = [
   // for testing
   'sampleTaskRemovedType',
+  'alerting:example.people-in-space',
 ];

6. Verify that the rule status is updated to unrecognized.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[ymao1]

@elasticmachine merge upstream

[ymao1]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[gmmorris]

Once this PR is merged will we not actually have any type in here?
It raises the question of whether we need the unusedTypes feature at all, no? 🤔

[pmuellr]

This reminds me of an issue I forgot to open up, when looking to disable apm telemetry. I believe when I looked into this, if apm was configured to disable telemetry, the existing telemetry task would eventually get marked as unrecognized, with no way to reverse that without explicitly deleting the task doc from the TM index. I think if we merge something like this, the existing code would then work fine.

Start here, where the apm plugin sets up telemetry stuff, including registering the telemetry task, in the plugin setup(), by calling createApmTelemetry() - but that call is made conditionally:

kibana/x-pack/plugins/apm/server/plugin.ts

Lines 67 to 94 in 8e8c4b8

	public setup(
	core: CoreSetup<APMPluginStartDependencies>,
	plugins: APMPluginSetupDependencies
	) {
	this.logger = this.initContext.logger.get();
	const config$ = this.initContext.config.create<APMConfig>();
	core.savedObjects.registerType(apmIndices);
	core.savedObjects.registerType(apmTelemetry);
	core.savedObjects.registerType(apmServerSettings);
	const currentConfig = this.initContext.config.get<APMConfig>();
	this.currentConfig = currentConfig;
	if (
	plugins.taskManager &&
	plugins.usageCollection &&
	currentConfig.telemetryCollectionEnabled
	) {
	createApmTelemetry({
	core,
	config$,
	usageCollector: plugins.usageCollection,
	taskManager: plugins.taskManager,
	logger: this.logger,
	kibanaVersion: this.initContext.env.packageInfo.version,
	});
	}

The actual task registration is in that createApmTelemetry() function:

kibana/x-pack/plugins/apm/server/lib/apm_telemetry/index.ts

Lines 37 to 64 in 8e8c4b8

	export async function createApmTelemetry({
	core,
	config$,
	usageCollector,
	taskManager,
	logger,
	kibanaVersion,
	}: {
	core: CoreSetup;
	config$: Observable<APMConfig>;
	usageCollector: UsageCollectionSetup;
	taskManager: TaskManagerSetupContract;
	logger: Logger;
	kibanaVersion: string;
	}) {
	taskManager.registerTaskDefinitions({
	[APM_TELEMETRY_TASK_NAME]: {
	title: 'Collect APM usage',
	createTaskRunner: () => {
	return {
	run: async () => {
	await collectAndStore();
	},
	cancel: async () => {},
	};
	},
	},
	});

It wouldn't surprise me to find other task registrations that are conditional like this. Again, I >think< with the change in this PR, such tasks would be ignored, until Kibana was later restarted and when down a conditional path to register the task again. And wouldn't end up in the unrecognized task black hole. Am I thinking about that right?

[ymao1]

It wouldn't surprise me to find other task registrations that are conditional like this. Again, I >think< with the change in this PR, such tasks would be ignored, until Kibana was later restarted and when down a conditional path to register the task again. And wouldn't end up in the unrecognized task black hole. Am I thinking about that right?

@pmuellr Yes, that's correct. With the change in this PR, those tasks would not get marked as unrecognized unless they are specifically added to the UNUSED_TYPES list. Instead task manager would just ignore them so they stay idle until a Kibana that does register the task type tries to claim them.

[ersin-erdal]

Don't we need to test the skipped tasks?

[ymao1]

Previously we maintained a list of claimableTaskTypes and skippedTaskTypes. These are all the types that this task manager knows about so if a task is not claimable or skipped, we mark is as unrecognized. Now we are just using claimableTaskTypes to claim and unusedTaskTypes to determine what to mark as unrecognized. Skipped task types will go into the else bucket and get a no-op.

[mikecote]

Changes LGTM, tested locally the following scenarios:

• Alerting rule with a task in "unrecognized" status returning to idle after upgrade
• Alerting rule with a task definition missing did not make the task have an unrecognized status
• Alerting rule with a task definition missing and part of REMOVED_TYPES caused the task status to become unrecognized

👍

[ymao1]

@elasticmachine merge upstream

[ersin-erdal]

Clean Code, group the arguments, love it :)

[ersin-erdal]

LGTM!

[ymao1]

@elasticmachine merge upstream

[ymao1]

@madirey @marshallmain

Tagging you on this PR for awareness. We adding an explicit registry of removed task types that will be marked as unrecognized. Adding alerting:siem.signals due to this PR that removed that rule type. There should be no change to the outcome of that PR (meaning old siem.signals rule types will still be marked as unrecognized)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 2dee057
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

✅ unchanged

History

• 💚 Build #23863 succeeded 09d30cd
• 💚 Build #22979 succeeded bda2fdd
• 💚 Build #21622 succeeded 0c40a92
• 💔 Build #21487 failed ed31d92
• 💔 Build #20759 failed 821bc7d
• 💚 Build #20717 succeeded 1a0453b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1

[tsullivan]

I just found out about this PR by investigating #120995 and finding I could no longer reproduce the Reporting issue. Thank you @ymao1!