Skip to content

Manually building KueryNode for Fleet's routes #75693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Sep 1, 2020
Merged

Manually building KueryNode for Fleet's routes #75693

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
8a490fd
Simple benchmark tests for kuery
kobelb Aug 21, 2020
0c23f97
Building manually is "better" still not free
kobelb Aug 21, 2020
3341219
Building the KueryNode manually
kobelb Aug 21, 2020
062aaf2
Removing benchmark tests
kobelb Aug 21, 2020
8bd226f
Another query is building the KueryNode manually
kobelb Aug 21, 2020
070552c
Empty strings are inherently falsy
kobelb Aug 21, 2020
225bb15
No longer reaching into the data plugin, import from the "root" indexes
kobelb Aug 25, 2020
0ee4f45
Using AGENT_ACTION_SAVED_OBJECT_TYPE everywhere
kobelb Aug 26, 2020
4cc972b
Merge remote-tracking branch 'upstream/master' into kuery-time
kobelb Aug 26, 2020
6da0f32
Adding SavedObjectsRepository#find unit test for KueryNode
kobelb Aug 26, 2020
7afb589
Adding KQL KueryNode test for validateConvertFilterToKueryNode
kobelb Aug 26, 2020
130d5cf
/s/KQL string/KQL expression
kobelb Aug 26, 2020
55051f8
Merge remote-tracking branch 'upstream/master' into kuery-time
kobelb Aug 26, 2020
4ebbe98
Updating API docs
kobelb Aug 26, 2020
97e19c0
Adding micro benchmark
kobelb Aug 27, 2020
aa14f7f
Revert "Adding micro benchmark"
kobelb Aug 27, 2020
205dd30
Merge remote-tracking branch 'upstream/master' into kuery-time
kobelb Aug 27, 2020
195a5a1
Adding an empty string filters test
kobelb Aug 31, 2020
f135722
Merge branch 'master' into kuery-time
elasticmachine Sep 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions src/core/server/saved_objects/service/lib/filter_utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,12 @@ const astFunctionType = ['is', 'range', 'nested'];

export const validateConvertFilterToKueryNode = (
allowedTypes: string[],
filter: string,
filter: string | KueryNode,
indexMapping: IndexMapping
): KueryNode | undefined => {
if (filter && filter.length > 0 && indexMapping) {
const filterKueryNode = esKuery.fromKueryExpression(filter);
if (filter && indexMapping) {
const filterKueryNode =
typeof filter === 'string' ? esKuery.fromKueryExpression(filter) : filter;
Comment on lines -34 to +36
Copy link
Contributor

@pgayvallet pgayvallet Aug 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: we are no longer ignoring empty string filters, but I guess this has no impact?

Copy link
Contributor Author

@kobelb kobelb Aug 31, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empty strings are still ignored, I added 195a5a1#diff-c5ab35755205b8adbf6cfc858d69ea3bR86-R88 to double-check this. JavaScript treats an empty-string as being falsy, so checking filter and filter.length > 0 is redundant, we only need to check filter.

Copy link
Contributor

@pgayvallet pgayvallet Sep 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait what? Kibana is not Java code?

Copy link
Contributor Author

@kobelb kobelb Sep 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, no... Maybe in the future though! New-new platform??

Copy link
Contributor

@pgayvallet pgayvallet Sep 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would need to find a more suitable codename if we switch to java. NewPlatformBijectiveAdapterFactory maybe.


const validationFilterKuery = validateFilterKueryNode({
astFilter: filterKueryNode,
Expand Down
5 changes: 4 additions & 1 deletion src/core/server/saved_objects/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ import { SavedObjectUnsanitizedDoc } from './serialization';
import { SavedObjectsMigrationLogger } from './migrations/core/migration_logger';
import { SavedObject } from '../../types';

// eslint-disable-next-line @kbn/eslint/no-restricted-paths
import { KueryNode } from '../../../plugins/data/common/es_query/kuery';

export {
SavedObjectAttributes,
SavedObjectAttribute,
Expand Down Expand Up @@ -81,7 +84,7 @@ export interface SavedObjectsFindOptions {
searchFields?: string[];
hasReference?: { type: string; id: string };
defaultSearchOperator?: 'AND' | 'OR';
filter?: string;
filter?: string | KueryNode;
namespaces?: string[];
/** An optional ES preference value to be used for the query **/
preference?: string;
Expand Down
33 changes: 31 additions & 2 deletions x-pack/plugins/ingest_manager/server/services/agents/actions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { Agent, AgentAction, AgentActionSOAttributes } from '../../../common/typ
import { AGENT_ACTION_SAVED_OBJECT_TYPE } from '../../../common/constants';
import { savedObjectToAgentAction } from './saved_objects';
import { appContextService } from '../app_context';
import { nodeTypes } from '../../../../../../src/plugins/data/common/es_query/kuery/node_types';

export async function createAgentAction(
soClient: SavedObjectsClientContract,
Expand All @@ -29,9 +30,20 @@ export async function getAgentActionsForCheckin(
soClient: SavedObjectsClientContract,
agentId: string
): Promise<AgentAction[]> {
const filter = nodeTypes.function.buildNode('and', [
nodeTypes.function.buildNode(
'not',
nodeTypes.function.buildNode(
'is',
`${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.sent_at`,
'*'
)
),
nodeTypes.function.buildNode('is', 'fleet-agent-actions.attributes.agent_id', agentId),
]);
const res = await soClient.find<AgentActionSOAttributes>({
type: AGENT_ACTION_SAVED_OBJECT_TYPE,
filter: `not ${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.sent_at: * and ${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.agent_id:${agentId}`,
filter,
});

return Promise.all(
Expand Down Expand Up @@ -78,9 +90,26 @@ export async function getAgentActionByIds(
}

export async function getNewActionsSince(soClient: SavedObjectsClientContract, timestamp: string) {
const filter = nodeTypes.function.buildNode('and', [
nodeTypes.function.buildNode(
'not',
nodeTypes.function.buildNode(
'is',
`${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.sent_at`,
'*'
)
),
nodeTypes.function.buildNode(
'range',
`${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.created_at`,
{
gte: timestamp,
}
),
]);
const res = await soClient.find<AgentActionSOAttributes>({
type: AGENT_ACTION_SAVED_OBJECT_TYPE,
filter: `not ${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.sent_at: * AND ${AGENT_ACTION_SAVED_OBJECT_TYPE}.attributes.created_at >= "${timestamp}"`,
filter,
});

return res.saved_objects.map(savedObjectToAgentAction);
Expand Down