-
Notifications
You must be signed in to change notification settings - Fork 8.6k
[Entity Store v2] Add CRUD API #252052
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
[Entity Store v2] Add CRUD API #252052
Changes from 73 commits
Commits
Show all changes
81 commits
Select commit
Hold shift + click to select a range
de04f1b
add boilerplate
kubasobon 5b2b0c6
add crud delete
kubasobon 37fb4d9
add upsert bulk
kubasobon 653726f
add entity schema
kubasobon 05f2d20
change schema to omit AC/Risk for now
kubasobon 232573f
generate temp schema and fix types
kubasobon 597fe1d
register routes
kubasobon a95d6e7
remove dependency on entityType
kubasobon b048751
delete files added by mistake
kubasobon fb77de0
Merge branch 'main' into entity-store-v2-crud
kubasobon b7ffde1
fix imports
kubasobon 24dd5f1
update with extra validation
kubasobon d05f012
drop EUID method
kubasobon ed80d21
add hashed ids
kubasobon 32184f5
update schema
kubasobon cf87382
rename to crud_client and reintroduce entityType
kubasobon ed7a859
update upsert & delete behavior
kubasobon e1c83e4
handle bulkUpsert entities
kubasobon da76347
handle request errors better
kubasobon d34f03e
add missing entity id field clause
kubasobon 28f6175
Merge branch 'main' into entity-store-v2-crud
kubasobon 94ebf32
Merge branch 'main' into entity-store-v2-crud
kubasobon 82bb6af
start working on Scout tests
kubasobon 2609597
add first scout test - working
kubasobon 0728382
update tests to reflect new routes
kubasobon a0675ed
change v2 api to internal
kubasobon 303d3a8
test force flag
kubasobon 39b2291
add upsert test
kubasobon 9c4011b
updates
kubasobon b923c65
fix bulk update test
kubasobon 67e65aa
add working scout test suite
kubasobon b0d445f
fix naming issue
kubasobon 567f0dd
delete entity id
kubasobon 290f091
Merge branch 'main' into entity-store-v2-crud
kubasobon 6cd8bf3
Changes from node scripts/lint_ts_projects --fix
kibanamachine 334d814
Changes from node scripts/regenerate_moon_projects.js --update
kibanamachine 25dec94
fix type errors
kubasobon b7b92ac
update last test
kubasobon d3459f3
Changes from node scripts/eslint_all_files --no-cache --fix
kibanamachine 1f84f43
add eslint marker
kubasobon ed896c0
Merge branch 'entity-store-v2-crud' of github.com:elastic/kibana into…
kubasobon 224ea5b
Merge branch 'main' into entity-store-v2-crud
kubasobon 70660e9
lowercase the attributes/relationships/behaviors etc.
kubasobon f9cdbdc
fix hanging promise issue
kubasobon 184b01c
fix ID/Hashed ID issues
kubasobon b93afbc
crud_client: use logger.debug
kubasobon a8fc7a4
crud_client: extract create/update methods, add explicit return values
kubasobon b505a38
crud_client: retry 3x on update conflict
kubasobon e3c3891
crud_api: remove redundant type casts
kubasobon 640870e
crud_api: move endpoints to a common directory
kubasobon 339a3d6
crud_api: extend async/bulk upsert test with forced extraction
kubasobon 1d52156
crud_api: remove two update results that never occur
kubasobon 9df544c
Changes from node scripts/eslint_all_files --no-cache --fix
kibanamachine d5f4add
crud_api: let upsert errors bubble up
kubasobon fdd4fbc
crud_api: update tests
kubasobon 7647f20
Merge branch 'entity-store-v2-crud' of github.com:elastic/kibana into…
kubasobon 71f5b72
update snapshots
kubasobon efc0f51
Changes from node scripts/eslint_all_files --no-cache --fix
kibanamachine e1499af
fix eslint issue
kubasobon 6fb8227
Merge branch 'entity-store-v2-crud' of github.com:elastic/kibana into…
kubasobon 2b4f0b0
crud_client: extract util funcs
kubasobon ffb61f0
crud_client: add utils.ts tests
kubasobon 82e3bab
Merge branch 'main' into entity-store-v2-crud
kubasobon 4cc1b08
crud_client: use update+doc_as_upsert
kubasobon 690395b
crud_api: throw 503 if not started for bulk
kubasobon 6114f34
crud_api: wait for shard sync on update
kubasobon 6ed1c02
crud_api: handle 503 for bulk api
kubasobon 86ddefa
crud_client: update transformDocForUpsert to be more readable and pre…
kubasobon bbe9e07
Merge branch 'main' into entity-store-v2-crud
kubasobon 322a612
crud_client: document sync/async upsert methods
kubasobon 0188a41
openapi: comment with script command
kubasobon 99cc470
Merge branch 'main' into entity-store-v2-crud
kubasobon bbdbdc6
openapi: make the comment clearer
kubasobon bc40d88
crud_client: apply review remarks
kubasobon 0d85afc
crud_client: make generic a const
kubasobon d6b2e4a
asset_manager: remove unused isInstalled function
kubasobon 5e00170
crud_client: rename util func for clarity
kubasobon 6edeb12
crud_client: filter bulk_upsert errors
kubasobon 5beb7cb
crud_client: delete based on EUID rather than hash
kubasobon 21e38e3
crud_client: update DELETE tests and check ids
kubasobon 94b69f3
Merge branch 'main' into entity-store-v2-crud
kubasobon File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
310 changes: 310 additions & 0 deletions
310
x-pack/solutions/security/plugins/entity_store/common/domain/definitions/entity.gen.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,310 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| /* | ||
| * NOTICE: Do not edit this file manually. | ||
| * This file is automatically generated by the OpenAPI Generator, @kbn/openapi-generator. | ||
| * | ||
| * info: | ||
| * title: Common Entities Schemas | ||
| * version: 1 | ||
| */ | ||
|
|
||
| import { z } from '@kbn/zod'; | ||
|
|
||
| export type EngineMetadata = z.infer<typeof EngineMetadata>; | ||
| export const EngineMetadata = z | ||
| .object({ | ||
| Type: z.string(), | ||
| }) | ||
| .strict(); | ||
|
|
||
| export type EntityRiskLevels = z.infer<typeof EntityRiskLevels>; | ||
| export const EntityRiskLevels = z.enum(['Unknown', 'Low', 'Moderate', 'High', 'Critical']); | ||
| export type EntityRiskLevelsEnum = typeof EntityRiskLevels.enum; | ||
| export const EntityRiskLevelsEnum = EntityRiskLevels.enum; | ||
|
|
||
| export type EntityField = z.infer<typeof EntityField>; | ||
| export const EntityField = z | ||
| .object({ | ||
| id: z.string(), | ||
| name: z.string().optional(), | ||
| type: z.string().optional(), | ||
| sub_type: z.string().optional(), | ||
| source: z.string().optional(), | ||
| EngineMetadata: EngineMetadata.optional(), | ||
| attributes: z | ||
| .object({ | ||
| privileged: z.boolean().optional(), | ||
| asset: z.boolean().optional(), | ||
| managed: z.boolean().optional(), | ||
| mfa_enabled: z.boolean().optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| behaviors: z | ||
| .object({ | ||
| brute_force_victim: z.boolean().optional(), | ||
| new_country_login: z.boolean().optional(), | ||
| used_usb_device: z.boolean().optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| lifecycle: z | ||
| .object({ | ||
| first_seen: z.string().datetime().optional(), | ||
| last_activity: z.string().datetime().optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| relationships: z | ||
| .object({ | ||
| communicates_with: z.array(z.string()).optional(), | ||
| depends_on: z.array(z.string()).optional(), | ||
| dependent_of: z.array(z.string()).optional(), | ||
| owns: z.array(z.string()).optional(), | ||
| owned_by: z.array(z.string()).optional(), | ||
| accesses_frequently: z.array(z.string()).optional(), | ||
| accessed_frequently_by: z.array(z.string()).optional(), | ||
| supervises: z.array(z.string()).optional(), | ||
| supervised_by: z.array(z.string()).optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| risk: z | ||
| .object({ | ||
| /** | ||
| * Lexical description of the entity's risk. | ||
| */ | ||
| calculated_level: EntityRiskLevels.optional(), | ||
| /** | ||
| * The raw numeric value of the given entity's risk score. | ||
| */ | ||
| calculated_score: z.number().optional(), | ||
| /** | ||
| * The normalized numeric value of the given entity's risk score. Useful for comparing with other entities. | ||
| */ | ||
| calculated_score_norm: z.number().min(0).max(100).optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| }) | ||
| .strict(); | ||
|
|
||
| /** | ||
| * The criticality level of the asset. | ||
| */ | ||
| export type AssetCriticalityLevel = z.infer<typeof AssetCriticalityLevel>; | ||
| export const AssetCriticalityLevel = z.enum([ | ||
| 'low_impact', | ||
| 'medium_impact', | ||
| 'high_impact', | ||
| 'extreme_impact', | ||
| ]); | ||
| export type AssetCriticalityLevelEnum = typeof AssetCriticalityLevel.enum; | ||
| export const AssetCriticalityLevelEnum = AssetCriticalityLevel.enum; | ||
|
|
||
| export type Asset = z.infer<typeof Asset>; | ||
| export const Asset = z | ||
| .object({ | ||
| id: z.string().optional(), | ||
| name: z.string().optional(), | ||
| owner: z.string().optional(), | ||
| serial_number: z.string().optional(), | ||
| model: z.string().optional(), | ||
| vendor: z.string().optional(), | ||
| environment: z.string().optional(), | ||
| criticality: AssetCriticalityLevel.optional(), | ||
| business_unit: z.string().optional(), | ||
| }) | ||
| .strict(); | ||
|
|
||
| /** | ||
| * A generic representation of a document contributing to a Risk Score. | ||
| */ | ||
| export type RiskScoreInput = z.infer<typeof RiskScoreInput>; | ||
| export const RiskScoreInput = z.object({ | ||
| /** | ||
| * The unique identifier (`_id`) of the original source document | ||
| */ | ||
| id: z.string(), | ||
| /** | ||
| * The unique index (`_index`) of the original source document | ||
| */ | ||
| index: z.string(), | ||
| /** | ||
| * The risk category of the risk input document. | ||
| */ | ||
| category: z.string(), | ||
| /** | ||
| * A human-readable description of the risk input document. | ||
| */ | ||
| description: z.string(), | ||
| /** | ||
| * The weighted risk score of the risk input document. | ||
| */ | ||
| risk_score: z.number().min(0).max(100).optional(), | ||
| /** | ||
| * The @timestamp of the risk input document. | ||
| */ | ||
| timestamp: z.string().optional(), | ||
| contribution_score: z.number().optional(), | ||
| }); | ||
|
|
||
| export type EntityRiskScoreRecord = z.infer<typeof EntityRiskScoreRecord>; | ||
| export const EntityRiskScoreRecord = z.object({ | ||
| /** | ||
| * The time at which the risk score was calculated. | ||
| */ | ||
| '@timestamp': z.string().datetime(), | ||
| /** | ||
| * The identifier field defining this risk score. Coupled with `id_value`, uniquely identifies the entity being scored. | ||
| */ | ||
| id_field: z.string(), | ||
| /** | ||
| * The identifier value defining this risk score. Coupled with `id_field`, uniquely identifies the entity being scored. | ||
| */ | ||
| id_value: z.string(), | ||
| /** | ||
| * Lexical description of the entity's risk. | ||
| */ | ||
| calculated_level: EntityRiskLevels, | ||
| /** | ||
| * The raw numeric value of the given entity's risk score. | ||
| */ | ||
| calculated_score: z.number(), | ||
| /** | ||
| * The normalized numeric value of the given entity's risk score. Useful for comparing with other entities. | ||
| */ | ||
| calculated_score_norm: z.number().min(0).max(100), | ||
| /** | ||
| * The contribution of Category 1 to the overall risk score (`calculated_score`). Category 1 contains Detection Engine Alerts. | ||
| */ | ||
| category_1_score: z.number(), | ||
| /** | ||
| * The number of risk input documents that contributed to the Category 1 score (`category_1_score`). | ||
| */ | ||
| category_1_count: z.number().int(), | ||
| /** | ||
| * A list of the highest-risk documents contributing to this risk score. Useful for investigative purposes. | ||
| */ | ||
| inputs: z.array(RiskScoreInput), | ||
| category_2_score: z.number().optional(), | ||
| category_2_count: z.number().int().optional(), | ||
| notes: z.array(z.string()), | ||
| criticality_modifier: z.number().optional(), | ||
| criticality_level: AssetCriticalityLevel.optional(), | ||
| /** | ||
| * A list of modifiers that were applied to the risk score calculation. | ||
| */ | ||
| modifiers: z | ||
| .array( | ||
| z.object({ | ||
| type: z.string(), | ||
| subtype: z.string().optional(), | ||
| modifier_value: z.number().optional(), | ||
| contribution: z.number(), | ||
| metadata: z.object({}).catchall(z.unknown()).optional(), | ||
| }) | ||
| ) | ||
| .optional(), | ||
| }); | ||
|
|
||
| export type UserEntity = z.infer<typeof UserEntity>; | ||
| export const UserEntity = z | ||
| .object({ | ||
| '@timestamp': z.string().datetime().optional(), | ||
| entity: EntityField, | ||
| user: z | ||
| .object({ | ||
| full_name: z.array(z.string()).optional(), | ||
| domain: z.array(z.string()).optional(), | ||
| roles: z.array(z.string()).optional(), | ||
| name: z.string(), | ||
| id: z.array(z.string()).optional(), | ||
| email: z.array(z.string()).optional(), | ||
| hash: z.array(z.string()).optional(), | ||
| risk: EntityRiskScoreRecord.optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| asset: Asset.optional(), | ||
| event: z | ||
| .object({ | ||
| ingested: z.string().datetime().optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| }) | ||
| .strict(); | ||
|
|
||
| export type HostEntity = z.infer<typeof HostEntity>; | ||
| export const HostEntity = z | ||
| .object({ | ||
| '@timestamp': z.string().datetime().optional(), | ||
| entity: EntityField, | ||
| host: z | ||
| .object({ | ||
| hostname: z.array(z.string()).optional(), | ||
| domain: z.array(z.string()).optional(), | ||
| ip: z.array(z.string()).optional(), | ||
| name: z.string(), | ||
| id: z.array(z.string()).optional(), | ||
| type: z.array(z.string()).optional(), | ||
| mac: z.array(z.string()).optional(), | ||
| architecture: z.array(z.string()).optional(), | ||
| risk: EntityRiskScoreRecord.optional(), | ||
| entity: EntityField.optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| asset: Asset.optional(), | ||
| event: z | ||
| .object({ | ||
| ingested: z.string().datetime().optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| }) | ||
| .strict(); | ||
|
|
||
| export type ServiceEntity = z.infer<typeof ServiceEntity>; | ||
| export const ServiceEntity = z | ||
| .object({ | ||
| '@timestamp': z.string().datetime().optional(), | ||
| entity: EntityField, | ||
| service: z | ||
| .object({ | ||
| name: z.string(), | ||
| risk: EntityRiskScoreRecord.optional(), | ||
| entity: EntityField.optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| asset: Asset.optional(), | ||
| event: z | ||
| .object({ | ||
| ingested: z.string().datetime().optional(), | ||
| }) | ||
| .strict() | ||
| .optional(), | ||
| }) | ||
| .strict(); | ||
|
|
||
| export type GenericEntity = z.infer<typeof GenericEntity>; | ||
| export const GenericEntity = z | ||
| .object({ | ||
| '@timestamp': z.string().datetime().optional(), | ||
| entity: EntityField, | ||
| asset: Asset.optional(), | ||
| }) | ||
| .strict(); | ||
|
|
||
| export const EntityInternal = z.union([UserEntity, HostEntity, ServiceEntity, GenericEntity]); | ||
|
|
||
| export type Entity = z.infer<typeof EntityInternal>; | ||
| export const Entity = EntityInternal as z.ZodType<Entity>; | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.