[Security Solution] Data quality dashboard persistence#173185
[Security Solution] Data quality dashboard persistence#173185semd merged 89 commits intoelastic:mainfrom
Conversation
semd
added
Team:Threat Hunting
|
Pinging @elastic/security-threat-hunting (Team:Threat Hunting) |
|
Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore) |
|
@elasticmachine merge upstream |
TinaHeiligers
left a comment
TinaHeiligers
left a comment
There was a problem hiding this comment.
Spaces should be listed under optionalPlugins and you'll have to implement the if spaces do ... pattern.
TinaHeiligers
left a comment
TinaHeiligers
left a comment
There was a problem hiding this comment.
"spaces" can be disabled and should be listed as an optional plugin. See https://github.com/elastic/kibana/blob/main/src/plugins/saved_search/kibana.jsonc
|
While desk testing commit At first this appeared to be related to two users (in the same space) creating results, however the issue is reproducable even when one user only has Setup
elastic_and_read_only_user_480p.mov |
|
The Data Quality page displays the Elasticsearch permissions required to check an index when a user doesn't have sufficient privileges to complete a check. For example, a user assigned the built-in
will see the following when they don't have sufficient privileges to complete a check:
in this state:
When a user with a role that has all the permissions to perform data quality checks, but doesn't have permissions to persist them, like the following role:
When a user logged-in the the role above checks an index, posts to the
Consider:
|
| 'meta.batchId': { type: 'keyword', required: true }, | ||
| 'meta.ecsVersion': { type: 'keyword', required: true }, | ||
| 'meta.errorCount': { type: 'long', required: true }, | ||
| 'meta.ilmPhase': { type: 'keyword', required: true }, | ||
| 'meta.indexId': { type: 'keyword', required: true }, | ||
| 'meta.indexName': { type: 'keyword', required: true }, | ||
| 'meta.isCheckAll': { type: 'boolean', required: true }, | ||
| 'meta.numberOfDocuments': { type: 'long', required: true }, | ||
| 'meta.numberOfFields': { type: 'long', required: true }, | ||
| 'meta.numberOfIncompatibleFields': { type: 'long', required: true }, | ||
| 'meta.numberOfEcsFields': { type: 'long', required: true }, | ||
| 'meta.numberOfCustomFields': { type: 'long', required: true }, | ||
| 'meta.numberOfIndices': { type: 'long', required: true }, | ||
| 'meta.numberOfIndicesChecked': { type: 'long', required: true }, | ||
| 'meta.numberOfSameFamily': { type: 'long', required: true }, | ||
| 'meta.sameFamilyFields': { type: 'keyword', required: true, array: true }, | ||
| 'meta.sizeInBytes': { type: 'long', required: true }, | ||
| 'meta.timeConsumedMs': { type: 'long', required: true }, | ||
| 'meta.unallowedMappingFields': { type: 'keyword', required: true, array: true }, | ||
| 'meta.unallowedValueFields': { type: 'keyword', required: true, array: true }, |
There was a problem hiding this comment.
Consider removing the meta prefix, or an alternative prefix, if the meta prefix, i.e. meta.isCheckAll in the KQL query, and the sum of meta.numberOfIndicesChecked in the example Lens visualization below doesn't, convey what meta represents:
Put differently, does the meta prefix describe an entity, like cloud, host, or user in the ECS Field Reference?
|
Consider the following annotated screenshot of a DQ result: Click to expand this JSON document from index `.ds-.kibana-data-quality-dashboard-results-default-2024.01.22-000001`
"indexName":".internal.alerts-security.alerts-default-000002",
"numberOfDocuments":16684,
"docsCount":19103,however the
Results is a child of rollup (i.e. The contents of the singular entry in It appears that Consider:
Other items for consideration (not shown in the screenshot above):
"merges":{
"current":0,
"current_docs":0,
"current_size_in_bytes":0,
"total":3,
"total_time_in_millis":1033,
"total_docs":1639,
"total_size_in_bytes":9556294,
"total_stopped_time_in_millis":0,
"total_throttled_time_in_millis":0,
"total_auto_throttle_in_bytes":20971520
},
"segments":{
"count":8,
"memory_in_bytes":0,
"terms_memory_in_bytes":0,
"stored_fields_memory_in_bytes":0,
"term_vectors_memory_in_bytes":0,
"norms_memory_in_bytes":0,
"points_memory_in_bytes":0,
"doc_values_memory_in_bytes":0,
"index_writer_memory_in_bytes":0,
"version_map_memory_in_bytes":0,
"fixed_bit_set_memory_in_bytes":128,
"max_unsafe_auto_id_timestamp":-1,
"file_sizes":{
}Put differently, should the results persisted to the document include raw application state? |
|
Summary ✅ Startup of Kibana was resilient to all error scenarios (in the data stream installation process) that were simulated ✅ Errors resulting from ✅ The Data Quality dashboard correctly displayed error toasts when it failed to read or write results, e.g. ✅ It was possible to recover from all simulated error states (after removing the test code) by restarting Kibana Details The following dev tools console commands: were used to delete the assets created by the After resetting the installed assets, simulated errors were thrown throughout the functions called by ✅ and other code paths. |
It appears it may be possible for users to enter a state where it's not possible to recover from a simulated error state via If the following simulated error is thrown in /**
* Updates the data stream mapping and total field limit setting
*/
const updateDataStreamMappings = async ({
logger,
esClient,
totalFieldsLimit,
indexNames,
}: UpdateIndexMappingsOpts) => {
throw new Error('simulated error in updateDataStreamMappings');
// Update total field limit setting of found indices
// Other index setting changes are not updated at this time
// await Promise.all(
// indexNames.map((indexName) =>
// updateTotalFieldLimitSetting({ logger, esClient, totalFieldsLimit, indexName })
// )
// );
// // Update mappings of the found indices.
// await Promise.all(
// indexNames.map((indexName) => updateMapping({ logger, esClient, totalFieldsLimit, indexName }))
// );
};The error above was simulated after the data stream was created (at startup, on a previous run). After:
Errors are NOT logged at startup: however after the next step:
The Data Quality page displays a |
@andrew-goldstein I didn't know it was possible to stop the "check all". I bet the aborted (empty) result is incorrectly persisted. let me check that logic. |
This is the expected behavior. This data stream is "space-aware", it can not be created on server start, it is created during runtime when we know the space the user is using. If there's an error in the creation it will pop up to the UI. |
We discussed that with @dhru42, we don't want to restrict the /results POST or GET from the UI, all users that can use the data quality dashboard should be able to persist the check results and read the persisted checks. We are already preventing data leaks by checking the user's index privileges against the results being retrieved. This criteria only affects this specific data quality dashboard API route, which is only used to retrieve the latest check results of authorized index patterns. Arbitrary queries to |
@semd and I paired to debug this, and observed that an index (instead of a data stream) with empty mappings was created around the time the error above started appearing. These observations were made via: It appears that Elasticsearch created the index as a side effect, but we weren't able to reproduce this behavior. ResolutionIf the Data Quality page displays an
|
Yes, both users consistently observed a |
| const userEsClient = services.core.elasticsearch.client.asCurrentUser; | ||
| const privileges = await userEsClient.security.hasPrivileges({ | ||
| index: [ | ||
| { names: patterns.split(','), privileges: ['all', 'read', 'view_index_metadata'] }, |
There was a problem hiding this comment.
Consider revising the implementation of this privileges check, because per the following screenshot:
the following privileges are required to check an index:
monitorormanageview_index_metadataread
Consider:
- The
allprivilege is not required - either
monitorormanageis required to check an index- In a future release,
readmay be expanded to bereadorread_cross_cluster, if the Elasticsearchstatsand_ilm/explainAPIs are enhanced to support CCS.
- In a future release,
| { names: patterns.split(','), privileges: ['all', 'read', 'view_index_metadata'] }, | ||
| ], | ||
| }); | ||
| const authorizedPatterns = Object.keys(privileges.index).filter((pattern) => |
There was a problem hiding this comment.
Consider an alternative authorization check, because per the following scenario, this check appears to return unexpected results when two users visit the Data Quality page, but have different index privileges for the same pattern.
Example: given a pattern from the Security Solution default, e.g. packetbeat-*, one user may have privileges to view all packetbeat-* data streams and indicies (i.e. .ds-packetbeat-8.9.2-2023.12.06-000001 and packetbeat-7.17.9-2024.01.08-000002), while another user may have privileges for ONLY data streams created by the 8.x version of Packetbeat (i.e. .ds-packetbeat-8.9.2-2023.12.06-000001), but not the 7.x version of Packetbeat.
To demonstrate the unexpected results, first add the following console.log statements to the authorization step, such that the Kibana server logs include the privileges and derived authorizedPatterns:
// Confirm user has authorization for the requested patterns
const { patterns } = request.query;
const userEsClient = services.core.elasticsearch.client.asCurrentUser;
const privileges = await userEsClient.security.hasPrivileges({
index: [
{ names: patterns.split(','), privileges: ['all', 'read', 'view_index_metadata'] },
],
});
console.log('--> privileges', JSON.stringify(privileges, null, 2));
const authorizedPatterns = Object.keys(privileges.index).filter((pattern) =>
Object.values(privileges.index[pattern]).some((v) => v === true)
);
console.log('--> authorizedPatterns', JSON.stringify(authorizedPatterns, null, 2));
if (authorizedPatterns.length === 0) {
return response.ok({ body: [] });
}Next, create a new role called ONLY_SOME_PACKETBEAT_ROLE, such that
- The user has access to all
auditbeat-*indices - The user has access to only some
packetbeat-*indices
per the following screenshot of the ONLY_SOME_PACKETBEAT_ROLE:
In the screenshot above, the following pattern ONLY matches data streams created by the 8.x version of Packetbeat:
packetbeat-8.*
As a result, users with the ONLY_SOME_PACKETBEAT_ROLE may view (and check), for example, a data stream created by the 8.x version of Packetbeat, with a backing index named:
.ds-packetbeat-8.9.2-2023.12.06-000001
but that same user may NOT check an index created by the 7.x version of Packetbeat, e.g. an index named:
packetbeat-7.17.9-2024.01.08-000002
Next create a user named somepacketbeat, and assign that user the ONLY_SOME_PACKETBEAT_ROLE, per the following screenshot:
Finally, create one more user named alsosuperuser, and assign that user the superuser role, per the screenshot below:
Navigate to Stack Management > Users. Per the screenshot below:
✅ The alsosuperuser user has the superuser role (just like the elastic user)
✅ The somepacketbeat user has the ONLY_SOME_PACKETBEAT_ROLE
Establish a baseline with the elastic user
-
Login to Kibana as the
elasticuser, who has thesuperuserrole -
Navigate to Security > Dashboards > Data Qualty
-
Click the
Check allbutton
Note that:
✅ The .alerts-security.alerts-default pattern contains results, per the screenshot below
✅ The auditbeat-* pattern contains results, per previous screenshot above
✅ The packetbeat-* pattern contains 6 pages of results, which include indices from both the 8.x and 7.x versions of Packetbeat, per the screeenshot below:
Also observe the Kibana server logs console.log output for the elastic user:
--> privileges {
"username": "elastic",
"has_all_requested": true,
"cluster": {},
"index": {
".alerts-security.alerts-default": {
"all": true,
"read": true,
"view_index_metadata": true
},
"auditbeat-*": {
"all": true,
"read": true,
"view_index_metadata": true
},
"logs-*": {
"all": true,
"read": true,
"view_index_metadata": true
},
"packetbeat-*": {
"all": true,
"read": true,
"view_index_metadata": true
}
},
"application": {}
}
--> authorizedPatterns [
".alerts-security.alerts-default",
"auditbeat-*",
"logs-*",
"packetbeat-*"
]
Compare the alsosuperuser user with the elastic user
-
Login to Kibana as the
alsosuperuseruser, who also has thesuperuserrole, just like theelasticuser -
Navigate to Security > Dashboards > Data Quality
Note that:
✅ As expected, Last checked: -- is displayed for the alsosuperuser user, because this user has never performed a data quality check
✅ As expected, the .alerts-security.alerts-default pattern contains results, because the alsosuperuser is viewing results created by the elastic user, per the screenshot below:
✅ As expected, the auditbeat-* pattern contains results (from the elastic user), per previous screenshot above
✅ As expected, the packetbeat-* pattern contains 6 pages of results (created by the elastic user), which include indices from both the 8.x and 7.x versions of Packetbeat, per the screeenshot below:
Also observe the Kibana server logs console.log output for the alsosuperuser user:
--> privileges {
"username": "alsosuperuser",
"has_all_requested": true,
"cluster": {},
"index": {
".alerts-security.alerts-default": {
"all": true,
"read": true,
"view_index_metadata": true
},
"auditbeat-*": {
"all": true,
"read": true,
"view_index_metadata": true
},
"logs-*": {
"all": true,
"read": true,
"view_index_metadata": true
},
"packetbeat-*": {
"all": true,
"read": true,
"view_index_metadata": true
}
},
"application": {}
}
--> authorizedPatterns [
".alerts-security.alerts-default",
"auditbeat-*",
"logs-*",
"packetbeat-*"
]
So far, the output demonstrates that:
✅ As expected, the alsosuperuser user may view all the persisted results from the elastic user
✅ As expected, the privileges and derived authorizedPatterns are the same for both the alsosuperuser and elastic user
Compare the somepacketbeat user with the elastic user (has unexpected results)
-
Login to Kibana as the
somepacketbeatuser, who has DIFFERENT permissions compared with theelasticandalsosuperuserusers -
Navigate to Security > Dashboards > Data Quality
Note that:
✅ As expected, Last checked: -- is displayed for the somepacketbeat user, because this user has never performed a data quality check, per the screeenshot below
✅ As expected, the .alerts-security.alerts-default pattern is not displayed, because the somepacketbeat user (unlike the elastic user) does not have access to the .alerts-security.alerts-default pattern
✅ As expected, the auditbeat-* pattern contains results (from the elastic user)
✅ As expected, the packetbeat-* pattern contains 3 pages of indices, (as opposed to 6 for the elastic and alsosuperuser users), because the ONLY_SOME_PACKETBEAT_ROLE role only allows access to data streams created by the 8.x version of Packetbeat
❌ NOT EXPECTED: The packetbeat-* results (created by the elastic user) are NOT shown for the somepacketbeat user, even though the somepacketbeat has access to the 8.x Packetbeat data streams
This unexpected behavior appears to be related to the following unexpected authorizedPatterns derived state in the Kibana server log for the somepacketbeat user:
--> privileges {
"username": "somepacketbeat",
"has_all_requested": false,
"cluster": {},
"index": {
"auditbeat-*": {
"all": false,
"read": true,
"view_index_metadata": true
},
"packetbeat-*": {
"all": false,
"read": false,
"view_index_metadata": false
}
},
"application": {}
}
--> authorizedPatterns [
"auditbeat-*"
]
❌ NOT EXPECTED: The authorizedPatterns in the Kibana server log output for the somepacketbeat user does NOT include Packeatbeat, even though the stats API (correctly) returns results for data streams created by the 8.x version of packetbeat.
It appears that Packetbeat is not included in authorizedPatterns because the privileges entry for Packetbeat in the console.log output:
"packetbeat-*": {
"all": false,
"read": false,
"view_index_metadata": false
}
},
has "read": false and "view_index_metadata": false.
It appears both read and view_index_metadata are false for the somepacketbeat user because the packetbeat-* pattern matches data streams created by the 8.x version of Packetbeat (e.g. .ds-packetbeat-8.9.2-2023.12.06-000001), which the user has access to, AND indices created by the 7.x version (e.g. packetbeat-7.17.9-2024.01.08-000002), which the somepacketbeat user does NOT have access to.
Conclusion: it appears that the current implementation of authorization returns incorrect results when the specified pattern is not applicable to ALL data streams and / or indices that match the pattern.
andrew-goldstein
left a comment
andrew-goldstein
left a comment
There was a problem hiding this comment.
Thanks @semd for creating the new reusable Data Stream Adapter package, @kbn/data-stream-adapter, and consuming it to persist results in the Data Quality Dashboard! 🙏
The new package is useful for this use case (Data Quality dashboard persistence), and for creating data streams in future use cases.
Desk testing
🟢 Plugins that create new data streams via the @kbn/data-stream-adapter package at startup (when Kibana server starts) are resilient to errors that may occur.
For example, at least 4 API calls are made (some in parallel) at startup to create the data stream (metadata, without a backing index), index template, and component templates required to persist Data Quality dashboard results.
✅ Any one of the 4 API calls in this example may fail. While desk testing, none of those (simulated) API failures will prevent Kibana from starting. Details of the failure (at startup) are logged to the Kibana log.
✅ For some use cases, like the Data Quality dashboard, the backing indices for a data stream may only be created on demand (when users access a page), because the newly-created indices are space-specific. Errors that occur when attempting to create space-specific backing indices (on demand) are also logged to the Kibana log.
✅ The Data Quality dashboard displays error toasts when errors occur while reading or writing persisted results
🟡 Consider merging this PR to make the @kbn/data-stream-adapter package available to PRs that may consume it, but for now, consider disabling the creation of the data streams required for persisting Data Quality results at startup. This approach would enable the content of the Data Quality checks to evolve in a follow-up PR without the need to migrate mappings and results from their current state (in this PR) to a future state. Similarly, this would allow for revisions to the authorization check that's specific to data quality results.
LGTM 🚀
|
@elasticmachine merge upstream |
Sure, I am going to disable the persistence layer of the data quality dashboard so we can merge and make the data stream adapter library available in main.
|
angorayc
left a comment
angorayc
left a comment
There was a problem hiding this comment.
Thanks for adding this feature, Sergi, decent work! Very clean code and well structured. 👍
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]Public APIs missing comments
Async chunks
Public APIs missing exports
Page load bundle
Unknown metric groupsAPI count
ESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: cc @semd |
* main: (520 commits) Update Kibana code editor dependencies (#171720) [SLOs] Hide view in app in slo alerts table in slo details page (#175441) [api-docs] 2024-01-25 Daily api_docs build (#175502) [DOCS] Add buildkite links to doc preview comments (#175463) skip flaky suite (#175443) [Security Solution][Timeline] refactor timeline modal save timeline button (#175343) [RAM] Stack Management::Rules loses user selections when navigating back (#174954) [Security Solution][Timeline] refactor timeline modal attach to case button (#175163) Upgrade EUI to v92.1.1 (#174955) [Fleet]: Beta label is shown inconsistently while selecting proxy under Fleet settings. (#170634) [Cloud Security] Rules Combo Box filters Custom component (#175175) skip flaky suite (#175407) [Security Solution][Timeline] refactor timeline modal open timeline button (#175335) [Embedded Console] Introduce kbnSolutionNavOffset CSS variable (#175348) [Console] disable access to embedded console without dev tools capability (#175321) fix(x-pack/reporting): use FIPS-compliant ID generator `uuidv4` in Reporting plugin (#174809) [Security Solution] Data quality dashboard persistence (#173185) [RAM][Observability] Add alert fields table to Observability flyout (#174685) test: add missing await for connector table disappearance (#175430) [RAM][Maintenance Window] Fix maintenance window FE types and transforms (#173888) ...
20 participants
Summary
issue https://github.com/elastic/security-team/issues/7382
Data Stream Adapter
This PR introduces the
@kbn/data-stream-adapterpackage, which is a utility library to facilitate Data Stream creation and maintenance in Kibana, it was inspired by the data stream implementation in the Alerts plugin.The library has two exports:
DataStreamSpacesAdapter: to manage space data streams. It uses thename-of-the-data-stream-<spaceId>naming pattern.DataStreamAdapter: to manage single (not space-aware) data streams.Usage examples in the package README
Data Quality Dashboard
The
DataStreamSpacesAdapterhas been integrated into the data quality dashboard to store all the quality checks users perform. The information stored is the metadata (also used for telemetry) and the actual data rendered in the tables.FieldMap definition here
Demo
DQDashboard_persistance_demo.mov