-
Notifications
You must be signed in to change notification settings - Fork 255
[Streams] Update data quality calculations #4566
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 7 commits
f3d6cd7
9ee906d
481b25d
60cb98d
17f87b9
31adbe5
4d948c3
266d7bf
ddfff6c
9ee8d7a
9e5a80d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,24 +4,42 @@ applies_to: | |
| stack: preview =9.1, ga 9.2+ | ||
| --- | ||
|
|
||
| # Manage data quality [streams-data-retention] | ||
| # Manage data quality [streams-data-quality] | ||
|
|
||
| After selecting a stream, use the **Data quality** tab to find failed and degraded documents in your stream. Use the following components to monitor the health of your data and identify and fix issues: | ||
| From the **Streams** page, use the **Data quality** menu to filter your streams by data quality status, then select a stream to examine it more closely. After selecting a stream, use the **Data quality** tab to find failed and degraded documents in your stream. | ||
|
|
||
| - **Degraded documents:** Documents with the `ignored` property usually because of malformed fields or exceeding the limit of total fields when `ignore_above:false`. This component shows the total number of degraded documents, the percentage, and status (**Good**, **Degraded**, **Poor**). | ||
| - **Failed documents:**: Documents that were rejected during ingestion because of mapping conflicts or pipeline failures. | ||
| - **Quality score:** Streams calculates the overall quality score (**Good**, **Degraded**, **Poor**) based on the percentage of degraded and failed documents. | ||
| - **Trends over time:** A time-series chart so you can track how degraded and failed documents are accumulating over time. Use the date picker to zoom into a specific range and understand when problems are spiking. | ||
| - **Issues:** {applies_to}`stack: preview 9.2`Find issues with specific fields, how often they've occurred, and when they've occurred. | ||
| Use the following components to monitor the health of your data and identify and fix issues: | ||
|
|
||
| ## Failure store | ||
| * **Degraded documents:** Documents from the last backing index of the stream with the `ignored` property, usually because of malformed fields or exceeding the limit of total fields when `ignore_above` is set to `false`. This component shows: | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually, the
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the review @ElenaStoeva! Would you mind taking another look to make sure I got that update correct?
mdbirnstiehl marked this conversation as resolved.
Outdated
|
||
| * Total number of degraded documents. | ||
|
mdbirnstiehl marked this conversation as resolved.
|
||
| * Percentage of degraded documents relative to the total document count from the stream's last backing index. | ||
|
mdbirnstiehl marked this conversation as resolved.
|
||
| * The data quality status (**Good**, **Degraded**, **Poor**). | ||
|
mdbirnstiehl marked this conversation as resolved.
|
||
| * **Failed documents:** Documents that were rejected during ingestion because of mapping conflicts or pipeline failures. This component shows: | ||
| * Total number of failed documents that correspond with this stream from within the specified time range in the date picker. Refer to [Failure store](#streams-data-quality-failure) for more information. | ||
| * Percentage of failed documents relative to the total document count from the stream's last backing index. | ||
|
mdbirnstiehl marked this conversation as resolved.
|
||
| * The data quality status (**Good**, **Degraded**, **Poor**). | ||
|
mdbirnstiehl marked this conversation as resolved.
|
||
| * **Quality score:** Streams calculates the overall quality score (**Good**, **Degraded**, **Poor**) based on the percentage of degraded and failed documents. Refer to [Data quality calculation](#streams-data-quality-calculation) for more information. | ||
| * **Trends over time:** A time-series chart so you can track how degraded and failed documents are accumulating over time. Use the date picker to focus on a specific range and understand when problems are spiking. | ||
| * {applies_to}`stack: preview 9.2` **Issues:** Find issues with specific fields, how often they've occurred, and when they've occurred. | ||
|
|
||
| A [failure store](../../../../manage-data/data-store/data-streams/failure-store.md) is a secondary set of indices inside a data stream, dedicated to storing failed documents. Instead of losing documents that are rejected during ingestion, a failure store retains it in a `::failures` index, so you can review failed documents to understand what went wrong and how to fix it. | ||
| ## Data quality calculation [streams-data-quality-calculation] | ||
|
|
||
| Streams calculates data quality as follows: | ||
|
|
||
| * **Good:** Both the **Degraded documents** percentage and the **Failed documents** percentage are 0. | ||
| * **Degraded:** Either the **Degraded documents** percentage or the **Failed documents** percentage are greater than 0 and less than or equal to 3. | ||
| * **Poor:** Either the **Degraded documents** percentage or the **Failed documents** percentage are greater than 3. | ||
|
|
||
| ## Failure store [streams-data-quality-failure] | ||
|
mdbirnstiehl marked this conversation as resolved.
|
||
|
|
||
| A [failure store](../../../../manage-data/data-store/data-streams/failure-store.md) is a secondary set of indices inside a data stream, dedicated to storing failed documents. Instead of losing documents that are rejected during ingestion, a failure store retains them in a `::failures` index, so you can review failed documents to understand what went wrong and how to fix it. | ||
|
|
||
| For example, for a stream called `my-stream`, Streams fetches all documents from the `my-stream::failures` index from within the specified time range in the date picker. | ||
|
|
||
| ### Required permissions | ||
|
mdbirnstiehl marked this conversation as resolved.
|
||
| To view and modify failure store in {{stack}}, you need the following data stream level privileges: | ||
| - `read_failure_store` | ||
| - `manage_failure_store` | ||
| * `read_failure_store` | ||
| * `manage_failure_store` | ||
|
|
||
| For more information, refer to [Granting privileges for data streams and aliases](../../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md). | ||
|
|
||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.