[Automatic Import] Error handling when uploading a file#191310
Merged
ilyannn merged 22 commits intoelastic:mainfrom Sep 3, 2024
Merged
[Automatic Import] Error handling when uploading a file#191310ilyannn merged 22 commits intoelastic:mainfrom
ilyannn merged 22 commits intoelastic:mainfrom
Conversation
ilyannn
added
backport:prev-minor
8.16 candidate
enhancement
ilyannn
changed the title
Contributor
|
Pinging @elastic/security-scalability (Team:Security-Scalability) |
bhapas
reviewed
Aug 27, 2024
...create_integration/create_integration_assistant/steps/data_stream_step/sample_logs_input.tsx
Outdated
Show resolved
Hide resolved
...create_integration/create_integration_assistant/steps/data_stream_step/sample_logs_input.tsx
Show resolved
Hide resolved
Contributor
|
@semd Can you take a look considering the UI changes for wordings etc., |
Contributor
Author
|
@elasticmachine merge upstream |
Contributor
Author
|
@elasticmachine merge upstream |
semd
reviewed
Sep 2, 2024
...create_integration/create_integration_assistant/steps/data_stream_step/sample_logs_input.tsx
Show resolved
Hide resolved
semd
reviewed
Sep 2, 2024
...create_integration/create_integration_assistant/steps/data_stream_step/sample_logs_input.tsx
Show resolved
Hide resolved
semd
reviewed
Sep 2, 2024
...e_integration/create_integration_assistant/steps/data_stream_step/sample_logs_input.test.tsx
Outdated
Show resolved
Hide resolved
Contributor
Author
|
@elasticmachine merge upstream |
ilyannn
commented
Sep 2, 2024
...e_integration/create_integration_assistant/steps/data_stream_step/sample_logs_input.test.tsx
Show resolved
Hide resolved
...create_integration/create_integration_assistant/steps/data_stream_step/sample_logs_input.tsx
Show resolved
Hide resolved
… into auto-import/better-errors
Contributor
Author
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]Async chunks
History
To update your PR or re-run it, just comment with: cc @ilyannn |
Contributor
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
Member
|
@ilyannn was manual backport created for this one? |
bhapas
pushed a commit
to bhapas/kibana
that referenced
this pull request
Sep 10, 2024
### Release note
Display better error messages for issues with logs sample file upload in
Automatic Import.
## Summary
Previously the user would be told about parse issues that occur after
the file is successfully uploaded. However in the following scenarios
the operation would silently fail without displaying a user-visible
error:
1. When the file fails to upload (e.g. when it is too big).
2. When the upload operation is aborted, e.g. programmatically.
Additionally in the following scenario the generic `CAN_NOT_PARSE`
message was displayed:
3. When the file is uploaded but the browser runs out of memory when
trying to parse it.
Additionally, in the following scenario the `EMPTY` message was
displayed:
4. When the file is too big for the V8 engine (e.g. Chrome) to create a
string so upload process returns an empty string.
Additionally:
5. When the user switches from the invalid file (with an error
displayed) to the valid file, the error from the invalid file was
displayed during the analysis of the new file.
After the changes in this PR, the following error types would be
displayed in these cases, respectively:
1. `CAN_NOT_READ_WITH_REASON`: _An error occurred when reading logs
sample: {reason}_
2. `CAN_NOT_READ_WITH_REASON`: _An error occurred when reading logs
sample: An ongoing operation was aborted, typically with a call to
abort()._ (reason is provided by the browser)
3. `TOO_LARGE_TO_PARSE`: _This logs sample file is too large to parse_
4. `TOO_LARGE_TO_PARSE`: _This logs sample file is too large to parse_
5. No error would be displayed during the analysis.
This covers part of elastic/security-team#9844
though the issues were discovered separately.
Note that the fix in item 3 does not work in Firefox as it throws an
`InternalError` rather than `RangeError`. A generic `CAN_NOT_PARSE`
message will continue to be displayed in that case. The fix in item 4 is
only relevant for Chrome.
On a slightly different note, we provide the following improvements to
the log sampling functionality introduced in
elastic#190407:
- Add documentation for the `parseLogsContents` and its special cases
- Refactor the `parseLogsContent` output fields into protocols that
clearly define their optionality
- Add tests for the functionality of sampling when the format cannot be
determined
- Fix so that the error message is displayed for the case where
`fileContent == null` in `onChangeLogsSample`
### Risk Matrix
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Due to the complexity of browser engines, some of them might produce
unexpected events, or events in unexpected order and we will confuse the
user with an incorrect error message. | Low | Low | Testing. |
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
(cherry picked from commit 16c2bfe)
Contributor
|
@ebeahan No. I triggered one now. |
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
bhapas
added a commit
that referenced
this pull request
Sep 10, 2024
…) (#192464) # Backport This will backport the following commits from `main` to `8.15`: - [[Automatic Import] Error handling when uploading a file (#191310)](#191310) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ilya Nikokoshev","email":"ilya.nikokoshev@elastic.co"},"sourceCommit":{"committedDate":"2024-09-03T11:39:05Z","message":"[Automatic Import] Error handling when uploading a file (#191310)\n\n### Release note\r\n\r\nDisplay better error messages for issues with logs sample file upload in\r\nAutomatic Import.\r\n\r\n## Summary\r\n\r\nPreviously the user would be told about parse issues that occur after\r\nthe file is successfully uploaded. However in the following scenarios\r\nthe operation would silently fail without displaying a user-visible\r\nerror:\r\n\r\n1. When the file fails to upload (e.g. when it is too big).\r\n2. When the upload operation is aborted, e.g. programmatically.\r\n\r\nAdditionally in the following scenario the generic `CAN_NOT_PARSE`\r\nmessage was displayed:\r\n\r\n3. When the file is uploaded but the browser runs out of memory when\r\ntrying to parse it.\r\n\r\nAdditionally, in the following scenario the `EMPTY` message was\r\ndisplayed:\r\n\r\n4. When the file is too big for the V8 engine (e.g. Chrome) to create a\r\nstring so upload process returns an empty string.\r\n\r\nAdditionally:\r\n\r\n5. When the user switches from the invalid file (with an error\r\ndisplayed) to the valid file, the error from the invalid file was\r\ndisplayed during the analysis of the new file.\r\n\r\nAfter the changes in this PR, the following error types would be\r\ndisplayed in these cases, respectively:\r\n\r\n1. `CAN_NOT_READ_WITH_REASON`: _An error occurred when reading logs\r\nsample: {reason}_\r\n2. `CAN_NOT_READ_WITH_REASON`: _An error occurred when reading logs\r\nsample: An ongoing operation was aborted, typically with a call to\r\nabort()._ (reason is provided by the browser)\r\n3. `TOO_LARGE_TO_PARSE`: _This logs sample file is too large to parse_\r\n4. `TOO_LARGE_TO_PARSE`: _This logs sample file is too large to parse_\r\n5. No error would be displayed during the analysis.\r\n\r\nThis covers part of https://github.com/elastic/security-team/issues/9844\r\nthough the issues were discovered separately.\r\n\r\nNote that the fix in item 3 does not work in Firefox as it throws an\r\n`InternalError` rather than `RangeError`. A generic `CAN_NOT_PARSE`\r\nmessage will continue to be displayed in that case. The fix in item 4 is\r\nonly relevant for Chrome.\r\n\r\nOn a slightly different note, we provide the following improvements to\r\nthe log sampling functionality introduced in\r\nhttps://github.com//pull/190407:\r\n\r\n- Add documentation for the `parseLogsContents` and its special cases\r\n- Refactor the `parseLogsContent` output fields into protocols that\r\nclearly define their optionality\r\n- Add tests for the functionality of sampling when the format cannot be\r\ndetermined\r\n- Fix so that the error message is displayed for the case where\r\n`fileContent == null` in `onChangeLogsSample`\r\n\r\n### Risk Matrix\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes |\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n| Due to the complexity of browser engines, some of them might produce\r\nunexpected events, or events in unexpected order and we will confuse the\r\nuser with an incorrect error message. | Low | Low | Testing. |\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"16c2bfe41e585fdcdd3e796166ac3e2a6367f1a7","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","enhancement","backport:prev-minor","v8.16.0","Team:Security-Scalability"],"number":191310,"url":"https://github.com/elastic/kibana/pull/191310","mergeCommit":{"message":"[Automatic Import] Error handling when uploading a file (#191310)\n\n### Release note\r\n\r\nDisplay better error messages for issues with logs sample file upload in\r\nAutomatic Import.\r\n\r\n## Summary\r\n\r\nPreviously the user would be told about parse issues that occur after\r\nthe file is successfully uploaded. However in the following scenarios\r\nthe operation would silently fail without displaying a user-visible\r\nerror:\r\n\r\n1. When the file fails to upload (e.g. when it is too big).\r\n2. When the upload operation is aborted, e.g. programmatically.\r\n\r\nAdditionally in the following scenario the generic `CAN_NOT_PARSE`\r\nmessage was displayed:\r\n\r\n3. When the file is uploaded but the browser runs out of memory when\r\ntrying to parse it.\r\n\r\nAdditionally, in the following scenario the `EMPTY` message was\r\ndisplayed:\r\n\r\n4. When the file is too big for the V8 engine (e.g. Chrome) to create a\r\nstring so upload process returns an empty string.\r\n\r\nAdditionally:\r\n\r\n5. When the user switches from the invalid file (with an error\r\ndisplayed) to the valid file, the error from the invalid file was\r\ndisplayed during the analysis of the new file.\r\n\r\nAfter the changes in this PR, the following error types would be\r\ndisplayed in these cases, respectively:\r\n\r\n1. `CAN_NOT_READ_WITH_REASON`: _An error occurred when reading logs\r\nsample: {reason}_\r\n2. `CAN_NOT_READ_WITH_REASON`: _An error occurred when reading logs\r\nsample: An ongoing operation was aborted, typically with a call to\r\nabort()._ (reason is provided by the browser)\r\n3. `TOO_LARGE_TO_PARSE`: _This logs sample file is too large to parse_\r\n4. `TOO_LARGE_TO_PARSE`: _This logs sample file is too large to parse_\r\n5. No error would be displayed during the analysis.\r\n\r\nThis covers part of https://github.com/elastic/security-team/issues/9844\r\nthough the issues were discovered separately.\r\n\r\nNote that the fix in item 3 does not work in Firefox as it throws an\r\n`InternalError` rather than `RangeError`. A generic `CAN_NOT_PARSE`\r\nmessage will continue to be displayed in that case. The fix in item 4 is\r\nonly relevant for Chrome.\r\n\r\nOn a slightly different note, we provide the following improvements to\r\nthe log sampling functionality introduced in\r\nhttps://github.com//pull/190407:\r\n\r\n- Add documentation for the `parseLogsContents` and its special cases\r\n- Refactor the `parseLogsContent` output fields into protocols that\r\nclearly define their optionality\r\n- Add tests for the functionality of sampling when the format cannot be\r\ndetermined\r\n- Fix so that the error message is displayed for the case where\r\n`fileContent == null` in `onChangeLogsSample`\r\n\r\n### Risk Matrix\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes |\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n| Due to the complexity of browser engines, some of them might produce\r\nunexpected events, or events in unexpected order and we will confuse the\r\nuser with an incorrect error message. | Low | Low | Testing. |\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"16c2bfe41e585fdcdd3e796166ac3e2a6367f1a7"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","labelRegex":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191310","number":191310,"mergeCommit":{"message":"[Automatic Import] Error handling when uploading a file (#191310)\n\n### Release note\r\n\r\nDisplay better error messages for issues with logs sample file upload in\r\nAutomatic Import.\r\n\r\n## Summary\r\n\r\nPreviously the user would be told about parse issues that occur after\r\nthe file is successfully uploaded. However in the following scenarios\r\nthe operation would silently fail without displaying a user-visible\r\nerror:\r\n\r\n1. When the file fails to upload (e.g. when it is too big).\r\n2. When the upload operation is aborted, e.g. programmatically.\r\n\r\nAdditionally in the following scenario the generic `CAN_NOT_PARSE`\r\nmessage was displayed:\r\n\r\n3. When the file is uploaded but the browser runs out of memory when\r\ntrying to parse it.\r\n\r\nAdditionally, in the following scenario the `EMPTY` message was\r\ndisplayed:\r\n\r\n4. When the file is too big for the V8 engine (e.g. Chrome) to create a\r\nstring so upload process returns an empty string.\r\n\r\nAdditionally:\r\n\r\n5. When the user switches from the invalid file (with an error\r\ndisplayed) to the valid file, the error from the invalid file was\r\ndisplayed during the analysis of the new file.\r\n\r\nAfter the changes in this PR, the following error types would be\r\ndisplayed in these cases, respectively:\r\n\r\n1. `CAN_NOT_READ_WITH_REASON`: _An error occurred when reading logs\r\nsample: {reason}_\r\n2. `CAN_NOT_READ_WITH_REASON`: _An error occurred when reading logs\r\nsample: An ongoing operation was aborted, typically with a call to\r\nabort()._ (reason is provided by the browser)\r\n3. `TOO_LARGE_TO_PARSE`: _This logs sample file is too large to parse_\r\n4. `TOO_LARGE_TO_PARSE`: _This logs sample file is too large to parse_\r\n5. No error would be displayed during the analysis.\r\n\r\nThis covers part of https://github.com/elastic/security-team/issues/9844\r\nthough the issues were discovered separately.\r\n\r\nNote that the fix in item 3 does not work in Firefox as it throws an\r\n`InternalError` rather than `RangeError`. A generic `CAN_NOT_PARSE`\r\nmessage will continue to be displayed in that case. The fix in item 4 is\r\nonly relevant for Chrome.\r\n\r\nOn a slightly different note, we provide the following improvements to\r\nthe log sampling functionality introduced in\r\nhttps://github.com//pull/190407:\r\n\r\n- Add documentation for the `parseLogsContents` and its special cases\r\n- Refactor the `parseLogsContent` output fields into protocols that\r\nclearly define their optionality\r\n- Add tests for the functionality of sampling when the format cannot be\r\ndetermined\r\n- Fix so that the error message is displayed for the case where\r\n`fileContent == null` in `onChangeLogsSample`\r\n\r\n### Risk Matrix\r\n\r\n| Risk | Probability | Severity | Mitigation/Notes |\r\n\r\n|---------------------------|-------------|----------|-------------------------|\r\n| Due to the complexity of browser engines, some of them might produce\r\nunexpected events, or events in unexpected order and we will confuse the\r\nuser with an incorrect error message. | Low | Low | Testing. |\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"16c2bfe41e585fdcdd3e796166ac3e2a6367f1a7"}}]}] BACKPORT--> Co-authored-by: Ilya Nikokoshev <ilya.nikokoshev@elastic.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Previously the user would be told about parse issues that occur after the file is successfully uploaded. However in the following scenarios the operation would silently fail without displaying a user-visible error:
Additionally in the following scenario the generic
CAN_NOT_PARSEmessage was displayed:Additionally, in the following scenario the
EMPTYmessage was displayed:Additionally:
After the changes in this PR, the following error types would be displayed in these cases, respectively:
CAN_NOT_READ_WITH_REASON: An error occurred when reading logs sample: {reason}CAN_NOT_READ_WITH_REASON: An error occurred when reading logs sample: An ongoing operation was aborted, typically with a call to abort(). (reason is provided by the browser)TOO_LARGE_TO_PARSE: This logs sample file is too large to parseTOO_LARGE_TO_PARSE: This logs sample file is too large to parseThis covers part of https://github.com/elastic/security-team/issues/9844 though the issues were discovered separately.
Note that the fix in item 3 does not work in Firefox as it throws an
InternalErrorrather thanRangeError. A genericCAN_NOT_PARSEmessage will continue to be displayed in that case. The fix in item 4 is only relevant for Chrome.On a slightly different note, we provide the following improvements to the log sampling functionality introduced in #190407:
parseLogsContentsand its special casesparseLogsContentoutput fields into protocols that clearly define their optionalityfileContent == nullinonChangeLogsSampleRelease note
Display better error messages for issues with logs sample file upload in Automatic Import.
Example
Trying to upload a 30GB file (new error highlighted) in Firefox:
Trying to upload a large file in Island, with the special code that uses 15x times the memory:
Trying to upload a 3GB file in Island, which apparently cannot get scanned for viruses:
Trying to upload a 1GB file in Chrome before the PR changes (note event attributes):
Trying to upload a 1GB file in Chrome after the PR changes:
The error from the previous invalid file is not cleared before the PR changes:
Adding a programmatic
abort()intoonprogresswhich is called in Chrome after reading 150MB gives us:Checklist
Risk Matrix