Skip to content

Make CSV upload limit configurable #8228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Bargs merged 1 commit into from
Sep 12, 2016
Merged

Make CSV upload limit configurable #8228

Bargs merged 1 commit into from
Sep 12, 2016

Conversation

@Bargs
Copy link
Contributor

@Bargs Bargs commented Sep 12, 2016

Kibana's Upload CSV feature isn't intended for gigantic import jobs, so
I originally set a sane default of 1GB. Some users exprssed a desire to
import slightly larger files, they should be able to import something
that's 1.1GB without being blocked by an arbitrary limit. So I've made
the limit configurable via kibana.yml.

This change includes a few pieces:

  • Added optional kibana.addDataMaxBytes key to kibana.yml with 1GB
    default
  • Set upload data route payload limit based on new config value
  • Updated help text in UI to use the dynamic config value on the parse
    csv page
  • Updated parse csv page to check file size and fail early if the
    selected file is too big

Resolves: #7671

screen shot 2016-09-12 at 3 02 42 pm

@Bargs Bargs added the review label Sep 12, 2016
@epixa
Copy link
Contributor

epixa commented Sep 12, 2016

Can you add this configuration value to the docs?

@Bargs
Make CSV upload limit configurable
0503aa8 
Kibana's Upload CSV feature isn't intended for gigantic import jobs, so
I originally set a sane default of 1GB. Some users exprssed a desire to
import slightly larger files, they should be able to import something
that's 1.1GB without being blocked by an arbitrary limit. So I've made
the limit configurable via kibana.yml.

This change includes a few pieces:
* Added optional `kibana.addDataMaxBytes` key to kibana.yml with 1GB
  default
* Set upload data route payload limit based on new config value
* Updated help text in UI to use the dynamic config value on the parse
  csv page
* Updated parse csv page to check file size and fail early if the
  selected file is too big

Resolves: elastic#7671
@Bargs
Copy link
Contributor Author

Bargs commented Sep 12, 2016

@epixa done

@epixa
Copy link
Contributor

epixa commented Sep 12, 2016

LGTM

1 similar comment
@tylersmalley
Copy link
Member

tylersmalley commented Sep 12, 2016

LGTM

w33ble
w33ble reviewed Sep 12, 2016
View reviewed changes
...ns/kibana/public/management/sections/indices/add_data_steps/parse_csv_step/parse_csv_step.js
const maxSampleRows = 10;
const maxSampleColumns = 20;

this.maxBytesFormatted = numeral(addDataMaxBytes).format('0 b');
Copy link
Contributor

@w33ble w33ble Sep 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't seem this is used anywhere. Did you mean to use it in the formatted error message on line 63?

Copy link
Member

@tylersmalley tylersmalley Sep 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's used in the view

Copy link
Contributor

@w33ble w33ble Sep 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Bargs
Copy link
Contributor Author

Bargs commented Sep 12, 2016

Thanks for the speedy review guys!

@Bargs Bargs merged commit d7b8477 into elastic:master Sep 12, 2016
@epixa
Copy link
Contributor

epixa commented Sep 12, 2016

@Bargs Please make sure this is backported to 5.x and 5.0

This was referenced Sep 12, 2016
Merged
Merged
elastic-jasper added a commit that referenced this pull request Sep 12, 2016
@elastic-jasper
Backport PR #8228
425c66b 
---------

**Commit 1:**
Make CSV upload limit configurable

Kibana's Upload CSV feature isn't intended for gigantic import jobs, so
I originally set a sane default of 1GB. Some users exprssed a desire to
import slightly larger files, they should be able to import something
that's 1.1GB without being blocked by an arbitrary limit. So I've made
the limit configurable via kibana.yml.

This change includes a few pieces:
* Added optional `kibana.addDataMaxBytes` key to kibana.yml with 1GB
  default
* Set upload data route payload limit based on new config value
* Updated help text in UI to use the dynamic config value on the parse
  csv page
* Updated parse csv page to check file size and fail early if the
  selected file is too big

Resolves: #7671

* Original sha: 0503aa8
* Authored by Matthew Bargar <[email protected]> on 2016-09-12T18:13:44Z
elastic-jasper added a commit that referenced this pull request Sep 12, 2016
@elastic-jasper
Backport PR #8228
7479654 
---------

**Commit 1:**
Make CSV upload limit configurable

Kibana's Upload CSV feature isn't intended for gigantic import jobs, so
I originally set a sane default of 1GB. Some users exprssed a desire to
import slightly larger files, they should be able to import something
that's 1.1GB without being blocked by an arbitrary limit. So I've made
the limit configurable via kibana.yml.

This change includes a few pieces:
* Added optional `kibana.addDataMaxBytes` key to kibana.yml with 1GB
  default
* Set upload data route payload limit based on new config value
* Updated help text in UI to use the dynamic config value on the parse
  csv page
* Updated parse csv page to check file size and fail early if the
  selected file is too big

Resolves: #7671

* Original sha: 0503aa8
* Authored by Matthew Bargar <[email protected]> on 2016-09-12T18:13:44Z
Bargs pushed a commit that referenced this pull request Sep 12, 2016
Merge pull request #8230 from elastic/jasper/backport/8228/5.x
c417cac 
[backport] PR #8228 to 5.x - Make CSV upload limit configurable
Bargs pushed a commit that referenced this pull request Sep 12, 2016
Merge pull request #8231 from elastic/jasper/backport/8228/5.0
b084431 
[backport] PR #8228 to 5.0 - Make CSV upload limit configurable
airow pushed a commit to airow/kibana that referenced this pull request Feb 16, 2017
@elastic-jasper
Backport PR elastic#8228
8011171 
---------

**Commit 1:**
Make CSV upload limit configurable

Kibana's Upload CSV feature isn't intended for gigantic import jobs, so
I originally set a sane default of 1GB. Some users exprssed a desire to
import slightly larger files, they should be able to import something
that's 1.1GB without being blocked by an arbitrary limit. So I've made
the limit configurable via kibana.yml.

This change includes a few pieces:
* Added optional `kibana.addDataMaxBytes` key to kibana.yml with 1GB
  default
* Set upload data route payload limit based on new config value
* Updated help text in UI to use the dynamic config value on the parse
  csv page
* Updated parse csv page to check file size and fail early if the
  selected file is too big

Resolves: elastic#7671

* Original sha: bdc2f855000d3d153e85c16deef81a17f853d69c [formerly 0503aa8]
* Authored by Matthew Bargar <[email protected]> on 2016-09-12T18:13:44Z


Former-commit-id: 7479654
airow pushed a commit to airow/kibana that referenced this pull request Feb 16, 2017
Merge pull request elastic#8230 from elastic/jasper/backport/8228/5.x
cc4caf4 
[backport] PR elastic#8228 to 5.x - Make CSV upload limit configurable

Former-commit-id: c417cac
weronikaolejniczak added a commit that referenced this pull request Jan 6, 2025
@weronikaolejniczak @kibanamachine @elasticmachine
[Stack Monitoring] Visual refresh changes (#204258)
b95211b 
## Summary

This PR introduces changes to `x-pack/plugins/monitoring` necessary for
the Visual Refresh project
(#199715):

- replacing `euiThemeVars` with `euiTheme` context
- replacing old tokens with `euiTheme`
- making sure all color palette functions are run within the context of
the `EuiProvider`

Additionally:

- I migrated Sass to `@emotion/react`
- I migrated `euiStyled` to `@emotion/react`
- I extended `emotion.d.ts` in `tsconfig.json` for typing of the EUI
theme

closes [#8228](elastic/eui#8228)

### QA

We need to test the critical paths in the Stack monitoring, paying close
attention to:

- [ ] color palette, visibility and contrast ratio of elements in
Amsterdam / Borealis

Specific paths:
- [ ] Monitoring time-series "Zoom out" button hover behavior -
`x-pack/plugins/monitoring/public/components/chart/monitoring_timeseries_container.tsx`
- [ ] Shard allocation (especially color mapping with shard type and
status):
- [ ]
`x-pack/plugins/monitoring/public/components/elasticsearch/shard_allocation/components/assigned.js`
- [ ]
`x-pack/plugins/monitoring/public/components/elasticsearch/shard_allocation/components/shard.js`
- [ ] Kuery bar suggestions and autocomplete field:
- [ ]
`x-pack/plugins/monitoring/public/components/kuery_bar/autocomplete_field.tsx`
- [ ]
`x-pack/plugins/monitoring/public/components/kuery_bar/suggestion_item.tsx`

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
viduni94 pushed a commit to viduni94/kibana that referenced this pull request Jan 23, 2025
@weronikaolejniczak @kibanamachine
@elasticmachine @viduni94
[Stack Monitoring] Visual refresh changes (elastic#204258)
169144a 
## Summary

This PR introduces changes to `x-pack/plugins/monitoring` necessary for
the Visual Refresh project
(elastic#199715):

- replacing `euiThemeVars` with `euiTheme` context
- replacing old tokens with `euiTheme`
- making sure all color palette functions are run within the context of
the `EuiProvider`

Additionally:

- I migrated Sass to `@emotion/react`
- I migrated `euiStyled` to `@emotion/react`
- I extended `emotion.d.ts` in `tsconfig.json` for typing of the EUI
theme

closes [elastic#8228](elastic/eui#8228)

### QA

We need to test the critical paths in the Stack monitoring, paying close
attention to:

- [ ] color palette, visibility and contrast ratio of elements in
Amsterdam / Borealis

Specific paths:
- [ ] Monitoring time-series "Zoom out" button hover behavior -
`x-pack/plugins/monitoring/public/components/chart/monitoring_timeseries_container.tsx`
- [ ] Shard allocation (especially color mapping with shard type and
status):
- [ ]
`x-pack/plugins/monitoring/public/components/elasticsearch/shard_allocation/components/assigned.js`
- [ ]
`x-pack/plugins/monitoring/public/components/elasticsearch/shard_allocation/components/shard.js`
- [ ] Kuery bar suggestions and autocomplete field:
- [ ]
`x-pack/plugins/monitoring/public/components/kuery_bar/autocomplete_field.tsx`
- [ ]
`x-pack/plugins/monitoring/public/components/kuery_bar/suggestion_item.tsx`

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Elastic Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@Bargs Bargs

Labels

review v5.0.0-beta1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Bargs @epixa @tylersmalley @w33ble