Increase file upload limit to 100 MB

[marvin-w]

Proposed change

As spoken with balloob this PR will increase the file upload limit via the new file upload API from 10 MB to 100 MB. It now also uses streaming to upload and reads the data in 1 MB chunks (if available) and writes them to the file.

Things that we should check as well:

• Are there any webservers (like nginx) in front of HA in any of the available installation methods that potentially need the client_max_body_size adjusted?

Type of change

• Bugfix (non-breaking change which fixes an issue)
• Dependency upgrade
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes #
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• The code has been formatted using Black (black --fast homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
• Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

• No score or internal
• 🥈 Silver
• 🥇 Gold
• 🏆 Platinum

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[probot-home-assistant]

Hey there @home-assistant/core, mind taking a look at this pull request as it has been labeled with an integration (file_upload) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[pvizeli]

another way for not jump 100x times into a thread is to use deque from functiontools which is none blocking, and write each junk into queue and write it inside the thread. So there would be only 1 context switch

or:
https://github.com/aio-libs/janus

[marvin-w]

another way for not jump 100x times into a thread is to use deque from functiontools which is none blocking, and write each junk into queue and write it inside the thread. So there would be only 1 context switch

or: https://github.com/aio-libs/janus

The point of a streaming file upload is to not write the whole content to RAM first. Yes, we could do this but it will increase the RAM usage for larger files and then we might as well just revert the whole streaming approach.

[pvizeli]

Correct, with Janus you stream it into the thread, so you don't have a context switch and waiting for a thread after each junk. You can also limit the buffer and it get wait on the async part until the thread processed the buffer

[balloob]

@marvin-w you can still achieve that with janus by calling join() on the Janus queue when it gets too full before adding more files.

https://github.com/aio-libs/janus/blob/master/janus/__init__.py#L602

[marvin-w]

@marvin-w you can still achieve that with janus by calling join() on the Janus queue when it gets too full before adding more files.

https://github.com/aio-libs/janus/blob/master/janus/__init__.py#L602

I gave it a shot, it is working. However, I'm not 100% sure if it is correct like that. Especially the future handling looks kind of dirty. Open for suggestions :).