Commit 6147a8e
authored
Fix memory exhaustion when downloading large files (#869)
## Description
This PR fixes issue #754 where the Kaggle API would exhaust system
memory when downloading large datasets.
### Problem
The HTTP client was not using `stream=True` when making requests for
file downloads. This caused the entire file content to be loaded into
memory before being written to disk, making the system unstable when
downloading large datasets.
### Solution
- Modified `KaggleHttpClient.call()` to detect file download response
types (`FileDownload` and `HttpRedirect`)
- Automatically enable streaming (`stream=True`) for these response
types
- The existing `download_file()` method already uses
`response.iter_content()` for chunked reading, which now works properly
with streaming enabled
### Changes
- **src/kagglesdk/kaggle_http_client.py**
- Added imports for `FileDownload` and `HttpRedirect` types
- Added logic to set `stream=True` in request settings for file
downloads
- **pyproject.toml**
- Removed `kagglesdk` from dependencies list
- Reason: `kagglesdk` source code is in `src/kagglesdk/` and should use
local code during editable install, not pull from PyPI
### Impact
This fix improves memory usage for:
- Competition file downloads (`competition_download_file`,
`competition_download_files`)
- Dataset downloads (`dataset_download_file`, `dataset_download_files`)
- Model downloads (`model_instance_version_download`)
- Kernel output downloads (`kernels_output`)
- Leaderboard downloads (`competition_leaderboard_download`)
### Testing
- ✅ Tested CLI: `kaggle competitions list` works correctly
- ✅ Verified streaming is enabled for file download requests
- ✅ Backward compatible - no changes to public API
Fixes #7541 parent 03367e7 commit 6147a8e
1 file changed
+8
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
| |||
81 | 83 | | |
82 | 84 | | |
83 | 85 | | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
84 | 92 | | |
85 | 93 | | |
86 | 94 | | |
| |||
0 commit comments