Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support resumable download #17

Merged
merged 2 commits into from
Oct 30, 2023
Merged

Support resumable download #17

merged 2 commits into from
Oct 30, 2023

Conversation

rosbo
Copy link
Contributor

@rosbo rosbo commented Oct 27, 2023 •
edited
Loading

  • Supports full model & single file downloads
  • Updated client to send HTTP "Range" header on resume.
  • Use a fixed path for the archive to allow resuming.
  • Setup logging infra (and use it to print the resuming state).
    Example:
$ kagglehub.model_download('keras-nlp/albert/keras/albert_extra_extra_large_en_uncased/1')
3%|███                | 25.0M/789M [00:00<00:13, [00:00<00:13, 59.6MB/s]

# CTRL+D to kill the process

$ kagglehub.model_download('keras-nlp/albert/keras/albert_extra_extra_large_en_uncased/1')
Resuming download from 29360128 bytes (798274426 bytes left)...
100%|█████████████████| 789M/789M [00:09<00:00, 84.7MB/s]

http://b/307572374

@rosbo rosbo requested a review from Philmod October 27, 2023 00:29
Philmod
Philmod approved these changes Oct 27, 2023
View reviewed changes
Copy link
Contributor

@Philmod Philmod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

rosbo added 2 commits October 30, 2023 15:53
@rosbo
Support resumable download
a42283a 
- Updated client to send HTTP "Range" header on resume.
- Use a fixed path for the archive to allow resuming.
- Setup logging

Example:

```
$ hatch run python -c "import kagglehub; print(kagglehub.model_download('keras-nlp/albert/keras/albert_extra_extra_large_en_uncased/1'))"
  3%|███▉                                                                                                                          | 25.0M/789M [00:00<00:13, 59.3MB/s]  4%|████▍                                                                                                                         | 28.0M/789M [00:00<00:13, 59.6MB/s]

$ hatch run python -c "import kagglehub; print(kagglehub.model_download('keras-nlp/albert/keras/albert_extra_extra_large_en_uncased/1'))"
Resuming download from 29360128 bytes (798274426 bytes left)...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 789M/789M [00:09<00:00, 84.7MB/s]
```

http://b/307572374
@rosbo
add tests
70c6e68
@rosbo rosbo force-pushed the resumable-download branch from cfdf251 to 70c6e68 Compare October 30, 2023 16:02
@rosbo rosbo merged commit cd20c78 into main Oct 30, 2023
@rosbo rosbo deleted the resumable-download branch October 30, 2023 16:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Philmod Philmod Philmod approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rosbo @Philmod