Reduce the number of commits in push_to_hub (#6269)

* Test single commit `push_to_hub` API

* Address review comments

* Fix

* Cleaner implementation

* Tests

* Fixes and more tests

* Remove comment

* Apply suggestions from code review

Co-authored-by: Quentin Lhoest <[email protected]>

* Split push in multiple commits

* Fix test

* Apply suggestions from code review

Co-authored-by: Quentin Lhoest <[email protected]>

* Fix type hint and style

* Oops :)

* Log commit number

* Apply suggestions from code review

Co-authored-by: Quentin Lhoest <[email protected]>

* Add logging messages to DatasetDict

---------

Co-authored-by: Quentin Lhoest <[email protected]>

Author: mariosasko

.github/workflows/ci.yml

@@ -64,7 +64,7 @@ jobs:
         run: pip install --upgrade pyarrow huggingface-hub dill
       - name: Install depencencies (minimum versions)
         if: ${{ matrix.deps_versions != 'deps-latest' }}
-        run: pip install pyarrow==8.0.0 huggingface-hub==0.14.0 transformers dill==0.3.1.1
+        run: pip install pyarrow==8.0.0 huggingface-hub==0.18.0 transformers dill==0.3.1.1
       - name: Test with pytest
         run: |
           python -m pytest -rfExX -m ${{ matrix.test }} -n 2 --dist loadfile -sv ./tests/

setup.py

@@ -131,7 +131,7 @@
     "aiohttp",
     # To get datasets from the Datasets Hub on huggingface.co
     # minimum 0.14.0 to support HfFileSystem
-    "huggingface-hub>=0.14.0,<1.0.0",
+    "huggingface_hub>=0.18.0",
     # Utilities from PyPA to e.g., compare versions
     "packaging",
     # To parse YAML metadata from dataset cards