api.upload_large_folder consumes api request limit even when verifying which files were previously uploaded

[FurkanGozukara]

Upload used code as below

from huggingface_hub import HfApi

api = HfApi()

api.upload_large_folder(repo_id="YourUserName/reponame", 
                        repo_type="model", 
                        folder_path=r"/home/Ubuntu/apps/StableSwarmUI/Models/Lora")

print(".\n.\nUPLOAD COMPLETED")

Describe the bug

Initial upload

10581 files, 304.8 GB

I waited few hours, it uploaded and committed like 9576, then i stopped execution and came back few hours later

However when I execute again, it checks which files were already uploaded and it consumes API limit immediately

The folder structure of uploaded repo like this. I think somehow upload of small files can be more optimized to not consume API request limit

Extension: 
  Total files: 1
  Average size: 4.00 KB
  Min size: 4.00 KB
  Max size: 4.00 KB

Extension: .metadata
  Total files: 10581
  Average size: 4.00 KB
  Min size: 4.00 KB
  Max size: 4.00 KB

Extension: .safetensors
  Total files: 44
  Average size: 6.35 GB
  Min size: 3.97 GB
  Max size: 6.46 GB

Extension: .json
  Total files: 3
  Average size: 4.00 KB
  Min size: 4.00 KB
  Max size: 4.00 KB

Extension: .toml
  Total files: 6
  Average size: 4.00 KB
  Min size: 4.00 KB
  Max size: 4.00 KB

Extension: .npz
  Total files: 5240
  Average size: 260.00 KB
  Min size: 260.00 KB
  Max size: 260.00 KB

Extension: .png
  Total files: 44
  Average size: 1.48 MB
  Min size: 1.29 MB
  Max size: 1.64 MB

Extension: .txt
  Total files: 42
  Average size: 4.00 KB
  Min size: 4.00 KB
  Max size: 4.00 KB

Extension: .jpg
  Total files: 5200
  Average size: 640.86 KB
  Min size: 220.00 KB
  Max size: 1.27 MB

Reproduction

No response

Logs

No response

System info

- huggingface_hub version: 0.25.0.dev0
- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Running in iPython ?: Yes
- iPython shell: ZMQInteractiveShell
- Running in notebook ?: Yes
- Running in Google Colab ?: No
- Token path ?: /home/Ubuntu/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: N/A
- Jinja2: 3.1.4
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: N/A
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: N/A
- pydantic: N/A
- aiohttp: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/Ubuntu/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/Ubuntu/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/Ubuntu/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

[2]:
{'huggingface_hub version': '0.25.0.dev0',
 'Platform': 'Linux-6.8.0-40-generic-x86_64-with-glibc2.35',
 'Python version': '3.10.12',
 'Running in iPython ?': 'Yes',
 'iPython shell': 'ZMQInteractiveShell',
 'Running in notebook ?': 'Yes',
 'Running in Google Colab ?': 'No',
 'Token path ?': '/home/Ubuntu/.cache/huggingface/token',
 'Has saved token ?': False,
 'Configured git credential helpers': '',
 'FastAI': 'N/A',
 'Tensorflow': 'N/A',
 'Torch': 'N/A',
 'Jinja2': '3.1.4',
 'Graphviz': 'N/A',
 'keras': 'N/A',
 'Pydot': 'N/A',
 'Pillow': 'N/A',
 'hf_transfer': 'N/A',
 'gradio': 'N/A',
 'tensorboard': 'N/A',
 'numpy': 'N/A',
 'pydantic': 'N/A',
 'aiohttp': 'N/A',
 'ENDPOINT': 'https://huggingface.co',
 'HF_HUB_CACHE': '/home/Ubuntu/.cache/huggingface/hub',
 'HF_ASSETS_CACHE': '/home/Ubuntu/.cache/huggingface/assets',
 'HF_TOKEN_PATH': '/home/Ubuntu/.cache/huggingface/token',
 'HF_HUB_OFFLINE': False,
 'HF_HUB_DISABLE_TELEMETRY': False,
 'HF_HUB_DISABLE_PROGRESS_BARS': None,
 'HF_HUB_DISABLE_SYMLINKS_WARNING': False,
 'HF_HUB_DISABLE_EXPERIMENTAL_WARNING': False,
 'HF_HUB_DISABLE_IMPLICIT_TOKEN': False,
 'HF_HUB_ENABLE_HF_TRANSFER': False,
 'HF_HUB_ETAG_TIMEOUT': 10,
 'HF_HUB_DOWNLOAD_TIMEOUT': 10}

[Wauplin]

(related to #2254 (comment))

[Wauplin]

thanks for reporting it @FurkanGozukara :)