Fix resumable_download for fully downloaded files

[flyingleafe]

When resumable_download is called and the file is fully downloaded, the requested range starts after the file ends, which yields HTTP error 416. It seems logical that in this case the exception should not be raised, but rather Lhotse should report that the file is already downloaded.

[rouseabout]

The file should only be considered fully downloaded WHEN the 416 error code is returned AND the file size returned in the Content-Range header matches that on local disk.

Not all servers include the Content-Range header with the 416 response, meaning that the server does not support resuming downloads.

Here is an older patch from my tree. Feel free to adapt.

diff --git a/lhotse/utils.py b/lhotse/utils.py
index 6cdb0f15..60d774af 100644
--- a/lhotse/utils.py
+++ b/lhotse/utils.py
@@ -471,7 +471,8 @@ def resumable_download(
         f.seek(file_size)
 
         # Open the URL and read the contents in chunks
-        with urllib.request.urlopen(req) as response:
+        try:
+            response = urllib.request.urlopen(req)
             chunk_size = 1024
             total_size = int(response.headers.get("content-length", 0)) + file_size
             with tqdm(
@@ -487,6 +488,10 @@ def resumable_download(
                         break
                     f.write(chunk)
                     pbar.update(len(chunk))
+        except urllib.error.HTTPError as err:
+            if err.code != 416 or err.headers.get("Content-Range", "") != f"bytes */{file_size}":
+                raise err
 
 
 def safe_extract(

[flyingleafe]

@rouseabout Fair enough. If the server does not support resumable downloads though, we should not simply fail but rather try to download from scratch with a request without range headers then. Updated PR accordingly.

[pzelasko]

LGTM, thanks!

[pzelasko]

Please just fix the CI checks

[flyingleafe]

@pzelasko only black was failing, should be ok now.