Automated solution to address 403 error downloading tLD.zip from akamaized

[jpiesing]

While doing a clean re-install, the import.sh script reports errors as follows;

Could not load http url: https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/tLD/2023-04-28/tLD.zip
Could not load http url: https://dash.akamaized.net/WAVE/vectors/cfhd_sets/15_30_60/tLD/2023-04-28/tLD.zip

Either the URL for the segment zip file in database.json is wrong or access to that file is blocked because downloading at URL fails as follows.

wget https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/tLD/2023-04-28/tLD.zip
--2023-06-02 16:20:51-- https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/tLD/2023-04-28/tLD.zip
Resolving dash.akamaized.net (dash.akamaized.net)... 81.243.1.66, 81.243.1.88
Connecting to dash.akamaized.net (dash.akamaized.net)|81.243.1.66|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-06-02 16:20:53 ERROR 403: Forbidden.

[nicholas-fr]

For larger files I believe https://dash-large-files.akamaized.net/ must be used.

The following work:
https://dash-large-files.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/tLD/2023-04-28/tLD.zip
https://dash-large-files.akamaized.net/WAVE/vectors/cfhd_sets/15_30_60/tLD/2023-04-28/tLD.zip

[jpiesing]

For larger files I believe https://dash-large-files.akamaized.net/ must be used.

The following work: https://dash-large-files.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/tLD/2023-04-28/tLD.zip https://dash-large-files.akamaized.net/WAVE/vectors/cfhd_sets/15_30_60/tLD/2023-04-28/tLD.zip

In that case it's a bug in database.json - the wrong URL is included.

[nicholas-fr]

OK, to determine which URL should be used: https://dash-large-files.akamaized.net/ is for files over 1.8GB.

[rbouqueau]

Can't we just use https://dash-large-files.akamaized.net to serve all files? Do you know what difference it would make?

[rbouqueau]

Can't we just use https://dash-large-files.akamaized.net/ to serve all files? Do you know what difference it would make?

[gitwjr]

Not resolved as of Aug 8, 2023.

[jpiesing]

Can't we just use https://dash-large-files.akamaized.net/ to serve all files? Do you know what difference it would make?

I have no idea as I wasn't involved when this was setup.

@wilaw Do you know the answer to Romain's question?

[jpiesing]

Manually editing database.json isn't safe as a long term solution so re-opening.

[gitwjr]

If this issue remains, it is a blocker.

[gitwjr]

Renamed issue. Remove blocker.

[nlsdvl]

See the pull request for an updated database file using 'https://dash-large-files.akamaized.net/'.