9. Update

To update the hashes, you need to run the default download command.

./hibp download

./hibp download --type=ntlm

Refer to the download examples wiki page for a complete list examples.

Updating only changed hashes

If you are using the database at multiple Geo locations and to provide a faster lookup you have copied the entire database to that specific Geo location. You can use this method to grab only the updated hashes in one location and instead of all Geo locations querying the main API, they can query your primary location and only download the updated hash files from the API. This can save a great deal of bandwidth and time.

To download updates from a different location pass the argument --update-source={site name}

./hibp download --update-source=https://your_main_site.com/pwned

Complete process is as follows:

• On your primary site, you will download the whole database or you can only download the etags using the following command

./hibp download --download-data=false --type=sha --async=100 --force=true --resume=false
./hibp download --download-data=false --type=ntlm --async=100 --force=true --resume=false

• The above 2 commands will only download the updated etags and save them in the logs directory.

./hibp compare --rebuild-index=true

• The above command will generate update.json and update.php in the data/updates/ directory by comparing all log files.

./hibp download --update-source=https://your_main_site.com/pwned --update-since=2024-06-25 --type=ntlm

• The above command will download any updates since midnight 25/06/2024 and are of type ntlm. If you do not provide the argument --update-since, the tool looks for and {type}lastupdate.txt file in the data/ directory. The last update file is generated when you run a full download of the database.
• You can put those 2 files in a shell scrip and run a cron every midnight to check for any updates.

How I am using compare and update tools

• I have this file created on my system

/var/www/html/# cat update.sh 
php /var/www/html/index.php download --download-data=false --type=ntlm --async=100 --force=true --resume=false
php /var/www/html/index.php download --download-data=false --type=sha --async=100 --force=true --resume=false
php /var/www/html/index.php compare --rebuild-index=true
echo "Cron job complete." | mail -s "Subject" -A /var/www/html/data/logs/cron.log {my_email}@yourdomain.com

• I execute the above shell script every might night AEST

crontab -e -u www-data
0 0 * * * /var/www/html/update.sh > /var/www/html/data/logs/cron.log

The above shell script, check for both SHA and NTLM etags and records any changes to them. Then I run compare and rebuild the index which stores the updated information in update.php, which you can access via https://apps.oyeaussie.dev/pwned/?since=2024-06-25&type=ntlm. You need to define since date else the json data will not be returned. If you do not provide the type, you will get both SHA and NTLM.

NOTE: The above service will be live and accessible via the internet, so anyone can use it. If you want to query my server using the CLI tool, enter the following command after 3AM AEST :-)

./hibp download --update-source=https://apps.oyeaussie.dev/pwned/ --update-since=2024-06-25 --type=ntlm

Using the above command, the download tool will download any updates that took place since 25/06/2024 and generate a list of Hashes that needs to be downloaded from the official API.