Skip to content

7.5 Examples: Lookup

Jump to bottom
Gurdeep Singh (Guru) edited this page Jun 30, 2024 · 1 revision

So, you have downloaded the 35+ gigs of hashed password files... now what?

The whole idea of downloading the database locally is to integrate them into your local system for faster lookup of breached password. I do not see someone who is running a blog with 10 users benefiting from downloading this whole database regularly when you can query the password online via haveIBeenPwned API website. Also, what is the point if your lookup is slower then making a direct query to the haveIBeenPwned API.

While I was playing around with different methods of looking up, I realized a few things

  • Each files have random amount of hashes.
  • Those hashes are not sorted in anyway, which makes it quite difficult to implement one method of lookup that will be perfect.
  • Sorting, indexing, caching and then finally looping through the rest of the hash will yield the best lookup result.

The process I think would be best to implement a fastest lookup

  • Download all hash files and take a backup.
  • Sort contents of each hash files SORT_DESC, so you can more breached hashes on the top. The reason, more hash breached means common password.
  • Cache hash that have been breached more than 500 times and remove them from the file using argument --remove-cached=true
  • Index hash that have been breached more than 100 times and remove them from the file using argument --remove-indexed=true
  • This will leave a slim version of the hash file and lookup can be performed by either loading the whole file in an Array (faster by more memory usage) or via Stream (slower by less memory usage). Depending on the hardware you have, you can choose the method of lookup.

The lookup tool

  • Lookup password is hidden to avoid shoulder surfers from looking at your password.
  • Lookup can be made for a password or for a hash string.
  • Lookup can be made using different methods to see their performance. Available methods: array, cache, index, stream, string
  • Depending on the password hash type you have downloaded, lookup can be made on SHA or NTLM hash.
  • Detailed result of time taken, memory consumed, hashfile information is shows at each lookup.
Note: We will be using password 123
SHA Hash for password 123 - 40BD001563085FC35165329EA1FF5C5ECBDBBEEF
NTLM Hash for password 123 - 3DBDE697D71690A769204BEB12283678

To lookup for a password (SHA)

./hibp lookup
This is a CLI tool to download, index, cache, sort and lookup pwned password. Type ./hibp --help for list of commands.

Enter Password: ***
+------------------------------------------+-----------+--------+---------+------+---------+--------+---------------------+-------------+
| Hash                                     | Hash File | Method | Count   | NTLM | Indexed | Cached | Time (s)            | Memory (kb) |
+------------------------------------------+-----------+--------+---------+------+---------+--------+---------------------+-------------+
| 40BD001563085FC35165329EA1FF5C5ECBDBBEEF | 40BD0.txt | Stream | 1783558 | No   | No      | No     | 0.00035905838012695 | 10.6 kb     |
+------------------------------------------+-----------+--------+---------+------+---------+--------+---------------------+-------------+

To lookup for a password (NTLM)

./hibp lookup --ntlm=true
This is a CLI tool to download, index, cache, sort and lookup pwned password. Type ./hibp --help for list of commands.

Enter Password: ***
+----------------------------------+-----------+--------+---------+------+---------+--------+--------------------+-------------+
| Hash                             | Hash File | Method | Count   | NTLM | Indexed | Cached | Time (s)           | Memory (kb) |
+----------------------------------+-----------+--------+---------+------+---------+--------+--------------------+-------------+
| 3DBDE697D71690A769204BEB12283678 | 3DBDE.txt | Stream | 1783558 | Yes  | No      | No     | 0.0014369487762451 | 10.59 kb    |
+----------------------------------+-----------+--------+---------+------+---------+--------+--------------------+-------------+

Note: The above password lookup is done using stream method, where PHP reads the whole file line by line. This saves up memory, but takes a lot of time to find the hash. If the hash is in row 1, the time taken will be too short. The above result is also taken when the password files were not sorted. I will sort the files and run the command again to show the output.

Password 123 is on line 4 in SHA hash file 40BD0.txt & is on line 387 in NTLM hash file 3DBDE.txt

./hibp sort --hashes=40BD0
This is a CLI tool to download, index, cache, sort and lookup pwned password. Type ./hibp --help for list of commands.

Sorting hash: 40BD0 (1/1)  100% [================================================================================] 0:00 / 0:00

./hibp sort --hashes=3DBDE --type=ntlm
This is a CLI tool to download, index, cache, sort and lookup pwned password. Type ./hibp --help for list of commands.

Sorting hash: 3DBDE (1/1)  100% [================================================================================] 0:00 / 0:00
./hibp lookup
This is a CLI tool to download, index, cache, sort and lookup pwned password. Type ./hibp --help for list of commands.

Enter Password: ***
+------------------------------------------+-----------+--------+---------+------+---------+--------+---------------------+-------------+
| Hash                                     | Hash File | Method | Count   | NTLM | Indexed | Cached | Time (s)            | Memory (kb) |
+------------------------------------------+-----------+--------+---------+------+---------+--------+---------------------+-------------+
| 40BD001563085FC35165329EA1FF5C5ECBDBBEEF | 40BD0.txt | Stream | 1783558 | No   | No      | No     | 0.00032186508178711 | 10.6 kb     |
+------------------------------------------+-----------+--------+---------+------+---------+--------+---------------------+-------------+

./hibp lookup --ntlm=true
This is a CLI tool to download, index, cache, sort and lookup pwned password. Type ./hibp --help for list of commands.

Enter Password: ***
+----------------------------------+-----------+--------+---------+------+---------+--------+---------------------+-------------+
| Hash                             | Hash File | Method | Count   | NTLM | Indexed | Cached | Time (s)            | Memory (kb) |
+----------------------------------+-----------+--------+---------+------+---------+--------+---------------------+-------------+
| 3DBDE697D71690A769204BEB12283678 | 3DBDE.txt | Stream | 1783558 | Yes  | No      | No     | 0.00033903121948242 | 10.59 kb    |
+----------------------------------+-----------+--------+---------+------+---------+--------+---------------------+-------------+

Look at the time different after sorting the files, especially the NTLM hash lookup. As password 123 breach count is 1783558. The password is on top of the file (row 1) after sorting.

Try the lookup tool and let me know if you have any question or you think there is a better way to lookup for a password.

Clone this wiki locally