Multiple logs related to the reverse DNS map #542
Comments
|
Indeed, I've noticed that the program loads the reverse DNS map every time it runs the function: get_service_from_reverse_dns_base_domain. Additionally, when the log level is set to info, this log is generated with each query. This has caused a slight slowdown when processing aggregated reports with 70k to 80k entries, predominantly from Google and Microsoft, extending the parsing time of such a report to approximately 12 to 15 minutes. As @seanthegeek mentioned in this comment on issue 500, the purpose is to keep the map updated. However, I believe that isolating this function and loading the map just once, then using a timedelta to refresh the DNS map if it's older than one day, could significantly improve parsing performance. This change would also reduce the frequency of file loading, thus decreasing disk IO if the map is stored locally, and lessen the number of times the map is retrieved from GitHub. |
|
My intent was to only have the file procesed at startup. I'll revisit the code on Monday. |
- Fix processing of SMTP-TLS reports (#549) - Skip invalid aggregate report rows without calling the whole report invalid - Some providers such as GoDaddy will send reports with some rows missing a source IP address, while other rows are fine - Fix Dovecot support by using the seperator provided by the IPMAP namespace when possible (PR #552 closes #551) - Only download `base_reverse_dns_map.csv` once (fixes #542) - Update included `base_reverse_dns_map.csv` - Replace University category with Education to be more inclusive - Update included `dbip-country-lite.mmdb`
|
thank you very much @N4v41 for the additional research and @seanthegeek for the patch. |
I upgraded to version 8.12.0 and when trying to import lots of repots, I'm seeing lots of log entries like this:
INFO - [utils.py:340] - Loading included reverse DNS map...Turning debug on, they always seem to be placed around these log entries:
As per the code, it seems like it loads every time the "base_reverse_dns_map.csv" file and makes a lookup.
Wouldn't it be better for the file to be loaded once and make all the lookups over the same instance?
I'm sorry but my programming knowledge isn't enough for me to know how to solve this.
Thank you very much.
The text was updated successfully, but these errors were encountered: