Upgrade of the MAxMind GeoLite country database fails with HTTP error 404

[mum-viadee]

Type of Issue

Error upgrading geoip database

Description

As the legacy version of the geoip database has been discontinued and is not longer available on the MaxMind webserver, the upgrade script "util/upgrade.py" gets an HTTP error 404 when started with --geoip.

Your Environment

• CRS version: v3.1.0
• ModSecurity version: 2.9.0-1 (Ubuntu package version)
• Web Server and version: apache 2.4.18
• Operating System and version: Ubuntu 16.04 LTS

Confirmation

[X ] I have removed any personal data (email addresses, IP addresses,
passwords, domain names) from any logs posted.

Console Output

./upgrade.py --geoip --crs
crs:
From https://github.com/Spiderlabs/owasp-modsecurity-crs

• branch HEAD -> FETCH_HEAD
  Already up-to-date.

geoip: HTTP Error 404: Not Found

[dune73]

Sorry for the inconvenience and thank you for pointing this out. Did not think of this so far.
We'll need to find a solution here.

Are you familiar with the work of @emphazer that allows you to transform the GeoIP database and continue to use it with ModSec 2.9? https://github.com/emphazer/GeoIP_convert-v2-v1

[emphazer]

@dune73 maybe we should discuss this problem Monday at chat to find a solution.

[dune73]

Yes, that's a plan. Please add it to the agenda.

I would not mind integrating your script into CRS so we could offer people a CRS solution to the pending problem. Not strictly in line, but as a helper script...

[lifeforms]

Hi @mum-viadee , as you have heard in the above comments, our project member @emphazer has created a tool to convert a modern GeoIP database to the legacy format which can be used with ModSecurity 2.x.

We have discussed if we can bring the conversion utility into the CRS tree and make it part of our upgrade script, so that all CRS users automatically have it. However for this moment we have decided against it, because it would take a lot of time to integrate the converter and make it releasable as part of CRS. Also, our upgrade script is probably not widely used and already outdated (it uses git pulls to update the CRS code base to our dev branch, which we do not encourage any longer for production installations). As a project we have very limited resources, so we will not further develop the upgrade script.

However, we invite you to try the converter at https://github.com/emphazer/GeoIP_convert-v2-v1 and hope that it works for you. We will schedule to make a blog post on coreruleset.org about the converter so that it will be of use to more people.

For now I am closing this issue, thank you for reporting it to us.

[JeffCleverley]

I am using modsec 3, nginx connector and having the same issue.

[bittner]

We're facing the same issue and may be willing to integrate the conversion process into the upgrade.py script.

Just for clarification, which one of the three databases should be downloaded and converted in that case? GeoLite2 City, GeoLite2 Country, GeoLite2 ASN (Sorry if that question is stupid)

[dune73]

We have talked about this in the project and decided, that MaxMind GeoIP support is not an essential part of CRS and that we are thus not including the conversion script Which is not to say that the conversion script or GeoIP are not important. We just do not want to maintain it ourselves.

Personally, I am using the GeoLite2 Country DB. That's mainly because the City DB is not very accurate in Switzerland, so the smaller size of the country DB seems preferable. Generally, mileage may vary.

[bittner]

What does that translate to? Shall we remove the upgrade.py script from this repository?

Note that I offered that we would maintain the script in the project.

[dune73]

@lifeforms is the guy behind the upgrade.py scipt. So his opinion matters. Personally, I am torn.