Add data endpoint that can parse webserver logs. #1483
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jpwhite4
added
enhancement
|
for reference: https://github.com/jpwhite4/xdmod-ood |
jpwhite4
force-pushed
the
weblogs
branch
2 times, most recently
from
996defa
to
8e71938
Compare
This adds a couple of libraries that can parse webserver log files, parse user agent strings and a geoip lookup library. The geoip library is Apache Licenced, but the corresponding geoip database is not redistributable so is not included. The WebServerLogFile endpoint has configuration settings to specify the log file format (and hence the records extracted) and a config setting for the path to the goeip database. If there is no geoip database then it will still work. The regular expression ingestor supports using the PHP preg_filter() function to transform data. See the xdmod-ood module for CI tests and usage etc.
Originally the code would nonly try to add location properties if the geoip file was present. This changes it so that the properties are always present even if a file is absent. This makes it easier to support sites that don't have access to the location file.
jsperhac
approved these changes
Mar 29, 2021
|
I eyeballed this PR and approved it prior to merging with 9.5. Looks like there are some tests failing, however. |
|
@jsperhac I believe that the test failures are due to the chromium update that's breaking the report generator. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a couple of libraries that can parse webserver
log files, parse user agent strings and a geoip lookup library.
The geoip library is Apache Licenced, but the corresponding
geoip database is not redistributable so is not included.
The WebServerLogFile endpoint has configuration settings to
specify the log file format (and hence the records extracted) and
a config setting for the path to the goeip database. If there
is no geoip database then it will still work.
See the xdmod-ood module for CI tests and usage etc.