Quick, light weight querying of log files stored on AWS S3.
- Downloads files from a S3 bucket for a single day.
- Concatenates all files into a single file.
- Unzips if necessary.
- Converts from a standard log format into JSON.
s3
- S3 website logcloudfront
- Cloudfront log (File from each edge server)cloudtrail
- Cloudtrail audit log
npm install spotcheck -g
# Create aws.json with your AWS API credentials.
{
"accessKeyId": "",
"secretAccessKey": ""
}
# Create report.json (see example below).
# Download data.
spotcheck download report.json
Optional Parameters:
--date [date] example: 04-08-2015
# Print counts from downloaded data.
spotcheck report [path to json file]
Parameters (must use one):
--format [format] example: s3
--field [field] example: user-agent
Example report.json
{
"Bucket": "",
"Format": "s3",
"Prefix": "",
"Output": "report.log"
}
Spotcheck is not intended to be a robust log parsing solution. It is designed more to make one off requests and queries easy, while leaving the heavy lifting up to more robust and full feature tools like: Loggly / Splunk / Hadoop / etc.