Skip to content

dmuth/splunk-aws-s3-server-accesslogs

Repository files navigation

Splunk for AWS S3 Server Access Logs

This an app I built using Splunk Lab to extract data from AWS S3 Server Access Logs and use that data for graphs and analysis.

Getting your AWS S3 Server Access Logs

You'll need to configure Server Access Logging in AWS S3. Once that's done, you can either pull down many small logfiles directly, or instead perform daily rollup on them with an AWS S3 Rollup app which I built specifically for this purpose.

Either way, you can use the aws CLI app to download all of your logs into logs/ directory and then concatenate the contents of each directory into a single file for that bucket with something like this:

  • aws s3 sync s3://my-accesslogs/rollup-day/ logs
  • cd logs/
  • for DIR in $(find . -type d); do cat $DIR/* > $DIR.txt; done
  • for DIR in $(find . -type d); do rm -rfv $DIR/* > $DIR.txt; done

Naturally, this is highly dependent on how you're storing logs.

Starting up Splunk Lab

Next, start up Splunk Lab with this command:

  • bash <(curl -s https://raw.githubusercontent.com/dmuth/splunk-aws-s3-server-accesslogs/master/go.sh)

The script will guide you through various settings you can send to Splunk Lab.

From there, you can go to https://localhost:8000, log into Splunk with the credentials you specified when starting it, and you should be able to search for data or view reports in dashbaords.

Known Issues

Q: I see an error about exceeding "the configured depth_limit"?

A: You'll need to increase that value in app/limits.conf. You can read more about that here.

Development

  • ./bin/devel.sh splunk
  • ./bin/build.sh
  • ./bin/push.sh

Additional Resources

Credits

Bugs/Contact

Here's how to get in touch with me:

About

Generate Splunk Dashboards for your AWS S3 Server Access Logs

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages