Ever wanted to build a complex Splunk environment for testing, which looks as close as possible to a production deployment? Need to test a Splunk upgrade? See how Splunk indexer- or search head clustering works? Or just need to verify some configuration changes? This is the right place for you! The aim of this framework is to produce a Splunk environment in a fast and convenient way for testing purposes or maybe also for production use. The created Splunk installation and setup follows best practices. There are many ways to configure a Splunk environment in terms of configuration file locations, so this is just another example how to do it.
- Splunk Platform Automator 2.2.2
- Support
- Features
- Installation
- Upgrade
- Removed Biased Language
- Building Windows Virtual Machine Template
- Framework Usage
- Known issues, limitations
- License
Note: This framework is not officially supported by Splunk. I am developing this on best effort in my spare time.
- Build complex, reproducible Splunk environments in one shot, including all roles available for Splunk Enterprise.
- Building Cluster Manager, Indexer Clusters, Deployer, Search Head Clusters, Deployment Server, Universal Forwarders, Heavy Forwarders, License Manager and Monitoring Console. All ready to use.
- Configuration done according best practices with configuration apps
- Splunk environment definition stored in one simple yaml file
- Example configuration files for different setups included
- Deployment and configuration done with Ansible
- Virtual hosts can be created by Vagrant
- Currently supports Virtualbox or AWS Cloud.
- Can deploy Splunk on existing hosts (virtual or physical)
- Developed and tested on MacOSX but should support Linux as well.
See the upcoming features in the Roadmap
Implemented changes are to be found in the Changelog
The Framework is currently tested on Mac OSX and Linux, but any other Unix, which is supported by Virtualbox, should work too.
- Make sure you have Python 3.6+ installed. If not available for your distro, you can build your own Python version.
- Download and install Vagrant. Is you are using the AWS plugin, see version hints here
- Install Ansible, I personally prefer Brew (on OSX) which makes it as easy as
brew install ansible. For supported Ansible versions check here - Create a folder called
Vagrantand change into it. - Download and extract a Splunk Platform Automator release here or clone from GitHub when using the master branch:
git clone https://github.com/splunk/splunk-platform-automator.git - Create a folder called
Software. - Download the tgz. archive for the Splunk Software and put in the
Softwaredirectory - Download Splunk Professional Services Best Practices Base Config Apps and extract them into the
Softwaredirectory - If you have a Splunk License file, link it to the name
Splunk_Enterprise.licinside theSoftwaredirectory.
Your directory structure should now look like this:
./Vagrant/splunk-platform-automator/...
./Vagrant/Software/Configurations - Base/...
./Vagrant/Software/Configurations - Index Replication/...
./Vagrant/Software/splunk-8.1.2-545206cc9f70-Linux-x86_64.tgz
./Vagrant/Software/splunkforwarder-8.1.2-545206cc9f70-Linux-x86_64.tgz
./Vagrant/Software/Splunk_Enterprise.lic
- Download and install Virtualbox.
- Install the Virtualbox plugin for Vagrant:
vagrant plugin install vagrant-vbguest
The Windows Subsystem for Linux does allow to run Splunk Platform Automator on Windows. It even allows to create virtualbox hosts from it on the windows host directly.
Execute the steps above. To allow vagrant to talk to virtualbox follow the steps below.
- Create /etc/wsl.conf and reboot WSL (
wsl --shutdown)
[automount]
options = "metadata"
- Enable WSL2 port forwarding by installing a vagrant plugin with:
vagrant plugin install virtualbox_WSL2 - Add Environment Variables in WSL (maybe to your
~/.bashrc)
export VAGRANT_WSL_ENABLE_WINDOWS_ACCESS="1"
export PATH="$PATH:/mnt/c/Program Files/Oracle/VirtualBox"
- Install the aws vagrant plugin:
vagrant plugin install vagrant-aws. The plugin is not maintained anymore and has issues with newer vagrant versions on OSX. The last working version of vagrant is 2.3.4 - Download the vagrant dummy box for aws:
vagrant box add aws-dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box - Generate AWS ACCESS Keys, described here
- Optional, but recommended:
- Add AWS_ACCESS_KEY_ID= as environment variable
- Add AWS_SECRET_ACCESS_KEY= as environment variable
- Create an ssh key pair described here and store the public key on your disk for later reference in the config file
- Create an AWS security group and name it for example 'Splunk_Basic' and add the following TCP ports
| Type | Protocol | Port Range | Source | Description |
|---|---|---|---|---|
| All TCP | TCP | 0 - 65535 | 172.31.0.0/16 | Allow all internal traffic |
| Custom TCP | TCP | 8000 | 0.0.0.0/0 | Splunk Web Interface |
| SSH | TCP | 22 | 0.0.0.0/0 | SSH to all hosts |
| Type | Protocol | Port Range | Destination | Description |
|---|---|---|---|---|
| All Traffic | All | All | 0.0.0.0/0 | Allow all traffic |
To upgrade your Splunk Platform Automator, just update your local code from the repo
git pull
From Splunk Platform Automator 2.0 the Framework does use an Ansible Inventory Plugin to build the inventory on the fly during execution. The local inventory directory does only hold minimum settings based on the virtualization you choose. The rest calculated in flight and not stored somewhere.
You can verify your inventory with
ansible-inventory --list --export
The steps here apply to all environments.
- You have to add the
pluginsetting to the top of your config file
# splunk_config.yml
plugin: splunk-platform-automator
The steps here only apply if your current environment is built on virtualbox.
- Cleanup unneeded entries from the ansible inventory
rm -rf inventory/group*
- Move the setting
start_ipin thegeneralsection to thevirtualboxsection.
The steps here only apply if your current environment is built on AWS.
- Cleanup unneeded entries from the ansible inventory
rm -rf inventory/*
- Build the config/aws_ec2.yml config file
vagrant status
- Get the GUID from config/aws_ec2.yml at
tag:SplunkEnvID:and add a tagSplunkEnvIDto every host in your AWS environment with that GUID - Create also a tag
SplunkHostnamefor every AWS host with the name of your hosts from the splunk_hosts section
If you have the aws cli available, this can be done with the following one liner
for machine in $(ls -1d .vagrant/machines/*); do aws ec2 create-tags --resources $(cat $machine/aws/id) --tags Key=SplunkHostname,Value=$(basename $machine) Key=SplunkEnvID,Value=$(grep "tag:SplunkEnvID:" config/aws_ec2.yml | cut -d: -f3 | tr -d " ") Key=Name,Value=$(basename $machine) --no-cli-pager; done
The steps here only apply if your current environment is not built with vagrant.
The process is not so traight forward, since I do not know how you built your ansible inventory. Basically, you have to make sure everything you defined in your inventory files is reflected in the splunk_config.yml file.
- Before you upgrade your splunk automator environment, you have to export the inventory to a file
ansible-inventory --list --export > inventory_1.txt
- Migrate all settings to the splunk_config.yml file
- Remove the complete inventory
rm -rf inventory/*
- After the upgrade and building of your splunk_config.yml, you can check the new inventory with the
ansible-inventorycommand and compare it with your dump from version 1.x
With Splunk 9.x some configuration terms have been renamed and are considered deprecated. The Base Config Apps habe been changed and need to be updated and some apps have been renamed. Please download the latest version. The Splunk Platform Automator Framework does taking care of this automatically when Splunk version 9.x or higher is installed. Existing installation are working as is but the settings should be changed manually in the apps.
Those settings have been changed:
- server.conf[clustering]
- mode =
managerorpeer - master_uri -> manager_uri
- mode =
- server.conf[clustermaster:<name>] -> [clustermanager:<name>]
- server.conf[license]
- master_uri -> manager_uri
- outputs.conf[indexer_discovery:<name>]
- master_uri -> manager_uri
Due to this some roles have been renamed in the config file. Please update your exsiting configs:
- cluster_master -> cluster_manager
- license_master -> license_manager
To build your own windows vagrant image follow Setup Windows Vagrant image
Run vagrant the first time to initialize itself and create needed directories. You must execute vagrant always in side the Splunk Platform Automator directory where the Vagrantfile sits, otherwise it will not work correctly. You will see the usage page, when executing vagrant without options.
cd splunk-platform-automator
vagrant
There is one single configuration file, where all settings for your deployment are defined. Copy one configuration file from the examples to config/splunk_config.yml and adjust the setting to your needs. For a standard setup you should be fine with most of the default settings, but there are a lot of things you can adjust for special cases. See the configuration description file, where all existing values are described.
AWS: See instruction here when deploying into Amazon Cloud. You can start with splunk_config_aws.yml for a simple environment. Copy splunk_idxclusters, splunk_shclusters and splunk_hosts sections from other examples for more complex deployments.
When building virtual machines (for virtualbox) the first time it will pull an os image from the internet. The box images are cached here: ~/.vagrant.d/boxes.
vagrant up
The vagrant up command only creates the virtual machines. To deploy Splunk afterwards, run this command:
ansible-playbook ansible/deploy_site.yml
To run both steps with one command use:
vagrant up; ansible-playbook ansible/deploy_site.yml
This will gracefully shutdown all the virtual machines.
vagrant halt
You can destroy all the virtual machines with one command.
vagrant destroy [-f] [<hostname>]
Ansible playbooks can be run over and over again. If the virtual machine is already built, you can rerun the playbooks on a certain host again. This can be needed if something fails and you fixed the error.
ansible-playbook ansible/deploy_site.yml [--limit <hostname>]
To login to one of the hosts just open the index.html file created in the splunk-platform-automator/config directory. You will find links to every role of your deployment.
If something changes along the way and you need to update the linkpage just call this playbook:
ansible-playbook ansible/create_linkpage.yml
Vagrant deployes an ssh key for the vagrant user to login without a password.
vagrant ssh <hostname>
Vagrant uses a dedicated user to work inside the virtual machines. The user name is vagrant and has sudo rights to switch to root or other users.
Splunk Enterprise is installed and run as user splunk. You can switch to this user by sudo su - splunk. For convenience, I have added some command aliases to the user vagrant and user splunk.
alias
You can copy files from your host system to the virtual nodes with the vagrant command. You need to install the vagrant plugin vagrant-scp to have this feature available. Check Vagrant Docs on how to do this.
vagrant scp <files> <target_on_dest> [vm_name]
vagrant scp ../app_dir/splunk-add-on-for-unix-and-linux_831.tgz /var/tmp uf
Splunk Platform Automator can talk to the AWS cloud and create virtual machines with Splunk in the cloud. Vagrant is using the plugin vagrant-aws for that. Follow these steps to setup Splunk Platform Automator for AWS. In the example there is a simple network setup, with only one Security group, covering all ports. More complex network setups should be possible, but make sure the host, where Splunk Platform Automator is running does have ssh access to all instances.
To prepare the configuration file for Amazon deployments
- Take the AWS example and fill in the values you like in the 'aws' section. You need at least:
- access_key_id, secret_access_key if not specified as ENV vars.
- keypair_name
- ssh_private_key_path
- security_groups
- you can use the new 'splunk_download' section in 'splunk_defaults', if you do not want to upload the splunk binaries from your host all the time. This will download them from splunk.com instead.
You can copy splunk_hosts and cluster configs from other example files to the AWS template to create more complex environments. There can be all configuration option used, which are described in the vargant-aws plugin. They can also set individually on the splunk hosts, if needed. Just add a aws: section to the host.
You can also use the ansible playbooks without vagrant. For that you have to create your virtual or physical machines by other means. You can use the ansible playbooks to
deploy the Splunk roles onto the existing servers. Specify the hostnames in the splunk_config.yml file in the splunk_hosts section.
Ansible needs to know where to connect to via ssh to run the playbooks. For this you need to create some custom variables in the
splunk_config.yml file.
As a minimum specify the ssh user for ansible and the ssh private key which has been deployed on the systems. This user must be able to elevate to the root user with sudo.
custom:
ansible_user: ansible
ansible_ssh_private_key_file: '~/.ssh/id_rsa'
If you have host specific variables the custom section can also be added on host level. This could be for example ansible_host if different from the hostname. Also check configuration description
You can verify things like this first with an ansible ping:
ansible -m ping all
And then some more ansible prerequisites with this playbook
ansible-playbook ansible/test_ansible_prereqs.yml
You can build your own python version, if you need a specific python version or your local one is outdated. Please install the needed development tools in order to be able to compile stuff.
This examples installs Python 3.9.9 into your home directory.
cd
mkdir tmp
mkdir -p ~/Python/3.9.9
cd tmp
wget https://www.python.org/ftp/python/3.9.9/Python-3.9.9.tgz
tar -xzf Python-3.9.9.tgz
cd Python-3.9.9/
./configure --prefix=~/Python/3.9.9 --with-ensurepip=install
make
make install
If you need a specific Ansible version you can create it inside a virtualenv environment. This can be useful when deploying older linux images, which too old python versions.
python3 -m venv ansible_414
source ansible_414/bin/activate
python -m pip install ansible==7.7.0 # (which is ansible 2.14.10)
You must install some additional modules for Splunk Platform Automator to work
python -m pip install jmespath # required for json_query calls
python -m pip install lxml # required for license file checks
python -m pip install boto3 # required for ec2 (aws) plugin
Check the ansible version.
ansible --version
If the version is not correct, open a new terminal and activate the virtual environment again with the command from above.
source ansible_414/bin/activate
- Ubuntu 20.04 fails on checking the systemd settings and needs a more current ansible version. See service_facts broken in Ubuntu 20.04
- ulimit settings not working on Ubuntu 14 (without systemd)
- Forwarding data from a universal forwarder to a heavy forwarder cannot be configured in the config file. This must be done manually after installation.
- Virtualbox: Virtual host startup does not respond sometimes, if it fails, recreate the host again.
- Virtualbox has some issues with clock time skew, when not using virtualbox additions. I added a workaround with forcing time clock sync every 5 minutes. A working internet connection on the Virtualbox host is needed.
- AWS: Due to security reasons the login page with the admin password information has been disabled and https is enabled with splunk's own self signed certs.
- AWS: OS images (AMI) do not have ntp configured by default. This will be added in Splunk Platform Automator later.
The following Ansible versions are tested and working with Splunk Platform Automator, but any newer version should work as well.
- ❌ Ansible 2.7.x (EOL)
- ❌ Ansible 2.8.x (EOL)
- ❌ Ansible 2.9.x (EOL)
- ✅ Ansible 2.10.x (EOL)
- ✅ Ansible 2.11.x (EOL)
- ✅ Ansible 2.12.x (EOL)
- ✅ Ansible 2.13.x (EOL)
- ✅ Ansible 2.14.x
- ✅ Ansible 2.15.x
- ✅ Ansible 2.16.x
Check the Ansible Support Matrix for the most current information.
Copyright 2022 Splunk Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.