Install Splunk Forwarder to Send Logs to SIEM

Written by Alex DiMuzio on the 23rd Aug 2021

Exported on 24-Aug-2021 17:14:21

Using Attune to install and configure a Splunk Universal Forwarder

This blueprint is used to install a Splunk Universal Forwarder on a host. The universal forwarder will monitor log files on the system in real-time and forward them to a Splunk Indexer for configuration.

Pre-Blueprint Attune setup
  1. On the Inputs tab, create a Linux node for the host you wish to install the Splunk forwarder on.
  2. On the Inputs tab, create Linux credentials to connect to the host you wish to install the Splunk forwarder on.
  3. On the Inputs tab, create a basic credential that you wish to use for the Splunk admin user.
  4. On the Inputs tab, create a basic text value to store the IP address of the Splunk Forwarder
Blueprint Steps
  1. Pull down the Splunk Forwarder and unpack it to the default install directory.
  2. Create a dedicated, limited access user for Splunk to run as
  3. The Splunk installation prompts for credentials for an admin user. Since we are trying to automate without user interaction, we need to set the credentials up in our Attune Inputs and pass the values into a file in the Splunk directory.
  4. The next group step does the following:
  5. Ensures that the Splunk application directory has the correct permissions,
  6. Ensures Splunk is started with the dedicated splunk user to avoid running it as root
  7. Configures Splunk to start upon server boot, so that the system can be restarted without us forgetting to start Splunk back up and losing log data.
  8. Configures outputs.conf to specify where we want to send our log data
  9. Configures inputs.conf to specify what data we want to monitor and send to our indexer

Parameters

Name Type Script Reference Default Value Comment
Host_Creds Linux OS Credential host_creds
Hostname Linux / Unix Server hostname
Splunk Admin Generic Credential splunkAdmin
SplunkIndexer Text splunkindexer

1 - Download and Install Splunk Forwarder

The first step in our blueprint starts by checking if the Splunk forwarder is already installed. If it isn't, it downloads the forwarder and unpacks it in the default directory, then cleans up the installer to save disk space.

The connection details have changed from the last step.

Login as user on node

Connect via SSH 
ssh user@hostname
This is a Bash Script make sure you run it with bash -l from a terminal session 
#check if Splunk Forwarder is already installed
if [ -d "/opt/splunkforwarder/bin" ]
then
    echo "Splunk Forwarder is already installed"
else

#download latest version of Splunk 
wget -O splunkforwarder-8.2.1-ddff1c41e5cf-Linux-x86_64.tgz 'https://d7wz6hmoaavd0.cloudfront.net/products/universalforwarder/releases/8.2.1/linux/splunkforwarder-8.2.1-ddff1c41e5cf-Linux-x86_64.tgz'

#unpack the downloaded installer .tgz to the default Splunk install directory /opt/splunkforwarder
sudo tar -xzf splunkforwarder-8.2.1-ddff1c41e5cf-Linux-x86_64.tgz -C /opt/

#remove installer
rm -f splunkforwarder-8.2.1-ddff1c41e5cf-Linux-x86_64.tgz
fi

2 - Create Local Splunk User

The next steps creates the dedicated Splunk user and group so we don't have to run Splunk as root

Login as user on node

Connect via SSH 
ssh user@hostname
This is a Bash Script make sure you run it with bash -l from a terminal session 
#add splunk user
if ( useradd splunk )
then
    echo "Successfully created splunk user"
else
    echo "User already exists"
fi


#add splunk group
if ( groupadd splunk )
then
    echo "Successfully created splunk group"
else 
    echo "Group already exists"
fi

3 - Create Admin Creds

Next we create the user-seed.conf file, which will store the admin password we want to use to log into and manage Splunk. This file will automatically get deleted after the password is applied for security purposes. We pass in the splunk admin basic credential input that we created.

This step has the following parameters

Name Script Reference Default Value
Splunk Admin {splunkAdmin.password} None

Login as user on node

Connect via SSH 
ssh user@hostname
This is a Bash Script make sure you run it with bash -l from a terminal session 
#create user-seed.conf file that Splunk accepts to set admin credentials without user interaction
sudo touch /opt/splunkforwarder/etc/system/local/user-seed.conf

#pass Splunk admin credentials into file
sudo cat <<EOF > /opt/splunkforwarder/etc/system/local/user-seed.conf
[user_info]
USERNAME = admin
PASSWORD = {splunkAdmin.password}
EOF

4 - Configure Splunk Forwarder

The following steps are grouped together as they all fall into the realm of application-specific Splunk configuration.

4.1 - Update Permissions and Accept License

The first thing we need to do is ensure the dedicated Splunk user we created owns all of the Splunk files, otherwise we will run into permission issues without running as root. Next, we start Splunk with the accept-license switch to avoid getting prompted to accept the Splunk user agreement. The third command will tell the server to start Splunk upon server startup, so we don't have to manually start the process every time.

Login as user on node

Connect via SSH 
ssh user@hostname
This is a Bash Script make sure you run it with bash -l from a terminal session 
#make splunk user the owner of splunk dir
chown -R splunk:splunk /opt/splunkforwarder

#start Splunk to accept license agreement
sudo runuser -l splunk -c "/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes"

#configure Splunk to start on boot
sudo /opt/splunkforwarder/bin/splunk enable boot-start

4.2 - Configure Splunk Outputs.conf

The outputs.conf file is where we configure our log destination. This can be a Splunk Indexer, or a Heavy Forwarder that can be used to transform the data before sending it into the indexer. In our case, we use a basic configuration to tell our forwarder to send our log data to the Splunk Indexer, which we configured in an input.

This step has the following parameters

Name Script Reference Default Value
SplunkIndexer {splunkindexer.value} None

Login as user on node

Connect via SSH 
ssh user@hostname
This is a Bash Script make sure you run it with bash -l from a terminal session 
sudo cat <<EOF > /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
defaultGroup=my_indexer

[tcpout:my_indexer]
server={splunkindexer.value}

[tcpout-server://{splunkindexer.value}:9997]
EOF

4.3 - Configure Splunk Inputs.conf

Finally, we need to tell our Splunk forwarder what data we want to monitor. We do that in the inputs.conf file, and here we are telling it to monitor the files /var/log/messages and /var/log/secure. The sourcetype value is a way we can separate our different log files to more easily search them.

At this point, your forwarder is configured. If you also ran the Splunk Indexer blueprint and configured your forwarder to send there, you can log into your Indexer and begin searching your log data. A simple search of "sourcetype=syslog OR sourcetype=linux_secure" will display the logs we configured above. Happy Splunking!

Login as user on node

Connect via SSH 
ssh user@hostname
This is a Bash Script make sure you run it with bash -l from a terminal session 
sudo cat <<EOF > /opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/secure]
disabled = false
sourcetype = linux_secure
 
[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
EOF