Linux Servers & Splunk – You Can’t Really Work Splunk Without Linux Commands, Can You?

By Tyler Phillips, Splunk Consultant

Today I will be going over some essential Linux commands to set up Splunk on a Linux server and troubleshoot.

Once you are in a Linux box we first need to make sure there is an /opt/splunk directory in this server. So, we are going to use the cd (change directory) command to see if it is there.

cd /opt/splunk

If you do not go into a Splunk box, then we now move to the mkdir (make directory) command to create it. We also are going to add a sudo before this command to make root perform the action.

sudo mkdir /opt/splunk

Now that there is a directory to place Splunk, let’s bring it in! We are going to install wget onto the server before moving to making a Splunk user.

sudo yum install wget

Next, we want to have a user to run Splunk so let’s use the adduser command to create a Splunk user and log into it.

sudo adduser splunk

sudo su – splunk

Let’s use that wget command to bring in the Splunk tgz. This command can be found on the Splunk downloads page for the version you would like. For this example, I will list the wget for 9.0.2: wget -O splunk-9.0.2-17e00c557dc1-Linux-x86_64.tgz

After downloading Splunk to your server there we need to untar the folder in the /opt/splunk/folder. Let’s run this command: sudo tar -xvzf wget -O splunk-9.0.2-17e00c557dc1-Linux-x86_64.tgz

Now we need to accept Splunk license agreements: cd /opt/splunk/bin

Then run this command to enable boot start and accept the license: sudo enable boot-start -user splunk –accept-license

Here you will be asked for the username and password you want to use for Splunk, so fill those in.

Now you should have a Web UI for Splunk!

Troubleshooting

Since this post is about using Linux commands to start and troubleshoot Splunk, let’s move on to some common items you may encounter.

  1. Upgrading vs. Re-Installing: while working on this post I decided to upgrade my local instance and ran into a problem. I used a different install method this time, so it wasn’t a true “upgrade” but instead more like a re-install.
  2. Holding ports: I ran into the problem where my old version of Splunk was still holding on to port 8000. I used the ps -ef | grep splunk command to find the splunk tasks still running. I used the kill command to stop those tasks. Here is my command for example: kill -9 20189. After stopping I was able to start my new version of Splunk with no problems.
  3. Finding available disks: A command recommended to me by a friend that I find helpful finds the available disks and their usage on the server. This command does just that: df -h. Nothing beats a friendly recommendation sometimes!
  4. Decrypting your pass4SymmKey: Another instance you might need a special command for is decrypting your pass4SymmKey. When you have forgotten your pass4SymmKey, copy the value down in the .conf file and go to the bin directory. Run this command with your value to show what you decrypted password is: ./splunk show-decrypted -value
  5. Data doesn’t show up: A command I used often in my training was the btool command. Anytime I had data not show up I would run the ./splunk btool outputs list command on my forwarders and indexers to make sure they were forwarding to the correct IPs. You can also add –debug to the end of the command to show the destination path to each line.
  6. Find your inputs: One more command that may help you is the ./splunk list inputstatus command. This command will help you find your inputs and the STUS where they were ingested. You can see the file size, the position, directory, how much of the file has been processed and its completion status.

That’s a good round of the basic commands you will need to set up Splunk and troubleshoot with a Linux box. I hope this helps you get started Splunking! You can contact me below for follow up: