Make Unstructured Data Searchable with a Multikv Command
Jeff Rabine | Splunk Consultant
Splunk works best with structured data. This blog post will cover how to make unstructured data searchable. In this real-world example, the customer wanted to use data in the second table of an unstructured log file. Changing the log format was not an option, and access to .conf files was not available, so all changes needed to happen at search time.
Raw log sample:
As you can see, there are two tables of data in the log. The first step is to remove the top table from the results since it’s unnecessary for this search. We will do that using the rex command to over-write _raw capturing only the data that we need.
The next step is to use the multikv to break the tables into separate events. This command will attempt to create fields and values from the table however, in our case, we removed the headers from the table because the formatting of our table was not clean. This caused the multikv command to not work properly. Since we removed the headers, we will set them to noheader=t.
Now, the last thing we need to do is create our field extractions, and then we can use the data however we please.
As you can see, we now have nice clean data!
Other uses of the multikv command:
Depending on your data, there are other ways to use the multikv command. Neither of these examples was able to make unstructured data searchable for our customer, but I recommend trying them with your data. Your success with the following examples will depend on how cleanly formatted your logs are.
In our example, we stripped out the headers of the table to make unstructured data searchable. You may be able to leave the headers. That would save you from extracting the fields with the rex command. Also, by default, the command will attempt to process multiple tables within the log, so you might just have to use the multikv command. After running this search, check to see if the correct fields were extracted.
index="fruitfactory" sourcetype="fruitfactory"
| multikv
You can also tell the command what row contains the headers of the table. This would allow you to always look for the headers on the first, second, etc row of the event. Again, check and see if the correct fields were extracted after running this command.
index="fruitfactory" sourcetype="fruitfactory"
| multikv forceheader=
Want to learn more about unstructured data or using the multikv command? Contact us today!