Using and Understanding Basic Subsearches in Splunk

Caroline Lea
June 19, 2020
01:49 pm

By: Brent Mckinney | Splunk Consultant

A subsearch in Splunk is a unique way to stitch together results from your data. Simply put, a subsearch is a way to use the result of one search as the input to another. Subsearches contain an inner search, who’s results are then used as input to filter the results of an outer search. The inner search always runs first, and it’s important to note that subsearches return a maximum of 10,000 results and will only run up to 60 seconds by default.

First, it’s good to understand when to use Subsearch and when NOT to use Subsearches. Generally, you want to avoid using subsearches when working with large result sets. If your inner search produces a lot of results, then applying them as input to your outer search could be inefficient. When working with large result sets, it will likely be more efficient to create fields using the eval command and performing statistical results using the stats command. Because subsearches are computationally more expensive than most search types, it is ideal to have an inner search that produces a small set of results and use that to filter out a bigger outer search.

Example:

Suppose we have a network that should only be accessed from those local to the United States. We’re interested in seeing a list of users who’ve successfully accessed our network from outside of the United States. We could build one search to give us a list of IP addresses from outside of the U.S., and another search could be used to give a list of all accepted connections. A subsearch could then be used to stitch these results together and help us obtain a comprehensive list.

First, we’d need to decide what our inner results should be, a list of all accepted connections, or a list of all non-U.S. IPs? Using the latter as an inner search would probably work best, as it should return a much smaller set of results.

Our inner search would look something like this, using the iplocation command to give us more info on the IP address field.

index=security sourcetype=linux_secure | stats count by ip_address | iplocation ip_address | search Country !=“United States” | fields ip_address

This essentially results in a list of IP addresses that are not from the U.S.

From here, we want to create another search to return a list of all accepted connections. This will be our outer search, and look something like this:

index=security sourcetype=linux_secure connection_status=accepted | dedup ip_address | table ip_address, Country

To Combine these, we can use the following subsearch format. Inner searches are always surrounded by square brackets, and begin with the search keyword. Here’s what our final search would look like:

Here, our inner search (enclosed in square brackets) would be run first and would return IP addresses that do not belong to the U.S. Those results would be used to filter out the outer search, with returns results of connections that were accepted by the network. Finally, the end of the outer search provides a table with the IP address and country for each result.

We have now obtained a list of IP addresses that have successfully accessed our network, along with the country that it was accessed from, all through the power of a Splunk subsearch!

Tips for troubleshooting if your subsearch is not producing desired results:

Ensure that the syntax is correct. Make sure that the entire inner search is enclosed in square brackets, and that it is placed in the appropriate place of the outer search.
Run both searches by themselves to ensure that they return the expected results independent of each other. Each search may need to be tuned a bit before combining them into a subsearch. Keep in mind that the results of the inner search are used as a filter for the outer search.
You can check into the Splunk job inspector to see if anything stands out that looks out of the ordinary. The normalizedSearch property helps in showing the results of the subsearch.

This covers the basics of subsearches within Splunk. It’s worth noting, however, that there are advanced commands available to use with subsearches to achieve specific results. These commands include append, which could be used to combine searches that run over different periods or join, which can take a field from an inner search, and correlate that field to events from an outer search. These take on similar syntax to run, and are worth trying out once you have down the basics!

Want to learn more about basic subsearches in Splunk? Contact us today!