Email Routing Using Sendemail in Splunk Enterprise Security
Email Routing Using Sendemail in Splunk Enterprise Security
By: Bruce Johnson | Practice Lead, Operational Intelligence
This was the use case scenario: Something went bump in the night. We needed to be able to send alerts from correlation searches to the security guards after hours and on weekends for a few specific correlation searches. Certain categories of activity (e.g, access violations, creating a new account, getting deleted, getting locked out, clearing security or system logs, using service accounts, etc.) needed to alert the after-hours team.
Now there are plenty of tools that do this very effectively (VictorOps among them). We needed something simple and, in Splunk, it really couldn’t be simpler.
The brute force method would have been to create correlation searches that run after hours and send to different email aliases. In other words, you have a different schedule to run a correlation search because you want that correlation search to route to different people, so create a duplicate search with different schedule settings. I suppose this would have been appropriate if the after-hours search had different levels of severity because of the timing, in which case I would have definitely taken that approach, but that was not the case. There is also no way to use cron to do conditionals. So I couldn’t do a single secondary search that would run on both after hours on weekdays and all hours on weekends (e.g. <*/15 0,1,2,3,4,5,6,19,20,21,22,23 * * 1-5> OR <*/5 * * * 6-7>). Practically speaking that would mean three different correlation searches – untenable for Splunkers like me that are aspirationally lazy (not very successful yet but someday).
What we needed was a means to determine whether a search result was run after hours or on weekends and set a flag. Then use a lookup to return the emails that we would route to and pass that as a parameter to the email action set up in the correlation search. This was just so much simpler than I thought it would be.
The lookup (mail_recipients.csv) for routing purposes at its simplest level:
| after_hours | |
| bjohnson@whitehouse.gov | 1 |
| bruce.johnson@tekstream.com | 0 |
| bruce.johnson@match.com | 1 |
I added other columns for userid, escalation level, cc, bcc and some fields that we might anticipate using should our routing need to be more complex, but for now we focused simply on the “after hours” use case. By The Way – the Sendresults app makes sending emails to a column dead simple but our use case was so basic, it really wasn’t needed. If you want to play with it: https://splunkbase.splunk.com/app/1794/
Here’s the search – formatted to use _internal instead of CIM or wineventlog for testing purposes. The sendemail is included for testing as well. All we want to do in the format of the correlation search is to set the routing to $result.recipients$ in the To field. This may not work if you have no errors in your environment (insert appropriate emoji).
In the final version I pulled out the code between the evals and the recipient creation and put it in a macro (stripping all the fields I used except for recipients. Then inserted the macro into every correlation search that needed the routing.
The eventual correlation searches just needed to insert the macro, ensure that the recipients field was in the final result, and change the routing on the email action to go to $result.recipients$ – simple but useful.
The eventual search looked similar to this…
Next up: Modify the search to use data models and to actually use the max hour for the search so that if the search results that come back have a mix of times that cross the current hour boundary, the most conservative path is chosen.
Want to learn more about email routing in Splunk Enterprise Security? Contact us today!
[pardot-form id=”17645″ title=”Blog – Bruce Johnson – Email Routing Using Sendemail in Splunk Enterprise Security”]