Troubleshooting Guide: Assets & Identities in Splunk Enterprise Security
By Kamal Dorairaj, Senior Splunk Consultant
Splunk Enterprise Security (ES) uses an asset and identity (A&I) information management system to correlate asset and identity information with events to provide context and enrich data. There are many docs and discussions on how to populate these A&I in Splunk ES but not many on how to troubleshoot A&I issues and validate the framework. This blog covers various actions as well as steps to assess and validate the A&I framework in Splunk ES.
1. Validate Search Preview
In Splunk ES, Navigate to Configure -> Data Enrichment -> Assets and Identity Management-> Search Preview.
All three search in this page run periodically to make sure that the ES framework is using the latest and updated assets and identities. For example: When we expand the “Identity_lookup_expanded” SPL, there will be two macros which are merged into identity and updated into the identity lookup file. Ideally when we open the search preview SPL, it should return zero events. Returning zero events means the KV store committed all the changes to the collections. It commits changes every 300 seconds which is configured under “Time” in the Miscellaneous Settings -> Global Settings.
2. Validate the Macros
Run the “identity_lookup_expanded” SPL in a separate search. Below is an example of the SPL.
In this `add_entity_source(“TS Identities”,”TS Identities”)` is the macro that pulling the Identity details. Open this macro in a separate search window and expand it. Here we can see all the lookup files the search references.
Navigate to Settings -> “Searches, Reports and Alerts” -> Search for the “TS Identities”. Click the Report -> Review the SPL -> The search in this report is populating the above file. The SPL can be run over 7 days or 30days to check when it’s being populated.
3. Validate Correlation Setup
Go to “Correlation Setup” under Asset and Identity Management -> Select “Enable for all sourcetypes” as in screenshot below.
By enabling this option, every single search run on ES goes thru the identity lookup expanded and asset lookup expanded.
Let’s say for example, you have user=bob then correlation will try check to if identity=bob then try to pull all the information from the identity and details found and use it to enrich the events. So, this setting is very important for the ES framework.
4. Validate Enforce Props
The “Enforce props” setting should be enabled under the Global Settings. If we do not enable this option, then the framework will be using stale lookups and not receiving the enrichments on ES searches. This setting is a must for RBA.
Enabling this option would bring more risk fields into the automatic lookups. If not enabled, then we are not running the latest version of the automatic lookups. Also, if there are any custom fields in the identity fields, those won’t be available in the automatic lookup. Basically, custom fields won’t be enriched in all the searches. Go to “Searches, Reports, and Alerts” -> Search for “SA-IdentityManagement” -> All the lookups for src, dest, user, src_user. On the backend, Splunk uses props to provide the enrichments for all these fields.
5. Validate Lookup Files
Permissions of all the lookup files needs to be reviewed. Go to Lookups -> Lookup definition -> search for the lookup -> make sure permissions looks accurate. If correct, then take note of the csv file name and navigate to Lookups -> Lookup files -> check permissions and confirm it also looks correct.
The permissions should be scoped as global.
If all permissions appear accurate in the UI then directly ssh into the server.
Navigate to cd /opt/splunk and search for the lookup file by executing the following command:
Next run this:
ls -al <path of the file name returns from the above> and less the file name.
Basically, looking for the csv files defined in any inputs.conf stanzas.
Next, check If any transforms are using it with the following:
6. Validate Thru Logs
To look for any errors or issues in the log file, run the below SPL:
index=_internal source=*entity*
Check the source field for a list of used sources. We can see ‘identity manager log’, ‘entity merge log’ and ‘identity correlation rest handler log’. All these logs are related to the assets& identity framework logs.
For Example: A customer experienced WLM blocking their searches. Using the above SPL, we were able to further troubleshoot our issue.
When the identity manager runs it sends logs for everything into identity manager log.
Index=_internal source=*identity_manager.log
In the source field narrow your SPL to investigate the splunkd_access log. All rest calls executed are written in this log file. Make sure all the values for the filed status are “200”.
Splunk ES runs *Audit -* scheduled searches every day, which are it’s all-health check searches.
Now, you can populate A&I in Splunk ES, but you also have a guide for troubleshooting issues and validating the framework. Splunk Enterprise Security is considered the most robust security platform out there, and with this set of steps you can ensure you’re getting the most from your instance.
Our team is available if you have additional questions, just inquire using the form below. Find more Splunk Technical Blogs here, and learn more about TekStream Splunk Services here. Happy Splunking!