Troubleshooting Null Field Values and Trailing Spaces

By: Jeff Rabine | Splunk Consultant

In my career as a Splunk Consultant, I have run across numerous occasions where I was thrown off by what I thought were null field values or trailing spaces where I didn’t expect spaces to exist. This can be confusing and even lead to inaccurate results. I will do my best to demonstrate these to you but in a real-world scenario, it’s never as obvious as a blog post.

First here is an example of a search that is counting IP Addresses with an “Unknown” status. As you can see the results show zero:

Empty String and Trailing Example

Without knowing the data, the person writing this search could easily be done here. However, when we take a closer look, we notice there are 9 events with blank entries for Known_Bad_IP. But how could that be?!  We did a fillnull!

field values and trailing known bad ip

The fillnull did not work because the fields are not null they contain empty strings. In this case, we can fix this by adding a replace command. To replace these empty string fields with Unknown:

field values or trailing spaces

As you can see now the search is working as expected.

In my next example, we are coalescing IP addresses from two sources. As you can see, we are sorting the IP_Address column but it’s not sorting correctly.

empty string and trialing coalescing IP addresses

If we do an eval we can count the length of the field and as you can see, we have 10 character IP Addresses that are counting out as 11 characters.

empty string and trailing eval character length

Now if we look at the raw data, we can see that for some reason there is a trailing space on some but not all the IP addresses.

empty string and trailing spaceon some IP addresses

To fix this issue we would need to either fix the existing field extraction or add a new inline field extraction. Another option would be to use the trim command to trim the trailing space.

empty string and trailing inline field extraction

Trim:

Again, these are just lab examples of field values or trailing spaces. I experienced similar issues in the real world, but the errors were hiding in millions and billions of events. Sometimes you will run into cases where reports or field extractions are accurate 99.9% of the time but once a month a log source comes in that doesn’t work with your field extraction and causes the report to have inaccuracies. I hope this article will remind you to know your data and never assume anything. Happy Splunking!

Want to learn more about any of these commands?  Contact us today!