Using Rsyslog with Splunk
By Dave Cheever, Splunk Consultant
While Splunk can receive syslog events directly, having a centralized logging location is considered best practice. There are inherent issues that come with sending data to syslog over UDP 514. First, UDP is a stateless protocol that does not guarantee delivery which can result in events getting dropped. When you restart Splunk the listener is not available, thus you will lose events while Splunk is down. Additionally, it is difficult to differentiate between the various datasets coming in. If all syslog data sources are coming into Splunk as “udp:514” or “syslog”, how can you easily distinguish the events you are analyzing? Rsyslog is a great way to integrate log aggregation while also addressing these concerns.
Modular Approach
Rsyslog allows for a modular approach by treating messages according to defined rules. For example, data sources can be sent to different ports allowing them to be separated into their own respective directories. This can then be utilized in the Universal Forwarder’s monitor input to collect the logs and send them to Splunk. With this method, your data flow will resemble the following and Splunk will now be able to segregate source types:
When rsyslog is installed, the main configuration file is located under /etc/rsyslog.conf. In there you will find that $ModLoad imtcp is commented out. This needs to be uncommented to allow for TCP syslog reception (also uncomment $ModLoad imudp if you’ll be using UDP). There are three key components within rsyslog.conf to specify data collection ports and store messages in separate directories:
- Inputs objects designate input sources to collect the messages. For this use case, you’ll want to be familiar with the imptcp input module mentioned above. This module provides the ability to receive syslog messages via TCP and allows for specifying which ports to send data to. Ideally, each technology will be collected on its own unique port.
- Filter conditions allow for property-based or expression-based filtering. This will check messages against specific conditions using a compare operation. All remote traffic coming in will pass through technology-specific filters to match against the corresponding data sources.
- Rulesets specify how rsyslog will process a message when the filter condition is true. Rules provide the ability to carry out specific actions such as write a message to disk or send to a remote location. In this case, the output should be written to a directory to be used in the root of the UF monitoring stanza.
Once messages are written to a local directory on the syslog server, the Universal Forwarder can now transmit this data to Splunk. The UF’s inputs configuration file will allow the events to be collected, and will also allow for setting the indexes, source types, and host segments of the data. Assigning the source type will help distinguish data sources as well as determine how event data will be formatted during the indexing process; be sure to use what is required by the respective TA. The inputs.conf stanzas should resemble something like this:
[monitor:///<log root>/<vendor name>/*/*.log]
index=foo
sourcetype=bar
host_segment=<the segment in the path that is your hostname>
There you have it! We hope this has answered your questions about Rsyslog. Please contact us if you would like further information.