Blog

Configuring Splunk-Connect-for-Syslog with HTTP Event Collector Using Podman

Cece Kintner
August 11, 2022
04:45 pm

By: Oluwaseun Oke

Splunk Connect for Syslog is an open source packaged solution for getting data into Splunk. It is based on the syslog-ng open source Edition (Syslog-NG OSE) and transports data to Splunk via the Splunk HTTP event collector (HEC) rather than writing events to disk for collection by a Universal Forwarder.

According to Splunk.github.io “Splunk Connect for Syslog is a containerized distribution of syslog-ng with a configuration framework designed to simplify getting syslog data into Splunk Enterprise and Splunk Cloud. Our approach is to provide a runtime-agnostic solution allowing customers to deploy using the container runtime environment of choice.”

For this demo we are going to use two (2) servers; Podman/SC4S server and Splunk (all-in-one) server.

Podman: Podman is a daemonless, rootless container engine developed which allows containers to be run by non-root users. The best recommendation is to use CentOS 8. As such there are certain configurations that are necessary to be ensure a seamless installation.

Splunk (aio): We can use the regular amazon linux instance or Centos 8 but we will be using the regular Amazon linux for this demo

Splunk Installation
Install Splunk and have it running using regular Splunk installation.
Configuring SC4S using Podman.

Podman Configuration

Please run the following commands below to update the Centos 8 with the necessary updates (please note that this only necessary for Centos 8) on the sc4s/podman server.

# cd /etc/yum.repos.d/
# sed -i ‘s/mirrorlist/#mirrorlist/g’ /etc/yum.repos.d/CentOS-*
# sed -i ‘s|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g’ /etc/yum.repos.d/CentOS-*

After which we are going to swap the centos-linux-repos to the centos-stream-repos (Appstream) for Centos 8 which is used for setting up the Podman and we are going to sync it the Appstream repos we are replacing.

# dnf –disablerepo ‘*’ –enablerepo=extras swap centos-linux-repos centos-stream-repos
# dnf distro-sync

******Switch to your Splunk Server****

Create Indexes¶

SC4S is pre-configured to map each sourcetype to a typical index. Since this is a new installation, it is best practice to create them in Splunk when using the SC4S defaults as shown below. Please note that SC4S can be easily customized to use different indexes if desired.

• email
• epav
• epintel
• infraops
• netauth
• netdlp
• netdns
• netfw
• netids
• netlb
• netops
• netwaf
• netproxy
• netipam
• oswin
• oswinsec
• osnix
• print
• em_metrics (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index)

Please go to the Splunk web and confirm if the indexes have been created.

Configure the Splunk HTTP Event Collector

We are going configure the Splunk HTTP Event Collector by creating a HEC token and make it global to help stream the data from the SC4S/Podman container to the Splunk Server indexers. It is strongly recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of HWFs. It is recommended that the “Selected Indexes” on the token configuration page be left blank so that the token has access to all indexes, including the “lastChanceIndex. If this list is populated, extreme care must be taken to keep it up to date, as an attempt to send data to an index not in this list will result in a 400 error from the HEC endpoint.

Switch to the SC4S/Podman Server to begin the installation of the Podman container

Need to enable and the update the powertools to help with the installation of the “Podman”

# sudo dnf config-manager –set-enabled powertools

# sudo dnf -y update

We going to confirm if we have the container-tools needed to install Podman

# sudo dnf module list | grep container-tools

After the confirmation, we can now install Podman

# sudo dnf install -y @container-tools

Always confirm the installation by checking the Podman version by running:

# podman version

CONGRATS!!!!!!

Now we would want to confirm that we can check the version of the OS from Podman:

# podman run -it –rm alpine sh

/ # cat /etc/os-release

Now you can get out of the Podman container to the OS level by typing:

# exit

Implement a Container Runtime and SC4S

The following script can be saved in a GitHub repository which can be called with “wget” in the cli:

#!/bin/bash
# 14/03/2022 John Barnett
# Script created on / for CentOS 8
# 21/07/2021 – Added TLS Remix, added TLS listener – note creates a default cert below so edit / remove as required
# 14/03/2022 – Updated default container pull to version 2
### Based on quick start here – https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/podman-systemd-general/
# Set URL and Tokens here
# SplunkCloud Example
# HEC_URL=”https://http-inputs-MYSTACKNAME.splunkcloud.com”
HEC_URL=”https://3.134.109.163:8088″
HEC_TOKEN=”e5f1032d-b0a2-4cd2-8010-2411ce6b7552″
#hostnamectl
#hostnamectl set-chassis server
#hostnamectl set-location rack1
#hostnamectl set-hostname sc4sbuilder
hostnamectl

################################################################################
########### Dont edit below here, unless you know what you are doing ###########
################################################################################
red=`tput setaf 1`
green=`tput setaf 2`
yellow=`tput setaf 3`
reset=`tput sgr0`
echo “${yellow}Check date and TZ below!${reset}”
date
echo “${yellow}Updating Firewall Rules${reset}”
#Show original state
firewall-cmd –list-all
#Splunk ports
firewall-cmd –zone=public –add-port=514/tcp –permanent # syslog TCP
firewall-cmd –zone=public –add-port=514/udp –permanent # syslog UDP
firewall-cmd –zone=public –add-port=6514/tcp –permanent # syslog TLS
firewall-cmd –zone=public –add-port=5425/tcp –permanent # syslog
firewall-cmd –zone=public –add-port=601/tcp –permanent # syslog
firewall-cmd –reload
#Check applied
firewall-cmd –list-all
dnf install -y conntrack podman
echo ”
## Edited with JB Splunk Install script by magic
net.core.rmem_default = 17039360
net.core.rmem_max = 17039360
” >> /etc/sysctl.conf
sysctl -p
echo ”
## Created with JB Splunk Install script by magic
[Unit]
Description=SC4S Container
Wants=NetworkManager.service network-online.target
After=NetworkManager.service network-online.target
[Install]
WantedBy=multi-user.target
[Service]
Environment=\”SC4S_IMAGE=ghcr.io/splunk/splunk-connect-for-syslog/container2:2\”
# Required mount point for syslog-ng persist data (including disk buffer)
Environment=\”SC4S_PERSIST_MOUNT=splunk-sc4s-var:/var/lib/syslog-ng\”
# Optional mount point for local overrides and configurations; see notes in docs
Environment=\”SC4S_LOCAL_MOUNT=/opt/sc4s/local:/etc/syslog-ng/conf.d/local:z\”
# Optional mount point for local disk archive (EWMM output) files
Environment=\”SC4S_ARCHIVE_MOUNT=/opt/sc4s/archive:/var/lib/syslog-ng/archive:z\”
# Uncomment the following line if custom TLS certs are provided
Environment=\”SC4S_TLS_MOUNT=/opt/sc4s/tls:/etc/syslog-ng/tls:z\”
TimeoutStartSec=0
ExecStartPre=/usr/bin/podman pull \$SC4S_IMAGE
ExecStartPre=/usr/bin/bash -c \”/usr/bin/systemctl set-environment SC4SHOST=$(hostname -s)\”
ExecStart=/usr/bin/podman run \\
-e \”SC4S_CONTAINER_HOST=\${SC4SHOST}\” \\
-v \$SC4S_PERSIST_MOUNT \\
-v \$SC4S_LOCAL_MOUNT \\
-v \$SC4S_ARCHIVE_MOUNT \\
-v \$SC4S_TLS_MOUNT \\
–env-file=/opt/sc4s/env_file \\
–health-cmd=”/healthcheck.sh” \\
–health-interval=10s –health-retries=6 –health-timeout=6s \\
–network host \\
–name SC4S \\
–rm \$SC4S_IMAGE
Restart=on-abnormal
” > /lib/systemd/system/sc4s.service
sudo podman volume create splunk-sc4s-var
sudo mkdir /opt/sc4s/
mkdir /opt/sc4s/local
mkdir /opt/sc4s/archive
mkdir /opt/sc4s/tls

echo ”
## Created with JB Splunk Install script by magic
# Output config
SC4S_DEST_SPLUNK_HEC_DEFAULT_URL=$HEC_URL
SC4S_DEST_SPLUNK_HEC_DEFAULT_TOKEN=$HEC_TOKEN
#Uncomment the following line if using untrusted SSL certificates
SC4S_DEST_SPLUNK_HEC_TLS_VERIFY=no
# TLS Config, for McAfee etc
SC4S_SOURCE_TLS_ENABLE=yes
SC4S_LISTEN_DEFAULT_TLS_PORT=6514
#SC4S_SOURCE_TLS_OPTIONS=tls1.2
#SC4S_SOURCE_TLS_CIPHER_SUITE=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
” > /opt/sc4s/env_file
echo “${yellow}Generating Cert for TLS${reset}”
openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -subj “/C=NZ/ST=NI/L=Home/O=SC4S Name/OU=Org/CN=sc4sbuilder” -keyout /opt/sc4s/tls/server.key -out /opt/sc4s/tls/server.pem
echo “${yellow}Your /opt/sc4s/env_file looks like this${reset}”
cat /opt/sc4s/env_file
echo “${yellow}Starting SC4S – This might take a while first time as the container is downloaded${reset}”
sudo systemctl daemon-reload
sudo systemctl enable –now sc4s
# Send a test event
echo “Hello MYSC4S” > /dev/udp/127.0.0.1/514
sleep 10
sudo podman logs SC4S
sudo podman ps
# Sleep to allow TLS to come up
sleep 20
netstat -tulpn | grep LISTEN
#### Use command below and then type to test
#openssl s_client -connect localhost:6514
#### Use command below for full tls test if required (adjust as needed)
#podman run -ti drwetter/testssl.sh –severity MEDIUM –ip 127.0.0.1 sc4sbuilder:6514

After the installation and configuration, we should be able to see the data coming in on the Splunk Server.

Steps to Creating a custom Index and Filtering

Custom Index creation

Create the custom index in Splunk before editing sc4s
Move to the sc4s directory: cd /opt/sc4s/local/context
Create a file “splunk_metadata.csv” to create custom indexes and events and place the following in file: vi splunk_metadata.csv
name of the application, index, name of the index (example is shown below)
vi splunk_metadata.csv
test_sourcetype, index, org_test_index
test_sourcetype, source, mytest

Save the file and restart sc4s service
# systemctl restart sc4s

Check the status of sc4s to confirm the update made to the splunk_metadata.csv file
# systemctl status sc4s

Filtering
Use case: Exclude a host “sc4s_vendor_product

Move to the sc4s directory: cd /opt/sc4s/local/context
This will involve 2 conf files in the “context” directory: “vendor_product_by_source.conf” and vendor_product_by_source.csv.
First edit the “vendor_product_by_source.csv” specify the filter and field name

vi vendor_product_by_source.csv

f_null_queue,sc4s_vendor_product,"null_queue"

and save the file

vi vendor_product_by_source.conf
filter f_null_queue {
host(‘192.168.12.1’)\
};

and save file

Restart sc4s service
# systemctl restart sc4s

Check the status of sc4s to confirm the update made to the splunk_metadata.csv file
# systemctl status sc4s

Use case: Exclude events with the word “deny”

Move to the sc4s directory: cd /opt/sc4s/local/context
This will involve 1 conf file in the “context” directory: “vendor_product_by_source.conf”

vi vendor_product_by_source.conf

filter f_null_queue {
message(‘deny’)
};

Save the file and restart sc4s service
# systemctl restart sc4s

Check the status of sc4s to confirm the update made to the splunk_metadata.csv file
# systemctl status sc4s

Additional Resources

Splunk Connect for Syslog
TechGlimpse Blog
TekStream Resources
Splunk
Podman

HAPPY SPLUNKING!!!!!!!!!!!!!