Securing Splunk 9.0

Implementing Splunk Web SSL and TLS Hostname Certificate Validation

This post focuses on how to secure Splunk Enterprise 9.0 deployments by implementing SSL for Splunk Web and configuring TLS certificate hostname validation. It focuses securing a simple deployment consisting of a single all-in-one Splunk instance (i.e.., responsible for indexing, searching, app and configuration deployment) and a solitary universal forwarder. Figure 1 below shows a diagram of the deployment used in this blog post.

Figure 1. Splunk Enterprise 9.0 deployment consisting of a single all-in-one instance (aio1) and a single universal forwarder (uf1)

The all-in-one Splunk instance (aio1) runs Splunk Enterprise version 9.0.2, and the universal forwarder runs Splunk Universal Forwarder 9.0.2. The operating system running on aio1 and uf1 is CentOS Stream release 8.

NOTE:

• To follow along with this blog post, you will need to obtain a copy of the public certificate of your organization’s Certificate Authority (CA) and the ability to either generate a public certificate or submit a certificate signing request (CSR) for your Splunk Enterprise servers and universal forwarders.

• It is assumed that you have backend access to your deployment (e.g., able to SSH into your hosts) and can execute Splunk management commands.

• The SPLUNK_HOME environment variable refers to the Splunk installation directory and may be different in your set up. For this blog series, SPLUNK_HOME will be set to:

/opt/splunk (aio1)

/opt/splunkforwarder (uf1)

• It is important to use the OpenSSL version shipped with Splunk Enterprise to avoid compatibility issues that may present themselves when using a non-Splunk OpenSSL version to generate private keys and public certificate requests. The version OpenSSL provided by Splunk can be accessed in the CLI using $SPLUNK_HOME/bin/splunk cmd openssl.

Preparing the Deployment and Gathering TLS Certificates

This section will focus on the configuring the Splunk Enterprise all-in-one instance aio1. The steps described in this section can be replicated on any Splunk instance and universal forwarder.

In this set up, we will need to create the folders and apps to hold our configurations. Create a directory called org_certs/ in the $SPLUNK_HOME/etc/auth directory to hold the TLS certificates used to secure connections.

Next, create the following folders within the newly created $SPLUNK_HOME/etc/auth/org_certs/ folder:

• root/ – will hold the public certificate of the Certificate Authority responsible for issuing certificates
• server/ – will hold the private key generated by and public certificate issued to the Splunk instance

Figure 2. Creating folders to hold CA public certificate and certificate issued to the server

Next, place a copy of the CA’s public certificate in $SPLUNK_HOME/etc/auth/org_certs/root. The name of the certificate file does not have to match the name in Figure 3 and can be anything you chose that describes its purpose. Ensure that the following with the CA public certificate is true:

• Account that Splunk is running as has ownership of the file
• CA public certificate is in PEM (.pem) format.

Figure 3. CA public certificate

Generating Private Keys

Navigate to $SPLUNK_HOME/etc/auth/org_certs/server and generate a RSA private key for the Splunk instance. When prompted, supply a password.

Figure 4. Generating RSA private key for Splunk instance

Remove the password from the generated private key file using the command in Figure 5. The reason for removing password protection from the private key file is that Splunk performance may degrade due to handshake operations timing out when having to decrypt the private key. The impact of this is typically felt with Splunk Web timing out and producing 500 Internal Server errors.

Figure 5. Remove password from generated private key file

Generating and Submitting a Certificate Signing Request (CSR)

For TLS certificate hostname validation to work properly, we will need to make sure that the common name (CN) of the public certificate obtained from the CA matches the actual hostname of the server and that any possible subject alternative names (SAN) are identified, e.g., if the hostname is aio1, a possible SAN would be aio1.example.com. To do this, create a custom SSL configuration file in $SPLUNK_HOME/etc/auth/server called cert.cnf and add the following content to it:

[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
# Replace the below with your organization's information
C = US
ST = Georgia
L = Atlanta
O = Tekstream   
OU = Splunk

# Replace with the server's hostname
CN = aio1

[req_ext]
subjectAltName = @alt_names

[alt_names]
# Replace with IP.2 with your server’s address.
# IP.1 is set to the loopback address in order to run Splunk CLI commands on the server
IP.1 = 127.0.0.1
# You can add the server IP address, e.g. IP.2 = 10.1.2.3
# Add any alternative names that the server is known by, e.g. DNS.1 = aio1.example.com

Next, save the configuration file and run the following command to generate the CSR to submit to the CA:

$SPLUNK_HOME/bin/splunk cmd openssl req -new -key server.key -config cert.cnf -out server.csr

(Optional) Check the content of the CSR you just created by running the following command to see the CN and SANs configured in the request:

Figure 6. Output of Certificate Signing Request (CSR) for all-in-one Splunk instance

Finally, submit the CSR to your CA and wait for the CA to issue a public certificate for the instance. When you receive the server’s public certificate, store it in $SPLUNK_HOME/etc/auth/org_certs/server and ensure that the account Splunk is running as has ownership and the ability to read the certificate file.

(Optional) Verify that the Subject Alternative Names show up in the server’s public certificate by running the following command, replacing the name of the certificate file with your own:

[splunk@aio1 server]$ splunk cmd openssl x509 -in server.pem -noout -text | grep -A 1 “Subject Alternative Name”

You should see an output similar to below:

X509v3 Subject Alternative Name:
 IP Address:127.0.0.1

Preparing the Public Certificate for the Splunk Instance

Once you receive the CA-issued public certificate, place it in the $SPLUNK_HOME/etc/auth/org_certs/server folder and ensure that Splunk can read the certificate file and that it is in PEM format. If the issued certificate is not in PEM format, you can perform one of the following commands based on the file extension to convert it to PEM.

To convert a certificate from CRT to PEM:

[splunk@aio1 server]$ $SPLUNK_HOME/bin/splunk cmd openssl x509 -in server.crt -out server.pem

To convert a certificate from DER to PEM:

[splunk@aio1 server]$ $SPLUNK_HOME/bin/splunk cmd openssl x509 -in server.der -out server.pem

To convert a certificate from PKCS7 to PEM:

[splunk@aio1 server]$ openssl pkcs7 -print_certs -in certificate.p7b -out certificate.cer

To convert a certificate from PKCS12 to PEM:

[splunk@aio1 server]$ $SPLUNK_HOME/bin/splunk cmd openssl pkcs12 -in server.p12 -out server.pem -clcerts -nokeys

With the server’s public certificate in PEM format, our next step will be to combine the server public certificate, the CA public certificate, and the server private key into a single PEM certificate file. Make sure you are within the $SPLUNK_HOME/etc/auth/org_certs/server directory and run the following to combine the certificate files:

[splunk@aio1 ~]$ cd /opt/splunk/etc/auth/org_certs/server/
[splunk@aio1 server]$ cat server.pem server.key ../root/ca.pem > server-combined.pem

The contents of the combined certificate file should look similar to what is below:

----- BEGIN CERTIFICATE -----
… (server public certificate) … 
----- END CERTIFICATE -----
----- BEGIN RSA PRIVATE KEY -----
… (server private key) …
----- END RSA PRIVATE KEY -----
----- BEGIN CERTIFICATE -----
… (CA public certificate) …
----- END CERTIFICATE -----

(Optional) Verify the information within the combined public certificate PEM file matches the original server public certificate PEM file by running the following:

# Create a temporary file containing the certificate info of the original certificate
[splunk@aio1 server]$ $SPLUNK_HOME/bin/splunk openssl x509 -in server.pem -text -noout > pemfile1

# Create a second temporary file containing the certificate info of the combined certificate
[splunk@aio1 server]$ $SPLUNK_HOME/bin/splunk openssl x509 -in server-combined.pem -text -noout > pemfile2

# Compare the two files to see if the are different:
# - If no output is generated, the certificates match and we can proceed
# - If there is output, you will need to recreate the combined certificate file
[splunk@aio1 server]$ diff pemfile1 pemfile2
[splunk@aio1 server]$

Before continuing, repeat the steps above for your universal forwarders to obtain the required certificate files. Make sure to provide the correct hostname for the universal forwarder when generating the CSR.

Securing Splunk with TLS Certificates

Before proceeding with the set up of SSL for Splunk Web and TLS certificate hostname validation, we will need to create three configuration apps, org_enable_splunkweb_ssl, org_enable_tls_validation, and org_indexer_base, within $SPLUNK_HOME/etc/apps on the Splunk server (aio1).

Create the configuration app folders and ensure they have the following directory structure:

Figure 7.Configuration app directory structures

For each app’s local/app.conf file, add the following content:

[install]
state = enabled

[package]
check_for_updates = false

[ui]
is_visible = false
is_manageable = false

Next, for each app’s metadata/local.meta file, add the following content:

[]
access = read : [ * ], write : [ admin ]
export = system

Configure SSL for Splunk Web, TLS Certificate Hostname Validation, and Receiving on the All-in-One Instance

Navigate to the local folder of the configuration app we created earlier to enable SSL for Splunk Web, $SPLUNK_HOME/etc/apps/org_enable_splunkweb_ssl/local. Add the following content to the web.conf file:

[settings]
# Enable SSL for Splunk Web
enableSplunkWebSSL = true

# Reference the server’s private key created earlier.
privKeyPath = $SPLUNK_HOME/etc/auth/org_certs/server/server.key

# Reference the server’s public key obtained from your CA
serverCert = $SPLUNK_HOME/etc/auth/org_certs/server/server-combined.pem

Next, navigate to the local folder of configuration app to enable TLS hostname validation, $SPLUNK_HOME/etc/apps/org_enable_tls_validation/local. Add the following content to the server.conf file:

[sslConfig]
# Reference the file that contains all root certificate authority certificates combined
sslRootCAPath = $SPLUNK_HOME/etc/auth/org_certs/root/ca.pem

# Turns on TLS certificate requirements
sslVerifyServerCert = true

# Turns on TLS certificate host name validation
sslVerifyServerName = true
serverCert = $SPLUNK_HOME/etc/auth/org_certs/server/server-combined.pem

# Turns on TLS certificate hostname validation for CLI. This is a required setting on
# any instance you will use the CLI to connect to a Splunk platform instance. E.g.,
# running ‘splunk reload deploy-server’ to deploy app bundles from a Deployment Server
# to Deployment Clients.
cliVerifyServerName = true

Next, navigate to the local folder of the configuration app to enable receiving on the server, $SPLUNK_HOME/etc/apps/org_indexer_base/local. Add the following content to the inputs.conf file:

# Enable listening on port 9996 for receiving events using SSL
[splunktcp-ssl://9996]

# Configure SSL settings
[SSL]
# Full path to the combined server certificate file.
serverCert = $SPLUNK_HOME/etc/auth/org_certs/server/server-combined.pem

# Force client to present an SSL certificate to authenticate
requireClientCert = true

# TLS/SSL version required for handshakes to be successful. 
# It is best practice to use the latest version of TLS, which is version 1.2 at the time of this writing.
sslVersions = tls1.2

# Comma-separated list of Common Name (CN)
sslCommonNameToCheck = uf1

# Optional, if Subject Alternative Names (SAN) are configured in client certificates
# sslAltNameToCheck = uf1.example.com

Restart Splunk on the server and verify whether you can log into Splunk Web. If you are unable to log in, check internal splunkd logs and revisit the steps outlined in this blog.

Configure Secure Forwarding on Universal Forwarder

Before continuing, make sure that the universal forwarder (uf1) has a private key, a public certificate, and the CA public certificate in $SPLUNK_HOME/etc/auth/org_certs, in the respective server/ and root/ folders.

On the universal forwarder, create a configuration app in the $SPLUNK_HOME/etc/apps folder called org_forwarder_outputs. The configuration app should have a similar directory structure as the apps we created earlier, with the only difference being configuration files within:

Figure 8. Configuration app directory structure

Navigate to $SPLUNK_HOME/etc/apps/org_forwarder_outputs/local and add the following content to the server.conf file:

[sslConfig]
sslRootCAPath = $SPLUNK_HOME/etc/auth/org_certs/root/ca.pem

Next, add the following content to the outputs.conf file:

[tcpout]
defaultGroup = primary_indexers

[tcpout:primary_indexers]
disabled = false
# Replace <IP-Address> with IP address of the all-in-one instance
server = <IP-Address>:9996

# SSL SETTINGS
clientCert = $SPLUNK_HOME/etc/auth/org_certs/server/server-combined.pem
sslVerifyServerCert = true
useClientSSLCompression = true
sslCommonNameToCheck = aio1

Finally, restart the Splunk on the universal forwarder. After Splunk restarts on the universal forwarder, you can verify that it is connecting to the all-in-one instance by checking the internal Splunk logs on the universal forwarder. You should see something like the following that will confirm communication between the two servers:

12-08-2022 01:19:43.439 +0000 INFO  AutoLoadBalancedConnectionStrategy [1032 TcpOutEloop] - Connected to idx=x.x.x.x:9996:2, pset=0, reuse=0. autoBatch=1

Another way to verify that the universal forwarder is forwarding events search for its internal logs from the all-in-one instance. Run the following SPL query over the last 15 minutes to verify this:

index=_internal host=”uf1”

Conclusion

There you have it. You have successfully configured TLS hostname validation between an all-in-one standalone Splunk server and universal forwarder. Let’s revisit the steps taken to achieve this goal:

• Created a directory, $SPLUNK_HOME/etc/auth/org_certs/ to hold our certificates for the CA and our Splunk instances.
• Prepared our certificates for configuring TLS hostname validation, making sure that the Common Name (CN) identified in each public certificate for the server matched the server’s hostname and included the loopback address as a Subject Alternative Name (SAN).
• Created three apps on the all-in-one instance and restarted the server:
• org_enable_splunkweb_ssl – enables SSL for Splunk Web
• org_enable_tls_validation – configure TLS hostname validation
• org_indexer_base – configures receiving on the all-in-one instance using our certificates
• Created a single app on the universal forwarder and restarted it:
• org_forwarder_outputs – tells the universal forwarder to use our certificates when sending events to the all-in-one instance
• Verified that our configuration was successful by checking the internal logs of each server and running a query from the all-in-one instance to confirm that it was receiving events from the universal forwarder.

For more information on TLS hostname validation, reference the following Splunk documentation: https://docs.splunk.com/Documentation/Splunk/9.0.2/Security/EnableTLSCertHostnameValidation