Securing Splunk Enterprise with SSL

If you already possess or know how to generate the needed certificates, you can skip this topic and go directly to the configuration steps.

a.     Create a new directory to work from when creating your certificates. In our example, we are using $SPLUNK_HOME/etc/auth/mycerts. (This ensures you do not overwrite the Splunk-provided certificates that reside in $SPLUNK_HOME/etc/auth)

b.     Create the root certificate:

First, you create a root certificate that serves as your root certificate authority. You use this root CA to sign the server certificates that you generate and distribute to your Splunk instances.

Generate a private key for your root certificate:

$SPLUNK_HOME/bin/splunk cmd openssl genrsa -aes256 -out myCAPrivateKey.key 2048

When prompted, create a password for the key.

When the step is completed, the private key myCAPrivateKey.key appears in your directory.

Generate and sign the certificate:

$SPLUNK_HOME/bin/splunk cmd openssl req -new -key myCAPrivateKey.key -out myCACertificate.csr

When prompted, enter the password you created for the private key in $SPLUNK_HOME/etc/auth/mycerts/myCAPrivateKey.key.

Provide the requested certificate information, including the common name if you plan to use common name checking in your configuration. A new CSR myCACertificate.csr appears in your directory.

Use the CSR myCACertificate.csr to generate the public certificate:

$SPLUNK_HOME/bin/splunk cmd openssl x509 -req -in myCACertificate.csr -sha512 -signkey myCAPrivateKey.key -CAcreateserial -out myCACertificate.pem -days 1095

When prompted, enter the password for the private key myCAPrivateKey.key.

A new file myCACertificate.pem appears in your directory. This is the public CA certificate that you will distribute to your Splunk instances.

c.    Create the server certificate:

Now that you have created a root certificate to serve as your CA, you must create and sign your server certificate.

Generate a key for your server certificate:

Generate a new RSA private key for your server certificate. In this example we are again using AES encryption and a 2048 bit key length:

$SPLUNK_HOME/bin/splunk cmd openssl genrsa -aes256 -out myServerPrivateKey.key 2048

When prompted, create a new password for your key. A new key myServerPrivateKey.key is created.

You will use this key to encrypt the outgoing data on any Splunk Software instance where you install it as part of the server certificate.

Generate and sign a new server certificate:

Use your new server private key myServerPrivateKey.key to generate a CSR for your server certificate.

$SPLUNK_HOME/bin/splunk cmd openssl req -new -key myServerPrivateKey.key -out myServerCertificate.csr

When prompted, provide the password to the private key myServerPrivateKey.key.

Provide the requested information for your certificate, including a Common Name if you plan to configure Splunk Software to authenticate via common name checking. A new CSR myServerCertificate.csr appears in your directory.

Use the CSR myServerCertificate.csr and your CA certificate and private key to generate a server certificate.

$SPLUNK_HOME/bin/splunk cmd openssl x509 -req -in myServerCertificate.csr -SHA256 -CA myCACertificate.pem -CAkey myCAPrivateKey.key -CAcreateserial -out myServerCertificate.pem -days 1095

When prompted, provide the password for the certificate authority private key myCAPrivateKey.key.
Make sure to sign this with your private key and not the server key you just created.

A new public server certificate myServerCertificate.pem appears in your directory.

d.    You should now have the following files in the directory you created:

• • • myServerCertificate.pem
    • myServerPrivateKey.key
    • myCACertificate.pem

Prepare your signed certificates for Splunk authentication:

Once you have your certificates, you must combine the server certificate and your keys into a single file that Splunk software can use.

a.   Create a single PEM file:
Combine your server certificate and public certificate, in that order, into a single PEM file.

cat myServerCertificate.pem myServerPrivateKey.key myCACertificate.pem > myNewServerCertificate.pem

b.   Once created, the contents of the file myNewServerCertificate.pem should contain, in the following order:

• • • The server certificate (myServerCertificate.pem)
    • The private key (myServerPrivateKey.key)
    • The certificate authority public key (myCACertificate.pem)

PART TWO: Securing Splunk Enterprise

What to do when you have your certificates. You can apply encryption and/or authentication using your own certificates for:

1. Communications between the browser and Splunk Web
2. Communication from Splunk forwarders to indexers
3. Other types of communication, such as communications between Splunk instances over the management port

1. Communications between the browser and Splunk Web:

Assuming that you have already generated self-signed certificates or purchased third-party certificates.

Make sure your certificate and key are available from your folder. In this example we are using $SPLUNK_HOME/etc/auth/mycerts/:

• $SPLUNK_HOME/etc/auth/mycerts/mySplunkWebCertificate.pem
• $SPLUNK_HOME/etc/auth/mycerts/mySplunkWebPrivateKey.key

Open or create a local web.conf file in $SPLUNK_HOME/etc/system/local/web.conf, or in any other application location if you’re using a deployment server.

Under the [settings] stanza, configure the path to the file containing the web server SSL certificate private key and the path to the PEM format Splunk web server certificate file.

The following example shows an edited settings stanza:

[settings] enableSplunkWebSSL = true privKeyPath = </home/etc/auth/mycerts/mySplunkWebPrivateKey.key > serverCert = </home/etc/auth/mycerts/mySplunkWebCertificate.pem >

Restart your Splunk software: # $SPLUNK_HOME/bin/splunk restart splunkd

You must now prepend “https://” to the URL you use to access Splunk Web.

2. Communication from Splunk forwarders to indexers:

Using your own certificates to secure Splunk communications involves the following procedures:

· Configuring indexers to use a new signed certificate

· Configuring forwarders to use a new signed certificate

Configuring indexers to use a new signed certificate:

Copy the server certificate and CA public certificate into an accessible directory on the indexer you want to configure.

$SPLUNK_HOME/etc/auth/mycerts/

Configure the inputs.conf file on the indexer to use the new server certificate.
Add the following stanzas to the $SPLUNK_HOME/etc/system/local/inputs.conf file, or the appropriate directory of any app you are using to distribute your forwarding configuration:

[splunktcp-ssl://9998]
disabled=0
[SSL]
serverCert=/opt/splunk/etc/mycerts/myServerCertificate.pem
#requireClientCert = true
#sslAltNameToCheck = forwarder.local

Note: Configure your indexers to use SSL on port 9998. Because you can continue to have the existing non-SSL port to use on 9997. Once we configure complete enterprise to use SSL on port 9998 and everything is reporting on 9998 port, we can then disable the 9997.

Indexers are configured with the Root CA cert used to sign all certificates. This can be achieved by editing the file server.conf in $SPLUNK_HOME/etc/system/local on your indexer(s).

[sslConfig]
sslRootCAPath = /opt/splunk/etc/mycerts/myCACertificate.pem

Restart the splunkd process:
# $SPLUNK_HOME/bin/splunk restart splunkd

Repeat the above steps on all the Indexers.

Configure forwarders to use a signed certificate:

Given we would be having huge number of forwarders; We need to use DS to push the certificates and the configuration files.
I would recommend to test it with few set of forwarders before deploying into all the clients.

Steps:
In the DS, create an app ‘customer_cert_outputs’ (use your own naming) with all the cert

files (.pem) in the /local folder. The app should also contain the outputs.conf file.

Make sure to refer the correct app path in the outputs.conf file.

[tcpout:group1]
server=indexer01:9998, indexer02:9998
disabled = 0
clientCert = /opt/splunk/etc/apps/customer_cert_outputs/local/myServerCertificate.pem

Push this app out to the forwarders! (Don’t forget to mark the app as “restart required” in the server class on the DS.)

To verify your SSL connections in Splunk Web, try the following command:

index=_internal source=*metrics.log* group=tcpin_connections | dedup hostname | table _time hostname version sourceIp destPort ssl

You can also splunkd.log to validate and troubleshoot your configuration.

Once we see the clients forwarding thru port 9998. We can leverage the whitelist/blacklist in the serverclass to push the app to all forwarders in a phased manner.

3. Securing inter-Splunk communication:

Distributed search configurations share search information, knowledge objects and app and configuration information over the management port.

Communication between search heads and peers relies on public-key encryption. Upon startup, Splunk software generates a private key and public key on your Splunk installation. When you configure distributed search on the search head, the public keys are distributed by search heads to peers and those keys are used to secure communication. This default configuration provides built-in encryption as well as data compression that improves performance.

It is possible to swap these generated keys out with your own keys, though the existing keys are generally considered adequate for most configurations.

Click here to connect with a Splunk Specialist or learn more about our Splunk Managed Services Here.

• Share This: