pass4SymmKey does not hash when pushed from Cluster Master
by Aaron Dobrzeniecki | Senior Splunk Consultant
When working with a pass4SymmKey, the goal is to get the password to hash when Splunk restarts on the servers you have placed them on. I have noticed while working with many customers that pushing a pass4SymmKey from the Cluster Manager WILL NOT hash the password on the indexers. The screenshots below are using Splunk 8.2.5 and I am pushing a pass4SymKey to the indexers.
NOTE, pushing passwords from a CM in Splunk version 9.0+ will hash the pass4SymmKey on the indexer AND cluster manager side.
On the Cluster Manager:
In newer versions of Splunk they have included a directory called manager-apps. As you can see above, I am using master-apps which you can still use in 8.2.5. In order to migrate over from using master-apps vs manager-apps, you need to move ALL of the apps in master-apps to manager-apps and apply the cluster bundle. NOTE! You cannot use master-apps and manager-apps at the same time.
I have applied the bundle to the indexers and this image shows the checksums matching!
Splunk Indexer 1:
Splunk Indexer 2:
Splunk Indexer 3:
As you can see from the multiple screenshots above, the pass4SymmKey did not hash on the indexers when pushed from the Cluster Manager. Also, in the first screenshot you can see that the password is also in plain text on the Cluster Manager. This imposes a security risk as well on the Cluster Manager. If an unauthorized user was able to gain access to the Cluster Manager, they could easily find this pass4SymmKey.
Now, the stanza I have used in my example is for cascading bundle replication from the Search Heads. Having a similar pass4SymmKey under that stanza on all indexers, allows the indexers to send each other knowledge object bundles originating from the Search Head. Pushing ANY pass4SymmKey in ANY stanza from the Cluster Manager will not hash the value. For more information on cascading bundles click here.
Solution/Resolution
An absolute way I have found to make sure the pass4SymmKey hashes is to place the stanza and pass4SymmKey in $SPLUNK_HOME/etc/system/local/server.conf and restart Splunk. I know what you must be thinking, “We have a large number of indexers that we would need to manually touch in order to place the pass4SymmKey in $SPLUNK_HOME/etc/system/local/server.conf”. IF YOU DO NOT have any sort of automation tool (puppet, ansible, bitbucket CICD, or even just a script that will reach out and touch every indexer), you will have to manually touch each box. If you do not have any automation tools but you have a person who can write a script to touch each box, try that before the manual process.
Manually Setting pass4SymmKey in Splunk
-Navigate to the CLI or backend of the indexers
-cd $SPLUNK_HOME/etc/system/local (Make sure you are the user that is running Splunk)
-If there is no current server.conf, you can create one
-vi server.conf
-Place the specific stanza and pass4SymmKey setting you will be using as below:
Note: **You will need to restart the indexers for the hashing to occur. In an indexer cluster you will want to perform a rolling restart instead of restarting each indexer manually. **
-Navigate to the GUI or the backend of the Cluster Manager
–GUI: Settings -> Indexer Clustering -> Edit (Top Right) -> Rolling Restart -> Begin Rolling Restart (check the widgets as necessary)
–Back End/CLI: Navigate to $SPLUNK_HOME/bin and run the following command
./splunk rolling-restart cluster-peers
The screenshots below will show you, that after manually placing my pass4SymmKey in $SPLUNK_HOME/etc/system/local/server.conf and performing a rolling restart on the cluster; the pass4SymmKey hashes!
In Conclusion
Splunk versions pre 9.0 WILL NOT hash the pass4SymmKey when pushed from the Cluster Manager. If you would like to apply a pass4SymmKey on versions before 9.0, you will either need to use an automation tool to reach every indexer, or touch each server manually as explained above. Please see all server.conf files from each version of Splunk after the push from the Cluster Master/Manager.
Splunk 8.2.5 Indexer
Splunk 8.2.5 Cluster Master
Splunk 9.0 Indexer
Splunk 9.0 Cluster “Manager”
Splunk 9.0.1 Indexer
Splunk 9.0.1 Cluster “Manager”
There you have it! If you still have questions about this, or any other Splunk item that you’d like help with, contact us! Submit your comments below and we will be in touch right away.