Operating a Splunk Environment with Multiple Deployment Servers

Operating a Splunk Environment with Multiple Deployment Servers

By: Eric Howell | Splunk Consultant

Splunk Environments come in all shapes and sizes, from the small single-server installation managing all of your Splunk needs in one easily-managed box, to the multi-site, extra complex environments scaled out for huge amounts of data and all the bells and whistles to get in-depth visibility and reporting into a wide variety of circumstances as suits functionally any use case you can throw at Splunk. And, of course, everything in between.

For those multi-site, or multi-homed environments, that many data centers require for any range of needs, managing your configurations begins to get complicated between the additional firewall rules, data management stipulations, and any other broad range of issues that might crop up.

Thankfully, Splunk Enterprise allows for your administrative team, or Splunk professional services, to set up a Deployment Server to manage the configurations (bundled into apps) for all of the universal forwarders, so long as they’ve been set up as deployment clients. In a complicated environment, you may find that you need two deployment servers to manage the workload, for any number of reasons. Perhaps you are trying to keep uniform configuration management systems in multiple environments, or perhaps you are aiming to spread the communication load across multiple servers for these deployments. Whatever the use case, setting up two (or more) deployment servers is not the heartache you may be worried about, and the guide below should be ample to get you on the right track.

Multiple Deployment Servers – Appropriate Setup

To set up multiple deployment servers in an environment, you will need to designate one of the Deployment Servers as the “Master” or “Parent” server (DS1). This is likely to be the original deployment server that houses all of the necessary apps, and is likely already serving as deployment server to your environment.

The use case below will allow you to service a multi-site environment where each environment requires the same pool of apps, but is small enough to be serviced by a single deployment server.

  1. Stand up a new box (or repurpose a decommissioned server, as is your prerogative)! Install Splunk on this new server. This will act as your second deployment server (DS2).
  2. The key difference between these servers is that DS2 will actually be a client of the DS1.
  3. Initial set up is minimal, but make sure that this server has any standard configurations the rest of your environment holds, such as an outputs.conf to send its internal logs to the indexer layer, if you are leveraging that functionality.
  • You will create a deployment client app on DS2. You could use a copy of a similar app that resides on one of your heavy forwarders that poll DS1 for configuration management, but you will need to make two key adjustments in deploymentclient.conf:

  • Once this change has been made, the apps that will be pulled down from DS1 will reside in the appropriate location on DS2 to be deployed out to any servers that poll it.
  • Restart Splunk on DS2
  • Next, you will need to navigate to the ForwarderManagement UI on DS1 and create a Server Class for your Slave or ChildDeployment Servers (DS2 in this case)
  • Add all apps to this new server class
    • Allowing Splunk to restart with these apps isfine, as changes made to the originating Deployment Server (DS1) will allow DS2
      to recognize that the apps that it holds have been updated and are ready for
      deployment.
  • Add DS2 to this Server Class
  • Depending on the settings you have configured indeploymentclient.conf on DS2 for its polling period (phoneHomeIntervalInSecs
    attribute), and how many apps there are for it to pull down from DS1, wait an appropriate amount of time (longer than your polling period, and more) and
    verify if the apps have all been deployed.
  • After this, updates made to the apps on DS1 will propagate down to DS2.

Alternative Use Case

If you are planning to leverage multiple deployment servers to service the same group of servers/forwarders, you will want to also copy over the serverclass.conf from DS1. If all server classes have been created through the web ui, the file should be available here:

$SPLUNK_HOME/etc/system/local/serverclass.conf

If this is your intended use case, you will also want to work with your Network Team to place the Deployment Servers behind a loadbalancer. If you do so, you’ll need to modify the following attribute in deploymentclient.conf in your deployment client app that resides on your forwarders to indicate the VLAN:

You will also need to make sure both Deployment Servers generate the same “checksums” so that servers polling in and reaching different DS servers do not redownload the full list of apps with each check-in.

To do so, you will need to modify serverclass.conf on both Deployment Servers to include the following attribute:

This attribute may not be listed by default, so you may need to include it manually. This can be included with the other attributes in your [global] stanza.

Want to learn more about operating a Splunk environment with multiple deployment servers? Contact us today!