Using clientName to Simplify Forwarder Management

By Jon Walthour, Technical Architect

Today, I want to introduce you to a new way of simplifying configuration management with Splunk’s deployment server/deployment client architecture. The goal is to get you out of the business of constantly adding new hostnames to or removing hostnames from whitelists or csv files in your serverclass.conf file on your deployment server. You can put all that on automatic with this approach.

Overview

As an overview, a deployment server manages the configuration for multiple deployment clients, usually forwarders, via a model where clients connect to a deployment server, identify themselves and then, based on that identification, download app bundles from the deployment server which the deployment server has been configured for them to receive. This determination by the deployment server of which apps a client downloads is usually done by matching the hostname of the client or its IP address against a list of values defined in a configuration file called “serverclass.conf” on the deployment server. These lists are in the form of one or more whitelist or blacklist settings in a “serverClass” stanza and are typlically comprised of hostnames and IP addresses, either in whole or in part with the use of wildcards (*). Whenever a new deployment client is added or an old one retired, a change to these listings must be made and the deployment server’s configuration reloaded.

Auto-Scale AWS Heavy Forwarders

Let’s say, for example, you need to auto-scale heavy forwarders to handle increased AWS event traffic. You can’t be constantly adding and removing ec2 instance names to one or more serverclasses in serverclass.conf. You need a way to have the deployment client communicate to the deployment server, letting it know “I’m an AWS heavy forwarder” and have the deployment server recognize that and send the client a listing of apps to install and update based solely on its classification as an “AWS heavy forwarder” without needing to have the particular hostname or IP address or IP address range detailed anywhere in serverclass.conf.

The answer here is to add that designation to the deployment client when the universal forwarder software is installed. This way, when the deployment client connects to the deployment server for the very first time, it identifies itself properly. This designation is defined in a setting called “clientName” in deploymentclient.conf in the “[deploynent- client]” stanza. It is recommended this setting be added to the copy of deploymentclient.conf in $SPLUNK_HOME/etc/system/local as it is an identifier for that particular server and will remain untouched by any changes made by the deployment server in $SPLUNK_HOME/etc/apps. Unlike “hostname” or “DNS name”, multiple deployment clients can have the same clientName. It’s just a categorization, a tag, a label to say, “I’m a web server” or “I’m a database server” or, in our example, “I’m an AWS heavy forwarder.” The actual value you use for clientName by itself is unimportant. What is important is that the “clientName” in deploymentclient.conf matches against one or more server classes on the deployment server to determine which apps are to be retrieved by the client from the server.

So, a deploymentclient.conf file in $SPLUNK_HOME/etc/system/local might look like this:

[deployment-client] clientName = aws-idm

and the serverclass.conf file on the deployment server might contain something like this:

[serverClass:aws-heavy-forwarders] whitelist.0 = aws-idm

[serverClass:aws-heavy-forwarders:app:Splunk_TA_aws]

Notice that you use the “clientName” value in serverclass.conf in the same way you would use hostname or IP address. So, you could also have a configuration in serverclass.conf that looks like this:

[serverClass:all-aws] whitelist.0 = aws-*
machineTypesFilter = linux-x86_64 [serverClass:all-aws:app:Splunk_TA_nix]

This configuration could be used to address all AWS instances running a Linux OS to distribute the Unix/Linux TA to them.

Mix and Match Identifiers

You can also mix and match the use of clientName with the other identifiers such as hostname or IP address. So, this example is also possible using a wildcarded clientName of “:oracle:”:

[serverClass:all-oracle] 
whitelist.0 = *:oracle:* 
whitelist.1 = oraclehost01 
whitelist.2 = oraclehost02 
whitelist.3 = 192.168.23.*

Remember that the order of these various identifiers like “clientName”, “hostname” and “IP address” are evaluated by the deployment server. “clientName” takes precedence over “hostname” or “DNS name”. So, you can use a phased in approach if you want to introduce this categorization system into an existing Splunk deployment. Add the clientName values alongside the existing hostnames and other identifiers. As deployment clients are configured with their clientName value, it is that value that will take precedence and eventually, when all clients have a clientName, the hostname can be removed from serverclass.conf to clean up the file. Just be careful during the transition. Just because clientName is part of the client’s identification of itself doesn’t mean that hostname or IP address is ignored. They are still part of the identification. The precedence just means that, should both identifiers exist in a serverClass stanza, the clientName setting will be decisive in determining the client’s membership in that serverClass.

clientName Values

The clientName value can be anything. You don’t need to segment it with dashes or colons. It could be just a string like “weblogic” or “database”. However, I have found in implementing this approach with customers that using segmentation in the value allows these clientNames to have multiple dimensions in them. You could, for example, identify a host as being in “Production” versus “Development”, in the “Finance” department or the “Sales” department, in the “Seattle”, “Chicago” or “New York” offices, all within the same clientName. For example:

newyork:sales:sqlserver:prod

By evaluating the segments, different apps can be sent to different hosts simply distinguished only by the value of one particular segment. For example, a serverClass to configure the collection of database log events from all the Microsoft SQL Server hosts could, using the example hostname above, employ a simple whitelist like:

[serverClass:mssqlservers] whitelist.0 = *:*:sqlserver:*

Or server classes could be configured to send prod hosts’ event data to a production indexer cluster while those in the development environment to the development cluster.

[serverClass:prod_forwarder_outputs] whitelist.0 = *:*:*:prod

[serverClass:dev_forwarder_outputs] whitelist.0 = *:*:*:dev

If you choose to take this approach, though, consider the nature of your infrastructure carefully and how Splunk is laid out in it. Then, design your “clientName” conventions to your best advantage.

In this article, you have seen how you can get out of the business of maintaining lists of server hostnames in serverclass.conf because of constantly onboarding new and removing decommissioned hostnames. With the “clientName” setting in deploymentclient.conf, you can put the management of configuring deployment clients on automatic.