How to Merge Two Multi-Site Indexer Clusters into One

  By: Jon Walthour  |  Team Lead, Senior Splunk Consultant

 

Problem: Take two multi-site indexer clusters and meld them into one with all the buckets from cluster A residing in and being managed by cluster B. Then, with all the buckets transferred to cluster B, cluster A indexer hardware can be decommissioned.

TL;DR: It is not possible to do this, because the buckets from cluster A will never be moved outside of cluster A’s storage by cluster B’s cluster manager.

Step 1 is to make the clusters identical in configuration. You’d have to:

  1. Ensure both clusters are running on the same version of Splunk.
  2. Ensure the indexes.conf files on both clusters were identical—they both contain all the indexes, that each index stanza is named the same in both (e.g. one indexer cluster can’t put its Windows Event Logs in an index named “windows” and the other put them in one named “wineventlog”). Configurations would need to be changed and directories renamed to make both clusters the same in terms of index names and locations of hot/warm and cold buckets. Bottom line: You need to be able to use the same indexes.conf in either cluster since, when they were merged, they would be.
  3. The contents of the controller apps and agent apps directories would also need to be identical in terms of contents and versions across both clusters. Again, they are eventually going to share the controller app’s contents. So, in preparation, they must be made identical.

Step 2: Turn cluster A from a multi-site indexer cluster into a single-site indexer cluster. The replicated buckets on site2 in cluster A are redundant and will get in the way. To do this, site search and replication factors in cluster A will need to be set to “total” values of “2” on each. Then, wait for all the fixup tasks to complete to get the cluster to meet site_replication_factor and site_search_factor. Finally, you would follow these steps to convert cluster A to a single-site indexer cluster.

Step 3: Now add single-site cluster A to multi-site cluster B as a new site. For instance, if multi-site cluster B has three sites, site1, site2, and site3, cluster A now becomes site4 of cluster B. You do this by:

  1. Edit server.conf on the cluster manager, adding “site4” to the “available_sites” attribute under the “clustering” stanza and restart Splunk.
  2. Add peers to the new site by running:
    splunk edit cluster-config -mode peer -site site4 -master_uri https://:8089 -repiication_port 8080 -secret
    splunk restart
  3. Add search heads by running:
    splunk edit cluster-config -mode searchhead -site site4 -master_uri https://:8089 -replication_port 8080 -secret
    splunk restart
  4. Point forwarders at the new cluster by adding the following to their server.conf:
    [general]
    site=site4

Step 4: In theory, then, you would next decommission site4, causing all its buckets to be dumped into the other three sites. Here’s the rub, though, and it’s a big one: the buckets created when site4 was cluster A won’t migrate off the site. Any buckets created when site4 was cluster A will continue to only exist on site4 and the cluster manager will never migrate them elsewhere into other sites.

This is clearly stated in the documentation for decommissioning a site of a multi-site indexer cluster:

If a peer on the decommissioned site contains any buckets that were created before the peer was part of a cluster, such buckets exist only on that peer and therefore will be lost when the site is decommissioned…Similarly, if the decommissioned site started out as a single-site cluster before it became part of a multisite cluster, any buckets that were created when it was a single-site cluster exist only on that site and will be lost when the site is decommissioned.

It is immaterial here that cluster A was once a multi-site indexer cluster. The condition is the same. They are buckets from a foreign cluster and the cluster manager will leave them segregated as long as they exist. So, the only way your end goal could be accomplished is if you could hold on to the hardware for one of the sites in cluster A until such time as all its buckets aged out to frozen and were, presumably, deleted. Then, you would be at a state where all the buckets in site4 were created after it became site4 and no buckets remained from when it was cluster A. Only at that time could you decommission site4 and reclaim the hardware without suffering any data loss.

Contact us for more help with merging multi-site indexer clusters: