Splunk Cloud App Vetting Made Simpler!
BY: Bryan Bollou | Splunk Consultant
Whether you want to get an app approved or create a new sourcetype in Splunk Cloud, the cloud vetting process can be an exhausting exercise in vetting involving you and the cloud ops team. This post will show you how to leverage the Splunk App Inspect API, as well as common pitfalls to avoid so that your app gets approved for installation.
From an on-prem Splunk instance access the Splunk GUI to create a new app on your instance. Set a name, folder name, set visibility to “No”, and the template to barebones.
Access the CLI of the machine you did this on through SSH or locally. Navigate into the directory where the template app was created; $SPLUNK_HOME/etc/apps.
This directory will be the template for creating the app. It is easier to start from this template as it will have the necessary default files and a structure to start from.
In this example we are looking to create a custom app that will modify a sourcetype in a Splunk Cloud deployment.
To figure out what props.conf changes need to be made for the sourcetype;
Get the raw data you are trying to parse.
Upload data through the “Add Data” section in Splunk.
Select a source and drop in the data file.
Now you can create a custom sourcetype.
Go to “Advanced” and enter the appropriate fields to correctly parse your sourcetype.
These can be new settings for; time_prefix, max_timestamp_lookade, time_format, etc.
Save the setting and check if the parsing is working as you want it. Hit “preview” to see the necessarily props.conf settings.
Example of resulting props.conf stanza:
[new_sourcetype_dreamcrush]
SHOULD_LINEMERGE=false
LINE_BREAKER=</Interceptor>([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=UTF-8
disabled=false
TIME_FORMAT=%Y-%m-%d</ActionDate>
TIME_PREFIX=<ActionDate>
Go back to the machine with the app template directory.
If a props.conf needs to be created, create, and set it to the proper permissions for Splunk custom apps (700 for directories and 600 for files). Append the setting you created to the $SPLUNK_HOME/etc/apps/appname/props.conf of your app.
Commands
cd $SPLUNK_HOME/etc/apps/appname/
touch props.conf
chmod 600 props.conf
vi props.conf
If this is a one-time custom app, such for creating a sourcetype, add the following stanza to the $SPLUNK_HOME/etc/apps/appname/app.conf.
[package]
check_for_updates = 0
Next, there should only be a default directory in the custom app. If you are modifying an app through the GUI however, it will create files in the local directory. Make sure you move the necessary files and merge the configs in the default directory. NOTE: You should also remove the bin directory, which would be where you place scripts, if it is not needed.
Commands
cd $SPLUNK_HOME/etc/apps/appname/
rm -rf local
rm -rf bin
As bare minimums, you only need the default and metadata directories.
Make sure the parent app directory has “execute” permissions.
The following commands will set all the files’ permissions to 600 and all directories’ permissions to 700 – the standard for Splunk custom apps. Run this inside the root folder of your app ($SPLUNK_HOME/etc/apps/appname/). You can replace “/path/to/base/dir” with a period “.” to get every file/directory in and under the working directory.
Commands
find /path/to/base/dir -type d -exec chmod 700 {} +
find /path/to/base/dir -type f -exec chmod 600 {} +
You can then compress the directory into an archive using a command such as tar. This is packaging the app.
NOTE: On a MacOS machine, there are temporary files created that will be denied by the cloud vetting process. To avoid this, a variable to not copy the temporary files (dot_underscore) must be set followed by the appropriate flag in the tar command.
Commands
export COPYFILE_DISABLE=1
tar –exclude=”.*” -czvf test_template.tar test_template
Now let’s check if our app would pass the cloud vetting process – get ready for tons of failures and cryptic messages! For this we will use the Splunk App Inspect API that is well documented here; https://dev.splunk.com/enterprise/docs/developapps/testvalidate/appinspect
For this article, we will be using the Postman Method.
Sign up/into Postman then download the application from; https://www.postman.com/downloads/
Import the Splunk AppInpsect Script from; https://dev.splunk.com/enterprise/docs/developapps/testvalidate/appinspect/useappinspectapi/ and import it into Postman.
Under your workspaces on the left you will see the AppInspect – Production collection.
You can then run the Authenticate script supplying your Splunk login and password to get an authorization token.
Depending on the type of validation you need, you can then use one of the “script” POST scripts to submit your app tar by selecting the app_package KEY.
You can then use the “status” GET script to generate the results of the validation based on the app you submitted.
Finally, you will use the “report” GET scripts to grab the report results in either HTML or JSON format. This report will tell you if your app would pass the vetting process if you were to submit it through Splunk Customer as an app install, for example. Issues that will block the app are listed as failures and issues that should be fixed put wouldn’t cause a block are listed as warnings.
If you followed the steps in this article. You will see that there are no failures, and this custom app would pass the cloud vetting.
There is, however, 1 warning;
check_kos_are_accessible
• The ‘admin’ role is not available to Splunk Cloud customers. Please consider also including the ‘sc_admin’ role if you want Splunk Cloud customer administrators to be able to access this knowledge object File: default.meta Line Number: 5
To change all the occurences of “admin” to “sc_admin” in the default.meta file use the following commands.
Commands
cd $SPLUNK_HOME/metadata
vi default.meta
:%s/admin/sc_admin
:wq
After making this change, if you compress the file and resubmit it through the Splunk AppInspect checks, it will have no failures/warnings.
This custom app can then be submitted to be implemented in your Splunk Cloud environment without any issues!
Depending on the type of custom app you are looking to vet, the error checks will be more numerous and can be cryptic. The error checks are listed here; https://dev.splunk.com/enterprise/reference/appinspect/appinspectcheck/ and you can see some of them are not very descriptive.
The following are errors that can be encountered when trying to vet a dashboard app for Splunk Cloud and their possible solutions.
1.
check_that_extracted_splunk_app_contains_default_app_conf_file
• Splunk App packages must contain a `default/app.conf file`. No `default/app.conf` was found in `[redacted]`.
An app.conf file must be created. Using a template from a barebones app from Splunk will do this but if one already exists in your local directory;
Commands
cd $splunkhome/etc/apps/appname/local
mv app.conf ../default/app.conf
2.
check_that_local_does_not_exist
• A 'local' directory exists in the app.
You need to move all the files from local into default. You can either merge the two if both are populated or remove the default directory and rename the local one.
Commands
rm -r default
mv local default
3.
check_for_bin_files
• This file has execute permissions for owners, groups, or others. File: local/data/ui/views/[redacted].xml
You need to change all fields to have 600 permissions and all directories to have 700 permissions.
Commands
find /path/to/base/dir -type d -exec chmod 700 {} +
find /path/to/base/dir -type f -exec chmod 600 {} +
4.
check_simplexml_standards_version
• Change the version attribute in the root node of your Simple XML dashboard local/data/ui/views/[redacted].xml to `<version=1.1>`. Earlier dashboard versions introduce security vulnerabilities into your apps and are not permitted in Splunk Cloud File: local/data/ui/views/[redacted].xml
In all your dashboards, the Splunk standard is to have it with <dashboard> or <form>.
Example
Old:
<?xml encoding=”UTF-8″?><version=”1.1″><form theme=”light”>
New:
<form theme=”light” version=”1.1″>
5.
check_for_sched_saved_searches_earliest_time
The saved search [redacted] doesn't contain dispatch.earliest_time.It is prohibited to specify scheduled searches that don't specify a dispatch.earliest_time in Splunk Cloud. File: default/savedsearches.conf Line Number: 165
All saved searches must have an earliest time. If a search is set to All-Time in the GUI. It will normally have a latest_time of “now()” and no earliest_time defined. You can fix this by going to the $SPLUNK_HOME/etc/apps/appname/default/savedsearches.conf and adding “dispatch.earliest_time = 1” under the appropriate search stanzas. This will set the start time at the start of unix time.
6.
check_for_valid_package_id
• The app.conf [package] stanza's has an invalide 'id' property: [redacted]. For the `id` property, it must contain only letters, numbers, `.` (dot), and `_` (underscore) characters, and can not endwith a dot character. Besides, some reserved names are prohibited. See https://docs.splunk.com/Documentation/Splunk/7.3.1/Admin/Appconf for details. File: default/app.conf Line Number: 6
check_that_directory_name_matches_package_id
• The `app.conf` [package] stanza has an `id` property that does not match the uncompressed directory's name. `app.conf` [package] id: [redacted] uncompressed directory name: [redacted]. File: default/app.conf Line Number: 6
In the $SPLUNK_HOME/etc/apps/appname/default/app.conf there is an id property under the [package] stanza. This has to exactly match the filename of the UNCOMPRESSED app package. It can also only contain letters, numbers, dot and underscore.
Example
Old:
[package]
id = Test® Template™ BETA
New:
[package]
id = TestTemplate
You would also have to rename the directory to match the id;
Commands
mv TestTemplateBETA TestTemplate
7.
check_for_local_meta
• Do not supply a local.meta file- put all settings in default.meta. File: metadata/local.meta
There should be no local.meta file. If there is one and it is populated, you should merge the local.meta into the default.meta. Make sure that you aren’t making changes to the app while you are packaging it, or a new local.meta file will be created.
Commands
cat default.meta local.meta > merged.meta
mv merged.meta default.meta
chmod 600 default.meta
8.
check_indexes_conf_properties
In stanza [redact]_initial_index, property coldPath should be $SPLUNK_DB/somedata_index/colddb, but is $SPLUNK_DB\somedata_index\colddb here. File: default/indexes.conf Line Number: 3Illegal property maxTotalDataSizeMB found in stanza somedata_index. Only properties [homePath, coldPath, thawedPath, frozenTimePeriodInSecs, disabled, datatype, repFactor] are allowed in default/indexes.conf. File: default/indexes.conf Line Number: 15
The file indexes.conf must have certain properties and must be written with forward slashes (/).
Example
Old:
[somedata_index]
coldPath = $SPLUNK_DB\somedata_index\colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB\somedata_index\db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB\somedata_index\thaweddb
New:
[somedata_index]
coldPath = $SPLUNK_DB/somedata_index /colddb
homePath = $SPLUNK_DB/ omedata_index /db
thawedPath = $SPLUNK_DB/somedata_index /thaweddb
9.
check_for_sched_saved_searches_latest_time
The saved search [redacted] doesn't contain dispatch.latest_time.It is better to add a dispatch.latest_time when specifying scheduled searches in Splunk Cloud. File: default/savedsearches.conf Line Number: 165
All saved searches should have a latest time. If a search is set to All-Time in the GUI. You can fix this by going to the $SPLUNK_HOME/etc/apps/appname/default/savedsearches.conf and adding “dispatch.latest_time = now()” (or whatever appropriate time frame) under the appropriate search stanzas.
10.
check_kos_are_accessible
In Splunk Cloud, customers have access to the sc_admin role instead of the admin role. Your customers will not be able to access knowledge objects where the only assigned role is admin. Please consider also including the sc_admin role for compatibility with Splunk Cloud. File: local.meta Line Number: 342
To fix this you must go into the appropriate file (default.meta in this case) and replace all instances of “admin” to “sc_admin” for the Splunk Cloud users.
Commands
cd $SPLUNK_HOME/etc/apps/appname/medata
vi default.meta
:%s/admin/sc_admin
:wq
11.
Check_that_extracted_splunk_app_does_not_contain_prohibited_directories_or_files
A prohibited file or directory was found in the extracted Splunk App: default/data/ui/views/._[redacted].xml
NOTE: On a MacOS machine, there are temporary files created that will be denied by the cloud vetting process. To avoid this, a variable to not copy the temporary files (dot_underscore) must be set followed by the appropriate flag in the tar command.
Commands
export COPYFILE_DISABLE=1
tar –exclude=”.*” -czvf appname.tar appname