Optimizing the Update VM Performance Query with Splunk

If you have integrated with Splunk to retrieve guest OS performance data, when Embotics® vCommander® refreshes VM performance for a machine (either manually for a single VM, or automatically with the nightly performance update), vCommander generates a query for the VM and sends it to Splunk to perform. This query can take a long time to complete, especially when every VM in your environment is being queried. To reduce the time taken and reduce the load on your Splunk server when performance data is being requested, you can configure Splunk to run these queries for all the hosts in your environment and store the results for vCommander to fetch later.

About performance sample size

Before configuring the saved searches in Splunk, you need to understand how vCommander treats the performance data it retrieves. For each performance counter (for example, Percent CPU), vCommander retrieves a daily sample from Splunk and detects how many days of performance data are available; the number of days vCommander retrieves is known as the sample size. vCommander uses both the sample size and the number of available samples to determine whether this performance data can be used to drive recommendations. For example, if a machine is only active on weekdays and you would like to make sure that a week’s data is present before applying it to recommendations, configure the sample size to be 7 and the minimum number of days in the sample to be 5.

You will need the sample size when configuring the searches in Splunk. The example queries in this article use the numbers given above: sample size = 7 days, and minimum number of samples = 5.

Configure Accelerated Search in Splunk

By making use of the Report Acceleration Summaries feature in Splunk, you can predefine a search that is run once for each performance counter that vCommander needs to refresh. You need to make sure that report acceleration has been enabled for each query, and that the query has been scheduled to run once a day (since vCommander summarizes the data daily, it is not necessary for Splunk to perform the query multiple times in a day; just make sure that the time the query is run is before the nightly performance job in vCommander).

NotePencil-smallThese instructions assume that you have already created and configured the data inputs for the hosts you need performance data for.
Percent CPU Performance Counter

On the Searches and Reports page in Splunk, create a new search with the following settings. See Customizing the Splunk Accelerated Search below if you need to specify different values.

Destination app

Search

Search name

vCommander-windows-cpu

Search

PercentProcessorTime=* Name=_Total latest=@d earliest=@d-7d | bucket _time span=5m | stats avg(PercentProcessorTime) as avgBucketValue, min(_time) as minBucketTime, max(_time) as maxBucketTime by host, _time | bucket _time span=1d | stats avg(avgBucketValue) as AVERAGE, max(avgBucketValue) as PEAK, min(minBucketTime) as minDayTime, max(maxBucketTime) as maxDayTimeUnadjusted by host, _time | eval maxDayTime = maxDayTimeUnadjusted + 300

Change the value of the earliest parameter to match your sample size. For example, to sample a month's data, change the earliest value to: earliest=@d-30d

Accelerate this search

Enabled

Summary range

Must be greater than the sample size you have chosen

Schedule this search

Enabled

Available Bytes Performance Counter

On the Searches and Reports page in Splunk, create a new search with the following settings. See Customizing the Splunk Accelerated Search below if you need to specify different values.

Destination app

Search

Search name

vCommander-windows-available-bytes

Search

AvailableBytes=* latest=@d earliest=@d-7d | bucket _time span=5m | stats avg(AvailableBytes) as avgBucketValue, min(_time) as minBucketTime, max(_time) as maxBucketTime by host, _time | bucket _time span=1d | stats min(avgBucketValue) as PEAK, min(minBucketTime) as minDayTime, max(maxBucketTime) as maxDayTimeUnadjusted by host, _time | eval maxDayTime = maxDayTimeUnadjusted + 300

Change the value of the earliest parameter to match your sample size. For example, to sample a month's data, change the earliest value to: earliest=@d-30d

Accelerate this search

Enabled

Summary range

Must be greater than the sample size you have chosen

Schedule this search

Enabled

Pages Output per Second Performance Counter

On the Searches and Reports page in Splunk, create a new search with the following settings. See Customizing the Splunk Accelerated Search below if you need to specify different values.

Destination app

Search

Search name

vCommander-windows-pages-output-per-sec

Search

PagesOutputPersec=* latest=@d earliest=@d-7d | bucket _time span=5m | stats avg(PagesOutputPersec) as avgBucketValue, min(_time) as minBucketTime, max(_time) as maxBucketTime by host, _time | bucket _time span=1d | stats avg(avgBucketValue) as AVERAGE, max(avgBucketValue) as PEAK, min(minBucketTime) as minDayTime, max(maxBucketTime) as maxDayTimeUnadjusted by host, _time | eval maxDayTime = maxDayTimeUnadjusted + 300

Change the value of the earliest parameter to match your sample size. For example, to sample a month's data, change the earliest value to: earliest=@d-30d

Accelerate this search

Enabled

Summary range

Must be greater than the sample size you have chosen

Schedule this search

Enabled

Run each search in Splunk first

Important: Make sure you run each saved search at least once in Splunk to generate the index. Otherwise, the index must be generated when vCommander executes the search for the first time, resulting in a very slow performance update.

Configure vCommander to use Splunk Accelerated Search 

Access through:

Configuration menu > System Configuration > Integration tab

Available to:

vCommander Role of Superuser

To configure vCommander to make use of the saved searches to update performance data:

1.On the Integration page, under Splunk Server, click Edit.

2.On the Splunk Server dialog, enable Accelerate Searches.

3.Click Test.

vCommander queries Splunk to determine whether everything is set up correctly.

4.Once the test has succeeded, click OK to save the configuration.

NotePencil-smallIf Splunk has been misconfigured and Accelerate Searches is enabled, vCommander will not be able to retrieve performance data from Splunk until the Splunk configuration has been corrected.

Customizing the Splunk Accelerated Search

Using Different Values for Destination App or Search Name

vCommander expects the Splunk values for Destination App and Search Name to match those in the tables above. If you cannot use the expected values (for example, if you need to point to a differently named app, or if you have a policy for search names), you can customize these values.

1.Create a splunk.properties file in the <vcommander_install_dir>/tomcat/common/classes directory.

2.Use the standard property file format, one property per line:

property1=value1

property2=value2

...

3.Add the following properties:

Property Name

Description

Default Value

splunk.properties.application

The name of the application where the searches have been created.

search

splunk.performance.windows.percent_cpu

The name of the saved search for reading the percent CPU performance counter.

vCommander-windows-cpu

splunk.performance.windows.available_bytes

The name of the saved search for reading the available bytes performance counter.

vCommander-windows-available-bytes

splunk.performance.windows.pages_out_per_sec

The name of the saved search for reading the pages output per second performance counter.

vCommander-windows-pages-out-per-sec

Customizing the Search Query

VMware takes the average value of every counter in a five-minute sampling period and uses that value for its calculations. Performance queries used by vCommander are likewise broken into five-minute intervals to match the way VMware retrieves performance data. vCommander identifies the peak value and the average value for the day; these values are used to drive rightsizing recommendations.

In order for vCommander to use both VM and Guest OS performance data to drive rightsizing recommendations, the two counters need to be comparable. Keep this in mind if you want to customize the search query.