Using Splunk Streamstats to Calculate Alert Volume (2024)

By Josh Neubecker|Published On: November 10th, 2020|

Dynamic thresholding using standard deviation is a common method we used to detect anomalies in Splunk correlation searches. However, one of the pitfalls with this method is the difficulty in tuning these searches. This is where the wonderful streamstats command comes to the rescue.

This Splunk tutorial will cover why tuning standard deviation searches is different from using a static threshold, how to use streamstats, and how we can use streamstats to get immediate feedback on alert volume.

Tuning Using Streamstats

1. Understanding the problem

With a static threshold search that runs over 60 minutes, calculating alert volume over 30 days is as simple as running the count by 60 minutes over 30 days. This is different with a dynamic threshold.

Typically, a standard deviation search will calculate a threshold based on the last 7 to 30 days to compare against the last hour of data. Running the same search to see approximately how many notables would be generated in 30 days will calculate the threshold differently than when it runs as a correlation search.

When running a correlation search, the threshold is based on historical data. Using the same search to calculate the alert volume for the whole 30 days the threshold will be based on historical, current, and future data for any given hour but the last.

This is where we can use streamstats to calculate the threshold based on the last 30 days for any given hour.

Still confusing? Let’s take a look at a few examples.

2. What does streamstats even do?

To understand how we can do this, we need to understand how streamstats works. In my experience, streamstats is the most confusing of the stats commands. I find it’s easier to show than explain. Let’s start with a basic example using data from the makeresults command and work our way up.

Example 1: streamstats without options

Copy to Clipboard

Using Splunk Streamstats to Calculate Alert Volume (1)

The streamstats command will run statistics as events come in. In this case, counting how many times each color appears and generating an incremental count for our testing.

Example 2: streamstats with a window

Copy to Clipboard

Using Splunk Streamstats to Calculate Alert Volume (2)

With a window, streamstats will calculate statistics based on the number of events specified. In this case, streamstats looks at the current event and the previous. This causes the count by color to be 1 for each event because the previous event is always a different color. A common expectation with streamstats is that the window by default would be separate for each color. To do this, see our next example.

Example 3: streamstats with a windows and global=false

Copy to Clipboard

Using Splunk Streamstats to Calculate Alert Volume (3)

When global=false a separate window is kept for each color. This is the behavior we need for testing alert volume.

3. How can we use streamstats to help us?

Let’s take a simplified standard deviation search for finding an anomalous amount of failed logins.

Copy to Clipboard

Running this search over 7 days will count the number of failed authentications by src for each hour. Then it will calculate an upper bound based on the average and standard deviation of the counts for each hour by src. Finally, it will only show events where the failure count for the last hour was above the upper bound.

Removing the time constraint would show anomalies for the entire time frame, but the results will be different then when the search runs every hour. To get a more accurate representation, we need to use streamstats to look at the previous 7 days for each individual hour.

Copy to Clipboard

Changing eventstats to streamstats and specifying a window of 168 (168 hours in a week since our time span is 1 hour) with global=false will calculate the threshold for each event based on the last 7 days of counts. Constraining the time to the past 7 days in the where clause and running the search over 14 days will ensure there’s a full history for each event. To exclude the current event from the threshold simply set current=false if desired.

Bonus Example: Creating an alert volume test for ESCU SMB Traffic Spike

Below shows the original search–taken from Splunk’s Enterprise Security Content Update app:

Copy to Clipboard

  • Change stats to streamstats window=168 (for 7 days of history) and global=false
  • In this search, the calculations are done on (maxtime, “-70m@m”) so set current=false
  • Remove `max(eval(if(_time >= relative_time(maxtime, “-70m@m”), count, null))) as count`. We want to keep the original count from each event
  • Add the time constraint `_time>relative_time(now(), “-7d”)` and run over 14 days

Putting all that together, here is the search:

Copy to Clipboard

Using this, we can change the threshold, filter noise, and immediately see what our changes will do.

Conclusion

Using this method, you can immediately see how many alerts would be generated from a standard deviation search. This allows for instant feedback to tune out sources, users, or failure reasons. Hopefully, this post helped you have a better understanding of how you can apply streamstats when creating and tuning alerts as well as other applications.

Share with your network!

About Hurricane Labs

Hurricane Labs is a dynamic Managed Services Provider that unlocks the potential of Splunk and security for diverse enterprises across the United States. With a dedicated, Splunk-focused team and an emphasis on humanity and collaboration, we provide the skills, resources, and results to help make our customers’ lives easier.

For more information, visitwww.hurricanelabs.comand follow us on Twitter@hurricanelabs.

Using Splunk Streamstats to Calculate Alert Volume (4)

Using Splunk Streamstats to Calculate Alert Volume (2024)

FAQs

What is the difference between stats eventstats and streamstats in splunk? ›

Eventstats calculates a statistical result same as stats command only difference is it does not create statistical results, it aggregates them to the original raw data. Streamstats command uses events before the current event to compute the aggregate statistics that are applied to each event.

How do streamstats work in Splunk? ›

Streamstats builds upon the basics of the stats command but it provides a way for statistics to be generated as each event is seen. This can be very useful for things like running totals or looking for averages as data is coming into the result set.

How to check alerts in Splunk? ›

You can view alerts from charts or in the Infrastructure Navigator by browsing related detectors.
  1. Open the Detector menu by clicking the bell icon at the top right corner of a chart.
  2. Select an active alert or hover over a detector and select View Active Alerts to see all active alerts for that detector.
May 28, 2024

What is the time range in Splunk alert? ›

The default time range is Last 24 hours. You can restrict the search to one of the preset time ranges, or use a custom time range.

What are the 4 types of searches in Splunk by performance? ›

How search types affect Splunk Enterprise performance
Search typeRef. indexer throughputPerformance impact
DenseUp to 50,000 matching events per second.CPU-bound
SparseUp to 5,000 matching events per second.CPU-bound
Super-sparseUp to 2 seconds per index bucket.I/O bound
RareFrom 10 to 50 index buckets per second.I/O bound

What is the difference between Tstats and stats command in Splunk? ›

tstats is faster than stats since tstats only looks at the indexed metadata (the . tsidx files in the buckets on the indexers) whereas stats is working off the data (in this case the raw events) before that command. Since tstats can only look at the indexed metadata it can only search fields that are in the metadata.

What is StreamStats used for? ›

StreamStats provides estimates of various streamflow statistics for user-selected sites by solving equations that were developed through a process known as regionalization.

What are streaming commands in Splunk? ›

A streaming command applies a transformation to each event returned by a search. For example, the rex command is streaming because it extracts and adds fields to events at search time.

How to improve Splunk performance? ›

Improve your searches
  1. Select an index in the first line of your search. ...
  2. Use the TERM directive. ...
  3. Use the tstats command. ...
  4. Avoid using table commands in the middle of searches and instead, place them at the end. ...
  5. Test your search string performance.
Apr 16, 2024

What are the two valid types of alerts in Splunk? ›

There are two alert types, scheduled and real-time.

How do I triage alerts in Splunk? ›

Triage incidents using incident review in Splunk Mission Control
  1. Review incidents.
  2. Sort incidents in the incident review table.
  3. Customize the incident review table.
  4. Perform bulk actions on incidents. Edit multiple incidents. ...
  5. Apply filters and save filtered views for incidents. Manage filtered views.
  6. Triage incidents.
Oct 18, 2023

Can Splunk generate alerts? ›

Alerts can trigger frequently because of similar search results or scheduling. If you have scheduled searches that run frequently and you do not want to be notified each time results generate, set the throttling controls to suppress the alert for a longer time period.

What is the lifespan of Splunk alerts? ›

Records of triggered alerts are available for twenty-four hours by default. You can configure this expiration time on a per-alert basis. For example, you can arrange to have the triggered alert records for an alert have a lifespan of seven days instead of twenty-four hours.

What is the alert limit for Splunk? ›

How many recipients can get an email alert through Splunk? What is the limit? In default it is 100(Number of recipients). maximum we can increase till 10000.

What is the alert rule in Splunk? ›

Alert rules use settings you specify for built-in alert conditions to define thresholds that trigger alerts. When a detector determines that the conditions for a rule are met, it triggers an alert, creates an event, and sends notifications (if specified).

What is the difference between events and statistics in Splunk? ›

The difference is that with the eventstats command aggregation results are added inline to each event and added only if the aggregation is pertinent to that event. let me know if this helps ! stats - Calculates aggregate statistics over the results set, such as average, count, and sum.

What is the use of Eventstats in Splunk? ›

The SPL2 eventstats command generates summary statistics from fields in your events and saves those statistics into a new field. The eventstats command places the generated statistics in new field that is added to the original raw events.

What is the difference between stats and transaction commands in Splunk? ›

Both the stats command and the transaction command are similar in that they enable you to aggregate individual events together based on field values. The stats command is meant to calculate statistics on events grouped by one or more fields and discard the events (unless you are using eventstats or streamstats).

What does stats mean in Splunk? ›

The SPL2 stats command calculates aggregate statistics, such as average, count, and sum, over the incoming search results set. This is similar to SQL aggregation. If the stats command is used without a BY clause, only one row is returned, which is the aggregation over the entire incoming result set.

Top Articles
Latest Posts
Article information

Author: Dean Jakubowski Ret

Last Updated:

Views: 6322

Rating: 5 / 5 (50 voted)

Reviews: 89% of readers found this page helpful

Author information

Name: Dean Jakubowski Ret

Birthday: 1996-05-10

Address: Apt. 425 4346 Santiago Islands, Shariside, AK 38830-1874

Phone: +96313309894162

Job: Legacy Sales Designer

Hobby: Baseball, Wood carving, Candle making, Jigsaw puzzles, Lacemaking, Parkour, Drawing

Introduction: My name is Dean Jakubowski Ret, I am a enthusiastic, friendly, homely, handsome, zealous, brainy, elegant person who loves writing and wants to share my knowledge and understanding with you.