Splunk | Steps to create an alert in splunk - Support and Troubleshooting

Splunk | Steps to create an alert in splunk Issue This article walks you through the process of creating alerts in splunk and be able to send out a notification email to a distribution list or a specific set of users.

Release Production instances.

Resolution Below are the steps to create an alert in splunk:

1) Navigate to Splunk using the following link:

https://search.servicenow.net/

2) In the Search page, enter the search string you are looking for example:

sourcetype=appnode_localhost_log instance= "OutofMemoryError: Metaspace"

and select the three different modes available: Fast Mode, Smart Mode and Verbose Mode and then hit the Green magnifying icon to search.

3) To create an alert select Save As and then select Alert as follows:

4) You will be presented with the following screen where you can enter additional details:

Title: Name of the alert

Description: Additional details of the alert

Permissions: Options are Private and Shared in App. Private indicates that only you have permission to view and edit the alert, it is not visible to other users. Shared in App indicates that the alert is available to other users in the searching and reporting app. The alert is visible to other users in this context. Depending on their permissions, other users can edit the dashboard.

Alert Type: Options available are Scheduled and Real-time and below are the details:

To create a CRON job, you will have to set the alert type to Scheduled and select Run on CRON Schedule and you will have to select the Time range( which is the time range you want the search query to look at everytime the CRON jobs executed)

The above image, shows an example where the CRON job executes every 15 minutes (value set in the Cron Expression field) and looks for the last 15 minutes of data (value set in the Time Range field).

Cron Expression: Customize alert scheduling using a time range and cron expression.

A cron expression is a data string of five fields separated by spaces. From left to right, the five cron fields have the following chronological value ranges:

Minute: 0-59 Hour: 0-23 Day of the month: 1-31 Month: 1-12 Day of the week: 0-6 (where 0 = Sunday) In cron expressions with an interval of /N , all values in the specified range that are intervals of N are used. If a number in the range is outside of the interval N , the value resets to 0.

Here are some example cron expressions.

*/5 * * * * Every 5 minutes. */30 * * * * Every 30 minutes. 0 */12 * * * Every 12 hours, on the hour. */20 * * * 1-5 Every 20 minutes, Monday through Friday. 0 9 1-7 * * The first 7 days of every month at 9 AM.

Trigger Conditions: The condition that triggered the alert.

Trigger alert when: The alert is triggered when a criteria is met and you can also add additional criteria where the number of results are greater or lesser than a particular value.

Trigger: Do you want this alert to trigger once or whenever the criteria is met.

Throttle: Use throttling to suppress alert triggering for a specific time period. Alerts can trigger frequently because of similar search results or scheduling. If you have scheduled searches that run frequently and you do not want to be notified each time results generate, set the throttling controls to suppress the alert for a longer time period. For real-time searches, if you configure an alert so that it triggers once when a specific triggering condition is met, you do not need to configure throttling. If the alert triggers for each result, you might need to configure throttling to suppress additional alerts.

When you select the Throttle checkbox, you indicate that alert notifications should be suppressed for the indicated period of time and (if you've chosen to trigger for each result) any specified field values. The default time period is 60 seconds, but you can change that to be any length of time you want. Both of these settings will help keep your e-mail inbox from being overwhelmed with alert notification e-mails.

Trigger Actions: This is the section that determines how the users are notified of this alert:

The fields in this section are self-explanatory, but note that we can use tokens in the e-mail subject and body to add specificity to the alert. For example, the subject and body fields are prefilled with text that uses the $name$ token, which will be replaced by the name of the search when the alert is sent.

You can even include information from the trigger search results themselves by using the $result. fieldname $ token and replacing fieldname with the name of the field to include --for instance, the field name that contains the user name of the user.

Once you save the alert you will be presented with the following where you can make changes to the permissions:

Related Links 1) Examples of Cron job alerts:

https://docs.splunk.com/Documentation/Splunk/7.2.6/Alert/CronExpressions

2) Best practices for alert scheduling:

https://docs.splunk.com/Documentation/Splunk/7.2.6/Alert/AlertSchedulingBestPractices