Alerts are not being created for huge number of events even though the event rules are being applied.
Steps to reproduce
- Navigate to the list of events with no alerts.
- In the events processing notes you can see event rules are being applied.
- Reprocess the same event for same message key, event gets processed but you do not see the alert being created/updated.
This is because relevant alert is in flapping state. Alerts won't get attached to the events if they are in flapping state.
You can find the relevant alert by navigating to the list of alerts in [em_alert] table and find the alerts with same message key. If you find the alert with same message key in Flapping state, you might need to avoid flapping state.
In order to test it, you can change the one of the alert state to open and reprocess the events. Now the Alert will get attached to the event.
You should play with the following properties to avoid flapping state.