Configuring this:
- when some monitor fails once, do nothing
- when it fails twice, send e-mail
- when it fails four times, send sms, send e-mail to additional four e-mail addresses
- when it fails five times, do a call using voip
For some services, it is common, that they fail sometimes (network traffic, ...). So I'm only interested, if the test fails five or more times.
F.e. I don't care, if load is temporarily 20. I do care, when it's more than certain threshold for 20 minutes.
Most monitoring systems (f.e. nagios, mon) can be configured to do this.
Now you can do all of this, it's not easy to implement, but you can do. Yo need to setup (for following this example), three alerts associated to a module. You setup a minimum of 2 and a max of 2, and associated response to send an email.
You setup another alert, with a minimum of 3 and a max of 3, with a response (a new one defined by you) that execute an script that sends an SMS and four emails to different address.
You setup the last alert, with a minimum of 4 and a max of 4, with a response (a new one defined by you) that execute an script to make a voIP call.
Is not simple, but it could be done. Really I don't know how to make it more easy to implement. Ideas be welcome :-)