Failed to send a notification through sms (SCOM 2007 R2)

SCOM notification for high priority and critical severity events was set to alert through SMS and email channels. This worked well, but occasionally, SCOM would generate an alert about not being able to send a notification through SMS gateway while email worked all the time.

SCOM sends 2 notifications about the SMS-ing issue:

1. Failed to send notification using server/device

Notification subsystem failed to send notification using device/server 'Standard  9600 bps Modem' over 'SMS' protocol to 'phone_number'.

2. Failed to send notification

Notification subsystem failed to send notification over 'SMS' protocol to 'phone_number'.

I tried adding the prefix “+ country code” as advised by Scott in How to configure SCOM 2007 to send SMS messages, but it did not help.

Then I checked the modem for speed configuration and status, and found no issues with it. Here is a useful document with loads of commands for configuring and checking modems:

GPRS AT Commands

Some examples:

AT+CPAS
Description: Returns the activity status of the mobile equipment. (p. 17)

AT+CSQ
Description: This command determines the received signal strength indication (<rssi>) and the channel bit error rate (<ber>) with or without a SIM card inserted. (p. 31)

AT+CBST?
Description: This command applies to both outgoing and incoming data calls, but in a different way. For an outgoing call, the two parameters (e.g., <speed> and <ce>) apply; whereas, for an incoming call, only the <ce> parameter applies. (p. 93) [check the speed configuration]

Since the (network) modem looked okay and the parallel email communication was working without issues, we concluded that there may be intermittent issues with the mobile network. So we decided to filter out the SMS related issues from the ‘Failed to send notification’ alert.

There are 2 alert generating NT event log monitoring rules:

1. Failed to send notification alerting rule

This rule creates alert every time notification subsystem fails to send notification using all configured devices/servers

Event log: Operations Monitor
Event Source: Health Service Modules
Parameter 1: $Target/ManagementGroup/Name$
Event ID: 31505

2. Failed to send through device alerting rule

This rule creates alert every time notification subsystem fails to send notification through certain device/server

Event log: Operations Monitor
Event Source: Health Service Modules
Parameter 1: $Target/ManagementGroup/Name$
Event ID: 31503

As these cannot be modified, they need to be disabled through an override. Then 2 new rules need to be created and configured for additional filtering.

Failed SMS notifications can be filtered out by using the parameter 5 that sets the word ‘SMS’ in the notification description in the alerts from both alerts. You can find the parameter in the rules’ Properties > Configuration > Response (view) > Alert Description.

The parameters can also be found under the events’ tab ‘Details’, as well as using log parser as explained by Marcus in Adjusting “failed to send notification using server/device”.

Once the original rules have been disabled, create 2 new rules and save them in your custom MP. Copy the configuration from the original rules:

Rule category: Alert
Rule target: Notification server

Under Configuration > Data sources > Expression insert a new line, set the Parameter Name to Parameter 5, Operator to Does not equal, and Value to SMS.

Copy the Alert suppression settings from Configuration > Responses > Alerting > Alert suppression. Modify priority and severity if necessary.