First off, we are sorry for the temporary outage in sending SMS / Email today. Our purpose is to reliably deliver your messages in a timely manner and this issue impacted that goal. We will use this incident to grow and build an even stronger network for all.
We’ve provided details below about the nature of this incident and how we’ve resolved the underlying problem.
Some messages sent during this incident would show “0 messages sent” in the Omnilert Timeline. These messages were then sent upon the resolution of the problem.
Omnilert’s engineering team was in the process of upgrading part of the network as part of routine maintenance. No service impact was anticipated.
In normal cases, the Omnilert network’s built-in redundancy would pick up message processing during these routine events to prevent any service disruption.
In this instance, there was an unforeseen race condition that was causing the redundant servers from processing messages as intended. The result was messages not processing out of the messaging queue, causing the delay experienced today.
We have identified the root cause and are putting additional alarms in place on both our primary and secondary networks that will alert us if this happens again. This should prevent any chance of messages queuing for any significant amount of time should such a situation recur.