Rainbow - [EMEA] Rainbow Hub: Intermittent issue with telephony – Incident details

The planned fixes and implementation of new defense mechanisms have been successfully deployed last night. The situation is now confirmed stable and remains under continuous observation. All services are operational. Our teams are still working diligently to fully identify the underlying causes and plan any additional preventive actions to ensure the long-term stability of the solution. 

A new overload incident occurred at 9:30 AM, caused by an atypical use of our telephone queue management component. Despite the corrective actions previously implemented, this incident once again triggered a cascading effect on other elements of the infrastructure, causing degradation for some clients. 

We quickly took measures to mitigate the impact of this incident, and most services were restored by 9:45 AM, with full restoration of queue functionalities by 10:30 AM. 

The issues met since Tuesday are all originated from the same atypical pattern and affecting the same components. A new fix is currently being validated for deployment this evening. We appreciate your understanding and patience. 

We noticed some latencies on call queues which could lead to failed calls.

We implemented a fix.

Our team is now monitoring the system

We are continuing to work on a fix for the incident impacting call queues.