Rainbow - [EMEA] Rainbow Hub: Intermittent issue with telephony – Incident details

Since our last communication, our teams have continued their thorough investigations and corrective actions. We have validated several hypotheses regarding the origin of the instabilities and identified additional necessary adjustments. Thanks to the efforts made so far, we have seen a significant improvement, and the services are now fully operational again. 

At the same time, we are strengthening our infrastructure to prevent any recurrence of incidents, particularly by stabilizing the high traffic flows observed since January 6th. We have also intensified our proactive monitoring to detect and address any residual issues in real-time. 

We remain fully committed until we can confirm that the actions taken in recent days have completely resolved the situation. 

We sincerely thank you for your patience and trust. 

On Tuesday, January 7th, starting at 10:00 a.m., we observed an unusual traffic pattern on Rainbow Hub within the EMEA region. This anomaly led to an unexpected server overload, causing service degradation and global instability for some customers. Our teams promptly began analyzing and addressing the incident, and managed restoring services by 12:20 p.m. 

However, similar issues reoccurred in the early afternoon, leading to similar service disruptions until stability was reestablished for all services around 5:30 p.m, then again on Wednesday 8th. 

Mitigation measures have been implemented, and our teams have enhanced the monitoring capabilities of the system. We remain fully committed to identifying and resolving the root cause of the incident to ensure a permanent solution. 

We will provide further communication once the final root cause analysis is completed. 

Update on the progress of handling Rainbow Hub incidents 

Our operations teams remain fully mobilized to deepen investigations into the causes of the instabilities encountered throughout the day. We are beginning to identify probable leads that could be the source of these incidents. 

At this stage, it is difficult to provide a single cause, as it is very likely that these problems are the result of a combination of several factors. This situation has been exacerbated by a significant increase in traffic on the platform since yesterday. 

We have already made adjustments to the infrastructure to manage this unusual traffic spike. At the same time, we continue our efforts to resolve the side effects contributing to these incidents. 

Our teams will remain engaged until services are restored to 100%. At this stage, services are operational except for the call queues. We will keep you informed of the progress of actions in the coming hours. 

We implemented a fix and are currently monitoring the result. All incoming outgoing calls are working well as well as standard group calls, according to our metrics and tests.

We are still facing Call queues incident, followed in a specific case. Please be sure the team is doing it's maximum to restore the service as soon as possible.

Our engineers have identified the issue affecting incoming calls and are working toward a fix. We will provide another update as soon as more information becomes available.