The failing hosts and network has been identified and isolated. The failover has been performed and service restored.

Please find here after the root cause analysis.

On 2022-09-20 at 13:37 CEDT, we experienced a major network failure due to OVHCloud, our infrastructure provider in France. In the context of continuous improvement, OVHcloud has migrated a peering configuration from a network equipment to another. During the maintenance window, a configuration change resulted in some external routes not being properly propagated, which resulted in links saturation. We noticed during a short period of time (14min) latencies and packet drops (degradation of service) externally to end users, but also internally between the different Rainbow servers.

Therefore, rainbow users may have experienced following issues:

• [EMEA] connect or use rainbow core services, participate to conferences
• [WW] Hybrid telephony services (control the PBX phones and make peer to peer calls)
• [WW] Use Rainbow hub cloud telephony services

While most of usages were restored at 14:15 CEDT, some rare users may have experienced issues up to 14:45 as some PBX agents took longer to reconnect than others Also, some servers showed an unwanted high error rate and were restarted manually.

We implemented a fix and currently monitoring the result.

We are continuing to work on a fix for this incident.

We see worldwide impacts on all services and core components because of an issue at our provider.

We are currently mitigating and restarting the services

Please find here after the root cause analysis.

On 2022-09-19 evening Rainbow desktop version 2.109.0 has been released to all users. This version should not have been released as some blocking issues where still under correction. One of those issues created a very high load on Rainbow servers, leading for all EMEA users, too difficulties to connect to rainbow.

The slowness was visible to users mainly in the morning between 8:00 and 9:20 CEDT.

As soon as identified, the desktop client has been updated to previous version: 2.107.0.

Please be sure we will review our staging procedure to avoid those mistakes. Also, a bottleneck has been identified in some internet facing servers. Its correction is already under test and should be able to support 4 times more load should an equivalent load occur again.

We just resolved the issue!

We just resolved the issue!

We just resolved the issue!

We just resolved the issue!