Rainbow - [EU] Rainbow Hub interruption of service – Incident details

Post‑Incident Review

Summary

A short service interruption occurred following a power loss affecting one server. The redundancy mechanism was automatically triggered, redirecting traffic to a secondary server.

Impact

Users experienced a service interruption for a few minutes. During this time, making or receiving new calls was not possible, while ongoing calls were not interrupted. No data loss occurred.

Root Cause

The primary cause of the incident was a power loss on the initial server. When failover occurred, the secondary server was unable to instantly synchronize all data due to abnormal behavior in the database system. This resulted in a short delay while the server scaled up and stabilized.

Resolution

The redundancy mechanism functioned as designed. Service was fully restored once the secondary server completed startup, data synchronization, and load handling.

Next Steps

• Improve capacity to ensure faster startup and data synchronization when taking full load during failover

• Optimize failover performance to better handle sudden data loads

We have corrected the recorded start time of the service interruption to reflect the exact moment the issue began. This adjustment ensures full accuracy in our incident timeline. Based on updated log analysis, we were able to more precisely determine when the incident started.

We now estimate the start of the incident at 11:53 AM CET.

The incident has been resolved and all services are operating normally.
The root cause analysis will be shared in a forthcoming post‑mortem.