Rainbow - [WW] Rainbow / Hub Softphony — Intermittent Connectivity Issues – Incident details

All systems operational

[WW] Rainbow / Hub Softphony — Intermittent Connectivity Issues

Resolved
Operational
Started about 13 hours agoLasted about 10 hours

Affected

Global (WW)

Major outage from 6:28 AM to 8:36 AM, Operational from 8:36 AM to 8:37 AM, Major outage from 8:37 AM to 12:54 PM, Partial outage from 12:54 PM to 1:27 PM, Degraded performance from 1:27 PM to 2:55 PM, Operational from 2:55 PM to 3:58 PM

[WW] Rainbow Hybrid PBX Telephony

Major outage from 6:28 AM to 8:36 AM, Operational from 8:36 AM to 8:37 AM, Major outage from 8:37 AM to 12:54 PM, Partial outage from 12:54 PM to 1:27 PM, Degraded performance from 1:27 PM to 2:55 PM, Operational from 2:55 PM to 3:58 PM

[WW] Rainbow Administration & Subscriptions

Major outage from 6:28 AM to 8:36 AM, Operational from 8:36 AM to 8:37 AM, Major outage from 8:37 AM to 12:54 PM, Partial outage from 12:54 PM to 1:27 PM, Degraded performance from 1:27 PM to 2:55 PM, Operational from 2:55 PM to 3:58 PM

Europe & Middle East (EMEA)

Major outage from 6:28 AM to 8:36 AM, Operational from 8:36 AM to 8:37 AM, Major outage from 8:37 AM to 12:54 PM, Partial outage from 12:54 PM to 1:27 PM, Degraded performance from 1:27 PM to 2:55 PM, Operational from 2:55 PM to 3:58 PM

[EMEA] Rainbow Core Services

Major outage from 6:28 AM to 8:36 AM, Operational from 8:36 AM to 8:37 AM, Major outage from 8:37 AM to 12:54 PM, Partial outage from 12:54 PM to 1:27 PM, Degraded performance from 1:27 PM to 2:55 PM, Operational from 2:55 PM to 3:58 PM

[EMEA] Rainbow Hub Voice Services

Major outage from 6:28 AM to 8:36 AM, Operational from 8:36 AM to 8:37 AM, Major outage from 8:37 AM to 12:54 PM, Partial outage from 12:54 PM to 1:27 PM, Degraded performance from 1:27 PM to 2:55 PM, Operational from 2:55 PM to 3:58 PM

Updates
  • Resolved
    Resolved

    The Rainbow service remains operational. OVH is still experiencing network issues in the Roubaix datacenter, which temporarily reduces our high‑availability capacity.
    Despite this, the platform is functioning normally, and we are maintaining reinforced monitoring to ensure service continuity.

    Our teams are closely following the situation with our provider and will take any additional measures needed.
    Thank you for your understanding while we worked through this incident.

  • Update
    Update

    Monitoring activities are still ongoing, and current indicators remain positive. Our IaaS provider has also made progress on resolving the underlying network issues. We will continue to observe the platform closely and provide further updates if necessary.

  • Update
    Update

    Services have been restored, and we continue to monitor the platform closely to ensure everything remains stable.
    If you experience any issue with Rainbow, a simple log out and log back in should help.

  • Monitoring
    Monitoring

    Services have been restored, and the vast majority of data is now available to users.
    Our teams are monitoring the platform closely to ensure full stability.

  • Update
    Update

    We continue to observe some instability on our infrastructure, but our teams have identified the contributing network issues and are actively working with our provider to address them.

    As part of our stabilisation efforts, several servers are being redirected to an alternate site to ensure a more reliable environment.
    These actions are already helping to improve the situation, and we remain fully mobilized to restore full service as quickly as possible. Further updates will follow as progress continues.

  • Update
    Update

    Telephony services are starting to recover, and PBX systems are reconnecting. Some users may still experience slow response times, but overall availability continues to improve. Our teams remain fully engaged and are closely monitoring the situation to support the ongoing recovery.

  • Update
    Update

    Most of our services are coming back online, and availability continues to improve, even though some users may still experience limited access. Our teams remain fully committed and are making steady progress toward full recovery.

  • Update
    Update

    Services are starting to be partially restored, although not all users may be able to access them yet.
    Our teams continue to work on completing the recovery.

  • Update
    Update

    To stabilise the service, a portion of user traffic has been temporarily reduced while our teams work on redistributing the load across our systems. This may prevent access to the platform for some users.
    We are actively working to restore normal access as quickly as possible.

  • Update
    Update

    Our teams continue to work actively on restoring the service. While the situation remains complex and no definitive recovery path has been confirmed yet, all necessary actions are being taken to move forward.

  • Update
    Update

    Despite no significant change for users at this time, our teams continue to work steadily on the resolution. Based on current progress, we expect full recovery before the end of the day.
    We will continue to keep you informed and share more details as soon as we make further progress.

  • Update
    Update

    We are performing additional actions on our infrastructure, but the service has not yet returned to normal. Our teams are continuing to work through the situation, and we will share further details as soon as new progress is made.

  • Update
    Update

    We are still completing the restart process, and some services may take a bit more time to fully stabilize. Our teams remain actively engaged, and we will provide a further update as soon as the next milestone is reached.

  • Update
    Update

    The system restart is still in progress. We will provide another update as soon as the operation is completed.

  • Update
    Update

    The system restart is still ongoing and is expected to take approximately one hour. We will provide another update as soon as the process is completed.

  • Update
    Update

    As part of the ongoing incident resolution, a full system restart will be performed. This will cause active sessions to be disconnected. This action is necessary to restore the service and stabilize the platform.
    Our teams will be closely monitoring the situation throughout this operation, and we expect this step to accelerate the return to normal service.

  • Update
    Update

    We are still experiencing degraded performance, but we are seeing ongoing progress as our teams actively manage elevated traffic levels as part of the mitigation efforts.
    We currently estimate full recovery within the next several hours and are working to shorten this timeframe as improvements continue.

  • Update
    Update

    We are continuing to make progress in restoring the service, although elevated load is currently slowing the process. Our teams remain fully mobilized, and we will provide further updates as the situation evolves.

  • Update
    Update

    Our teams are actively working on restoring the service. Services are partially recovered, although we continue to observe latency. We will share further updates as progress is made.

  • Identified
    Identified

    We’re facing a major incident caused by unexpected database latency. Our engineers are investigating and working to restore normal performance. We sincerely apologize for the inconvenience and will keep you updated.

  • Investigating
    Investigating
    We are currently investigating this incident.