Rainbow - Global outage on Rainbow authentication – Incident details

All systems operational

Global outage on Rainbow authentication

Resolved
Operational
Started 6 months agoLasted about 17 hours

Affected

Global (WW)

Major outage from 10:13 AM to 10:15 AM, Degraded performance from 10:13 AM to 10:15 AM, Major outage from 10:15 AM to 10:24 AM, Degraded performance from 10:15 AM to 10:24 AM, Major outage from 10:24 AM to 3:07 AM

[WW] Rainbow Hybrid PBX Telephony

Major outage from 10:13 AM to 3:07 AM

[WW] Rainbow Administration & Subscriptions

Degraded performance from 10:13 AM to 10:24 AM, Major outage from 10:24 AM to 3:07 AM

Europe & Middle East (EMEA)

Operational from 10:13 AM to 11:16 AM, Major outage from 11:16 AM to 3:07 AM

[EMEA] Rainbow Core Services

Operational from 10:13 AM to 11:16 AM, Major outage from 11:16 AM to 3:07 AM

[EMEA] Rainbow Hub Voice Services

Operational from 10:13 AM to 11:16 AM, Major outage from 11:16 AM to 3:07 AM

Updates
  • Resolved
    Resolved

    Our engineers have monitored the fix and confirmed Rainbow authentication errors have been resolved. All services are now operating normally at this time.

  • Monitoring
    Monitoring

    As you may have noticed, today we experienced a significant disruption affecting our database systems, which impacted the main collaboration services of Rainbow.

    While we are still performing a root cause analysis to understand the full extent of the incident, we can share the following preliminary details:

    - Service Restoration: Most of our services have now been restored. However, we are aware that some operations related to Rainbow (such as user creation or bubble creation) performed between 5 AM and 12 PM on Friday may have been lost.

    - Data Integrity: No user data or exchanged data has been lost.

    Our team is actively monitoring each error and working diligently to resolve any remaining issues. We will continue to provide updates as more information becomes available.

  • Update
    Update

    A fix has been identified, and according to our initial tests, we should be able to restore 95% of services within a few minutes.

  • Update
    Update
    We are continuing to work on a fix for this incident.
  • Update
    Update

    Our team is working around the clock to restore the service as soon as possible

  • Update
    Update

    We are continuing to work on a fix for this incident.
    We are still working on the missing components.

  • Update
    Update

    We are continuing to work on a fix for this incident.

  • Update
    Update

    We are continuing to work on a fix for this incident.
    We're still fixing some routing components to secure the service.

  • Update
    Update

    We are continuing to work on a fix for this incident.
    We're still working on the internal component faulty.

  • Update
    Update

    We are continuing to work on a fix for this incident.
    We identified missing components and work on it.

  • Update
    Update

    We are continuing to work on a fix for this incident.
    Some central components are now correctly restored. We're still working on the backbone to restore the service.

  • Update
    Update

    We are continuing to work on a fix for this incident.

    Still ongoing procedure.

  • Update
    Update

    We are continuing to work on a fix for this incident.
    the procedure is still ongoing and work as expected.

  • Update
    Update

    We are continuing to work on a fix for this incident.
    Restoring the service is still ongoing.

  • Update
    Update

    We are continuing to work on a fix for this incident.
    The restore procedure is still ongoing.

  • Update
    Update
    We are continuing to work on a fix for this incident.
  • Update
    Update

    We identified this incident. We're applying a procedure to restore the service.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    we identified a global issue when authentication is required (rainbow users, file transfer, pbx connections)

  • Investigating
    Investigating
    We are currently investigating this incident.