Rainbow - Global outage on Rainbow authentication – Incident details

All systems operational

Global outage on Rainbow authentication

Resolved
Operational
Started 4 months agoLasted about 17 hours

Affected

Global (WW)

Major outage from 10:13 AM to 10:15 AM, Degraded performance from 10:13 AM to 10:15 AM, Major outage from 10:15 AM to 10:24 AM, Degraded performance from 10:15 AM to 10:24 AM, Major outage from 10:24 AM to 3:07 AM

[WW] Rainbow Hybrid PBX Telephony

Major outage from 10:13 AM to 3:07 AM

[WW] Rainbow Administration & Subscriptions

Degraded performance from 10:13 AM to 10:24 AM, Major outage from 10:24 AM to 3:07 AM

Europe & Middle East (EMEA)

Operational from 10:13 AM to 11:16 AM, Major outage from 11:16 AM to 3:07 AM

[EMEA] Rainbow Core Services

Operational from 10:13 AM to 11:16 AM, Major outage from 11:16 AM to 3:07 AM

[EMEA] Rainbow Hub Voice Services

Operational from 10:13 AM to 11:16 AM, Major outage from 11:16 AM to 3:07 AM

Updates
  • Resolved
    Resolved

    Our engineers have monitored the fix and confirmed Rainbow authentication errors have been resolved. All services are now operating normally at this time.

  • Monitoring
    Monitoring

    As you may have noticed, today we experienced a significant disruption affecting our database systems, which impacted the main collaboration services of Rainbow.

    While we are still performing a root cause analysis to understand the full extent of the incident, we can share the following preliminary details:

    - Service Restoration: Most of our services have now been restored. However, we are aware that some operations related to Rainbow (such as user creation or bubble creation) performed between 5 AM and 12 PM on Friday may have been lost.

    - Data Integrity: No user data or exchanged data has been lost.

    Our team is actively monitoring each error and working diligently to resolve any remaining issues. We will continue to provide updates as more information becomes available.

  • Identified
    Update

    A fix has been identified, and according to our initial tests, we should be able to restore 95% of services within a few minutes.

  • Identified
    Update
    We are continuing to work on a fix for this incident.
  • Identified
    Update

    Our team is working around the clock to restore the service as soon as possible

  • Identified
    Update

    We are continuing to work on a fix for this incident.
    We are still working on the missing components.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Update

    We are continuing to work on a fix for this incident.
    We're still fixing some routing components to secure the service.

  • Identified
    Update

    We are continuing to work on a fix for this incident.
    We're still working on the internal component faulty.

  • Identified
    Update

    We are continuing to work on a fix for this incident.
    We identified missing components and work on it.

  • Identified
    Update

    We are continuing to work on a fix for this incident.
    Some central components are now correctly restored. We're still working on the backbone to restore the service.

  • Identified
    Update

    We are continuing to work on a fix for this incident.

    Still ongoing procedure.

  • Identified
    Update

    We are continuing to work on a fix for this incident.
    the procedure is still ongoing and work as expected.

  • Identified
    Update

    We are continuing to work on a fix for this incident.
    Restoring the service is still ongoing.

  • Identified
    Update

    We are continuing to work on a fix for this incident.
    The restore procedure is still ongoing.

  • Identified
    Update
    We are continuing to work on a fix for this incident.
  • Identified
    Update

    We identified this incident. We're applying a procedure to restore the service.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    we identified a global issue when authentication is required (rainbow users, file transfer, pbx connections)

  • Investigating
    Investigating
    We are currently investigating this incident.