Rainbow - [EMEA | DE] [Major Issue] Bubbles & Conferencing issue – Incident details

All systems operational

[EMEA | DE] [Major Issue] Bubbles & Conferencing issue

Resolved
Major outage
Started over 4 years agoLasted about 13 hours

Affected

Europe & Middle East (EMEA)

Major outage from 7:57 AM to 8:45 PM

[EMEA] Rainbow Conferencing

Major outage from 7:57 AM to 8:45 PM

Germany (DE)

Major outage from 7:57 AM to 8:45 PM

[DE] Rainbow Conferencing

Major outage from 7:57 AM to 8:45 PM

Updates
  • Resolved
    Resolved

    The issue has been resolved. All systems are functional again.

    The Root Cause Analysis is available on the "Help Center".

  • Update
    Update

    We implemented a more exhaustive fix and currently monitoring the result. Bubbles creation is operational again and login through Android devices is restored.

  • Update
    Update

    While we were implementing a contingency plan to prevent outages and consolidate the stability of the Rainbow infrastructure, a critical problem in our Latin America infrastructure has arisen, because a third-party service provider suffered a major outage due to cooling system failure. This problem had a snowball effect that was impacting users around the world. Rainbow connections, bubbles and conferences were affected. Most services were restored at 13:00 CET for users in EMEA, and NAR & APAC regions are being updated. Our operation team is working with the local IaaS provider to restore services for CALA users. Updates have been provided here on status.openrainbow.com page. Our Operations and R&D teams are fully focused to fix this inconvenience as soon as possible. A detailed RCA will follow soon in our Help Center "here".

  • Update
    Update

    Existing bubbles are fully functional but creation of new ones are still causing issues. Android clients still have troubles connecting.

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result. Bubbles are operational again.

  • Update
    Update

    We are continuing to work on a fix for this incident. Rainbow Bubbles are still not operational.

  • Identified
    Identified

    A possible root-cause has been identified and we're working on a fix for this incident.

  • Investigating
    Investigating

    We have observed some production systems anomalies and Rainbow Operations team is currently investigating this incident.