All systems operational

[EMEA | DE] [Major Issue] Bubbles & Conferencing issue

Resolved
Major outage
Started over 3 years ago Lasted about 13 hours

Affected

Europe & Middle East (EMEA)
[EMEA] Rainbow Conferencing
Germany (DE)
[DE] Rainbow Conferencing
Updates
  • Resolved
    Resolved

    The issue has been resolved. All systems are functional again.

    The Root Cause Analysis is available on the "Help Center".

  • Monitoring
    Update

    We implemented a more exhaustive fix and currently monitoring the result. Bubbles creation is operational again and login through Android devices is restored.

  • Monitoring
    Update

    While we were implementing a contingency plan to prevent outages and consolidate the stability of the Rainbow infrastructure, a critical problem in our Latin America infrastructure has arisen, because a third-party service provider suffered a major outage due to cooling system failure. This problem had a snowball effect that was impacting users around the world. Rainbow connections, bubbles and conferences were affected. Most services were restored at 13:00 CET for users in EMEA, and NAR & APAC regions are being updated. Our operation team is working with the local IaaS provider to restore services for CALA users. Updates have been provided here on status.openrainbow.com page. Our Operations and R&D teams are fully focused to fix this inconvenience as soon as possible. A detailed RCA will follow soon in our Help Center "here".

  • Monitoring
    Update

    Existing bubbles are fully functional but creation of new ones are still causing issues. Android clients still have troubles connecting.

  • Monitoring
    Monitoring

    We implemented a fix and currently monitoring the result. Bubbles are operational again.

  • Identified
    Update

    We are continuing to work on a fix for this incident. Rainbow Bubbles are still not operational.

  • Identified
    Identified

    A possible root-cause has been identified and we're working on a fix for this incident.

  • Investigating
    Investigating

    We have observed some production systems anomalies and Rainbow Operations team is currently investigating this incident.