Rainbow - Rainbow connection issue – Incident details

All systems operational

Rainbow connection issue

Resolved
Degraded performance
Started about 1 year agoLasted about 3 hours

Affected

Germany (DE)

Degraded performance from 7:00 AM to 10:20 AM

[DE] Rainbow Core Services

Degraded performance from 7:00 AM to 10:20 AM

Updates
  • Monitoring
    Monitoring

    We applied a fix and so far, all telemetry counters are back to normal.
    We do not see any errors anymore:

    • Hybrid telephony services are back to normal.
    • No more slowness on conversations usages

    We will continue to monitor the service to be sure the situation is completely resolved.

  • Resolved
    Resolved

    Rainbow is operating normally.

    During the incident, in order to limit the impact, rainbow operations team redirected a part of the openrainbow.com trafic to French datacenter. We noticed that some users were not able to connect any more, probably due to network firewall misconfiguration.

    It is mandatory to whitelist any Rainbow server IP as described below as in case of rare circumstances, we may need to redirect the Rainbow trafic to a failover datacenter during some upgrades or incidents.

    https://support.openrainbow.com/hc/fr/articles/115000301750-What-Are-Rainbow-Network-Requirements-EN-

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

    Currently, end users may experience following issues:

    • Loose of hybrid telephony features
    • Conversations are slow to load
  • Investigating
    Update

    We are currently investigating this incident.

    We have seen that impacts are on hybrid telephony. We are still currently in investigation.

  • Investigating
    Investigating

    Users may experience a slowness on connections/disconnections.

    We are currently investigating this incident.