[APAC] Rainbow Media Relays experiencing partial outage

Show current status

Resolved
Rainbow connection issue

Started
August 23, 2023 at 7:00:00 AM
Status
Resolved after about 3 hours

Impact

Degraded performance
Affected
Germany (DE)
[DE] Rainbow Core Services
  • Monitoring
    Monitoring

    We applied a fix and so far, all telemetry counters are back to normal.
    We do not see any errors anymore:

    • Hybrid telephony services are back to normal.
    • No more slowness on conversations usages

    We will continue to monitor the service to be sure the situation is completely resolved.

  • Resolved
    Resolved

    Rainbow is operating normally.

    During the incident, in order to limit the impact, rainbow operations team redirected a part of the openrainbow.com trafic to French datacenter. We noticed that some users were not able to connect any more, probably due to network firewall misconfiguration.

    It is mandatory to whitelist any Rainbow server IP as described below as in case of rare circumstances, we may need to redirect the Rainbow trafic to a failover datacenter during some upgrades or incidents.

    https://support.openrainbow.com/hc/fr/articles/115000301750-What-Are-Rainbow-Network-Requirements-EN-

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

    Currently, end users may experience following issues:

    • Loose of hybrid telephony features
    • Conversations are slow to load
  • Investigating
    Update

    We are currently investigating this incident.

    We have seen that impacts are on hybrid telephony. We are still currently in investigation.

  • Investigating
    Investigating

    Users may experience a slowness on connections/disconnections.

    We are currently investigating this incident.

See how we figure out uptime