Rainbow - Rainbow connection slowness – Incident details

All systems operational

Rainbow connection slowness

Resolved
Degraded performance
Started almost 3 years agoLasted about 1 hour

Affected

Global (WW)

Degraded performance from 6:00 AM to 7:25 AM

[WW] Rainbow Hybrid PBX Telephony

Degraded performance from 6:00 AM to 7:25 AM

[WW] Rainbow Administration & Subscriptions

Degraded performance from 6:00 AM to 7:25 AM

Europe & Middle East (EMEA)

Degraded performance from 6:00 AM to 7:25 AM

[EMEA] Rainbow Core Services

Degraded performance from 6:00 AM to 7:25 AM

[EMEA] Rainbow Hub Voice Services

Degraded performance from 6:00 AM to 7:25 AM

Updates
  • Resolved
    Resolved

    Please find here after the root cause analysis.

    On 2022-09-19 evening Rainbow desktop version 2.109.0 has been released to all users. This version should not have been released as some blocking issues where still under correction. One of those issues created a very high load on Rainbow servers, leading for all EMEA users, too difficulties to connect to rainbow.

    The slowness was visible to users mainly in the morning between 8:00 and 9:20 CEDT.

    As soon as identified, the desktop client has been updated to previous version: 2.107.0.

    Please be sure we will review our staging procedure to avoid those mistakes. Also, a bottleneck has been identified in some internet facing servers. Its correction is already under test and should be able to support 4 times more load should an equivalent load occur again.

  • Investigating
    Investigating

    We are currently investigating this incident.