Rainbow - Intermittent issues with call queues (mitigated) – Incident details

All systems operational

Intermittent issues with call queues (mitigated)

Resolved
Operational
Started 25 days agoLasted 4 days

Affected

Europe & Middle East (EMEA)

Degraded performance from 7:00 AM to 9:51 PM, Operational from 9:51 PM to 11:38 AM

[EMEA] Rainbow Hub Voice Services

Degraded performance from 7:00 AM to 9:51 PM, Operational from 9:51 PM to 11:38 AM

Updates
  • Resolved
    Resolved

    The observations made on Monday morning confirm the stability of the queuing solution.

    The incident is now considered closed.

    The root cause has been identified as a race condition related to the download and caching mechanisms of audio files configured for queues, which has been fixed.

  • Update
    Update

    Our monitoring today has confirmed that the queuing solution is now stable.

    We will definitely close the incident after the next business day.

  • Monitoring
    Monitoring

    The fix has been implemented, we we’ll keep monitoring the production on Friday morning

  • Update
    Update

    Tests are still ongoing, and the update still planned tonight

  • Identified
    Identified

    ALE is aware of random call queue instability.

    The team is actively working on it.

    A fix has been identified and is being tested.

    The patch is planned to be implemented on production tonight.