Intermittent issues with call queues (mitigated)

Resolved

Operational

Started 2 months agoLasted 4 days

Affected

Europe & Middle East (EMEA)

[EMEA] Rainbow Hub Voice Services

Updates

Resolved
March 03, 2025 at 11:38:46 AM
Resolved
March 03, 2025 at 11:38:46 AM
The observations made on Monday morning confirm the stability of the queuing solution.
The incident is now considered closed.
The root cause has been identified as a race condition related to the download and caching mechanisms of audio files configured for queues, which has been fixed.
Update
February 28, 2025 at 4:33:52 PM
Update
February 28, 2025 at 4:33:52 PM
Our monitoring today has confirmed that the queuing solution is now stable.
We will definitely close the incident after the next business day.
Monitoring
February 27, 2025 at 9:51:00 PM
Monitoring
February 27, 2025 at 9:51:00 PM
The fix has been implemented, we we’ll keep monitoring the production on Friday morning
Update
February 27, 2025 at 3:16:08 PM
Update
February 27, 2025 at 3:16:08 PM
Tests are still ongoing, and the update still planned tonight
Identified
February 27, 2025 at 7:00:00 AM
Identified
February 27, 2025 at 7:00:00 AM
ALE is aware of random call queue instability.
The team is actively working on it.
A fix has been identified and is being tested.
The patch is planned to be implemented on production tonight.

Rainbow - Intermittent issues with call queues (mitigated) – Incident details