Rainbow - [Middle-East] Audio & Video Degraded Quality โ€“ Incident details

All systems operational

[Middle-East] Audio & Video Degraded Quality

Resolved
Operational
Started about 1 month agoLasted about 1 month

Affected

Middle East (ME)

Degraded performance from 5:30 AM to 10:06 AM, Operational from 10:06 AM to 3:39 PM

[Middle-East] Rainbow Conferencing

Degraded performance from 5:30 AM to 10:06 AM, Operational from 10:06 AM to 3:39 PM

[Middle-East] Rainbow Media Relays

Degraded performance from 5:30 AM to 10:06 AM, Operational from 10:06 AM to 3:39 PM

Updates
  • Resolved
    Resolved

    We understand the inconvenience this incident may have caused and thank you for your patience.

    Our teams have worked on our infrastructure to address the issues experienced by customers, and a fix was successfully deployed on May 6th. The service should now be back to normal.

    To ensure optimal performance, we recommend verifying your firewall configuration to allow access to the full Rainbow infrastructure. Please review and apply the required network rules as described in the following documentation:
    ๐Ÿ‘‰ https://help.openrainbow.com/hc/en-us/articles/23942019777170-Check-Rainbow-Network-Requirements

    Failure to update firewall rules may result in connectivity issues or service disruption.

    If you continue to experience any degradation, please open a ticket with our Customer Care team so we can assist you as quickly as possible.

    We will continue to monitor the situation closely.

  • Update
    Update

    Update โ€“ Revised Availability Timeline and Traffic Routing

    As part of the ongoing regional infrastructure recovery, following a drone strike that impacted critical infrastructure in the region, negotiations with our IaaS provider to finalize and prepare the new production environment in Dubai and Doha are taking longer than initially anticipated.

    We recognize that this incident has been ongoing since April 1st and has exceeded initial expectations.

    As of today, we estimate that May 20 represents the next key milestone for service restoration activities, corresponding to an approximate twoโ€‘week delay compared to our previous update.

    Current status

    • The new regional infrastructure is still in preparation.

    • All media traffic is now strictly routed to our European infrastructure.

    • The incident status remains Partially Degraded Performance.

    Customer impact

    Customers located in the Middle East and adjacent countries may continue to experience:

    • Increased latency

    • Degraded audio and video quality

    • Intermittent call setup or media issues

    Required customer action

    Due to the enforced redirection of traffic to Europe, customers must ensure that their network and firewall configurations allow access to the full Rainbow infrastructure.

    Please review and apply the required network rules as described in the following documentation:
    ๐Ÿ‘‰ https://help.openrainbow.com/hc/en-us/articles/23942019777170-Check-Rainbow-Network-Requirements

    Failure to update firewall rules may result in connectivity issues or service disruption.

    Next steps

    • Infrastructure preparation and alignment with the IaaS provider are ongoing.

    • We will communicate as soon as the new infrastructure is made available to us and ready for final preparation and migration.

    • In all cases, a further status update will be provided no later than mid-May, even if no major change has occurred.


    We appreciate your patience and will continue to communicate transparently until full service restoration is completed.

  • Update
    Update

    Update โ€“ Regional Infrastructure Rebuild in Progress

    Following our previously communicated recovery plan, we are proceeding with the rebuild of a new production data center in Dubai to restore stable regional service.

    We remain within the recovery timeline already shared. The new Dubai-based environment is currently being prepared and is targeted to be available within the next 10 to 15 days.

    Impact scope and observations:

    • The issue is primarily expected to impact customers in Middle East countries.

    • However, we have also received limited reports of service impact from some customers in India and adjacent countries in Africa.

    Important notes and next steps:

    • Until the migration is completed, media traffic continues to be routed through other regions. As a result, some customers have reported call failures, inability to establish calls, or other malfunctioning features, in addition to degraded media quality or increased latency.

    • The new data center deployment is actively in progress.

    • Service activation will require network and firewall configuration updates on the customer side to allow traffic to the new infrastructure.

    • Detailed technical information (including IP ranges, endpoints, and configuration requirements) will be shared as soon as it becomes available, to allow customers to prepare in advance.

    We will continue to provide regular progress updates and will communicate clearly once the new environment is ready and traffic migration is scheduled.

    Thank you for your patience and understanding.

  • Update
    Update

    Update โ€“ Infrastructure Recovery Plan

    Due to the ongoing unavailability of our primary infrastructure in the AWS Bahrain region datacenter, and constraints related to the current regional situation, we are unable to restore service from this data center in the short term.

    As a result, we have initiated the deployment of a new production instance in an alternative data center outside the Middle East region to restore long-term service stability.

    Current status and next steps:

    • The new environment is being identified, then built and validated.

    • This process is expected to take approximately one month.

    • In the meantime, media traffic will continue to be routed through other regions to maintain service availability, which may result in temporary impacts on media quality or latency for some users.

    We will continue to share progress updates and will communicate once the new instance is ready and traffic migration is completed.

    Thank you for your patience and understanding.

  • Identified
    Identified

    Following an incident affecting the AWS data center in Bahrain on April 1st (see AWS Health Dashboard: https://health.aws.amazon.com/health/status), media traffic was temporarily redirected to APAC and European regions to ensure service continuity.
    As a result, some users may experience a temporary decrease in media quality during this period.

    A full recovery is planned within one week.