Duration: 20 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

Jun 2, 09:15:54 GMT+0
Investigating - Some customers have reported issues on Rainbow Hub. Our technical teams are analyzing the issue to identify the root cause and assess the full impact. We will provide an update shortly. .

Jun 2, 09:26:31 GMT+0
Identified - Our teams have identified the origin of the incident. Mitigation actions are in progress, including cleanup and restart of the impacted component. Service recovery is underway, and we will continue to keep you informed..

Jun 2, 09:32:52 GMT+0
Monitoring - **Service restoration in progress** Services are currently being restored. Please note that it may take some time before full service is fully restored for all customers. Our teams remain fully engaged to complete the recovery as quickly as possible..

Jun 2, 09:35:38 GMT+0
Resolved - This incident has been resolved. If you continue to experience issues, please restart your Rainbow phone or softphone application. Thank you for your patience and understanding throughout this incident..

Jun 3, 07:15:16 GMT+0
Postmortem - # Post-Incident Report Incident Window: June 2, 2026 - 11:09 – 11:30 CEST ### What happened? Yesterday, we experienced a temporary issue affecting one component of our Rainbow Hub infrastructure. A switchover to the backup system was automatically triggered, but it did not start as expected, which led to a short service disruption. ### Impact During this period, Rainbow Hub users in Europe were unable to make calls. ### Resolution Our teams acted immediately to stabilize the affected system and safely restore service. Full service was restored at 11:30 CEST. ### Root cause The issue was linked to an internal database system inconsistency that impacted service availability and prevented the failover mechanism from operating as expected. This type of inconsistency had been observed previously, but only under a very specific and rarely encountered combination of conditions. A correction addressing this class of issue was already in preparation and planned for production update. ### What we’re doing? The fix addressing this issue will be deployed on our European infrastructure on the evening of June 3rd, as part of a more general worldwide fix in all regions. While we have strong confidence this will prevent a recurrence, we are continuing to monitor closely. We sincerely apologize for the inconvenience caused and thank you for your patience and continued trust..

Duration: 2 hours and 29 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

Jun 1, 13:58:11 GMT+0
Investigating - We are currently investigating this incident..

Jun 1, 14:10:55 GMT+0
Investigating - We’re currently experiencing a loss of telephony for some customers, with intermittent disruptions affecting others. We understand the impact this may have and sincerely apologize for the inconvenience. Our teams are actively investigating and working to restore full service as quickly as possible..

Jun 1, 14:58:37 GMT+0
Identified - **Ongoing investigation** We are observing an intermittent network issue affecting both our data centers in Roubaix and Limburg. We are working closely with our IaaS provider to analyze the situation and identify the root cause. Our teams remain fully engaged to restore full stability as quickly as possible..

Jun 1, 15:30:12 GMT+0
Identified - Our teams remain fully mobilized and are continuing in-depth analysis with our IaaS provider. Work is progressing, and we are focused on restoring full service stability..

Jun 1, 15:42:41 GMT+0
Identified - The network has been stabilized, but the service is still impacted. Our teams are now focusing on the application layer and are fully engaged in restoring full service as quickly as possible..

Jun 1, 15:50:58 GMT+0
Identified - **Service restoration in progress** Services are currently being restored. Please note that it may take some time before full service is fully restored for all customers. Our teams remain fully engaged to complete the recovery as quickly as possible..

Jun 1, 15:57:38 GMT+0
Monitoring - **Service restored** Service has now been restored. Our teams are continuing to monitor the situation closely to ensure ongoing stability. If you continue to experience issues, please restart your Rainbow phone or softphone application. Thank you for your patience and understanding throughout this incident..

Jun 1, 14:28:55 GMT+0
Investigating - **Increase in impact** We are currently seeing an increase in the number of customers affected. All European users are impacted. We understand how critical this service is and sincerely apologize for the continued inconvenience. At this time, the root cause has not yet been identified, but our teams are actively investigating and working urgently to resolve the issue..

Jun 1, 16:27:05 GMT+0
Resolved - This incident has been resolved..

Jun 2, 08:28:53 GMT+0
Postmortem - ## Post-Incident Report **Incident Window:** June 1st, 2026 – 15:45 to 18:00 CEST ### Summary On June 1st, 2026, we experienced a network incident that led to **Rainbow Hub service disruption in Europe**, with up to **\~50% traffic loss**. During this period, some users may have experienced **intermittent connectivity issues**, up to **full service disruption** at peak impact. ### Root Cause The incident occurred in the context of our execution of the **Rainbow Infrastructure upgrade plan and the connection of a new data center.** An infrastructure-level network configuration inconsistency resulted in network instability and significant packet loss affecting Rainbow Hub components. ### Resolution Our teams identified the issue and worked in coordination with our infrastructure provider. The misconfigured routes were corrected and the network stabilized. **Packet loss was resolved by 17:20 CEST**, with **full service restoration completed at 18:00 CEST**. ### Preventive Actions To prevent recurrence, we are implementing the following improvements: * Strengthening **configuration validation processes**, including multi-level and cross-DC checks * Adjusting **change scheduling** to reduce user impact while ensuring adequate monitoring * Enhancing **monitoring and alerting**, particularly for routing anomalies, link flapping, and infrastructure changes * The ongoing **infrastructure upgrade plan** will furthermore result in better platform resilience to network instabilities.

Duration: 2 hours and 44 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

May 4, 10:08:19 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

May 4, 12:50:29 GMT+0
Resolved - The incident has been resolved and all services are operating normally. The root cause analysis will be shared in a forthcoming post‑mortem..

May 4, 13:00:11 GMT+0
Resolved - We have corrected the recorded start time of the service interruption to reflect the exact moment the issue began. This adjustment ensures full accuracy in our incident timeline. Based on updated log analysis, we were able to more precisely determine when the incident started. We now estimate the start of the incident at **11:53 AM CET**..

May 4, 09:53:00 GMT+0
Investigating - May 4, 2026 at 12:06 PM CET Some customers have reported an interruption of Rainbow Hub service. Our technical teams are analyzing the issue to identify the root cause and assess the full impact. We will provide an update shortly. .

May 5, 08:41:23 GMT+0
Postmortem - ## Post‑Incident Review **Incident Window:** May 4st, 2026 – 11:53 to 12:08 CEST ### Summary A short service interruption occurred following a **power loss affecting one server**. The redundancy mechanism was automatically triggered, redirecting traffic to a secondary server. ### Impact Users experienced a service interruption for a few minutes. During this time, making or receiving new calls was not possible, while ongoing calls were not interrupted. No data loss occurred. ### Root Cause The primary cause of the incident was a power loss on the initial server. When failover occurred, the secondary server was unable to instantly synchronize all data due to abnormal behavior in the database system. This resulted in a short delay while the server scaled up and stabilized. ### Resolution The redundancy mechanism functioned as designed. Service was fully restored once the secondary server completed startup, data synchronization, and load handling. ### Next Steps * Improve capacity to ensure faster startup and data synchronization when taking full load during failover * Optimize failover performance to better handle sudden traffic spikes.

Duration: 1 hour

Affected Components: [DE] Rainbow Core Services, [DE] Rainbow Media Relays, [DE] Rainbow Conferencing

Apr 15, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 15, 19:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. New load balancers will be added will be added with following IP Addresses: \- 141.95.121.152 .[openrainbow.com](http://openrainbow.com) \- 141.95.121.153 [rdd.openrainbow.com](http://rdd.openrainbow.com) \- 141.95.121.157 .[openrainbow.com](http://openrainbow.com) \- 141.95.121.158 [rdd.openrainbow.com](http://rdd.openrainbow.com) Please ensure they are trusted on your firewall. Current load balancers remain reachable: haproxy-external-lim3-prod-1 \- 54.37.95.41 .[openrainbow.com](http://openrainbow.com) \- 51.38.111.143 [agent.openrainbow.com](http://agent.openrainbow.com) \- 54.37.194.109 [rdd.openrainbow.com](http://rdd.openrainbow.com)haproxy-external-lim3-prod-2: \- 51.195.21.11 .[openrainbow.com](http://openrainbow.com) \- 51.195.55.209 [rdd.openrainbow.com](http://rdd.openrainbow.com).

Apr 15, 20:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: [APAC] Rainbow Media Relays, [APAC] Rainbow Core Services, [APAC] Rainbow Conferencing, [APAC] Rainbow Hub Voice Services

Apr 15, 14:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. A new load balancer will be added with following IP Addresses: \- 15.235.188.232 .[openrainbow.com](http://openrainbow.com) \- 15.235.188.233 [agent.openrainbow.com](http://agent.openrainbow.com) \- 15.235.188.234 [rdd.openrainbow.com](http://rdd.openrainbow.com) \- 2402:1f00:8008:8f00::1 .[openrainbow.com](http://openrainbow.com) \- 2402:1f00:8008:8f00::2 [agent.openrainbow.com](http://agent.openrainbow.com) \- 2402:1f00:8008:8f00::3 [rdd.openrainbow.com](http://rdd.openrainbow.com) Please ensure they are trusted on your firewall. Current load balancers remain reachable: haproxy-external-sgp1-prod-1 (Note: **decommissioning planned by end of may**): \- 139.99.122.102 .[openrainbow.com](http://openrainbow.com) \- 139.99.4.245 [agent.openrainbow.com](http://agent.openrainbow.com) \- 139.99.2.105 [rdd.openrainbow.com](http://rdd.openrainbow.com) \- 2402:1f00:8001:266::1 .[openrainbow.com](http://openrainbow.com) \- 2402:1f00:8001:266::2 [agent.openrainbow.com](http://agent.openrainbow.com) \- 2402:1f00:8001:266::3 [rdd.openrainbow.com](http://rdd.openrainbow.com) haproxy-external-sgp1-prod-2: \- 15.235.144.86 .[openrainbow.com](http://openrainbow.com) \- 139.99.4.244 [agent.openrainbow.com](http://agent.openrainbow.com) \- 139.99.80.249 [rdd.openrainbow.com](http://rdd.openrainbow.com) \- 2402:1f00:8001:1356::1 .[openrainbow.com](http://openrainbow.com) \- 2402:1f00:8001:1356::2 [agent.openrainbow.com](http://agent.openrainbow.com) \- 2402:1f00:8001:1356::3 [rdd.openrainbow.com](http://rdd.openrainbow.com).

Apr 15, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 15, 15:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: [ANZ] Rainbow Hub Voice Services, [ANZ] Rainbow Media Relays, [ANZ] Rainbow Conferencing, [ANZ] Rainbow Core Services

Apr 15, 12:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. A new load balancer will be added with following IP Addresses: \- 51.161.144.88 .openrainbow.com \- 51.161.144.89 [rdd.openrainbow.com](http://rdd.openrainbow.com) Please ensure they are trusted on your firewall. Current load balancers remain reachable: haproxy-external-syd-prod-1 (Note: **decommissioning planned by end of may**): \- 139.99.148.135 .openrainbow.com \- 139.99.161.35 [agent.openrainbow.com](http://agent.openrainbow.com) \- 139.99.177.193 [rdd.openrainbow.com](http://rdd.openrainbow.com) haproxy-external-syd2-prod-1: \- 139.99.212.212 \*.openrainbow.com \- 139.99.212.213 [agent.openrainbow.com](http://agent.openrainbow.com) \- 139.99.212.214 rdd.openrainbow.com.

Apr 15, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 15, 13:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours

Affected Components: [WW] Rainbow Hybrid PBX Telephony, [WW] Rainbow Administration & Subscriptions, , Global (WW) →

Apr 9, 19:00:00 GMT+0
Identified - As part of the preparation for a future infrastructure upgrade, our hosting provider **OVH** will apply a **network configuration update** on link aggregation components. This operation will be performed **interface by interface** to avoid any full service interruption. During the maintenance scheduled for **April 9, 2026, between 21:00 and 23:00 (CEST)**, the following **temporary impacts on Rainbow services** may be observed: * A **brief interruption of connections** * **Temporary slowdowns** on some services, including possible **audio and video quality degradation** in case of limited bandwidth * **Telephony services may be unavailable for several minutes** If telephony services are impacted, affected users will be informed via the message **“Telephone services currently unavailable”**, visible in their telephony settings. No long‑lasting disruption is expected, and services should return to normal once the operation is completed..

Apr 9, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 9, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 9, 19:00:00 GMT+0
Identified - OVH, our underlying IaaS provider, will conduct an exceptional infrastructure maintenance on **April 9, 2026 at 21:00 CEST**. This operation targets core platform components and may result in **temporary service degradation or brief interruptions** across our systems. The potential impact is **global (WW)**. We have implemented preventive safeguards to **minimize any operational impact** during the maintenance window. Our team will supervise the event end‑to‑end and respond immediately to any anomaly. Thank you for your understanding..

Duration: 2 hours and 57 minutes

Affected Components: [CALA] Rainbow Hub Voices Services

Apr 6, 11:59:00 GMT+0
Investigating - We are currently investigating this incident..

Apr 6, 13:47:39 GMT+0
Identified - We are continuing to work on a fix for this incident..

Apr 6, 14:25:32 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 6, 14:56:27 GMT+0
Resolved - This incident has been resolved..

Affected Components: [Middle-East] Rainbow Media Relays, [Middle-East] Rainbow Conferencing

Apr 1, 05:30:00 GMT+0
Identified - Following an incident affecting the **AWS data center in Bahrain** on April 1st (see AWS Health Dashboard: <https://health.aws.amazon.com/health/status>), media traffic was temporarily redirected to **APAC and European regions** to ensure service continuity. As a result, some users may experience a **temporary decrease in media quality** during this period. A full recovery is planned **within one week.**.

Apr 9, 07:13:47 GMT+0
Identified - Update – Infrastructure Recovery Plan Due to the ongoing unavailability of our primary infrastructure in the **AWS Bahrain region** datacenter, and constraints related to the current regional situation, we are unable to restore service from this data center in the short term. As a result, we have initiated the deployment of a **new production instance in an alternative data center outside the Middle East region** to restore long-term service stability. **Current status and next steps:** * The new environment is being identified, then built and validated. * This process is expected to take **approximately one month**. * In the meantime, media traffic will continue to be routed through other regions to maintain service availability, which may result in **temporary impacts on media quality or latency** for some users. We will continue to share progress updates and will communicate once the new instance is ready and traffic migration is completed. Thank you for your patience and understanding..

Apr 21, 08:31:39 GMT+0
Identified - **Update – Regional Infrastructure Rebuild in Progress** Following our previously communicated recovery plan, we are proceeding with the rebuild of a new production data center in **Dubai** to restore stable regional service. We remain within the recovery timeline already shared. The new Dubai-based environment is currently being prepared and is **targeted to be available within the next 10 to 15 days**. **Impact scope and observations:** * The issue is **primarily expected to impact customers in Middle East countries**. * However, we have also received **limited reports of service impact from some customers in India and adjacent countries** in Africa. **Important notes and next steps:** * Until the migration is completed, media traffic continues to be routed through other regions. As a result, **some customers have reported call failures, inability to establish calls, or other malfunctioning features**, in addition to degraded media quality or increased latency. * The new data center deployment is actively in progress. * Service activation will require **network and firewall configuration updates on the customer side** to allow traffic to the new infrastructure. * Detailed technical information (including IP ranges, endpoints, and configuration requirements) will be shared **as soon as it becomes available**, to allow customers to prepare in advance. We will continue to provide regular progress updates and will communicate clearly once the new environment is ready and traffic migration is scheduled. Thank you for your patience and understanding..

Apr 28, 09:01:08 GMT+0
Identified - ### Update – Revised Availability Timeline and Traffic Routing As part of the ongoing regional infrastructure recovery, following a drone strike that impacted critical infrastructure in the region, negotiations with our IaaS provider to finalize and prepare the new production environment in Dubai and Doha are taking longer than initially anticipated. We recognize that this incident has been ongoing since April 1st and has exceeded initial expectations. **As of today, we estimate that May 20 represents the next key milestone for service restoration activities, corresponding to an approximate two‑week delay compared to our previous update.** **Current status** * The new regional infrastructure is still in preparation. * **All media traffic is now strictly routed to our European infrastructure.** * The incident status remains **Partially Degraded Performance**. **Customer impact** Customers located in the Middle East and adjacent countries may continue to experience: * Increased latency * Degraded audio and video quality * Intermittent call setup or media issues **Required customer action** Due to the enforced redirection of traffic to Europe, **customers must ensure that their network and firewall configurations allow access to the full Rainbow infrastructure**. Please review and apply the required network rules as described in the following documentation: 👉 <https://help.openrainbow.com/hc/en-us/articles/23942019777170-Check-Rainbow-Network-Requirements> Failure to update firewall rules may result in connectivity issues or service disruption. **Next steps** * Infrastructure preparation and alignment with the IaaS provider are ongoing. * **We will communicate as soon as the new infrastructure is made available to us and ready for final preparation and migration.** * **In all cases, a further status update will be provided no later than mid-May**, even if no major change has occurred. We appreciate your patience and will continue to communicate transparently until full service restoration is completed..

May 13, 15:39:59 GMT+0
Resolved - We understand the inconvenience this incident may have caused and thank you for your patience. Our teams have worked on our infrastructure to address the issues experienced by customers, and a fix was successfully deployed on **May 6th**. The service should now be back to normal. To ensure optimal performance, we recommend verifying your firewall configuration to **allow access to the full Rainbow infrastructure**. Please review and apply the required network rules as described in the following documentation: 👉 <https://help.openrainbow.com/hc/en-us/articles/23942019777170-Check-Rainbow-Network-Requirements> Failure to update firewall rules may result in connectivity issues or service disruption. If you continue to experience any degradation, please open a ticket with our Customer Care team so we can assist you as quickly as possible. We will continue to monitor the situation closely..

Jun 16, 07:26:42 GMT+0
Identified - \[Middle-East\] Upcoming deployment of Dubai regional media infrastructure Following the media quality issues experienced in the Middle East region, Rainbow is planning the deployment of a new regional media infrastructure located in Dubai. This deployment is intended to improve regional media handling and provide better audio/video quality, lower latency, and improved conference media connectivity for customers located in the Middle East and adjacent regions. The new Dubai-based infrastructure will mainly concern media-related services, including: * Audio/video media traffic * Conference media traffic * TURN / media relay services No customer impact is expected during this deployment. No customer action is required at this stage. A detailed communication will be published later with: * The planned deployment date * The technical details to authorize the new infrastructure, including IP ranges and DNS information * The expected impact, if any, and any required customer / partner actions Customers and partners using restrictive firewall policies should be prepared to update their network rules once the technical details are shared. We will share more information as soon as the deployment schedule and technical details are confirmed..

Duration: 2 days, 4 hours and 32 minutes

Affected Components: [WW] Rainbow Hybrid PBX Telephony, , [APAC] Rainbow Media Relays, [EMEA] Rainbow Core Services, [APAC] Rainbow Core Services, [ANZ] Rainbow Hub Voice Services, [CALA] Rainbow Conferencing, [ANZ] Rainbow Media Relays, [CALA] Rainbow Core Services, [WW] Rainbow Administration & Subscriptions, , [DE] Rainbow Core Services, [NA] Rainbow Hub Voices Services, [NA] Rainbow Conferencing, [CALA] Rainbow Hub Voices Services, [NA] Rainbow Media Relays, [DE] Rainbow Media Relays, [CALA] Rainbow Media Relays, [EMEA] Rainbow Media Relays, , , [DE] Rainbow Conferencing, [APAC] Rainbow Conferencing, [EMEA] Rainbow Conferencing, [EMEA] Rainbow Hub Voice Services, [ANZ] Rainbow Conferencing, [APAC] Rainbow Hub Voice Services, [ANZ] Rainbow Core Services, [NA] Rainbow Core Services, North America (NA) → Caribbean & Latin America (CALA) → Australia-New Zealand (ANZ) → Asia-Pacific (APAC) →

Mar 31, 06:28:19 GMT+0
Investigating - We are currently investigating this incident..

Mar 31, 06:55:34 GMT+0
Identified - We’re facing a major incident caused by unexpected database latency. Our engineers are investigating and working to restore normal performance. We sincerely apologize for the inconvenience and will keep you updated..

Mar 31, 07:25:24 GMT+0
Identified - Our teams are actively working on restoring the service. Services are partially recovered, although we continue to observe latency. We will share further updates as progress is made..

Mar 31, 07:59:03 GMT+0
Identified - We are continuing to make progress in restoring the service, although elevated load is currently slowing the process. Our teams remain fully mobilized, and we will provide further updates as the situation evolves..

Mar 31, 08:32:59 GMT+0
Identified - We are still experiencing degraded performance, but we are seeing ongoing progress as our teams actively manage elevated traffic levels as part of the mitigation efforts. We currently estimate full recovery within the next several hours and are working to shorten this timeframe as improvements continue..

Mar 31, 08:37:39 GMT+0
Identified - As part of the ongoing incident resolution, a full system restart will be performed. This will cause active sessions to be disconnected. This action is necessary to restore the service and stabilize the platform. Our teams will be closely monitoring the situation throughout this operation, and we expect this step to accelerate the return to normal service..

Mar 31, 08:58:38 GMT+0
Identified - The system restart is still ongoing and is expected to take approximately one hour. We will provide another update as soon as the process is completed..

Mar 31, 09:32:03 GMT+0
Identified - The system restart is still in progress. We will provide another update as soon as the operation is completed..

Mar 31, 10:01:38 GMT+0
Identified - We are still completing the restart process, and some services may take a bit more time to fully stabilize. Our teams remain actively engaged, and we will provide a further update as soon as the next milestone is reached..

Mar 31, 10:26:20 GMT+0
Identified - We are performing additional actions on our infrastructure, but the service has not yet returned to normal. Our teams are continuing to work through the situation, and we will share further details as soon as new progress is made..

Mar 31, 11:00:59 GMT+0
Identified - Despite no significant change for users at this time, our teams continue to work steadily on the resolution. Based on current progress, we expect full recovery before the end of the day. We will continue to keep you informed and share more details as soon as we make further progress..

Mar 31, 11:34:07 GMT+0
Identified - Our teams continue to work actively on restoring the service. While the situation remains complex and no definitive recovery path has been confirmed yet, all necessary actions are being taken to move forward..

Mar 31, 12:07:10 GMT+0
Identified - To stabilise the service, a portion of user traffic has been temporarily reduced while our teams work on redistributing the load across our systems. This may prevent access to the platform for some users. We are actively working to restore normal access as quickly as possible..

Mar 31, 12:28:21 GMT+0
Identified - Services are starting to be partially restored, although not all users may be able to access them yet. Our teams continue to work on completing the recovery..

Mar 31, 12:54:31 GMT+0
Identified - Most of our services are coming back online, and availability continues to improve, even though some users may still experience limited access. Our teams remain fully committed and are making steady progress toward full recovery..

Mar 31, 13:27:03 GMT+0
Identified - Telephony services are starting to recover, and PBX systems are reconnecting. Some users may still experience slow response times, but overall availability continues to improve. Our teams remain fully engaged and are closely monitoring the situation to support the ongoing recovery..

Mar 31, 13:56:36 GMT+0
Identified - We continue to observe some instability on our infrastructure, but our teams have identified the contributing network issues and are actively working with our provider to address them. As part of our stabilisation efforts, several servers are being redirected to an alternate site to ensure a more reliable environment. These actions are already helping to improve the situation, and we remain fully mobilized to restore full service as quickly as possible. Further updates will follow as progress continues..

Mar 31, 14:31:20 GMT+0
Monitoring - Services have been restored, and the vast majority of data is now available to users. Our teams are monitoring the platform closely to ensure full stability..

Mar 31, 14:55:50 GMT+0
Monitoring - Services have been restored, and we continue to monitor the platform closely to ensure everything remains stable. If you experience any issue with Rainbow, a simple log out and log back in should help..

Mar 31, 15:31:12 GMT+0
Monitoring - Monitoring activities are still ongoing, and current indicators remain positive. Our IaaS provider has also made progress on resolving the underlying network issues. We will continue to observe the platform closely and provide further updates if necessary..

Apr 2, 11:00:01 GMT+0
Resolved - This incident has been resolved..

Mar 31, 15:58:57 GMT+0
Resolved - The Rainbow service remains operational. OVH is still experiencing network issues in the Roubaix datacenter, which temporarily reduces our high‑availability capacity. Despite this, the platform is functioning normally, and we are maintaining reinforced monitoring to ensure service continuity. Our teams are closely following the situation with our provider and will take any additional measures needed. Thank you for your understanding while we worked through this incident..

Apr 2, 11:00:00 GMT+0
Postmortem - **Post‑mortem Update** – Apr 02, 13:00 AM CEST Following the application of OVH’s fix yesterday at approximately 22:00 CEST, all data centers are now fully operational. All services have returned to normal and continue to be closely monitored. \=== Apr 01, 2026 10:07 AM **Incident Report: Root Cause Analysis – Rainbow Data Center** **1\. Executive Summary** Our Rainbow Data Center provider experienced significant service degradation impacting our Rainbow services in our different regions due to a network routing anomaly at our primary hosting provider. Incident was caused by the instability of IP Routing between the different Data Center virtual router resulted in an active/passive switch flapping scenario that ultimately isolated database nodes and interrupted client connections. All services have now been restored by routing traffic to an alternate data center, and we are currently in strict monitoring mode. **2\. Timeline of Events** * **31/03, 08:00:** Flapping issue escalated on OVH Data Center, leading to a split-brain condition on our database infrastructure. Service disruption became widespread. * **31/03, 08:00 – 10:00:** The team assessed all available options to recover service without resorting to a full platform restart, which was considered a last resort given the risk of a broader outage. Despite these efforts, the underlying network instability could not be resolved through targeted intervention alone. * **31/03, 11:00 - 11:30:** The engineering team attempted a first restoration. This was unsuccessful as the network continued to flap between DCs. * **31/03, 12:00:** A second restart was initiated, attempting to distribute the load across different data centers. * **31/03, 13:00:** Our teams executed a major restoration by isolating faulty datacenter network elements. * **31/03, 13:30-16:30:** Rainbow services were recovered and users were getting progressively access to their services in the different regions * **Current Status:** All IP addresses from Roubaix were successfully switched to the Gravelines data center, and services have been restored. We are now in a dedicated monitoring phase and working with OVH Team to ensure DC Roubaix stability. **3\. Root Cause Description** The service disruption was triggered by a malfunction in the network at our data center, which is operated by our infrastructure provider, OVH. We observed repeated VRRP flapping, preventing incoming traffic from reliably reaching the active destination server. This persistent routing instability ultimately disrupted database synchronization, resulting in a split-brain scenario in which database nodes operated independently without maintaining consensus. As the VRRP protocol is managed externally by the hosting provider, direct intervention at the switch level was not possible on our end, requiring the implementation of an architectural bypass to restore service. **4\. Corrective Actions and Next Steps** We are accelerating the expansion Rainbow Service with new IaaS provider to avoid full dependency on OVH Cloud network solution – A detailed timeline will be shared before end of next week **Invitation: Partner Webinar on Rainbow Solution Stability** To ensure full transparency and ongoing collaboration, we will invite our partner community to a dedicated technical webinar next week. Invitation to follow before end of this week..

Duration: 50 minutes

Affected Components: [WW] Rainbow Hybrid PBX Telephony, [WW] Rainbow Administration & Subscriptions, , [EMEA] Rainbow Hub Voice Services, Global (WW) →

Mar 16, 21:00:22 GMT+0
Identified - We are continuing to work on a fix for this incident..

Mar 16, 21:50:22 GMT+0
Resolved - This incident has been resolved..

Duration: 40 minutes

Affected Components: [WW] Rainbow Hybrid PBX Telephony, [EMEA] Rainbow Hub Voice Services, [WW] Rainbow Administration & Subscriptions, [DE] Rainbow Core Services, [EMEA] Rainbow Core Services, , , Global (WW) → Germany (DE) →

Feb 16, 09:11:00 GMT+0
Identified - We experienced some slowness on the European platform between 10:10 and 10:30 CET, which resulted in degraded service access in certain areas and on some features. This was due to partial congestion caused by an update rollout from our IaaS provider. Service is back to normal. We keep monitoring closely.

Feb 16, 09:51:15 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 25 minutes

Feb 9, 07:30:00 GMT+0
Investigating - We are currently investigating this incident..

Feb 9, 07:45:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Feb 9, 07:55:00 GMT+0
Resolved - This incident has been resolved..

Feb 10, 11:00:00 GMT+0
Postmortem - ### Root Cause Analysis – Service Degradation on February 9th Incident Window : 07:20–08:55 CET (end users impact between 08:30-08:55 CET) Services Impacted : Outbound and inbound call establishment for a subset of customers in France and Germany Current Status : Fully resolved ### Summary of the Incident On February 9th, a service degradation affected voice call processing on our platform. Some customers experienced failed call attempts for approximately 30 minutes. The issue has since been fully resolved, and the platform is operating normally. ### Customer Impact • Some customers experienced call setup failures for approximately 30 minutes. • No impact on other services or call legs already established. • No further issues have been detected since restoration. ### What Happened (Timeline) 07:20 CET – Automatic Failover • One of our proxy servers became unavailable. • The platform automatically switched over to the redundant proxy as expected. • At this stage, only a few calls (4–5) in ringing phase were lost. • Services were nominal after the switchover. 08:30 CET – Crash on the Redundant Proxy • The newly active proxy experienced a crash. • Service degradation became visible to some customers (call failures). • During restart, our team detected data inconsistencies in the proxy environment. This inconsistency prevented the automated switchover to the secondary instance. • Checks and clean-up of database were required before safely restarting the service to prevent further data corruption and repeat failure. 08:55 CET – Full Service Restoration • The proxy was restarted once data integrity was secured. • All services stabilized and no further issues were observed. ### Root Cause Preliminary analysis indicates that data corruption occurred during the initial switchover between proxies. This corruption prevented a clean restart of the redundant proxy and ultimately triggered the crash. The underlying cause appears linked to anomalies in the proxy cluster’s data synchronization process during the failover. ### Corrective Actions Taken • Ensured the integrity of all proxy data before restart • Restored the proxy and validated call flow end‑to‑end • Performed full health checks across the platform ### Preventive Actions (In Progress) We are implementing the following measures: 1\. Hardening of failover procedures to avoid data inconsistencies during switchover 2\. Improve continuous monitoring of proxy cluster data consistency to detect abnormal state earlier and isolate faulty instances.

Duration: 5 hours and 46 minutes

Affected Components: [DE] Rainbow Core Services, [CALA] Rainbow Core Services, [EMEA] Rainbow Core Services, [APAC] Rainbow Core Services, [ANZ] Rainbow Core Services, [NA] Rainbow Core Services

Feb 4, 13:50:49 GMT+0
Monitoring - **Service is partially restored.** All Rainbow Core features are available except call logs..

Feb 4, 14:33:41 GMT+0
Resolved - Services are restored, RCA will be shared soon..

Feb 4, 08:47:35 GMT+0
Investigating - We are currently investigating this incident..

Feb 4, 09:05:51 GMT+0
Identified - We are continuing to work on a fix for this incident..

Feb 4, 09:34:39 GMT+0
Identified - We are continuing to work on a fix for this incident..

Feb 4, 09:49:51 GMT+0
Identified - We identified peak traffic problem, currently rebalancing the traffic.

Feb 4, 11:32:09 GMT+0
Identified - The root cause has been identified, and recovery actions are currently in progress.Service restoration in underway, and systems are being progressively brought back online.

Feb 4, 12:08:29 GMT+0
Monitoring - Core services have been restored and are now under enhanced monitoring.Service restoration is being progressively completed across all user accounts. Our teams remain fully engaged to ensure complete recovery and platform stability..

Feb 4, 13:08:43 GMT+0
Monitoring - **Service restoration continues across remaining affected accounts.** Our teams are actively finalizing recovery for the remaining accounts and continue monitoring platform stability closely..

Feb 6, 11:00:00 GMT+0
Postmortem - Post-Incident Report Incident Window: Feb 4th 8:30 AM- 5:00 PM CET 1\. Incident Timeline & Recovery · 08:30 AM CET: Initial instabilities detected on European core infrastructure. · 09:45 AM CET: Global service disruption confirmed. · 10:30 AM CET: Traffic temporarily suspended by our ALE operations team to allow for a safe, synchronized server reinitialization. · 12:00 PM – 02:00 PM CET: Services progressively restored for most impacted users, with the exception of Call-Log service · 02:00 PM – 05:00 PM CET: Close Monitoring of the situation and stability of services · 05:00 PM CET: All services including Call-Log fully restored. 2\. Scope of Impact · Rainbow Collaboration: Chat, Conference, and Webinar services in Europe, Middle East and Africa. · Rainbow Hub: Softphony services in Europe, Middle East and Africa. · Rainbow Hybrid Telephony: Worldwide services (Click-To-Call, Softphone). 3\. Technical Summary The incident originated from a network maintenance operation performed overnight between February 3rd to Feb 4th on an XMPP server cluster node. Although the standard restoration was completed at 04:00 AM CET on February 4th, subsequent network instabilities at 08:30 AM CET caused a desynchronization within the cluster, which ultimately led to the incident. To resolve the resulting service outage, our teams performed a controlled restart of all affected systems. We temporarily blocked traffic to prevent the recovering cluster from being overwhelmed. We then allowed the servers to fully resynchronize before restoring normal service to users. 4\. Current Status As confirmed above, all services are now fully operational from 05:00 PM CET. 5\. Corrective actions Our in-depth analysis of this incident has led us to take additional actions to further strengthen the long-term stability of the platform in the coming weeks. · Adding communication nodes to create a more robust cluster architecture. This will further reduce the risk of synchronization failures during periods of unforeseen network instabilities. · Upgrading database nodes’ hardware. This will accelerate system recovery during incidents. We value your partnership and your patience. If you have immediate concerns or require further support, please contact our support team directly..

Duration: 7 hours and 43 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

Feb 4, 15:54:50 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Feb 4, 16:12:33 GMT+0
Resolved - This incident has been resolved..

Feb 4, 08:29:11 GMT+0
Investigating - We are currently investigating this incident. A few amount of user cannot make external calls.

Feb 4, 10:20:27 GMT+0
Identified - We are continuing to work on a fix for this incident..

Duration: 32 minutes

Jan 29, 12:58:52 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 29, 13:21:05 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 29, 13:24:44 GMT+0
Monitoring - Some customers still impacted, fix is ongoing.

Jan 29, 13:31:07 GMT+0
Resolved - This incident has been resolved..

Duration: 20 minutes

Affected Components: [WW] Rainbow Hybrid PBX Telephony

Jan 22, 21:20:00 GMT+0
Investigating - We are currently investigating this incident..

Jan 22, 21:40:00 GMT+0
Resolved - This incident has been resolved..

Jan 22, 22:09:48 GMT+0
Resolved - This incident has been resolved..

Duration: 8 minutes

Affected Components: [DE] Rainbow Core Services, [DE] Rainbow Conferencing, , Germany (DE) →

Jan 20, 20:44:00 GMT+0
Investigating - Several network loss on DC provider, causing some users reconnections.

Jan 20, 20:52:00 GMT+0
Resolved - This incident has been resolved. No more reconnection issues observed.

Duration: 9 hours and 3 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

Dec 24, 16:28:14 GMT+0
Postmortem - On December 24, 2025, between 05:30 and 14:00 CET, some Rainbow Hub companies experienced inbound and outbound call issues. The incident was caused by a temporary network disruption, which led to an inconsistent NAT port handling on an intermediate component of the Rainbow Hub infrastructure. This affected several trunks, while the majority of services remained operational. Corrective actions were applied, and the issue was fully resolved after a configuration update on the affected intermediate server. All services have since been fully restored, and no other Rainbow services were impacted..

Dec 24, 05:00:00 GMT+0
Investigating - We are currently investigating this incident..

Dec 24, 07:46:11 GMT+0
Resolved - This incident has been resolved..

Dec 24, 08:18:12 GMT+0
Investigating - We are currently investigating this incident..

Dec 24, 08:44:23 GMT+0
Investigating - We are continuing to work on a fix for this incident.

Dec 24, 09:28:26 GMT+0
Identified - Several fixes have been applied, we monitor the system.

Dec 24, 09:47:51 GMT+0
Monitoring - We are currently monitoring the result.

Dec 24, 10:50:32 GMT+0
Monitoring - Remaining cases are identified, we work on fixes.

Dec 24, 14:02:35 GMT+0
Resolved - This incident has been resolved. We'll provide RCA soon.

Duration: 2 hours and 18 minutes

Affected Components: [APAC] Rainbow Media Relays

Dec 15, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 15, 10:18:19 GMT+0
Completed - Maintenance has completed successfully..

Dec 15, 08:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. During maintenance operation, server turn-sgp1-prod-1 (Media relay) will be down, following IP addresses will be unreachable: \- 139.99.120.111 \- 2402:1f00:8001:6f::1 During maintenance, call will be established on closest Media relays servers (please ensure they are trusted in end customer firewall): \- BHR: 15.184.158.216 / 157.175.244.247 \- TOK: 169.56.36.231 / 2401:c900:1001:287::2.

Duration: 3 hours and 32 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

Nov 25, 18:50:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Nov 25, 19:13:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Nov 25, 15:43:00 GMT+0
Investigating - We are currently investigating this incident..

Nov 25, 19:15:00 GMT+0
Resolved - This incident has been resolved..

Duration: 7 minutes

Affected Components: [EMEA] Rainbow Core Services

Nov 16, 04:36:26 GMT+0
Identified - Some users may encounter difficulties receiving notification on their mobile device when idle, leading to incoming calls not present for instance. We are working on a fix. In a meantime, to not loose any incoming call, you may use or other desktop or web applications on a PC..

Nov 16, 04:43:52 GMT+0
Resolved - This incident has been resolved..

Affected Components: [APAC] Rainbow Media Relays, [APAC] Rainbow Conferencing, [APAC] Rainbow Core Services

Nov 13, 08:50:00 GMT+0
Resolved - This incident has been resolved..

Nov 13, 09:10:00 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour

Affected Components: [WW] Rainbow Hybrid PBX Telephony

Oct 27, 08:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. Step1: 9:00 am (France time), Rainbow team will drain following load balancers: haproxy-external-gra3-prod-3 (178.32.125.32, 178.32.125.56, 178.32.125.46 will be removed from DNS answer) haproxy-external-rbx8-prod-3 (178.32.126.147, 178.32.126.16, 178.32.126.196 will be removed from DNS answer) Step2: 10:00 am (France time), a gentle reconnection (based on PBX country) of OXE cloud-agent during non-working hours will be performed. At this stage, if connections are established on above load balancer, a reconnection will happen (3xx answer returned by load balancer) and PBXs connected upon to load balancer will initiate a new connection which establish on available load balancers. Step3: Rainbow team will monitor the remaining connections on haproxy-external-gra3-prod-3 and haproxy-external-rbx8-prod-3\. Once all connections are closed to null, Rainbow team will proceed for OS migration. A dedicated maintenance will be published in Rainbow status page..

Oct 27, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 27, 09:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours

Affected Components: [WW] Rainbow Hybrid PBX Telephony

Oct 20, 05:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. Step1: 7:00 am (France time), Rainbow team will proceed for maintenance on following load balancers. No impact for customers, they are drained and removed from DNS pool: haproxy-external-gra3-prod-2 178.32.125.107, 178.32.125.135, 178.32.125.149 haproxy-external-rbx8-prod-2 178.32.125.161, 178.32.125.168, 178.32.125.241 Step2: 9:00 pm (France time), load balancers back in service in DNS pool Caution: next maintenance will occur by October 27th at 9:00 am (France time), following load balancers will be drained and removed from DNS pool: haproxy-external-gra3-prod-3 178.32.125.32, 178.32.125.56, 178.32.125.46 haproxy-external-rbx8-prod-3 178.32.126.147, 178.32.126.16, 178.32.126.196 Key points: \- Ensure all load balancers' IP addresses are trusted in customers' firewall \- Ensure customers use FQDN [agent.openrainbow.com](http://agent.openrainbow.com) instead of IP Addresses \> Note: you may also use Rainbow Pilot: <https://pilot.openrainbow.com/networkRequirements> to obtain the Network requirements..

Oct 20, 05:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 20, 07:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 22 minutes

Affected Components: [WW] Rainbow Hybrid PBX Telephony

Oct 15, 22:42:00 GMT+0
Investigating - We are currently investigating this incident..

Oct 15, 22:48:41 GMT+0
Identified - We are continuing to work on a fix for this incident..

Oct 15, 23:03:41 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours

Affected Components: [WW] Rainbow Hybrid PBX Telephony

Oct 15, 19:00:00 GMT+0
Identified - Step1: 9:00 pm (France time), Rainbow team will add 2 new load balancers in France, please ensure they are trusted in customer's firewall: \- 178.32.126.211, 178.32.126.97 (\*.[openrainbow.com](http://openrainbow.com)) \- 178.32.126.251, 178.32.127.14 ([agent.openrainbow.com](http://agent.openrainbow.com)) \- 178.32.126.70, 178.32.127.143 ([rdd.openrainbow.com](http://rdd.openrainbow.com)) \> Note: you may also use Rainbow Pilot: <https://pilot.openrainbow.com/networkRequirements> to obtain the Network requirements. Step2: 9:30 pm (France time), following load balancers will be drained: haproxy-external-gra3-prod-2 (178.32.125.107, 178.32.125.135, 178.32.125.149 will be removed from DNS answer) haproxy-external-rbx8-prod-2 (178.32.125.161, 178.32.125.168, 178.32.125.241 will be removed from DNS answer) Step3: 10:00 pm (France time), a gentle reconnection (based on PBX country) of OXE cloud-agent during non-working hours will be performed At this stage, if connections are established on above load balancer, a reconnection will happen (3xx answer returned by load balancer) and PBXs connected upon to load balancer will initiate a new connection which establish on available load balancers. Step4: Rainbow team will monitor the remaining connections on haproxy-external-gra3-prod-2 and haproxy-external-rbx8-prod-2\. Once all connections are closed to null, Rainbow team will proceed for OS migration. A dedicated maintenance will be published in Rainbow status page. Key points: \- Ensure all load balancers' IP addresses are trusted in customer's firewall \- Ensure customers use FQDN instead of IP Addresses \- Reconnection timers must be defined properly to ensure seamless connection.

Oct 15, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 15, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours

Affected Components: [CALA] Rainbow Core Services

Oct 14, 09:00:00 GMT+0
Completed - Maintenance is now in progress.

Oct 14, 07:00:00 GMT+0
Identified - We are planning for a maintenance during that time. During maintenance operation, servers haproxy-external-sao-prod-1 / haproxy-external-sao-prod-2 (load balancers) will be down, following IP addresses will be unreachable: * 169.57.174.106 (\*.[openrainbow.com](http://openrainbow.com)) * 169.57.207.27 (\*.[openrainbow.com](http://openrainbow.com)).

Oct 14, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 hour

Affected Components: [NA] Rainbow Core Services

Oct 8, 10:00:00 GMT+0
Completed - Maintenance is now in progress.

Oct 8, 09:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. During maintenance operation, server haproxy-external-bhs-prod-1 (load balancer) will be down, following IP addresses will be unreachable: * 54.39.227.90 (\*.[openrainbow.com](http://openrainbow.com)).

Oct 8, 09:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 hour

Affected Components: [NA] Rainbow Core Services

Oct 8, 08:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. During maintenance operation, server haproxy-external-vin1-prod-3 (load balancer) will be down, following IP addresses will be unreachable: * 135.148.197.15 (\*.[openrainbow.com](http://openrainbow.com)) * 135.148.197.18 ([agent.openrainbow.com](http://agent.openrainbow.com)) * 135.148.197.19 ([rdd.openrainbow.com](http://rdd.openrainbow.com)).

Oct 8, 08:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 8, 09:00:00 GMT+0
Completed - Maintenance is now in progress.

Duration: 1 hour

Affected Components: [NA] Rainbow Core Services

Oct 8, 08:00:00 GMT+0
Completed - Maintenance is now in progress.

Oct 8, 07:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. During maintenance operation, server haproxy-external-vin1-prod-2 (load balancer) will be down, following IP addresses will be unreachable: * 135.148.197.14 (\*.[openrainbow.com](http://openrainbow.com)) * 135.148.197.16 ([agent.openrainbow.com](http://agent.openrainbow.com)) * 135.148.197.17 ([rdd.openrainbow.com](http://rdd.openrainbow.com)).

Oct 8, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 hour

Affected Components: [APAC] Rainbow Core Services

Oct 7, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 7, 13:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. During maintenance operation, server haproxy-external-syd2-prod-1 (load balancer) will be down, following IP addresses will be unreachable: * 15.235.144.86 (\*.[openrainbow.com](http://openrainbow.com)) * 139.99.4.244 ([agent.openrainbow.com](http://agent.openrainbow.com)) * 139.99.80.249 ([rdd.openrainbow.com](http://rdd.openrainbow.com)) * 2402:1f00:8001:1356::1 (\*.[openrainbow.com](http://openrainbow.com)) * 2402:1f00:8001:1356::2 ([agent.openrainbow.com](http://agent.openrainbow.com)) * 2402:1f00:8001:1356::3 ([rdd.openrainbow.com](http://rdd.openrainbow.com)).

Oct 7, 14:00:00 GMT+0
Completed - Maintenance is now in progress.

Duration: 1 hour

Affected Components: [APAC] Rainbow Core Services

Oct 7, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 7, 13:00:00 GMT+0
Completed - Maintenance is now in progress.

Oct 7, 12:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. During maintenance operation, server haproxy-external-sgp1-prod-1 (load balancer) will be down, following IP addresses will be unreachable: * 139.99.122.102 (\*.[openrainbow.com](http://openrainbow.com)) * 139.99.4.245 ([agent.openrainbow.com](http://agent.openrainbow.com)) * 139.99.2.105 ([rdd.openrainbow.com](http://rdd.openrainbow.com)) * 2402:1f00:8001:266::1 (\*.[openrainbow.com](http://openrainbow.com)) * 2402:1f00:8001:266::2 ([agent.openrainbow.com](http://agent.openrainbow.com)) * 2402:1f00:8001:266::3 ([rdd.openrainbow.com](http://rdd.openrainbow.com)).

Duration: 1 hour

Affected Components: [ANZ] Rainbow Core Services

Oct 6, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 6, 13:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. During maintenance operation, server haproxy-external-syd2-prod-1 (load balancer) will be down, following IP addresses will be unreachable: * 139.99.212.212 (\*.[openrainbow.com](http://openrainbow.com)) * 139.99.212.213 ([agent.openrainbow.com](http://agent.openrainbow.com)) * 139.99.212.214 ([rdd.openrainbow.com](http://rdd.openrainbow.com)).

Oct 6, 14:00:00 GMT+0
Completed - Maintenance is now in progress.

Duration: 1 hour

Affected Components: [ANZ] Rainbow Core Services

Oct 6, 12:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. During maintenance operation, server haproxy-external-syd-prod-1 (load balancer) will be down, following IP addresses will be unreachable: * 139.99.148.135 (\*.[openrainbow.com](http://openrainbow.com)) * 139.99.161.35 ([agent.openrainbow.com](http://agent.openrainbow.com)) * 139.99.177.193 ([rdd.openrainbow.com](http://rdd.openrainbow.com)).

Oct 6, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 6, 13:00:00 GMT+0
Completed - Maintenance is now in progress.

Duration: 57 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

Oct 1, 11:46:00 GMT+0
Investigating - We are currently investigating this incident..

Oct 1, 12:00:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Oct 1, 12:35:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Oct 1, 12:43:00 GMT+0
Resolved - This incident has been resolved..

Duration: 10 minutes

Affected Components: , , [EMEA] Rainbow Core Services, [WW] Rainbow Hybrid PBX Telephony, Europe & Middle East (EMEA) → Global (WW) →

Oct 1, 11:45:00 GMT+0
Investigating - We are currently investigating this incident..

Oct 1, 11:55:00 GMT+0
Resolved - This incident has been resolved..

Duration: 2 hours and 25 minutes

Affected Components: , [EMEA] Rainbow Core Services, [WW] Rainbow Hybrid PBX Telephony, [EMEA] Rainbow Hub Voice Services, Global (WW) →

Oct 1, 05:05:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Oct 1, 05:05:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Oct 1, 07:30:00 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 17 minutes

Affected Components: [EMEA] Rainbow Core Services, [WW] Rainbow Hybrid PBX Telephony

Sep 24, 16:28:44 GMT+0
Monitoring - Systems are back to service, we continue to monitoring..

Sep 24, 16:53:51 GMT+0
Resolved - This incident has been resolved..

Sep 24, 15:37:15 GMT+0
Investigating - We are currently investigating this incident..

Sep 24, 16:04:44 GMT+0
Identified - We are continuing to work on a fix for this incident and keep you informed..

Sep 24, 16:11:31 GMT+0
Identified - Impacted services are back to service, proceeded to reconnection.

Duration: 35 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

Sep 24, 16:05:00 GMT+0
Resolved - This incident has been resolved..

Sep 24, 15:40:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Sep 24, 15:30:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 hours and 14 minutes

Sep 16, 19:43:30 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Sep 16, 19:15:00 GMT+0
Investigating - We are currently investigating this incident..

Sep 16, 21:29:05 GMT+0
Resolved - This incident has been resolved..

Duration: 6 hours

Affected Components: [APAC] Rainbow Core Services

Sep 16, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 16, 10:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. During move operation, server haproxy-external-sgp1-prod-2 (load balancer) will be down, following IP addresses will be unreachable: * 15.235.144.86 (\*.openrainbow.com) * 139.99.4.244 (agent.openrainbow.com) * 139.99.80.249 (rbd.openrainbow.com).

Sep 16, 16:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour and 44 minutes

Affected Components: [WW] Rainbow Hybrid PBX Telephony

Mar 23, 20:01:44 GMT+0
Identified - Maintenance is now in progress..

Mar 23, 20:00:44 GMT+0
Identified - In order to perform necessary component maintenances, we need to restart some services used for PBX connection. PBX may disconnect once or twice in the coming hours with possible loss of telephony features during a few minutes..

Mar 23, 21:45:40 GMT+0
Completed - Maintenance is now complete..

Duration: 10 minutes

Affected Components: [WW] Rainbow Hybrid PBX Telephony

Mar 16, 20:03:31 GMT+0
Identified - In order to perform necessary component maintenances, we need to restart some services used for PBX connection. PBX may disconnect once or twice in the coming hours with possible loss of telephony features during a few minutes..

Mar 16, 20:29:36 GMT+0
Identified - Maintenance is now in progress..

Mar 16, 20:40:00 GMT+0
Completed - The maintenance is now finished..

Duration: 48 minutes

Affected Components: [WW] Rainbow Hybrid PBX Telephony

Feb 2, 21:45:00 GMT+0
Completed - The operation has been performed.

Feb 2, 20:56:59 GMT+0
Identified - In order to perform necessary component maintenances, we need to restart some services used for PBX connection. PBX may disconnect once or twice in the coming hours with possible loss of telephony features during a few minutes..

Duration: 2 hours

Affected Components: [DE] Rainbow Core Services

Nov 7, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 7, 20:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. This maintenance concerns infrastructure updates to address security vulnerabilities.

Nov 7, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 2 hours

Affected Components: [DE] Rainbow Core Services

Nov 5, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 5, 20:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time. This maintenance concerns infrastructure updates to address security vulnerabilities.

Nov 5, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 7 hours

Affected Components: [EMEA] Rainbow Conferencing, [EMEA] Rainbow Media Relays, [EMEA] Rainbow Hub Voice Services, [WW] Rainbow Hybrid PBX Telephony, [EMEA] Rainbow Core Services

Jul 28, 12:00:00 GMT+0
Identified - We are performing some backend maintenances in order to improve rainbow stability. Some users in EMEA datacenter may experience some disconnections. Also, it is possible to have a loss of telephony services during a few minutes..

Jul 28, 19:00:00 GMT+0
Completed - Maintenance has completed successfully..

Duration: 5 hours and 47 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

Jun 30, 12:44:45 GMT+0
Completed - All migration tasks have been performed successfully.

Jun 30, 07:00:00 GMT+0
Identified - On **Sunday 2024-06-30**, we will perform some upgrade activity to improve **Rainbow HUB Services** on **EMEA** and **DE** Region. **Note:** The window is longer than usual because we are taking advantage of this operation to carry out several updates in a row. But rest assured, the real impact of this operation for users is identical to others of this type: * Cloud telephony services may be disturbed for the duration of the period and calls will be disconnected for approximately 10 minutes over the same period. * The administration of cloud telephony services may ALSO be disturbed during the operation..

Jun 30, 06:57:27 GMT+0
Identified - Operation will start in a few minutes.

Duration: 14 hours and 50 minutes

Affected Components: [EMEA] Rainbow Hub Voice Services

Jun 29, 12:18:57 GMT+0
Identified - <https://support.openrainbow.com/hc/fr/articles/19786325080978-Upgrade-on-Saturday-June-29-2024>.

Jun 29, 07:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Jun 29, 07:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 29, 21:49:52 GMT+0
Completed - Maintenance has completed successfully..