Our Data Center and Cloud Engineers, along with upstream providers have successfully isolated the root cause of the network performance degradation affecting a subset of users in Australia and New Zealand - and put measures in place to resolve the ongoing speed issues.
Routing updates in the Sydney Data Center that were put in place last week to cope with a number of carrier outages had caused some route looping in and out of Sydney. In some cases these routes meant customers were transiting through 3 to 4 times as many interconnects as usual. These problematic routes have now all been corrected overnight.
Monitoring this morning has confirmed that service levels are back to normal for affected users - however we will continue to monitor the environment to ensure availability and reliability for our customers.
Further works over the coming 24/48 hours are planned to put in place some additional ability for us to handle this type of event better in future - however no impact to clients is expected from these works.
Again - we’d like to apologise for the inconvenience and appreciate your patience as we worked to resolve this issue.
Our Engineering Team and Data Center engineers have now completed a reroute of a number of services around affected network paths and this appears to have resolved a majority of issues for most affected clients.
We are continuing to work with the Data Center to identify and mitigate any further issues and will update this thread as that work progresses.
Work is continuing to get affected services routed around affected infrastructure.
During the course of the next 30 minutes - as this work is completed - there may be small windows of interruption to services as changes are made to bring everything back online.
We’ll update this thread again as soon as this work is completed.
Our data center and engineers are continuing to work on speed issues impacting a subset of our customers this morning.
Further investigation throughout the weekend and this morning is showing some major changes to carrier routing paths within Australia and New Zealand causing significant transit and bandwidth congestion for some ISP’s into the Sydney data center.
Our data center are working with their providers to get this resolved as quickly as possible. We will provide updates as work progresses.
We apologise for any inconvenience caused and thank you for your patience while we work to get a resolution in place.
We are aware of slow connection issues for some customers this morning. Our cloud team are investigating this with the utmost urgency. We will provide further updates when possible.