simPRO Outage

Postmortem

Our data center provider has informed us that the reason for the outage was due to an erroneous network setting unintentionally rolling out to their production environment. This setting change caused large portions of the network in each data center to fail, taking the simPRO cloud offline until network connectivity was restored. They immediately became aware of the problem and worked to resolve it as quickly as possible by patching and correcting the configuration. Our cloud engineering team is working with our provider to ensure there are mitigation strategies in place to prevent a similar occurrence in the future. We apologise for the interruption to service, and can assure all our customers that we place very high priority on the availability of our platform.

Resolved

All systems have continued to operate effectively after the outage with no further impact expected. We are awaiting a “Reason for Outage” document from our data center provider to determine our next steps to ensure the continued availability of the simPRO cloud platform. We apologise for the inconvenience caused today.

Monitoring

All services have been restored globally for all customers. Our cloud engineering team are closely monitoring the systems to ensure they are operating effectively. We will be following up with our data center provider for more information about today’s outage. We apologise for the impact today that our customers faced.

Problem Identified

UK based customers are now back online. simPRO Service and USA customers should be sorted very shortly.

Problem Identified

simPRO Enterprise for Australian and New Zealand based customers has been restored. Our data center provider is continuing to resolve the issue for our Service, USA and UK based customers.

Problem Identified

We’re still waiting for our data center provider to fix the issue that is affecting the simPRO cloud. We are regularly following up with them and they are actively working on the resolution as quickly as possible.

Problem Identified

We’re still following up with our data center provider about the resolution. Once again we apologise for this outage.

Problem Identified

Our data center provider has let us know the problem should be fixed soon. We will follow up here with any further updates as they become available.

Investigating

Our data center provider is continuing to work on fixing the underlying problem, we apologise for any inconvenience this outage has caused.

Updated

We are continuing to escalate this with our data center provider to be resolved as soon as possible, please stand by as we work with them to resolve the outage.

Investigating

Our data center is currently experiencing a failure with internal networking. The root cause is still being investigated by our upstream providers. Our cloud engineers are working with our data center to restore services as quickly as possible. Further updates will be provided here as they become available.

3 Affected Services: