Service has been restored for the affected customers. Our cloud engineering team continue to monitor the situation to ensure continued operation. We apologise for the interruption. We will work with the data centre to understand what went wrong with the hardware host and put plans in place to mitigate the risk of this occurring in the future.
Our data centre has let us know it could be another 1 or 2 hours before services are restored for the affected customers. At simPRO we run three redundant systems on different hardware to ensure maximum availability, unfortunately the earlier maintenance window removed two of the three nodes for an hour and when they came back up after the maintenance we need to sync the data back on to them. This is a routine operation. It was at this time that the 3rd node went offline with the hardware issue so we can’t utilise the 2 other nodes until it comes back online. We understand the serious impact of this outage on the affected customers (roughly 5% of users) and apologise for the inconvenience. There is no risk of data loss.
Our data centre has informed us there is a hardware issue on a critical host that failed to come back online after the maintenance earlier this morning. They are actively migrating the affected instances off that host and services will be restored as quickly as possible. They can’t give us an ETA currently. We apologise for the inconvenience caused.
Our cloud engineering team is aware of an issue impacting a small subset of users. This has been caused by a maintenance window overrun by our upstream provider. We are working with them to get this resolved asap. Further updates will be provided here as they become available.