Cluster 1

Connectivity issues

Resolved

No further issues have been reported, so we will be marking this as resolved. Our cloud engineers continue to keep a very close eye on our monitoring systems. Our investigations point to the issue being large volumes of traffic on network connections outside our infrastructure and outside our control, causing high levels of packet loss for customers trying to access simPRO. The packet loss automatically triggers retry logic in both the network layer and our software to ensure no data is lost which causes slow connections and other issues with connectivity. We will be working over the weekend to mitigate the impact of this happening. We apologise for any inconvenience caused and thank you for your patience while we work through the problems.

Monitoring

There continues to be various spikes of network congestion outside the simPRO infrastructure that is impacting some customers from being able to reliably connect to our systems, however for the past few hours the congestion has eased and connectivity has been restored. We are continually monitoring our systems to ensure they are running smoothly and all internal traffic is unaffected. We apologise for the inconvenience and we take this matter very seriously.

Investigating

The network congestion has peaked again, we are continuing to investigate and mitigate where possible.

Monitoring

Our investigations showed the cause of the issues was inbound network congestion outside our data center, causing all traffic entering simPRO to slow down. We are working with our data center provider on the cause and solutions. The congestion is easing and connectivity is restoring for customers. Our cloud team is closely monitoring the situation, however our systems were operating without issue during the congestion window.

Investigating

We are aware of connectivity issues this morning. Our cloud engineering team is working to restore service as quickly as possible.