2017-09-29 09:05 UTC
We encountered a minor incident in the last number of hours. Users may have seen intermittent failures on some of their actions. Impacted systems are now stable and we are actively monitoring them while we investigate the root cause. We will provide further details when the root cause has been identified.
This issue arose as a result of two behaviours coincided. This impacted one third of our systems resulting in intermittent failures. Firstly, there was high usage on our systems. Secondly, our antivirus system attempted to allocate resources for a definition update. This failed due to the high usage consuming system resources. Our antivirus rapidly wrote to log files consuming hard disk space. Ordinarily our monitoring would have alerted us to an impending issue but the granularity of the monitoring failed to provide sufficient notice. The hard disk on impacted systems was consumed. We have implemented a number of configuration changes to our antivirus and monitoring improvements to prevent this issue arising again.