At around 20:10 on 10th July 2018 we encountered unusual traffic originating from customer equipment on a network service connected at our Dunsfold data centre.
ImpactThis caused the control plane on our Dunsfold WAN routers to become overloaded resulting in a period of disruption at our Dunsfold site which lasted approximately 55 minutes.
Recovery and Remedial actionsThe nature of this event resulted in traffic bypassing the configuration that we had implemented to prevent such a level of impact. As an immediate action we have isolated a portion of our network infrastructure from this service so that other services should not be affected should this incident reoccur. Additionally we are evaluating configuration changes in our lab environment under simulated incident conditions; we are engaging specialist contractors to review our network configuration and are in contact with our router vendor support channel.
ConclusionWhile we had measures in place to mitigate against this type of network event, the nature of the traffic affected our WAN routers in an unexpected way. Our WAN routers should have both capability and capacity to manage an event of this nature without it impacting other services. We have implemented temporary changes to safeguard the operation of our services should the incident re-occur and will deploy further infrastructure changes once the permanent recommended solution has been identified and tested.
Should you have any questions following the incident please contact email@example.com and we will respond to your email during normal working hours or if it is of a technical nature please log into your Memset account to submit a support ticket.