Platform Performance

Uptime Impact: 38 minutes and 27 seconds
Resolved
Resolved

We believe this issue to be resolved. If you are still having issues, please contact us on support@burstfire.net

Avatar for Anthony Allen
Anthony Allen
Recovering

Traffic is being released back into the platform and mitigation is being monitored closely. Customers will be seeing services restore.

Avatar for Anthony Allen
Anthony Allen
Identified

The traffic volumes remain largely mitigated, but our supplier is bringing online additional capacity to ensure they have unusually high levels of overhead as further mitigation at their core. Capacity is added routinely during normal volumes of traffic, but not when under the pressure of current volumes, hence restricting parts of their network while they continue to do so. Customers will be seeing traffic flow momentarily as this happens, but we do not expect a full resolution until this work is complete.

Avatar for Anthony Allen
Anthony Allen
Identified

Our provider was seeing traffic typical of a substantive DDOS attack. These are usually managed routinely and daily, however in this case the traffic is inordinate in scale.

They are confident the mitigation is effective and the NOC team is now monitoring. Traffic and packets per second peaked at close to 10x normal volumes. Note that proxy solutions were disabled for a period of 30 seconds or so during mitigation, and as a result, hardware may need to re-register by rebooting devices. This would impact inbound calling but not outbound calling.

Further mitigation was implemented at 12:32 and expect all inbound calls to resume without the need to re-register devices within the next minute.

Our supplier continues to mitigate the traffic and monitor, but all metrics have returned to normal. A more detailed post-mortem will be published once monitoring has been completed and teams have been fully debriefed.

Avatar for Anthony Allen
Anthony Allen
Investigating

We are investigating reports of outbound and inbound call failures.

Our teams are investigating as a matter of urgency. Further updates to follow.

Avatar for Anthony Allen
Anthony Allen
Began at:

Affected components
  • Telephony
    • Burstfire Switchboard (Phone)
    • Callswitch (VoIP)