Loud noise brings down banking data centre, destroys hardware
Mon 12 Sep 2016
ING Bank saw its Romanian payment and ATM services grind to a halt after a routine fire extinguisher test forced its data centre operations offline for 10 hours.
A loud noise emitted from inert gas being released during the planned extinguisher system drill cut off the bank’s main facility in Bucharest, Romania, but also caused irrevocable damage to dozens of its hard drives.
Head of ING Retail Banking, Daniel Llano, apologised [Romanian] for the inconvenience and informed customers that the ‘serious’ technical fault had been caused by Inergen flooding.
Inergen is a type of fire extinguisher system that uses gas instead of the traditional foam or liquid sprays. Inergen, typically deployed in enclosed areas, is compressed inside cylinders and spread evenly through nozzles to extinguish fires. The gas alternative is preferred to liquid or foam for data centre application as these systems would damage the machinery and computing equipment. Inergen gas also does not decrease the overall temperature within the data centre in the same way as other fire suppression systems.
In this instance, the Inergen gas was released through the nozzles, but the pressure was too high, causing a sound exceeding 130dB. The noise, similar to that of a ‘running jet engine’, created large vibrations which spread to the casing protecting the hard drives – and destroyed the internal hardware.
In addition to damage to the storage devices, card transactions, ATM withdrawals, internet banking, internal email, and the bank’s website were also affected for local customers. ING has said that it will reimburse costs to all customers who were forced to use other bank’s ATMs to withdraw money during the blackout.
Services have now been restored, but ING has been forced to rely on a backup data centre. The bank has launched a diagnostics and analysis investigation into the disaster to prevent similar incidents in the future.