Resonance from the fire alarm brought down 30% of the servers in the Digiplex data center near Stockholm


The Linux kernel turns off the hard drive after 120 seconds of sound at the resonant frequency through the Edifier r19u USB speaker. The speaker is turned on for about a quarter of the power (less than 100 mW) and is located at 20 cm from the HDD, directed to the table to enhance vibration. Shot from the video with the demonstration of the HDD killer

As you know, any HDD is subject to vibrations due to fluctuations in ambient air. HDD manufacturers specify the maximum permissible vibration level for each model, and the hard disk itself is often tried to be placed in a vibration-protected container made of rubber or other insulating material. Despite these efforts, the HDD can still work as a microphone and record ambient sounds .

To disable the HDD, you need to send a sound to it at a frequency that resonates with the frequency of the HDD. The hdd-killer program perfectly demonstrates how this happens in practice. If the sound is loud enough, then the system will soon shut down the device with an I / O error, and the hard disk itself may be permanently damaged.

Sometimes a similar occurrence occurs unintentionally. For example, in September 2016, the Romanian data center ING Bank suspended work for 10 hours after fire drills. Dozens of hard drives are out of order due to the loud sound of inert gas released from cylinders under high pressure. The sound was very powerful (more than 130 decibels), and in fact it is known that even shouting on hard drives is impossible - this increases the latency of access to the HDD.

About the same thing happened on April 18, 2018 in the Swedish data center Digiplex , which serves stock exchange transactions. Because of the incident, even had to stop trading on the Nasdaq Nordiq in Northern Europe, writes Bleeping Computer .

Like two years ago in Romania, it all started with a fire alarm - and the loud sound of inert gas being released from high-pressure cylinders. Sensitive drives did not sustain this sound - and soon about a third of all servers in the data center ceased to function. Reportedly, the trading systems of the Nasdaq Nordic and Finnish banks FIM Bank and OP Bank Group worked on these servers.

In addition to the servers of the exchange, servers for high-frequency trading were destroyed, which some banks installed as close as possible to the exchange servers to minimize ping. In this case, this strategy was unsuccessful.

Due to the incident, Nasdaq Nordic was unable to open at the scheduled time at 8:00 am local time, when all the stock exchanges began to operate in this area. Traders from Sweden, Finland, Denmark, Iceland, and also three Baltic countries suffered. For some reason, the Nasdaq in Norway has opened and worked, reports the publication.

It was possible to partially restore the systems and restart trading on the Nasdaq Nordic only at 3:00 pm on the same day, although according to the rules of the exchange, it is obliged to restore the system from a backup copy within two hours. Nasdaq representatives told the Finnish media that the backup system will be used until Monday, until they replace the affected equipment in the data center and verify that the backups have valid copies of all the working systems from all the participating banks.

A Digiplex spokesman said that Nasdaq rents only racks at the data center, but runs on its own servers. Nasdaq management explained that there were not enough servers in all of Sweden at that moment to replace the failed ones. I had to import new servers from abroad. This explains the delay in restoring the system from backups: “The planes have just arrived,” said Lauri Rosendahl, executive director of the Nasdaq Nordiq, in a comment to the Finnish newspaper Helsingin Sanomat the day after the incident. Until the new equipment is installed, the exchange services do not work to the full. In particular, trading is not available to all traders, writes Helsingin Sanomat.

Digiplex is one of the largest data centers in Northern Europe, with thousands of servers in a 20,000 m² building. It is located in the Upplands district of Vesby, near Stockholm. It is curious that within two days after the fatal fire alarm it was not clear whether it was caused by a real fire or turned out to be false. Depends on this, who will pay compensation to the victims and for what article.

Source: https://habr.com/ru/post/411697/


All Articles