From the outside, a data center (DC) looks like an unshakable fortress: autonomous power supply, sealed halls, multi-level monitoring. We are used to clouds, banking and streaming simply working 24/7. But behind this stability stands a complex engineering ecosystem where the failure of a single node can trigger a cascade reaction that automation does not always manage to intercept in time.
Power infrastructure: when redundancy does not save
Power is both the foundation and the most vulnerable point. A standard Tier III scheme предусматривает several independent feeds from the city grid, arrays of batteries (UPS) and diesel generators. The task of the UPS is to “hold” the load for 10-15 minutes while the generators spin up to operating speed and take over the system.
Problems begin where the logic of redundancy meets physics. For example, during load switching a resonance effect or a short circuit can appear in the distribution board itself. If the авария happens at the level of the main busbar, even ten generators will not help – the energy simply cannot be delivered to the racks. Sometimes the quality of the fuel also fails: if the diesel has stood too long or contains impurities, the generators may not reach nominal power at the critical moment.
Thermal runaway and the inertia of cooling
Modern high-density servers generate a colossal amount of heat. The air-conditioning system is not just a “cooler”, but a complex network of chillers, fan coils and pumps. In many halls the principle of hot and cold aisle isolation is used so that the air does not mix.
If a refrigerant leak appears in the cooling circuit or the circulation pumps stop, the temperature in the hall begins to rise instantly. Under high load the critical mark is reached in a matter of minutes. Then the protection automation triggers: servers start to throttle (reduce their frequency), and then simply shut down so that the hardware does not melt. Bringing such a system back quickly is not possible – the equipment must cool down evenly to avoid microcracks in the boards.
Human factor in the technology stack
Even a perfectly designed data center is operated by people. Most large-scale outages of recent years were not caused by fires but by configuration errors. A planned firmware update of a network switch or a change in BGP routing tables can “cut off” a DC from the outside world within seconds.
Mistakes in access-rights management are especially dangerous. One incorrect automation script, launched with elevated privileges, can delete logical partitions on storage arrays across several availability zones at once. Recovery after such incidents usually takes hours, sometimes days, because of the enormous volumes of information that must be restored from backups.
Physical impact: from natural forces to missiles
We tend to evaluate risks in terms of cybersecurity, but a data center is first of all a physical object. A flood, an earthquake, or even a simple fire in a neighboring building that damages a backbone cable can stop an entire region.
A telling example was the incident in the AWS ME-CENTRAL-1 region (UAE). Corporate reports usually use vague phrases about “external impact” or “foreign objects entering the facility”. In this case it was a direct hit by an Iranian missile. When infrastructure receives such physical damage, no software can “repair” hardware remotely. Fire and sparks inside a sealed zone mean the automatic activation of gas fire-suppression systems, which displace oxygen and stop all processes.
Geo-redundancy as the only way out
Understanding that no single facility is protected one hundred percent, architects move to the concept of Multi-AZ (Multiple Availability Zones). This means distributing services between different sites located tens of kilometers from each other.
If one data center goes offline because of a power grid failure or physical destruction, traffic is automatically redirected to neighboring locations. But another challenge appears here – data synchronization. The latency between sites must remain minimal so that databases can update in real time. Without this, switching to the backup site will lead to the loss of part of the transactions, which is unacceptable for the financial sector.
Absolute reliability does not exist. There is only an acceptable level of risk and the cost of reducing it. The story with Amazon in Dubai once again reminded the market: digital services exist only as long as the walls that house them remain intact.
Leave a Reply