Alarms at the Columbus Control Centre

Flight Director Tom Uhlig on the the warning sounds heard at ground control at the Columbus Control Centre in Germany:

Caution and warning tool. Credit: DLR (CC-BY 3.0)

My colleagues on night shift had expected a quiet night monitoring the Columbus space laboratory without disruptions. But sometimes things turn out differently than expected – and that is why we are constantly alert at the Columbus Control Centre…

Adrenaline always rises in the control room when the discreet but penetrating tone is heard that signifies to the flight controller that an alarm on the Space Station has sounded – it somehow always reminds me of “Das Boot”. We immediately glance at our big centre display where the “Caution and Warning” tool runs continuously to display Space Station alarms. “Yellow alert” – fortunately not so bad. Our view then moves to the text at the end “– COL” indicating that it is our concern…

A “– JEM” would put the Japanese on high alert, a “– LAB” or “– N2” is the domain of our NASA colleagues.

It might come as a surprise but alarms are common as we often trigger them on purpose when we reconfigure a system, for example, but they rarely concern our control centre – tonight we are not so lucky: “DMS Mass Memory Unit (MMU1) Failure – COL” looms in large letters. All the International Space Station control centres and the astronauts on the Station themselves are now all waiting for us: what must be done? How serious is the alarm? Do the astronauts need to take any measures?

The Space Station alarm system has three levels – or even four, if you include the harmless “Advisories”.

https://soundcloud.com/esaops/international-space-station-siren?in=esaops/sets/iss_alarms

Most dangerous are the “Emergencies” – emergencies that directly threaten the astronauts and where immediate countermeasures are necessary. These fall into three categories: something is burning, a sudden drop in pressure is detected that could bring the crew into danger  or the air could contain toxic substances. Emergencies are shown in red and announced in the Space Station by an unpleasant beep. On dedicated “Caution and Warning Panels” the astronauts also see the corresponding alarm flash: “FIRE”, ” ΔP/Δt”  to give a physical representation of “pressure drop”, or “ATM” for atmosphere. Their laptops will show more information – comparable to our screens in the control room – as they will display where the alarm was triggered as well as whether it was triggered manually by a colleague or triggered automatically by a smoke detector, for example.

https://soundcloud.com/esaops/international-space-station-warning-alarm?in=esaops/sets/iss_alarms

The middle-level alerts are the “Warnings“. These require an immediate response, because they mean an important component of the Space Station has a problem. These are also marked in red and their alarm is no less alarming. Based on the error code that is displayed, the astronauts and control centres can quickly reach for the appropriate procedure to follow as a first response to Warnings.

https://soundcloud.com/esaops/international-space-station-caution-alarm?in=esaops/sets/iss_alarms

Columbus Flight Control Team in K4. Credit: DLR (CC-BY 3.0)

The lowest level are the yellow “Cautions” – they also have their own sound and flashing lights in the Space Station, unless we have not “inhibited” or “suppressed” them. When a caution is “inhibited” it is not processed by the Space Station’s central computer. When Cautions are “suppressed” an alarm is triggered on the laptop or our display, but lights and tones that would wake astronauts at night are not turned on. Of course, very specific guidelines exist to decide when an alarm event can be turned off or suppressed.

Cautions are not time-critical, and the astronauts only need to do something if control centres are out of contact with the Space Station for some reason and cannot analyse the data or send corrective commands.

For the night shift, the Caution alarm for the Columbus Control Centre meant that our main memory in the Data Management System (DMS) had stopped interacting with the central computer, which activated a second, fail-safe memory that sent the error message. The system continues to work without major problems – we have lost one fail-safe – also called a “loss of redundancy” – but another error would really get us into trouble. So the night shift spent the rest of the evening looking at the log file for a detailed error analysis – the adrenaline level quickly dropped…

How is our “Mass Memory Unit” today? The experts who know the computer inside out looked at the problem and have come to the conclusion that space radiation changed the memory that caused the error. The solution is simple: reroute the contents of the defective memory via the second memory unit and reconfigure the Columbus systems.