Last week, Frederic Hemberger posted a query in Twitter

Frederic asked:

Do I know anybody working at @nasa or @esa? Saw a documentary about mission control and have a monitoring/interface question. During a mission, there are probably hundreds or thousands of metrics to be monitored and visualized. Are there some (publicly available) guides, best practices, etc. how to set up an interface, that all important metrics are clearly recognizable (and actionable) even in stressful or dangerous situations?

He added:

Looking at tweets like https://twitter.com/NASA_Johnson/status/907732062823518208 …, it seems like “just add more screens” 🙂

We asked ESA’s Bruno Sousa, Spacecraft Operations Manager for the Cluster mission, if he could reply. Bruno’s excellent response is below. 

Often people ask us how do we know, from the hundreds or thousands of engineering parameters that are continuously measured and recorded on-board a spacecraft, that something isn’t just right?

This is one of the core problems of spacecraft monitoring, and one for which there are several approaches for solving.

On the one hand, spacecraft today are made to be very self-aware. Not only do they measure and record data, but they can scan through it and check whether any of its parameters is outside what they consider to be a nominal range. Sometimes, they actually combine the logical value of multiple parameters to make a more complete, system-wide assessment. And when something is spotted that is not right, the spacecraft has the ability to isolate the problem on its own, when its location is clear, by, for example, switching to a redundant unit. Or it will issue a warning to the ground, asking mission controllers to have a closer look at it.

Or, when in serious doubt about the source of the problem, the spacecraft may decide to enter what we call a ‘safe mode’ – this is a mode that minimises resource consumption but maximises safety and the chance to communicate with Earth.

On the other hand, because spacecraft computing power is limited and not as advanced as that available on the ground, we sometimes replicate this monitoring behaviour by sorting and checking the data as it is received back on Earth. Our knowledge of the expected behaviour of each unit under different operating conditions is coded into a database and used systematically for checking.

Upon an ‘out-of-limit’ condition our monitoring system will also raise an alarm to the operators. A ‘soft out-of-limit’ will indicate a dangerous trend, and a ‘hard out-of-limit’ will indicate a failure has occurred.

In spite of the fact that each spacecraft collects a great amount of data, there are generally a few dozen parameters that give an overall indication of the state of the system. These are normally monitored in live displays before the operators, and are the ones that appear in photos of control room dashboards.

On 17 March 2015, Spacecraft Operations Manager Bruno Sousa (standing, looking up) and the Cluster flight control team watched closely as the Cluster-1 satellite performed a 17-minute orbital correction manoeuvre. The team operates the four-satellite Cluster mission from ESA’s Space Operations Centre, ESOC, Darmstadt, Germany. Credit: ESA

On 17 March 2015, Spacecraft Operations Manager Bruno Sousa (standing, looking up) and the Cluster flight control team watched closely as the Cluster-1 satellite performed a 17-minute orbital correction manoeuvre. The team operates the four-satellite Cluster mission from ESA’s Space Operations Centre, ESOC, Darmstadt, Germany. Credit: ESA

New technologies in the fields of data mining and pattern recognition are now being regularly used to analyse parameter value trend evolutions and predict – in advance – that a failure might be developing in the near future, therefore giving the operators extra time to react and prepare a different operational strategy.

These techniques work especially well with units that have a limited life time and exhibit a recognisable form of degradation before reaching their end-of-life, like the lasers on gyros.

Satellites present a unique challenge because they are always remote and one can’t just get to them and recalibrate whateven is being measured or have a good look at what is really going on. So a solid knowledge database and good data-analysis skills are fundamental to make overall sense of what is actually happening on board at any given time.

A body of knowledge, guidelines and best-practices in this topic is being continuously developed in specialised conferences and workshops, such as SpaceOps – a bi-annual meeting that joins professionals from the domain of spacecraft operations from all over the world, and in which ESA plays a key role.

Editor’s note: Thank you, Bruno, for an informative update! Follow Cluster direct from mission control via @ESA_Cluster. Frederic’s original query in Twitter, 22 Sep 2017