Royal Astronomical Society award and 18 years of Cluster

The four Cluster satellites are now old enough to vote and have a driver’s license in most countries of the world, in spite of the fact that they have, in fact, been happily ‘driving’ themselves well above most countries for the last 18 years.

Today’s post was contributed by Cluster Spacecraft Operations Manager Bruno Sousa. On 11 January, the UK’s Royal Astronomical Society awarded its 2019 Group Achievement Award to the Cluster Science and Operations Teams. Follow the mission in Twitter via @esa_cluster.

Beginnings

The Cluster mission got off to a very rough start in 1996, with the first quartet being destroyed along with their rocket during the unsuccessful Ariane 5 maiden flight from Kourou, in French Guiana. After being rebuilt and re-launched, in the summer of 2000, the satellites have gone on to exceed everyone’s most optimistic predictions, and they continue to produce the most amazing observations and data on the fundamental physics of the space between the Sun and Earth.

The rate of scientific publications from Cluster data has not significantly lowered over the years and in fact, with the launch of newer missions like NASA’s Magnetospheric Multiscale (MMS) mission, the Van Allen Probes mission and Themis, or China’s Double Star, Cluster’s relevance has, if anything, only become more important.

From Cluster’s unique vantage point, it is possible today to do complementary science observations together with all these other missions, enriching the quality of the data, and the quality of results for all.

Aging like port wine

If there is one feature that can define the Science Operations Workgroup of Cluster, it’s that they never felt complacent or happy to just do more of the same. Every extension of the mission up to now has been carefully planned to address new and exciting science targets. And from the mission operations side, all our efforts have been toward ensuring the success of those campaigns by carefully managing flight control challenges and optimising resources.  

It seems almost paradoxical that the older the spacecraft, and the fewer the resources available to keep the mission flying, the more data are produced. In fact, 2018 was the year the mission produced the largest-ever volume of data with the least amount of ground-station tracking time, one of the unavoidable costs to flying any mission. It seems as though the mission ages like port wine: the older, the better.

Orbital evolution – steady changes in the four satellites’ pathways around Earth – has been a major contribution to this, and with the apogee (the point in a satellite’s orbit when it is highest above Earth) at its lowest, we have benefitted from a favourable configuration. In addition, constant improvement of our weekly planning and scheduling coupled with innovative techniques like ‘Multiple Satellite per Aperture’ (allowing two or more satellites to be tracked simultaneously with the same ground station – we are the first mission to use it routinely at ESA), and better management of resources, has allowed for additional optimisation.

From surviving to thriving

From 2008 to 2012, the laws of physics and celestial mechanics dictated that the spacecraft were experiencing a nasty dip into the Van Allen belts, tyre-shaped belts of highly energetic electrons and protons that are trapped by Earth’s magnetic field. This caused solar array degradation, leading to power loss and, eventually, the on-board batteries failed and had to be permanently shut down. The loss of power meant that heating had to be sparse and the high-power amplifier had to be sacrificed, thus leading to an increase in the time required for downloading data.

In 2011, the spacecraft engineers faced an exhausting year with almost uninterrupted eclipses (when our satellites receive no sunlight so generate no power) every single orbit, requiring enormous effort to manage powering down and then switching on and reconfiguring everything back to full operational status. With all hands available doing shifts to cope with the workload, little time was left to implement improvements.

But necessity is the mother of creativity and soon the engineers got around to automating many of these command-intensive tasks. Further, simplifications have meant that we have reduced the time required to recover the four spacecraft from four to one hour per spacecraft. Today, when everything goes smoothly, which is almost always, we can finish the task in two hours doing a pair of spacecraft at one time, needing just four or five mouse clicks. 

Automation and optimisation

The last seven years have seen Cluster change from a mission that had been fighting against adversity to survive, to a mission at the forefront of optimisation and automation of operations. This has been in great measure possible due to a team of well-practiced and smart engineers, who were already highly motivated to achieve that transformation, as a result of a culture and philosophy put in place by my predecessor, Jürgen Volpp.

With the advent of automation and the pressure to optimise resources, less manual work has led, over the years, to a reduction in the size of the team available to conduct real-time operations. From an initial pool of nine spacecraft controllers covering 24 hrs/day, year-round operations in the control room plus three analysts to cover our database management and mission-planning needs, we have gradually but systematically reduced to four spacecraft controllers and no analysts, with part of the work simplified, automated or shared among the also-trimmed pool of spacecraft operations engineers, which now totals six, including the Spacecraft Operations Manager.

With the reduction in staffing, we improved shift and station planning to optimise the collocation of controllers and ground contacts. We’ve also extended automation to allow for ‘hands-off’ operations when no staffing of the consoles was possible. Automation has also been progressively built-in to some of our recurrent anomaly alerts, whose signature we can identify, and therefore provide a systematic and timely reaction.

Cutting-edge tools

Clusterweb has been one of the tools emerging from the creativity and skills of the Cluster team that has helped to dramatically improve planning and fleet supervision. It began its evolution in 2009.

In 2016, the team opted for a full-scale re-engineering of the tool, resulting in a new, modern and highly configurable timeline plotting engine, now called OPSWEB, currently in use by five other teams.

Operations teams have always prototyped any small tools they needed. What makes this development stand out is the scale and scope of development achieved. It wasn’t just another tool done in Excel or Java by a trainee; instead, a professional approach was used, making full use of Scrum, a very iterative development approach, combined with enlarging the development team through small voluntary contributions across the organisation (at its peak, seven people were working on it simultaneously) and supported by a state-of-the-art development and integration environment, a flexible and modular architecture and a modern technology stack. The result thus far achieved, is, by all standards, remarkable and on par with the best to be found in industry.

Incubating expertise

Engineers who have worked with Cluster operations, either because they have had to deal with the complex eclipse operations or because they have had to help sort out the radiation- and age-related equipment glitches, have traditionally become very comfortable dealing directly with the spacecraft, and have evoled into experts who tackle problems in an autonomous and responsible way. They have also become very pro-active in improving the overall operations setup, whether by improving flight procedures, automation scripts or deploying new tools.

This has meant that, over the years, Cluster operations alumni have found their way into the newest and most complex missions flown by ESA at the ESOC mission control centre, like Bepi, ExoMars and Juice. Several have also found their way into key positions at Eumetsat and in new space companies.

Cluster has become a ‘school for operations’ at ESOC, and managing the turnover of the team and the propagation of the needed skills, experience and mind-set has been a tremendous challenge. At the same time, we continue striving to produce ever bigger and more complete sets of science data.

It is, therefore, perhaps no big wonder that earlier this month, the UK’s Royal Astronomical Society, when honouring the extraordinary scientific output of this mission, also emphasised the role of operations in the society’s recent announcement of the 2019 Group Achievement Award to the Cluster mission.

A mission that not so long ago was at risk of being discontinued has instead continued shining as a backbone data provider for the geophysics community and a role model for effective and efficient mission operations.