Understanding Philae’s wake-up: behind the scenes with the Philae team

Since Rosetta’s lander Philae first woke up from hibernation and called ‘home’ on 13 June, the teams at the Lander Control Center (LCC – DLR), the Science Operations and Navigation Center (SONC – CNES), the Max-Planck Institute (MPS – Göttingen) and the Institute for Particle and Nuclear Physics (Wigner Research Centre for Physics – Budapest) have been working with ESA’s Rosetta Mission Operations Centre (RMOC – ESOC) and the Rosetta Science Ground Segment (RSGS – ESAC), and in close cooperation with the Philae and Rosetta scientists, to establish regular and predictable contacts with Philae, and to resume scientific measurements.

This blog post has been written by Koen Geurts, Philae technical manager, and Cinzia Fantinati, Philae operations manager (both from the LCC at DLR), and gives a detailed insight into the work being done by the teams.

Following Philae’s deployment by Rosetta to the surface of Comet 67P/Churyumov-Gerasimenko on 12 November 2014, the lander operated for 2.5 days before falling into hibernation at its final landing site, Abydos. The fundamental issue was a lack of sunlight to charge Philae’s secondary batteries.

Due to the large number of unknowns with respect to Philae’s final landing site, not least the lander’s orientation with respect to the local topography, it was difficult to precisely predict when Philae might wake-up again as the comet approached the Sun and the strength of the sunlight increased. Another major concern was the ability of the hardware to survive the very low temperatures, well below the –55°C qualification temperature, expected during the likely several month long hibernation phase.

Previous posts have described the essential conditions that must be met for Philae to wake up and boot: an internal temperature above –45°C and more than 5.5W of power. Based on measurements made before Philae entered hibernation in November and models based on its likely location and the comet’s orbit, it was decided to commence wake-up campaigns in March 2015 during selected periods where Rosetta’s orbital geometry was favourable to communication.

No signals were heard during those periods, and at a regular meeting of the Rosetta Science Working Team (SWT) in Rome on 11–12 June, a discussion was held on providing further support to making contact with Philae. The SWT approved a 2-week search campaign, during which Rosetta’s trajectory would be optimised for possible contacts with Philae.

The campaign was due to start mid July. Little did anyone know that Philae was ready to talk to us the next day!

Link establishment procedure
In order to establish communication with Philae, Rosetta’s on-board Electrical Support System Processor Unit (ESS), responsible for the interface between the two probes, must be switched on and configured in “Research Mode”. In this mode, the ESS transmitter is switched on and sending signals with a specific pattern that are recognised by Philae. In parallel, the ESS receiver is switched on in order to listen out for any possible replies.

When either of the receivers on Philae hears a signal from Rosetta, the on-board software performs a check to see if there is enough power to switch on one of its two redundant transmitters. If so, the transmitter is turned on and returns a signal with a specific pattern, signalling the ESS on Rosetta that it is ready to establish a two-way link. This done, Philae immediately starts uplinking science and housekeeping data that are stored in its on-board mass memory (MM). Once the two-way link is established, Rosetta can also send new command sequences for Philae to load and execute, initiating scientific measurements, for example.

The nominal operations mode foreseen for Philae during comet operations was to turn on its receivers periodically. However, there were concerns that this might lead to brief contact opportunities being missed. Thus, shortly before Philae went into hibernation last November, the operations team configured its on-board software to switch on both of its redundant receivers whenever the rechargeable secondary battery reached a minimum cell voltage of 3.4V or if sufficient surplus solar power was available. The aim was to maximise the chances of contact being made after Philae came out of hibernation.

First lander contact
At 20:28:11 UTC on 13 June 2015, Philae and Rosetta managed to establish their first two-way link since November, for a duration of just 78 seconds. In this time, a total of 343 packets of housekeeping telemetry (TM) data, including information from the thermal, power, and on-board computer subsystems, were transmitted from Philae to Rosetta. One TM packet comprises 141 16-bit words or 2256 bits, and thus the total amount of data transmitted was just under 100 Kbytes. There is a distinction between “real-time” TM, which provides live information on the status of various systems at the time of the link, and “stored” TM, information from some point in the past that has been stored in the on-board MM.

The order in which Philae’s TM is transmitted to Rosetta is linked to how it interacts with the MM. Not only “old” TM is stored in the MM as you would expect, but real-time TM generated during an on-going communication also goes via a buffer to Philae’s MM. Since the MM operates on a First-In-First-Out (FIFO) principle, the older, stored TM gets transmitted first, before the real-time TM. Completely clearing a full MM takes 40 minutes.

Thus, if the link is stable but relatively short, it is not possible to obtain real-time status information, as those TM packets remain queued in the MM.

However, in the case of a very unstable and short (e.g. less than 20–30 seconds) link, the behaviour is different. In this instance, the on-board software routes the real-time TM from the buffer straight to the communications unit for transmitting to Rosetta, completely bypassing the MM.

Decoding when each piece of TM was generated is tricky. Each time Philae has enough power to reboot, the on-board clock starts from zero, and true time synchronisation to UTC can only occur when a link is made with Rosetta. In the absence of contacts with Rosetta, an alternative way of keeping track of the time when the TM is generated is needed. To do so, the on-board software computes a “comet day counter”, which is stored in non-volatile memory, by counting in units of 744 minutes (i.e. 12.4 hours, the rotation period of the comet as specified in the software) and adding this to a predefined starting date of 28 November 2014 at 00:00:00. This gives an approximation of the UTC time, but only under the assumption that Philae is operating continuously or reboots at every local sunrise.

This date offset was randomly selected in the weeks before landing as a placeholder and was supposed to be updated in the hours after landing, based on predictions of the first wake-up on the comet surface. Due to the unexpected nature of Philae’s landing, the parameter was never updated.

The data sent to Rosetta during the first connection on 13 June contained stored TM with comet day counts of 2, 19, and 20. In addition, 6 real-time TM packets with count 95 were obtained. If Philae had been operating continuously since the landing in November 2014, then comet day 95 would have occurred on 16 January 2015. But since these ‘count 95’ TM packets actually correspond to 13 June, that offers the opportunity to work backwards and see when Philae actually first came back to life again, even if it was unable to communicate with Rosetta at the time.

The real-time TM packets indicated that Philae rebooted at 20:24:48 UTC on 13 June, less than four minutes before contact was received by Rosetta. If that moment corresponds to count 95, then count 20 must have occurred 75 x 744 minutes earlier, i.e. 38.75 Earth days earlier, on 6 May. Similarly, count 19 was 12.4 hours earlier on 5 May, and count 2 was on 26 April.

There are some caveats here. First, this arithmetic does not take into account the possibility of a reboot due to a power bus collapse, which the on-board software cannot distinguish from a “real” comet morning. However, by correlating the daily temperature increase seen in TM from consecutive days with the comet day count, such “fake” reboots can in principle be identified.

Also, although the MM operates on a FIFO mechanism, it is not clear why there are gaps in the downlinked TM: for example, the jump from count 2 to 19. While the date count is written to the non-volatile memory immediately after boot, subsequent housekeeping and science data are stored in volatile RAM and only transferred to non-volatile memory once sunset is detected via a decrease in incoming solar power.

One possible explanation then is that comet sunset occurred too quickly on those days, and the rechargeable battery was not able to provide the necessary 30–60 second buffer until the saving of TM into the non-volatile memory was complete. Conversely, the jump from 20 to 95 is understood as being due to the fact the count 95 TM were generated in real-time on the day of the connection, while there was not enough time to downlink any possible stored TM from comet day counts 21–94.

The downlinked TM for comet days 2 and 19 span the full period between local sunrise and sunset as seen by Philae, thus giving a direct indication of the duration of sunlight at that location. The TM from comet day 20 is incomplete, however, as the link to Rosetta was broken before it could all be transmitted.

In addition to the reconstruction of the actual UTC boot times, the engineering housekeeping data contained in the downlinked TM packets were also evaluated, including the internal and external temperatures, the incident solar power, the state of the battery, general currents and voltages, and the quality and configuration of the communication link. In particular, the latter information came to play an important role in the weeks to come.

Immediate Rosetta trajectory changes
Obviously, the first signal received on 13 June triggered a series of teleconferences, discussions, and need for immediate decision taking. Together with the Philae consortium, key people from the ESA teams at RMOC and RSGS, the Mission Manager, and the Project Scientist started evaluating and assessing Philae’s status. It was immediately clear that Rosetta’s trajectory needed to be optimised in order to be compatible with Philae communication, and steps were agreed to prioritise this activity.

The comet latitude range where Philae was thought to be was defined and constrained to be within 0° and +55°N, and processed for implementation by the RMOC Flight Dynamics and Flight Control teams. This sudden change was not without impact on the already foreseen and scheduled Rosetta science activities. For example, the pointing of Rosetta for scientific observations was strongly constrained during predicted communication windows, so the RSGS team had to quickly evaluate and modify the baseline planning in liaison and agreement with the instrument teams. All in all, this required a huge effort by everyone involved, with the combined objective of restarting Philae’s science measurements.

Subsequent contacts between Philae and Rosetta
The second contact with Philae occurred one Earth day (i.e. 2 comet days) later on 14 June at 21:22:47 UTC. This time, the duration between the first and final contact was 04:04 minutes, although frequent link interruptions occurred in this period. A total of just 26 TM packets were received, all real-time, with comet day count 97.

There was then a longer delay until the third contact occurred on 19 June at 13:20:33 UTC. This time it was for a significantly longer duration of 18:53 minutes, but it was again frequently interrupted. A total of 180 packets were received, comprising a mix of day count 107 (i.e. real-time) data and data with day count 96 generated during the previous contact. The real-time data allowed us to determine the configuration of Philae’s communication hardware for the first time, with transmitter TX1 and receiver RX2 in use.

The fourth contact, on 20 June at 13:55:25 UTC, had a total duration of 31:01 minutes with many interruptions. This time, a total of 744 packets were transferred containing stored TM from comet days 21 to 25 and 97, and real-time data from the current comet day 109. For the first time, this allowed the reconstruction of several sequential comet days. Again, the link was established with TX1 and RX2.

The fifth contact occurred on 21 June at 02:32:50 UTC and lasted 11:25 minutes, with a 10:37 minute interruption. A total of 294 TM packets were transferred with comet day count 25 to 27; no real-time TM was received.

The sixth contact occurred on 24 June at 17:23:48 and lasted 17:11 minutes with frequent link interruptions. A total of 83 real-time TM packets were received with comet day count 118. Analysis of the combined real-time data and the sparse stored data relayed back to Earth during these six contacts showed that Philae’s internal temperatures were steadily increasing, the battery was being charged, and the comet days were getting longer, all consistent with increasing solar illumination and seasonal variations en-route towards perihelion on 13 August.

Rosetta’s trajectory vs Philae contacts
The six contacts between 13 and 24 June clearly provided much less data than hoped for, and the focus of the teams was on improving this situation.

During this period, Rosetta was at a distance of approximately 200 km from Comet 67P/C-G (and Philae), mainly as a consequence of trying to avoid problems with Rosetta’s star trackers in the increasingly dusty coma environment around the comet. Based on the telemetry parameters indicating link quality, this appeared to be at the very edge of the operational range for making good contact with Philae.

 

Philae’s antennas are aligned with the direction of the spacecraft’s +Z axis, i.e. pointing vertically up through the body. The greater the angle between Rosetta and +Z axis, the weaker the signal, and ideally, Rosetta would fly over Philae’s location within a 20° half-cone around +Z to ensure the strongest contact.

However, Philae’s orientation at its final  landing site, Abydos, is such that the +Z axis is in fact directed into the comet. In practice, this makes it impossible for Rosetta to fly over and see Philae at angles of less than 30° from the +Z axis, with clear consequences for the strength of the communications signal.

Another way to improve the signal strength is to reduce the distance between the two spacecraft: roughly speaking it should improve with the inverse square of the distance. Mindful of the concerns regarding the star trackers and the dusty environment, the RMOC team nevertheless pushed Rosetta towards its safe operational limits, stepping down the distance to the comet every 3–4 days in late June and early July. For example, the contact made on 24 June was at a distance of 180 km, while the next contact was at 155 km on 9 July, as described in more detail later in this post.

In addition to the varying distance, the trajectory of Rosetta and the rotation of 67P/C-G meant that Rosetta appeared over the region where Philae is thought to be, at different comet latitudes each comet day. Over these few weeks, the range in latitudes between 0° and +55°N was covered several times.

As communication between Rosetta and Philae did not occur every comet day, it was hypothesised that there was a dependency on latitude. However, contacts did not take place at the same latitudes each time, and thus local topographical features may also obstruct the signal in some cases, complicating the analysis. In addition, it is not clear why the contacts – when established – were so heavily interrupted. For the short periods during which a link was established, the signal was relatively strong, again making it hard to come up with a simple explanation.

Overall, it was proving hard for the team to come up with reliable predictions of when good contacts would be made.

Hardware issues and CONSERT switch-on
The limited data retrieved from Philae’s MM during the sparse contacts provided the engineers with yet another puzzle. The data contained several gaps and during the contact of 24 June, the real-time TM indicated that one of the MM boards was ‘empty’, although no data were actually transferred from it. In addition, the housekeeping data showed a higher than expected current on the MM input. Apparently the MM electronics boards were not functioning as reliably as before.

Furthermore, the real-time TM received during the 19 June and 20 June contacts showed that only one of the two receivers (RX2) was switched on, even though the configuration was set to use both receivers. On the other hand, data stored from the May period and downlinked on 20 June showed that both receivers had been powered on at that time, as expected.

Further inspection of the data gave a clear picture of the situation: RX1 had suffered a short circuit on its input and was switched off by the hardware overcurrent protection. The failure concerned the team, but this is the reason why the lander has redundant units available. At that time, there was no indication of additional hardware issues on the receiver side, i.e. all currents and voltages were nominal.

However, after a two-week silence following the 24 June contact, the team had to at least entertain the possibility that perhaps RX2 had also failed.

Failure of both receivers would clearly terminate the Philae part of the mission, as no commands can be sent to it and it cannot be instructed to do anything. Another possibility for the lack of contact in this period was that the signals being sent from Philae to Rosetta were too weak. The power of Philae’s transmitter is only half that of the one on-board Rosetta, and it might have been that Philae’s reply signal would be too distorted by the comet environment for Rosetta’s ESS to recognise it.

The team worked hard to find ways of excluding one or other possibility. For example, perhaps Earth-based observations with a very large radio telescope or one of the radio-sensitive scientific instruments on-board Rosetta might be able to detect a signal from Philae in the appropriate frequency range. After further study, this turned out not to be possible.

In the end, it was decided to try using the CONSERT instrument, designed to make measurements of the interior of the comet by beaming radio signals back and forth between Rosetta and Philae and, importantly, using separate, independent antennas from the communications system.

The problem, of course, was that CONSERT was not switched on. Fortunately, in addition to the standard way of establishing a link as described earlier in this article, commands can also be sent to Philae “in the blind” using the so-called TC Backup Mode (TCBM). In this way, a small number of commands are loaded into Rosetta’s ESS and transmitted repeatedly over several hours in a specific pattern that should cause them to be recognised and processed by Philae’s on-board software, without the need to establish the two-way link.

The idea was to use the TCBM to command CONSERT to turn on and then broadcast signals from the CONSERT unit on Rosetta towards the lander. If the CONSERT signals were received on-board Philae and sent back to and received by Rosetta, that would have confirmed that the RX2 receiver on Philae was working, otherwise the command to turn on CONSERT would never have been received and acted upon.

TCBM was designed as a fall-back solution to communicate with Philae in case nothing else worked, but it comes with limitations. In particular, in its current situation, Philae’s software is configured to focus on battery charging and maintaining internal temperatures, not to operate the scientific payload. Also, it is not possible to guarantee that all commands are received, while conversely, the repeated receipt and execution of the same command might cause problems. Therefore, before an attempt could be made to use TCBM to turn on CONSERT, modified commands (essentially software routines) to avoid repeated execution had to be developed, tested extensively, and validated on a ground reference model of Philae.

When Philae receives a command telling it to turn on e.g. CONSERT, the on-board software instructs the power subsystem to switch on CONSERT. In addition, it loads a sequence of telecommands from an internal memory that must be transferred to CONSERT at the right time and only once, the so-called timeline. Thus, a simple telecommand telling Philae to switch on CONSERT triggers several different actions. For this reason, the on-board software and CONSERT get confused if the same command is received more than once. This was avoided by integration of the command in a software routine, which checks if the command is received for the first time or not.

The first attempt to use TCBM to turn on CONSERT was made on 5 July, but failed as no CONSERT signal from Philae was detected. A second attempt was made on 9 July and was followed by a big surprise, as a full two-way link was made between Philae and Rosetta at 17:45 UTC. It lasted for a total duration of 22 minutes and with an uninterrupted period of approximately 12 minutes – the best so far! A total of 246 packets were received, all from that comet day, both real-time and stored in the MM.

Curiously, however, the CONSERT receiver on Rosetta did not detect a signal from Philae at the same time. The TM downloaded during the contact revealed that the commands sent in TCBM had been received and CONSERT had indeed switched on, but that its boot sequence had stopped after roughly 6 minutes, before CONSERT started transmitting a signal. The unit then remained on, but not transmitting.

As the engineers looked more closely at the data, the mystery deepened. These showed that TCBM commands were being received from Rosetta shortly after Philae booted when exposed to enough sunlight, but it still took 35 minutes before the two-way link was established.

In principle, the on-board software should switch on one of Philae’s two transmitters as soon as a signal is received, in order to establish the two-way link. However, if the link is not made within 3 minutes and signals continue to be received from Rosetta, the first transmitter unit is declared faulty, and the software switches on the other one.

It appears as though transmitter unit TX1 was commanded on briefly, but timed-out, causing a switch to TX2. No data were available from TX1 in that short window to confirm whether the unit was drawing current, i.e. actually transmitting. By contrast, once TX2 was commanded on, approximately 10 minutes elapsed before it too timed-out, but data taken in that period showed that no current was being drawn, suggesting that TX2 was suffering from a short circuit. This would cause a brief spike in the current after the unit was turned on, triggering a limiter and turning the unit off again. However, the spike would be too short to be seen in the current measurement.

When the switch back to TX1 occurred, it took the unit roughly 17 minutes to power up and finally begin transmitting to Rosetta, establishing the two-way link. It is not understood why it took so long to boot and whether this behaviour is reproducible, although it is known that the normal 3 minute time-out was over-ridden by the presence of TCBM signals: this resets the time-out, allowing more time for the unit to boot.

Based on analysis and the possible failure of TX2, the engineers decided to focus future efforts on using just TX1. This meant developing and validating a special software patch that would cause Philae to select TX1 only, in tandem with the TCBM signal to reset the time-out and provide enough time for the unit to boot.

The TCBM would be used to upload the patch, so no two-way link would be needed required. The patch would be stored in the volatile memory of the software, so it would need to be resent at every communication window. This was meant to allow switching back to the normal configuration without the need to again reconfigure Philae. Unfortunately, there have not been any further contacts since 9 July, so it has not been possible to confirm if this patch has been uploaded. It must also be noted that the distance between Rosetta and Comet 67P/C-G (and Philae) was increased due to the increasingly dusty environment as the comet approached perihelion.

Last excursion through the favourable latitude range prior to going south
As described above, Rosetta had been flown around the comet in a constrained latitude range after Philae first made contact in June, in order to maximise the chances of further communication with the lander. However, that meant that other regions of the comet of increasing scientific interest, most notably the southern hemisphere, were not accessible.

It was therefore agreed that after 25 July, Rosetta’s trajectory would change to allow it to fly over regions on the southern hemisphere, allowing the scientific instruments on the orbiter to investigate this part of the comet that had recently moved into local “summer”. That would however then make it impossible to contact Philae.

Until then, the Philae team decided to make use of the remaining contact opportunities between the two spacecraft to continue using the TCBM to send commands to the lander in the blind. These commands included the reconfiguration of the on-board software to initiate renewed scientific measurements, using the by-then charged secondary battery. This would have the advantage that once a new contact was made, new scientific data would be immediately available.

The downside is that due to the limitations of TCBM, only “pre-existing” on-board sequences could be commanded: for example, a 2-hour block of measurements by ROMAP, SESAME, MUPUS, COSAC, and Ptolemy, with a CIVA panorama image, such as the one performed on 13 November. More importantly, perhaps, the use of TCBM mode meant that the software configuration was being done without any feedback, potentially leaving Philae in an undefined status.

Current status and future outlook
On 25 July, Rosetta was moved towards the southern hemisphere of Comet 67P/C-G as planned. During the following weeks, communication between the two spacecraft was not expected to be possible unless there had been a dramatic shift in Philae’s orientation. Nevertheless, Rosetta’s ESS was kept on, transmitting signals towards the comet, and listening out for possible return signals from Philae.

Just before perihelion on 13 August, Rosetta flew back over the northern hemisphere again, but this time in the afternoon terminator plane, meaning that any possible contact with Philae would take place in the hours before sunset at the lander’s location, as opposed to the morning contacts made in June and early July. At that time, no contact was made.

Between mid-August and the beginning of September Rosetta flew over the southern hemisphere again, at a distance of about 400 km – no contact was expected during that period.

Since the beginning of September, Rosetta is once again flying over the northern hemisphere at a similar distance. Although the chances for contact at such distances are not very high, Rosetta will nevertheless continue to listen for Philae. On 23 September, Rosetta will depart on a far excursion trajectory to reach a distance up to 1500 km, which is incompatible with Philae contacts. However, the far excursion is scientifically extremely interesting as it will fly in the terminator plane and will be trying to detect the comet’s bow shock. Moreover, it was never anticipated that the Philae part of the mission would continue until after perihelion. (Editor’s note: we’ll provide more details about the 1500 km excursion in a later blog post.)

In parallel, the engineers dedicated the time since the last contact to conduct in-depth tests with the power and communication hardware in order to reproduce the observed behaviour. Together with the still ongoing data analysis, the pieces of the puzzle are being meticulously put together to reconstruct the scenarios as they were seen during the contacts. The goal is to define the most promising strategy to contact Philae, to be applied after the far excursion when Rosetta is expected to approach the comet at distances much more favourable for communication. From a thermal and power perspective Philae should be able to operate until the end of 2015, so the attempts to get a signal will continue at least until then. The impact of the increased comet activity around perihelion is clearly unknown: only time will tell if and how Philae survived this.

Many people from all of the Philae and Rosetta teams have worked very hard over the past few months to try and return Philae to full operational status, and these efforts will certainly continue. Current thinking is that the problems being experienced with Philae’s communications hardware are probably due to the very low temperatures experienced by the lander in the months immediately following its landing at the dark Abydos location.

But as communications were re-established on 13 June and then intermittently on several occasions since then, it is hoped that the constantly changing thermal conditions on Comet 67P/C-G will make it possible for the hardware to return to a more stable state, to re-establish contact, and to continue Philae’s unprecedented scientific measurements from the surface of the comet, a key part of Rosetta’s overall mission.

Summary of abbreviations used in this article:
ESS: Electrical Support System Processor Unit
MM: Mass-memory
TM: Telemetry
FIFO: First-In-First-Out
TCBM: Tele-Command Back-up Mode
RAM: Random-access memory
UTC: Coordinated Universal Time
LCC: Lander Control Center
SONC: Science Operations and Navigation Center
RMOC: Rosetta Mission Operations Centre
RSGS: Rosetta Science Ground Segment
DLR: Deutsches Zentrum für Luft- und Raumfahrt
CNES: Centre National d’Études Spatiales
ESOC: European Space Operations Centre
ESAC: European Space Astronomy Centre
SWT: Science Working Team