NASA astronauts Tracy Dyson and Jeanette Epps operated from space a team of robots on Earth in the third session of the Surface Avatar experiment, a collaboration between ESA and the German Aerospace Center (DLR).
The robot team, consisting of ESA’s Interact rover, and DLR’s Rollin’ Justin and robot dog Bert, worked in a simulated martian environment at the DLR site in Oberpfaffenhofen, Germany. The astronauts operated the robots remotely from the International Space Station. Controlling multiple robots simultaneously is an important step forward for future missions involving complex tasks in the extreme environment of space.
ESA’s Interact and DLR’s Rollin’ Justin exchanging samples during the session. Credit: ESA/DLR
Robots, assemble
This session of the experiment consisted of three different protocols where the astronauts had to operate the robots simultaneously to complete tasks set by the engineers at ESA and DLR. The tasks are designed to anticipate exploration experiments on planetary surfaces such as the Moon and Mars.
A first protocol made use of Bert, DLR’s robot dog. Tracy and Jeanette had to navigate the terrain to pick up screws with its magnetised arm and place them in a container. In a second protocol, the astronauts operated Justin to re-arrange sample tubes, but with the twist that the operation was made to fail on purpose by the engineers, to see how the robot would react and how the astronaut could solve the issue.
NASA astronaut Jeanette Epps operating the robots in Oberpfaffenhofen from the International Space Station. Credit: ESA/NASA
The third protocol, which took place in July, involved a more complex choreography of the robotic fleet. Instead of a list of steps to undertake, astronauts received a high-level task: use ESA’s interact rover to collect samples and bring these to a lander mock-up where Justin would be waiting to pick these up and analyse them.
“This protocol marks a significant advancement, showing that astronauts can operate the different robots simultaneously and are also able to use our interface to problem-solve throughout the task without help from the ground, a really important asset when you are far away in space and have a delay in communication,” says Rute Luz, the robotic engineer in the ESA team who created the protocol.
ESA’s Interact rover and its robotic arm picking up samples at Oberpfaffenhofen. Credit: ESA/DLR
The interface of Surface Avatar
Astronauts control the robots through an interface on their laptop – made from scratch by the combined ESA and DLR teams – using two controllers: a joystick and the Sigma.7 device.
The Sigma controller offers seven degrees of freedom – all directions as well as rotation – and haptic feedback that allows the user to feel when the robot’s arm bumps into an object. The device is made by the Swiss company Force Dimension and has applications in telemedicine for remote surgery.
The controller has already been tested in the microgravity conditions of the International Space Station by Samantha Cristoforetti and project astronaut Marcus Wandt in January of this year during Axiom Mission 3.
ESA project astronaut Marcus Wandt took part in a session of Surface Avatar earlier this year while on the International Space Station. Credit: ESA
“At the start of the experiment, we do a ‘haptics check handshake’ – someone on the ground shakes the hand of the robot on Earth and we check that the astronaut on the International Space Station feels it through Sigma.7,” shares Neal Lii, the principal investigator of the experiment from DLR’s Institute of Robotics and Mechatronics.
“This tradition helps the user get a feel for the device and the 850-millisecond delay between moving the controller on the International Space Station and seeing the robot move on Earth through their laptop. A handshake is also symbolic of the long-standing collaboration between the teams at ESA and DLR,” adds Thomas Krueger, who leads the Human-Robot Interaction team responsible for ESA’s part of the experiment.
NASA astronaut Tracy Dyson operating the robots in Oberpfaffenhofen from the International Space Station. Credit: ESA/NASA
The interface is designed to be as intuitive as possible, with live video feed from the robots and buttons to switch between the different views available, for example a first-person perspective from each robot or exterior to the robots. The user can click on pre-determined actions such as picking up a sample or use the controllers to manually move the robots and their arms.
Multitasking is also possible. The astronaut can make use of different capabilities and levels of autonomy in the robot, for example to control one robot directly while the other is completing a task autonomously.
The graphical user interface that the astronaut sees during the experiment. On the large screen, we see Justin’s perspective as it picks up the rock from the handover station. Below, we can see ESA’s Interact rover through Bert’s perspective. The interface also shows a minimap of the terrain and a command panel on the right. Credit: ESA/DLR
Next steps
Surface Avatar is an iterative process, with each session giving the engineering teams valuable feedback from astronauts on how to refine the robot-operation interface. In the next months, the engineers will discuss how to improve the process and what they want to test in the final session next year.
“Robotics play a crucial role in every phase of lunar exploration outlined in the Terrae Novae strategy. From initial uncrewed missions to later crewed missions, robots will provide essential support to astronauts. Looking ahead, fully autonomous robots and partially AI-driven rovers could handle tasks independently,” underlines Didier Schmitt, the team leader from the ExPeRT group which coordinates the development of new technologies for future exploration missions.
The success of these experiments paves the way for future space exploration, allowing future astronauts to operate robots on the lunar or martian surface from the safety of a lander or orbiting base, such as the Gateway.
ESA’s Terrae Novae exploration programme is leading Europe’s human journey into the Solar System using robots as precursors and scouts. Exploring space is about travelling farther and coming back with new experiences and knowledge to help us on Earth. Humankind will benefit from the new discoveries, ambitions, science, inspiration and challenges. Credit: ESA-O. Pâques