Last Thursday, a team of robots explored a simulated martian landscape in Germany, guided by an astronaut aboard the International Space Station. This was the final session of the Surface Avatar experiment, a joint initiative between ESA and the German Aerospace Center (DLR) to investigate how astronauts can remotely control robotic teams to carry out complex tasks on the Moon and Mars.
This latest session took place at the DLR site in Oberpfaffenhofen and introduced new levels of autonomy, decision-making and realism, bringing Europe one step closer to seamless human-robot collaboration in space exploration.
“Developing European expertise in robotic autonomy and teleoperation is essential for future lunar and planetary missions,” shares Benoit Pouffary, who leads ESA’s ExPeRT studies and technology team. “Surface Avatar is a cornerstone in building that capability.”
The four robots – ESA’s Interact and Spot, and DLR’s Justin and Bert.
Credit: DLR-E. Hellerslien
Challenge one: retrieving martian samples
The first part of the session simulated an exploration scenario on Mars to retrieve scattered sample containers across a rugged terrain. NASA astronaut Jonny Kim operated two robots – ESA’s robot dog Spot and DLR’s humanoid Rollin’ Justin – working together to retrieve and deliver the containers to a lander.
NASA astronaut Jonny Kim in ESA’s Columbus module on the International Space Station, performing the third session of the ESA/DLR Surface Avatar experiment.
Credit: ESA/NASA
NB: this image is a screenshot from footage taken on the International Space Station; no high-resolution version is currently available.
Spot navigated the environment autonomously to locate and collect the samples. Once a container was delivered to a handover station, Jonny used a combination of direct control and pre-set commands to guide Justin in transferring it to the lander.
This setup allowed Jonny to delegate tasks and focus on higher-level decisions, an evolution from earlier sessions where robots like ESA’s Interact required full teleoperation. The team also introduced new elements to increase complexity, such as a cover on the sample station that Spot had to identify and manipulate.
ESA’s robot dog Spot placing a sample into the handover station.
Credit: DLR-E. Hellerslien
The team deliberately made the task complex to test Spot’s ability to recognise its own mistakes, recover, and retry during autonomous operations. “Spot failed to grab a container but realised the mistake, tried again, and succeeded,” says Sant Brinkman, the ESA robotic engineer who developed Spot’s code. “That ability to detect and recover adds a new layer of robustness compared to earlier sessions.”
Spot also monitored the handover station, waiting for Justin to remove the previous sample before autonomously searching for the next one.
DLR’s Justin picking up a sample from the handover station.
Credit: DLR-E. Hellerslien
Challenge two: robotic speleology
DLR’s robot dog Bert inside the cave.
Credit: DLR-E. Hellerslien
The second scenario simulated a cave exploration. ESA’s Interact rover transported DLR’s small robot dog Bert to a cave entrance, where a boulder blocked the way. Jonny used Interact’s robotic arm to remove the obstacle and deploy Bert onto the surface.
A simulated malfunction followed: Bert detected a damaged leg after landing. Jonny had to retrain the robot dog’s walking algorithm in real time before Bert could continue into the cave, where it detected signs of martian ice.
This scenario tested how operators respond to unexpected challenges and adapt robotic systems on the fly. The astronaut was free to choose between immersive views and strategic planning, offering a glimpse into how future missions might balance autonomy and human oversight.
ESA’s Interact wheeled robotic arm.
Credit: DLR-E. Hellerslien
Intuitive design
Astronauts control the robot through a custom-built interface developed by ESA and DLR, using a joystick and a haptic-feedback device that can move and rotate in any direction. Jonny quickly adapted to the system, highlighting the importance of designing tools that feel intuitive and responsive.
The interface offers both a first-person perspective and a top-down map view, giving the operator flexibility between immersive control and strategic oversight. “Inspired by real-time strategy games, our interface lets users smoothly switch between views to teleoperate one robot while delegating tasks to more autonomous ones,” explains Rute Luz, ESA robotic engineer and lead on the interface. “Having two robots move simultaneously was once our dream, but thanks to the autonomy Sant implemented in Spot and the new user interface tools, this capability is a reality and a major step forward for the experiment.”
Interface that the astronaut uses during the Surface Avatar experiment, here in the top-down strategic map view. On the bottom are the camera views the user can also switch to, as well as the commands that can be used.
Credit: ESA-Human-Robot Interaction team.
Physical design also played a key role in ensuring smooth operations. “We designed the sample tubes to remain upright no matter how they were rotated,” explains Emiel Den Exter, the ESA industrial design engineer who worked on the experiment hardware as well as the user interface. “This stability makes it easier for the robots to handle the samples reliably.”
ESA’s robot dog Spot picking up a sample.
Credit: DLR-E. Hellerslien
Behind the scenes, ESA Graduate Trainee Nicole Roshardt ensured stable and secure communication between the robots on Earth and Jonny on the International Space Station. “We had to make sure the connection stayed reliable despite delays or data loss, and that the system remained protected throughout the session,” she explains.
What’s next?
Over the course of four sessions, the Surface Avatar team has refined its approach to human-robot interaction, improving both teleoperation and the delegation of tasks to increasingly autonomous systems. Drawing on astronaut feedback and engineering insights, the experiment has helped define how robotic teams can be managed effectively as we explore space.
“Surface Avatar has been a true team effort, combining astronaut experience, robotic engineering and interface design to push the boundaries of human-robot collaboration,” says Thomas Krueger, ESA robotic engineer and lead of the experiment. “Each session has helped us evolve the system, making it more capable, flexible and ready for future exploration scenarios.”
Joint ESA/DLR Surface Avatar team with the four robots.
Credit: ESA-F. Malavasi
The team will return to ESA’s LUNA facility after summer, where Spot and Interact will be tested in lunar-like conditions, including dusty terrain and simulated spacesuit operations. These tests aim to define a catalogue of tasks for future Moon missions and assess how robotic systems perform in more realistic environments. The team is also exploring collaborations with ECSAT, ESA’s site in the United Kingdom to expand into geology and spectrometry analysis. Meanwhile, the haptic controller used in the experiment remains aboard the International Space Station and could support future remote operations.
As space agencies prepare for the next era of exploration, the ability to coordinate intelligent robotic teams will be essential. The technologies developed through Surface Avatar are designed to support a wide range of mission scenarios – from surface exploration to remote tasks in challenging environments – laying the foundation for astronauts and robots to collaborate effectively in future missions.
An astronaut accompanied by Spot at the LUNA inauguration show. Credit: ESA/DLR.
Surface Avatar experiment sessions:
Preliminary session: Samantha Crisoforetti, 2022
Session 1: Frank Rubio, 2023
Extra session: Marcus Wandt, 2024
Session 2: Jeanette Epps, Tracy Dyson, 2024
Session 3: Jonny Kim, 2025