Scientists Develop Application to Control Robots Using Apple Vision Pro VR Headset.
Researchers have created an application for the Apple Vision Pro mixed reality headset that allows users to control a robot simply through head and hand gestures. This application can be used for remote control in various situations, particularly in disaster-stricken areas.
An application named “Tracking Streamer” tracks human movement – particularly how the head, wrist, and fingers move – and transmits this data via Wi-Fi to the robot for control. (Photo: Younghyo Park).
The application, called “Tracking Streamer,” monitors human movement, especially the movements of the head, wrist, and fingers, and sends this data over a Wi-Fi connection to the robot on the same network. The connected robot then translates the data into corresponding movements.
The researchers published their work in a paper recently released on Github, an open-source code repository. This system tracks 26 points on the hand and wrist, as well as distinct points on the head. It also records spatial data – for instance, how high you are above the ground.
Application developer Younghyo Park, a Ph.D. candidate at the Massachusetts Institute of Technology (MIT), shared a video on social media platform X demonstrating the system in action.
In a short video clip, co-author Gabe Margolis, a graduate student at MIT, controls a four-legged robot using his hands and body movements.
Margolis gestures to command the robot to approach a closed door, rotate the doorknob with a gripping tool, and enter. In another scene, he signals for the robot to open a trash can lid and toss something into it. At one point, he also bends down, prompting the robot to mimic his action.
Apple launched the Vision Pro VR headset in February 2024, aiming to assist users in their daily lives with immersive experiences. Just before the launch, scientists released research to investigate how well VR headsets perform when used in everyday scenarios.
They found that these devices have latency issues, which limit peripheral vision and cause distortions similar to a funhouse mirror. They cautioned against wearing VR headsets while driving or walking down stairs.
Park and Margolis speculated that some users might wear the Apple Vision Pro all the time, like glasses, which would provide more data to teach the robot how humans move, allowing it to follow suit.
