In a neuro-robotics lab at the University of Southern California, a small mechanical hand heard a melody for the first time and played it back in a single attempt, without any sheet music, pre-loaded scores, or weeks of supervised training and practice (via USC Viterbi).
The system is called the Musician Hand. It has four fingers, each moved by a tendon connected to a small electric motor, mirroring how muscles actually pull tendons in a human hand. It was built by doctoral candidate Hesam Azadjou under the direction of Professor Francisco Valero-Cuevas.
How did the robot actually learn?
Through a technique called “motor babbling,” the exploratory trial-and-error process by which humans learn to control their limbs. Without knowing about the concept of hands, keyboards, and music, the robot discovered how its movements could produce sounds by pressing the keys of a piano.
For the first two minutes, the robotic hand pressed random keys, analyzing the relationship between finger movements and the sounds produced by the piano. From that brief experience, it mapped the connection between motor commands (the finger movements) and the results (the sounds).
When the Musician Hand heard the melody of around 30 notes for the first time, designed by Los Angeles-based composer and music theorist Richard Tuttobene and called “Robo Algo,” it converted the audio into a spectrogram, used neural networks to identify the notes, and then generated the required commands to reproduce the sequence, right on the first attempt.
What makes this more than a musical party trick?
The researchers also conducted a blind audition, wherein two judges assessed the Musician Hand alongside four trained pianists without knowing which one was the robot, and there were times when they couldn’t tell the difference between the humans and the robot.
By contrast, untrained adults couldn’t even replay the first dozen notes. Azadjou points out that the human brain solves complex motor problems by using less than 100 watts of power, which is roughly equivalent to a laptop charger, but conventional AI might need megawatts of power for the same task.
This kind of lean, efficient, and experience-based learning could actually be applied to robotic devices people can wear and use on a daily basis, such as exoskeletons for Parkinson’s patients and personalised rehabilitation robots. The research was supported by the NSF and DARPA.