AJMF9: Robotic Musician Does More Than Keep Beat

Georgia Tech’s Gil Weinberg contemplated ways to program machines to produce acoustic sounds and improvise and play music, then combined his background in computer science with his passion for music to create a robotic musician, Shimon.

Shimon’s improvisational skills will be featured Friday, March 9, at the Atlanta Science Festival’s opening night, an event incorporated into the ninth Atlanta Jewish Music Festival. The evening will showcase Shimon’s first performance with a rock band, which will include a drummer, a guitar player and Weinberg on bass.

“I really like the fact that the festival is expanding its arts and music and think it’s great to feature the program at Georgia Tech’s Center for Music Technology,” said Weinberg, who serves as the center’s founding director.

Get The AJT Newsletter by email and never miss our top stories Free Sign Up

As a musician, Weinberg detected that his music was missing acoustic sounds. To fill the void, he began experimenting with different algorithms until he came across body cognition. The program enables robots to generate acoustic sounds fabricated from body movements and appear as if they feel and connect with the music.

Weinberg’s first prototype, Haile, plays various percussion instruments, such as the power drum, and recognizes rhythm, tempo, beat and syncopation. After he posted a video of Haile online, the National Science Foundation reached out to Weinberg to submit a proposal to develop a robot to help people appreciate and understand robotics.

The proposal led to the creation of Shimon, which is an extension of Haile with the ability to recognize melodies, harmonies and visual cues.

Weinberg began developing Shimon in 2008. He added arms in 2009 and a bobbing head with a camera inside in 2010, enabling Shimon to recognize human movements.

Even after 10 years, Shimon remains a work in progress, Weinberg said. Mechanical challenges such as programing the machine to do things that humans can’t were among the biggest hurdles.

“Identifying where we can create something unique, where we can’t compete with humans and where we will have robots that inspire and push us to uncharted territories is tricky because we don’t want people to think we are replacing musicians,” Weinberg said.

To update Shimon’s programs, Weinberg invites students every year to create algorithms for him and program different ways the robot can interact with people.

Shimon has several demos that portray his abilities, such as his responses to human movements through physical cues. For example, if Weinberg raises his arm high, Shimon expects that a loud noise is imminent.

Based on the analysis of improvisations by jazz musicians such as Thelonious Monk and Miles Davis, Shimon can generate a sequence of notes that jazz musicians play in a particular style. The improvisations are built on rules but also on data, Weinberg said, which is known as “style modeling.”

“I can tell Shimon to play me something that is 30 percent Monk, 40 percent Coltrane, 10 percent Miles and maybe 20 percent my own music, and Shimon will respond and create a mash-up that is very interesting and unique,” Weinberg said. “This is really exciting for me because we can play and create genres that have never been created before.”

Weinberg does not focus on concerns either that machines could be a threat to humans in the future or that increasingly intelligent and aware machines could be treated unethically by people.

“I know some people say that artificial intelligence will take jobs and that it is dangerous, but whenever I think of different sciences, I focus on whether a human is capable of performing a certain act, and if not, can they be surprised, inspired and learn from it?” he said. “This is the most interesting aspect of robotics for me and the direction I choose to follow.”