Hyphen Hub community artist Janet Biggs partners with Shimon, the Musician Robot

By Malrey Head Wednesday, April 24, 2019 Atlanta, GA

original article from https://gtcmt.gatech.edu/news/shimon-savery-film-score

Robot Shimon can now add film composer to his list of accomplishments.

The marimba-playing robot’s talents are well-known. So much so, that a few years ago he was invited to compose for a film. He completed the work in the spring of 2018.

But despite all his talents, Shimon can’t work alone. Richard Savery was tapped to join him. Savery is a Ph.D. student at the Center for Music Technology in the College of Design at Georgia Tech.

He wrote the algorithms to give Shimon the knowledge to compose and produce the score for Space Between Fragility Curves. The film is part of a large-scale installation comprised of projected video images on two large screens. It combines scenes from a refugee camp and the Mars Desert Research Station in Utah.

Savery said that overall he worked about six months with Shimon to create the score, described at times as "minimal and a bit mysterious" at other times as "very lively." (View a clip: Courtesy of Janet Biggs.)

Savery is an experienced composer having written music for film and used technology to program for film. But writing the music and programming a robot to create this music was a first for him. In fact, he said he thinks this might be a first ever.

As a researcher at the Center for Music Technology, he said his work focuses on creating robots to help researchers answer questions about how artificial intelligence (AI) can help humans understand and create music and benefit robots as well.

Savery and Shimon Get to Work

Shimon was invited to compose the score by the film’s director, American artist Janet Biggs.

Savery said Biggs gave them great freedom to compose. To help them start, she gave Savery an early version of the film for Shimon to view.

It included footage she had made at the Mars Desert Research Station in Hanksville, Utah.

Savery said before creating any music, he first worked out how Shimon could watch the film. Shimon had done lots of computer vision material, and had done a some with a camera, he said. For this task, he programmed Shimon to track colors and people.

Savery said Shimon composed his music based on emotions and the people. He programmed Shimon to recognize people, such as Biggs, who is in the film.

Second was linking to musical generation. In that process, Shimon takes in other musical compositions and based on algorithms creates something like it. Other parts of his score are just rhythms based on algorithms, or computer rules, from Savery.

Working in the Couch Building at Georgia Tech, they spent about two days doing the final recording.

Savery enjoyed the experience, saying he was “really very happy with the way it came out, and had a great experience working with Janet.”

Before he started composing, Shimon watched a version of Biggs' film. Photo Courtesy: Janet Biggs

Biggs Meets Shimon’s Creator

Biggs said she wanted to work with Shimon after meeting Gil Weinberg, founding director of the Georgia Tech Center for Music Technology and the creator of Shimon, at a conference in 2016 in Cancun.

Weinberg, she said, gave an exciting talk about Shimon and the work he and his students are doing.

“I was so inspired, that I tracked down Gil during the conference and asked if he would be interested in working together. When asked if Shimon would be able to watch a film and compose a score for it, Gil’s response was, ‘Not yet!’,” she said.

But by fall 2017, Shimon was ready. The robot had been traveling quite a bit, and Weinberg said he was waiting for Shimon to return to Georgia Tech to allow Savery enough time to research and develop a worthy approach for interaction with Biggs’ film.

Savery has been working with Shimon since the fall of 2017 when he came to Georgia Tech. Shimon though has been around since 2008, and has performed around the world.

In choosing Savery to work with Shimon, Weinberg remarked on Savery’s familiarity with Shimon and his computational skills.

He described Savery as “an accomplished composer and an improviser with a strong record of previous work in music composition for films and games. He was therefore perfectly positioned to bring both his technical and artistic skills to create a novel approach for robotic musical composition.”

But this was the first film experience for Shimon, Weinberg said.

Biggs Incorporates Shimon Into Her Art

Biggs is known primarily for her work in video, photography, and performance. She lives and works in Brooklyn, New York. Biggs' work often includes images of individuals in extreme landscapes or situations.

She had been commissioned by two museums in the Canary Islands to create some new work. Separate to that, she asked Weinberg to work with Shimon. Savery said once he started working on it, it lined up with Biggs’ commission, so it all came together.

Biggs explained, “I was producing a new body of work that focused on the far ends of the spectrum of human movement in pursuit of new possibilities. I filmed at a Yemeni refugee camp in the Horn of Africa and at the Mars Desert Research Station in Hanksville, Utah, exploring my conviction that innovation, desire, and curiosity exist no matter what the given resources or support systems; that searching for the unknown is intrinsic to humanity. A hyper-technological marimba-playing robot with AI not only fit the project aurally and visually, but added to it conceptually.”

She then produced three video installations for the project, which is titled Like Walking on Mars.

It includes a two-channel piece, Space Between Fragility Curves, in which Shimon appears and performs his score; a three-channel installation, Weighing Life Without a Scale; and a four-channel work, Seeing Constellations in the Darkness Between Stars.

s

Savery Makes a Presentation

Biggs’ film project Like Walking on Mars premiered in the fall at the museums.

Savery attended the premier at the museums and gave a presentation, where he talked about the new algorithms he wrote to give Shimon the knowledge to compose and produce the score of Space Between Fragility Curves.

Despite Richard’s Work, Shimon is officially listed as the composer; Savery is listed as Shimon Musical and Visual Artificial Intelligence. But while Shimon is the composer, he could not do it without Savery.

Shimon also composed for another of Biggs’ films, Seeing Constellations in the Darkness Between Stars.

That music was played by Jason Barnes, an amputee who regularly works with Weinberg. A drummer who lost his lower right arm in an accident, Barnes now plays with a robotic prosthetic arm created at the Center for Music Technology.

While Shimon composed that score also, Barnes was able to improvise with his prosthetic arm, thus making their work more like a collaboration.

Asked if Shimon delivered what she wanted, Biggs said, “it was impossible to anticipate or expect specific results since Shimon made new compositional choices each time he watched my video.”

But, she said, “I couldn’t be happier with both the process of working with Richard and Shimon, and the final outcome.”