June 26, 2019 –
Mari Kimura uses the MUGIC™ device while performing
When the classically trained violinist Mari Kimura plays her instrument these days, her masterful bow hand creates sound from two sources: an acoustic violin, and a wireless motion-sensor device known as MUGIC™.
Kimura’s concept of using a motion sensor for performance was born in Paris, at the Institute for Research and Coordination in Acoustics/Music, where Kimura was a composer in residence in 2010. She collaborated with engineers to develop the system of motion tracking for a violinist’s bowing, enabling the performer’s arm motions to provide control of a computer’s musical activity.
Kimura then took her creation to New York for help from a media artist and fashion designer, and finally landed in California at UC Irvine’s CALIT2. Kimura has shepherded MUGIC™’s development and proof of concept for nearly a decade.
Her device is comprised of a glove-worn wireless sensor linked to a mobile software interface. The system interprets movements and translates them into performance actions. Kimura wears the glove on her bow hand, where the sensor extracts and interprets expression from her gestures while she plays the violin. It seamlessly and intuitively can change the tempo and pitch of the recorded accompaniment coming from the computer. Modulating accompaniment, adding sound effects, controlling images and video on projection systems, and managing stage lighting are a few of the applications of the MUGIC™ system that have been demonstrated and used for real performances.
The Japanese-born violinist became a professor of music in UCI’s Claire Trevor School of the Arts Integrated Composition, Improvisation, and Technology (ICIT) program in 2017. Kimura is a pioneer composer, programmer, researcher and performer of music involving the interactive use of computers in live performance. And she is at the forefront of violinists who are extending the technical and expressive capabilities of the instrument.
In the early ‘90s while a student at Julliard, she invented “subharmonics,” an extended bowing technique that allows the performer to play notes below the violin’s normal tonal range without changing the tuning. She currently serves as the founding chair of the Future Music Lab at the Atlantic Music Festival and has taught a graduate course in interactive computer music performance at the Juilliard School since 1998.
“Mari came to us with this idea of trying to use motion to augment musical performance,” said Michael Klopfer, CalPlug technical director and faculty mentor for two multidisciplinary program (MDP) teams that have worked with Kimura on MUGIC™. The MDP program is a collaboration between CALIT2 and the Undergraduate Research Opportunities Program (UROP) that gives undergraduates an opportunity to experience hands-on multidisciplinary research.
Kimura came to the MDP program with demo boards wrapped in a glove. With Klopfer’s guidance, students developed a common printed circuit board (PCB) and low form-factor housing, which provide long battery life and a reliable connection while allowing the device to be worn comfortably during performances. “The device has come a long way,” Klopfer says.
Taking ideas from concept to early production is how we bring the dreams of researchers to reality at CALIT2,” he adds. “We have developed products using music as a tool to heal and expanding the impact of music is an area of interest for us.”
Working with Kimura and Klopfer, the MDP student teams have been deeply involved in the development process. The 2018 team helped with the device hardware and prototype development, while the 2019 team is involved in software and user interface development as well as the design-for-manufacture process to prepare the device for commercial production.
Electrical engineer Andrew Dylan Begey ’18, project lead of the first MDP team, was responsible for designing the device’s initial printed circuit board. “Our goal was to integrate several different commercial products all onto a single PCB and add more functionality, such as inductive battery charging and battery-level monitoring. In the end, two revisions of the device were completed and proved functional.”
Begey says working on MUGIC™ helped him obtain his current position at Northrop Grumman. “Not only were they impressed by the PCB that I brought to my interview, but I also learned important skills related to working on multidisciplinary teams toward common project goals. I have done multiple PCB schematic and layout designs at my job, which I learned in this project.”
Kimura says CALIT2 is helping her make a better, faster, smaller device. “There are functions that I don’t have and I want.”
Klopfer and Mark Micchelli, one of Kimura’s graduate students, are working with the second MDP team on the software side to develop more features to allow enhanced control capabilities. These include allowing the user to customize settings via a smartphone, and configure and use the device right out of the box.
Klopfer says, “We are taking information gathered on our first prototype and using it to develop the next iteration and give Mari a viable product ready to market.”
Kimura participated recently in an experimental performance project called Your Ocean, My Ocean (YOMO) where she played her violin and MUGIC™. The 80-minute performance featured dancers and musicians onstage, with a larger virtual cast appearing in video projections, integrated with footage of ocean and coastal locations and fragments of animated films, all evoking aspects of natural beauty and environmental degradation. Kimura improvised her violin playing in a duet with prerecorded sounds of a breeching whale, using MUGIC™ to control the whale sounds. Following a successful run at UCI’s Contemporary Arts Center in February 2019, the project was presented at Brown University’s Granoff Center last month.
In the meantime, Kimura is working with the other arts disciplines to explore MUGIC™’s potential for additional aspects of performance. “I am working with theater people, dancers, directors and other musicians, including Annie Loui, a professor of drama at the Claire Trevor School of the Arts and an expert on mime.” One of Kimura’s graduate students who is interested in theater and hand motion will work on the project with her and Loui next quarter.
UCI music professor Christopher Dobrian, a composer of instrumental and electronic music, says that a number of music faculty are working on the issue of real-time interactive use of computers in live performance, but Kimura is the only one working with a sensor and integrated control system. “Obviously she has to do all these things to make the violin sound properly, but she also has to be conscious of what her movements are doing in the way that she’s got it mapped to what the computer does. I think that’s a really fruitful area of research, trying to see how the movements that one makes when naturally playing an acoustic instrument can be sensed and translated into meaningful control of other sounds.”
“I am passionate about this project,” Kimura says. “I want others to have a good time with it.”
— Lori Brandt
