The SonicMove project of the University of Eastern Finland, VTT Technical Research Centre of Finland and Aalto University examines the use of human body movements in the synthesis of voice and sensation feedback. In cooperation with Dance Theatre Minimi, sonification is applied to a choreographic stage performance.
The latest work by Dance Theatre Minimi called Biodatasonaatti (“Biodata sonata”) premiered in Kuopio in mid-November. The data produced by motion sensors attached to the dancers is converted into sound that the viewer hears when watching the performance.
"Inertial sensors (IMUs) are used to continuously measure the positions and overall acceleration of dancers," says Project Researcher Paavo Vartiainen from the Department of Technical Physics at the University of Eastern Finland.
“We started the project by defining certain critical data streams for sound design. Minimi’s performance uses the same equipment as we use in measurement and analysis motion in our research. In many cases, the purpose of research is to accurately analyse the movement of body parts. As sonification was being developed for the use of the dance theatre, the requirement for reliable, robust and easy to use on-stage software was emphasised.
The use of motion sensors is not new in itself, but in Minimi's dance performance, sensor data streams were offered to Minimi's sound designer and visual designer without delay.
Mikko Makkonen, the artistic director of Minimi and the choreographer for Biodatasonaatti, sees plenty of opportunities in combining art and technology.
The artistic equivalent produced from data can be anything.
Mikko Makkonen
Artistic director of Minimi and the choreographer for Biodatasonaatti
"Various accelerations or position changes produce strong modulations. Music adds even more to the performance. However, it may be difficult for the viewer and listener of the performance to understand which voice comes from who”, Makkonen says.
An application was developed for Biodatasonaatti that receives the signals from nine motion sensors and transmits them without delay to the sound and light designer's software throughout the performance. The data produced by the dancers is so variable that the challenge in the sound design process has been producing repeatability and curbing the amount of data.
According to Vartiainen, modifying sound based on the position and movement of performers may also be a way of improving the accessibility of art. Wearable or handheld sensors can also be used to engage the audience. Especially by using sensors on smartphones, audiences of the stage performance could be involved in editing the sound of the performance. Such further development requires cooperation between measurement technology, sound designers and artists.
“Different types of motion sensors are used in sports and rehabilitation games, for example. One of the goals of the SonicMove project is to develop more realistic sound effects for games using motion sensors in a versatile manner.
Sonification can also be used in rehabilitation in the future; a few studies on its applications have already been conducted internationally. For example, promising research evidence on this has already been obtained in the rehabilitation of upper limbs post-stroke,” says Docent Hanna Pohjola from the University of Eastern Finland.
Sonic Move - Creative and Expressive Sonification of Human Movement
The Sonic Move - Creative and Expressive Sonification of Human Movement project jointly operated by the University of Eastern Finland, VTT Technical Research Centre of Finland and Aalto University examines the use of human body movements in the synthesis of voice and sensation feedback and its use in dance and rehabilitation applications. Body movements can be measured very accurately, for example with multi-camera systems. When motion data is combined with sound synthesis, sound environments can be created in which, for example, the speed of hand movement can be combined with the pitch of the sound. In this way, body movement can be sensed through both sight and hearing. The aim is to apply this multisensory solution to the project as a means of expression in dance art and possibly as part of neurological rehabilitation.
The project involves cooperation with professional dancers and choreographers. The aim is to produce dance performances in which voice and touch feedback are an essential part of the experience. A key part of the project is sound design, which, together with choreographers and dancers, aims to find different sound tones and styles as well as meanings for movements.
The sound and touch feedback solutions produced in the project can be utilised in the work of dance choreographers in the future and in the research and development of business partners. For example, the solutions can increase the immersion of dance and make the products of Finnish companies in the sector more competitive. In addition, the project brings together a cross-disciplinary expert group that benefits from each other.
Read more on Minimi’s blog! Part 5 of the blog is being published on January 2024, in this part, team members discuss their journey with this unique artistic process.
First set of performances of the Biodata Sonata was at Kuopio City Theatre on 15 November - 9 December 2023. Info on the upcoming performances by Minimi is found at https://minimi.fi/en_GB/events/main/programme/
