Overview
“Reimagine the unheard miracles in an alternate expression”
Imagine a world where no mellifluous birdsong, soft-spoken words, or majestic melodies exist. How poignant to miss these sounds! Yet, 448 million globally endure this quietude due to auditory impairment.
In 2021, a group of Tsinghua University students aspired to bridge this void. They sought to ‘play’ a symphony in lights, enabling those with hearing loss to partake in the symphony’s wonder, in a one-to-one sensory dance.
Role
Design Prototyper, Programmer
Team
Yuchen Yao, Wen Zhou, Yuewen Zhang
Duration
2021 Mar. (2 months)
Director
Yingqing Xu, Tsinghua Future Lab
Category
Accessibility, Visualization, Orchestra
Ethos
This is an exploration of how do we create accessibility for hearing loss people to engage in classic music.
HMW: How might we provide the visual feedback soundscape for hard of hearing people to experience music?
Concept
Experiments
Converting MIDI Music to Light Signal
To convert music into visual signals, we first extract the notes from MIDI format compositions. We have developed a Python class that encapsulates attributes like note duration and intensity, and can extract these attributes from MIDI music through methods.
Next, we linearly map the notes to the RGB color space and introduce a random factor to create a flowing effect in the music.
Smoothening Light Effects
To reduce sudden changes in lighting and create a flowing light effect, we applied a smoothing effect to the lighting.
Each frame is influenced by the previous 19 frames, processed through a convolution function, resulting in a new image. Due to the application of the convolution function, the lighting in this image will show more continuous changes.
Prototype
Special Performance
Using this prototype, a special performance was conducted in the Xinya Experimental Theater. This performance played the famous symphonic poem “Vltava” (Die Moldau) through the light installation.
The audience wore noise-proof earmuffs during the symphony performance and felt the music solely through the light show.
Special Thanks to:
Prof. Yingqing Xu
Head of the Future Lab
Tsinghua University
Jiaxiong Hu
for help with RaspberryPi
