foraudiovisual
audiovisual / stereo audio
⟪KOREANISCHE PERSPEKTIVEN⟫
Elektroakustisches Konzert,
HFK BREMEN,
Bremen, Germany
(Research grant from Seoul Foundation of Arts and Culture)
People fear what they don't understand.
Some people fear not being understood.
Beings with otherness are mainly represented as monsters. Their voices have been considered incomprehensible. A new language is needed in order to reveal their true selves that have been hidden by their bizarreness and to properly speak out. After dismantling the existing language system’s [consonant + vowel] structure, the part corresponding to the consonant is replaced with sound to create a new language form of [sound + vowel], 'Monster Voice'. In this audiovisual piece, excerpt of the Maja Lee Lanvad’s poem She is angry was used as the text content.
• Techniques Employed
SuperCollier
Ableton Live
TouchDesigner
Blender
Production Details
By creating a sound-based language called “monster voice,” I aimed to provide a new language for those who have been labeled as “monsters” to have their voices heard. This new language, which replaces the [consonant + vowel] structure with a [sound + vowel] structure, is a crucial tool for representing the voices of various minorities, especially women. An audio-reactive audiovisual film was created, featuring a 3D object that spoke excerpts from the poem “She is Angry” in the new language of 'monster sounds.'
As someone with a chronic pain disease, the experience of trying to communicate my pain to others made me question whether it is possible to communicate pain and whether the language I speak is understood. This personal experience extended to the question of how to effectively communicate the voices of those considered abnormal, outside of the normal group, in a social structure where the voices/opinions of those considered normal are often accepted. The idea was to create a language for monsters, not the language used by the people who were used to treating them as monsters, and to use it as an example of how to use research to communicate that difference is something to be celebrated, not feared.
I wanted to create a new language by replacing the 'consonants' with monster sounds without breaking the existing language system of consonants and vowels. The sounds of 'monsters' have been considered animalistic, supernatural, hysterical, and barbaric, so I used sound-synthesized electronic sounds as consonant sounds to make them aggressive and strong.
I chose to work with audiovisuals rather than sound because the description of the monster's appearance has been largely visualized. I wanted to create a relationship between the viewer and the speaker in the audiovisuals by directly presenting the monster. Also, I wanted the text spoken by the speaker in the video to be consistent with the theme of the work, so I used excerpts from the poetry book “She is Angry” by Korean-Danish poet Maja Lee Langvad.
• Goals
- Creating a new kind of language by organizing consonants and vowels in a musical way to create monstrous sounds.
- Consonants: using noise to represent other, often considered wrong, sounds.
-
Vowels: using formant synthesis to recreate the sounds of existing vowels.
-
Translating text into the new sound language: the text itself becomes music, with only the vowels sounding from the existing language, creating a structure where the female monster reads it.
-
Creating an audiovisual piece in which the artist's 3D-modeled face responds to sound.
-
Distorting facial elements to respond to different sound parameters.
The sound synthesis for the consonants involved a process of synthesizing different sounds using SuperCollider. This approach led to the creation of a set of sound objects (SynthDefs) that correspond to each alphabet consonant. These sound materials were then used in Ableton Live to compose the musical elements of the song concept, employing basic tape processing techniques such as pitch shift, time stretch, ring modulation, and more.
The vowel sounds were synthesized using formant synthesis techniques. The existing formants of a, e, i, o, and u are widely known, and the goal was to synthesize these vowels by breaking them down phonetically, a process that required linguistic exploration to find formant frequencies and bandwidths.
Both consonants and vowels were created based on the International Phonetic Alphabet (IPA) chart and based on how they are pronounced. For instance, the first sentence, “She is angry.” was transformed into “ʃiː ɪz ˈ æŋgri.” to synthesize the sounds corresponding to each consonant and vowel.
• Creating Interactive Visuals
The female monster figure, serving as the audiovisual narrator, was created from a 3D scan of my own face. The 3D scan was transformed into a looping FBX object using Blender, making a grimace and angry expression.
Most of the work was done in TouchDesigner, where the 3D objects are increasingly deformed with noise. The deformations in TouchDesigner are cued to match the parts of the audio track, divided by text sentence numbers from the excerpted poem. To achieve this interaction, I used a native plugin, TD Ableton, that directly integrates Ableton Live and TouchDesigner. This plugin enables the detailed parameters of the visual noise in each scene to change in response to the sound parameters in Ableton. For example, for audio produced in stereo, the left face increases in size as the left channel volume increases, and the right face increases in size as the right channel volume increases, making the stereo experience intuitive.
• Score
*The original text and its phonetics are shown below.
She is angry.
ʃi ɪz ˈæŋgri.
She is angry at herself for being angry.
ʃi ɪz ˈæŋgri æt hərˈsɛlf fɔr ˈbiɪŋ ˈæŋgri.
She is angry at herself for blaming herself that she is angry.
ʃi ɪz ˈæŋgri æt hərˈsɛlf fɔr ˈbleɪmɪŋ hərˈsɛlf ðæt ʃi ɪz ˈæŋgri.
SHE IS ANGRY to be told she is angry.
She is well aware that she is angry.
She just not need to be told.
ʃi ɪz ˈæŋgritubitoʊld ʃi ɪz ˈæŋgri.
ʃi ɪz wɛl əˈwɛr ðæt ʃi ɪz ˈæŋgri.
ʃiʤʌst nɑt nid tubitoʊld.
There is nothing wrong to say that she is angry.
Who would not be angry if they had the same insight that she has?
ðɛr ɪz ˈnʌθɪŋ rɔŋ tuseɪðæt ʃi ɪz ˈæŋgri.
huwʊd nɑt biˈæŋgri ɪf ðeɪhæd ðəseɪm ˈɪnˌsaɪt ðæt ʃihæz?