“sound AS space” — Interview with Gerriet K. Sharma from spæs lab Berlin

Please tell us a little bit about your research, your journey as an artist/technologist, what first got you into spatial audio and how you arrived to start spæs Lab.
 

I was trained as a classical guitarist from an early age, playing solo and in all kind of bands and improvisation ensembles as a teenager. Jazz and Improv with electrified strings and electronics also became the ingredients of my performance practice.

The guitar was my main instrument and research tool that helped me, for so long, hold contact with the world “out there”. In the middle of the 90s, this connection was suddenly disconnected, voiceless, and mute. It did not make any sense to me to play the guitar anymore. There was no resonance that I could pick up and articulate. Thus, I had to put it away and never since took it up again. I also stopped listening to recorded music for quite a while. 

However, at the beginning of the 2000s, sound art and installation, with its inherent motivations to create and interact with spaces, caught my attention, so I started with sight-specific sound interventions and went to the Academy of Media Arts in Cologne where I also introduced myself to programming and multichannel-loudspeaker spatialisation of sound. I started experimenting with WFS, V-BAP and all kinds of irregular loudspeaker set-ups and was overwhelmed by the potential. However, from the beginning, I was also very critical of the huge gap between technical achievements, marketing myths and artistic utopia. 

From then on, synthesis was at the core of my music, even when thinking about conventional instruments. It also became clear to me that space as a parameter of music is not only the most fascinating aspect in terms of artistic conception but also in regard to the social implications of sound and music in the 21st century.

I started touring and recording with electronic improvisation ensembles, was involved in audiovisual installation collaborations, and worked for TV, theatre and cinema.

During my computer music and composition studies at IEM/KUG, I  regularly worked with loudspeaker half-spheres and Ambisonics. These exercises helped me to develop my personal artistic approaches. In parallel, I further investigated ideas of the sculpturality of sound and the question of the instrumentality of loudspeaker arrays. 

In Graz, scientist and acoustician Dr. Franz Zotter introduced me to the  Icosahedral beamforming loudspeaker (IKO) prototype. I had the immediate impression that the potential of this tool, conceived originally as a reproductive playback system, could be artistically immense and instrumental, one that could add a spatial quality to electronic music and sound that was not just sensational and technologically state-of-the-art but also aesthetically intriguing and very challenging in its ephemeral presence.

As a matter of fact, it was the first time since I had to let go of my former instrument that there was something that could resemble what was lost and missed for so long. To understand this tool better, I had to read and learn everything available at the time (2009) about spatialisation with loudspeaker arrays to be able to work in close contact with scientists and engineers for the coming ten years.

After my PhD “Composing with Sculptural Sound Phenomena in Computer Music”, I was appointed Edgard Varése professor at the Electronic Music Studio of the TU Berlin, where I started teaching my experiences and approaches in a programme that I called “Spatial Practices”. It was a multidisciplinary class with students from music, sound studies, musicology, engineering and architecture and colleagues involved from composition, media theory, geography and cognitive linguistics.

Therefore, the Lab for Spatial Aesthetics in Sound (spæs) Berlin was the consequent continuation of these experiments and events. In 2020 we founded the lab with studios at an old firefighting station in Wedding and the former East German radio station, Funkhaus Berlin.

© spæs Lab Berlin

In 2020 we had the chance to build the labs in two locations in Berlin.

spæs identifies with multiple challenges in the advancement of spatial media. These challenges are related to the creation and design of media content and the development of tools, technologies, techniques, and methods that are being used.

Ultimately, spatial audio technologies are just the tools to translate ideas into concrete listening experiences—a too often forgotten fact—but spatial sound is rarely discussed, evaluated, and reflected with all the multifaceted dimensions of space. Aware of the structural limitations of academic and commercial institutions, we are establishing an independent lab dedicated to the interdisciplinary study of spatial aesthetics in sound.
We were stunned at how many people made contact after a very short time proposing collaborations, and applying for residencies and internships, and I really hope we can establish something like a community for artistic research in this field.

© spæs Lab Berlin

You dedicated a lot of studies around the IKO loudspeakers system during your research at IEM in Gratz (AUS). Let’s start by explaining what is IKO and its potential.
 

I was working with multichannel systems like WFS and Ambisonics half-spheres for a while. One day, I stumbled into a presentation for audio engineering students of the so-called “icosahedron loudspeaker” at the Institute for Electronic Music and Acoustics Graz. I was mesmerized by the sculpturality of the IKO-sound objects. They remind you of the phenomena you can create with other multichannel systems. At the same time they are very different as you are not surrounded by loudspeakers, and your perception is triggered by the activated space itself. One does not longer point the finger to the source as a point, a direction, or merely move from A to B, but can describe an area with both arms and hands. One can observe fields of different sizes and gradations and reconstruct cloud formations and streaks of penetrating sounds. These extensions can be shaped and leave a three-dimensional impression in auditory perception. 

This loudspeaker array consists of a wooden icosahedron (5th platonic solid) with a loudspeaker chassis placed on each of its 20 sides. The cables are located within the housing. The object, mounted on a stand, can be flexibly positioned in a room and is connected to a sound-generating computer system via a multicore cable with a converter and amplifier unit. 

The IKO provides bundled sounds in completely freely adjustable spatial directions. This sound bundling should initially introduce an improvement in the quality of acoustic measurements. Using the holophony, natural sound generators can be simulated and the paths of the sound reflection modelled. This result was born from the initial intention of simulating musical instruments in their lower registers along with their omnidirectional emission, both tonally correct and strongly. In addition, the resulting device is also suitable for the use of innovative acoustic room surveying. Since 2016, the Graz-based start-up company sonible has taken over series production and international distribution after developing a revised prototype.

By applying the beamforming algorithm developed by Franz Zotter in Ambisonics (3. Order) it is possible to project tightly bundled sounds onto floors, ceilings and walls so that one not only hears sounds at or from the source itself (such as a violin or most radio devices), but at projected and reflective point (mirror source). Here, individual loudspeakers are weighted by superposition so that sound projection in a desired (main)-direction is possible. Constructive interferences in the desired direction are formed during sound propagation, while the individual signals superimpose themselves where little or no sound propagation is desired. These "beams" can be freely adjusted not only in their angles but also in their position. Thereby, the mirror sources may even be mobilized and superimposed on the reflective surfaces.

In 2009 I started to develop sculptural sound models and musical pieces for IKO. This work motivated the scientists at IEM to investigate the phenomena IKO produced, as they were not sure how to explain and describe them. Our mutual interest in developing IKO culminated in the 4-year research project “Orchestrating Space by Icosahedral Loudspeaker” (OSIL) from 2014 – 2018 funded by the Austrian Science Fund FWF.

In this period, I constantly worked on compositions with IKO for different architectures and in dialogue with other instruments and performance 

situations. Today, IKO is my main instrument. Even when I compose for 8-Channel rings or half-spheres, I start thinking about the sculpturality of my compositions from an IKO perspective, and quite often, I start with ideas on IKO now and transfer them to other loudspeaker environments.

Research was always part of artistic development. However, new digital technologies liquefied almost all aspects of music making: tools, instruments, working environments, venues, audience situations, composition and performance. Space as a parameter of music was also always a subject. However, we have tools now available that help us to investigate and create acoustic spaces in detail and reliably in terms of CPU, mobility, and reciprocity, and we can record, store and archive these phenomena for further investigations and form and connect knowledge bases, highly important for the development of aesthetic concepts, i.e. personalized musical statements rather than sounding demonstrations of technically advanced tools.

However, originally spatial audio was created for the reproduction of a certain idea of “reality” or even “naturalness”, whereas art and music produce cultural objects and environments, quite likely very different from this idea. 

To understand the influence and power of all kinds of technologies on our lives (e.g. transport, communities, entertainment), it is crucial that as artists, we learn the fundamental concepts, impulses and agendas behind these developments as our work is concerned with the consequences. We have to make ourselves familiar with the tools and turn them into instruments.

Thus, knowing technical details, scientific fundaments, and historical backgrounds are crucial for contemporary and future artistic expressions. Like in the past, when painters and sculpturers started to study muscles and tissues, we must dissect the fabric of a mediatized world to develop artistic expressions as variations and catalysts of these concepts. 

Please note: I am not saying the guitar, the violin, and the drum kit are obsolete! However, instruments resonate with the world in which they are used. Once the world starts changing its properties, behaviours, shapes and sounds, different instruments and artistic utilisations of those are necessary to observe and create this resonance. By this, we are trying to understand the potentials and variabilities of human perception - better.

There are many fields, to name only a few: Sonification has been around for a while. The fact that our hearing can detect the slightest frequency changes in sounds might help understanding vast data streams better than by visualizations, the common graphs and screen abstractions. The sonification tools are advanced, translating data into sound textures. Spatialisation of these textures is almost unexplored in this field, and I am convinced this would open a new dimension for how we conceive and perceive sonification and, therefore, complex data.

IKO, with its ability to activate and reveal acoustic properties of buildings, could also be a great tool for the better understanding of architecture. Nowadays, you can simulate all kinds of visual aspects in software programmes. However,  the simulation of the acoustic properties of a building supporting the design (!), not just emulating the acoustic consequences of visual decisions, is highly underdeveloped. Actually, a huge field is yet to be explored in aural architecture.

Car industry has been aware of sound as an important design and construction enhancement factor for decades. Therefore, you find the most advanced sound research labs in the high-security facilities of these companies. 

Moreover, sound also became a subject of the health and well-being industries. Our world is undeniably getting louder almost every month! We urgently must find ways to reduce the loudness and certain frequencies in our living environments. Thus, knowledge of how sound can be absorbed, masked, eliminated, and composed is fundamental for any scientific research in these fields. Only imagine developing different ways of making concrete for road surfaces based on the idea of sound reduction, absorption and spatial sound composition. Tuning the asphalt as if it is a spatialized music track is possible.

© spæs Lab Berlin

Absolutely, but be careful here. Both disciplines tend to believe that they (alone) can explain the world. Sometimes you find even representatives from both disciplines thinking they are working on something like a universal truth, whereas it is always about finding out what is less false. An artistic investigation is quite likely something very different from a scientific one. Scientific discourse seeks to eliminate ambiguity in its terminology and definitions. An artistic discourse would on the contrary often seek to be as polyvalent as possible, suggesting a network of meanings or implications. Thus, the scientific ideal is more often than not alien to an aesthetically oriented discourse. However, there is also a need for some intersubjective agreements in the aesthetic field so that musical phenomena can be meaningfully discussed in words. We are talking about two diametrically opposed cultures: on the one hand, we find the exact sciences of acoustics, computer science and engineering sciences, which primarily define the conditions for sound reproduction and develop corresponding tools for the composition. 

On the other hand, there is a culture of perception of music with the ear. While the first is strongly characterized by clearly defined terms and corresponding verbalization concepts shared by a community of specialists, the aural, musical aspect,  which yields musical thinking and musical perception, cannot fall back on consistent terminology, as far as loudspeaker music and—here especially—space-sound composition are concerned. Thus we have a serious communication problem.

© Urbiks Music Berlin

However, the question of how people are affected by certain (spatialized) sounds is hardly investigated or is based on cinematic or orchestral cliches. Here, artists working in the field could contribute offering fundamentally different approaches. With practices and theoretical knowledge from their sometimes unconventional or unpredictable working processes, they can contribute to debates normally based on measurements not concerned with the perceiving body living outside of the laboratory (or simulation). Aesthetics is not something that comes into play when you are thinking of operating on your nose or cheeks. It is the field taking into account that human beings make experiences with their senses (!) and base their knowledge and decisions e.g. composing, performing, archiving - on these experiences. Example: When you take the tech sheet of a new microphone, it might perfectly match the sheet of a microphone built in the 50s. So, it is statistically true you can use this microphone instead as long as you put it in the same position. However, experience in the field will tell you that the microphones from the 50s sound very different. This opinion is not just superstition but something that comes from years of experience in the field and needs further and alternative  investigation. 

I was talking about Sonification earlier. Here you find this gap even more extreme: There are almost 30 years of publications on practices dealing with complex data sets, sorting out the most important and reliable bits to be transferred into sounds and sound textures. Most of these sonifications sound like water, wind, midi musical instruments, and toys. There is hardly any research on listening habits, socializations, and what people make out of certain sounds. Moreover, there is hardly any research on listening training and schooling people in sound environments to concentrate their natural listening abilities on distinguished synthetic sound textures. As a matter of fact, these are the everyday realms of musicians and sound artists…

At the moment, the Metaverse is a metaphor for a department store with a lot of new ephemeral digital toys. Anyone can come up with a definition here to advertise anything. Thus, I can neither predict how this kind of world production will develop nor how it will sound. However, it is no wonder that a single (!) company that wants to overrun the existing VR market is first working on a head-tracked spatial audio environment, even affecting the worldwide production of music, turning the switch from stereo to spatial and offering tons of fashionable headphones as gates to this “new world”. This operation is unprecedented in the history of music! In other words: The spatiality of sound has climbed another level of importance in our mediatized societies.

I sincerely hope that artists understand the seriousness of the situation, as others own the mainframe already. Moreover, I hope that artists will understand this as great opportunity rethinking their processes and finding different ways for creation, distribution and communication. Example: At the moment, we work with a well-known DJ on spatialisation concepts for Ambisonics. However, from the first moment, she understood that she will have to question the physical position of her turntables and herself as a performer.

In 3D audio, when used as an instrument and a listening environment, the perception is easily distracted by single visual cues, whereas your ears shall reach out 360°to perceive the full, detailed potential of the instrument. Maybe the DJ as a visual phenomenon has to disappear? Furthermore, her hands are working all the time on the gear in front of her, but how to actively and artistically spatialize her sound? Does she need pedals or a new interface to steer the show with her feet? This process might change the concept of her musicianship, performance, production, technical setup, and maybe even distribution. This thought is only a tiny spot on the field hidden underneath the terminology of marketing and venues, trying to make people believe that this is the next cool thing. 

In short: 3D sound is the ingredient to render the Metaverse “alive” creating presence and personality. We need artists who can form this terra incognita, a place of surprises and demanding (not just smooth and pleasant) new concepts for alternative social engagement and future cultural pathways.

spæs Lab is currently working with Atmoky, a company shortlisted in 2022 as one of the most important companies for development of the metaverse. They are creating algorithms for the spatialisation of sounds, especially in AR and VR environments. As a model, they created a virtual dome for the web browser in which you can place virtual sound sources and navigate with your keyboard and ears binaurally. I was the first artist to test this setup, contributing to this promising environment with spatial sound compositions. I have tested a lot of different propagation patterns with a variety of spatial textures, and I found it intriguing how fast I was “in”, my perception melting into this altered spatial reality within my browser. Currently, we investigate this further by applying more complex environments and music.

Since 2021 spæs has been collaborating with Volst, a Dutch software company that developed Odio-App, a fascinating 3D-Audio App utilizing head-tracking and virtual interactive sound sources. Binaural soundscapes are treated like formerly tracks or songs with individually adjustable sound sources in your headphone world. We developed “liquid architectures I. – X.” a series of ten soundscapes exploring the overlaps of technically induced realities in our everyday lives. We tried out this App daily trying to understand how virtual soundscapes in headphones alter and change our perception of trains, houses, places and communicative situations between human beings. We worked on a lot of different interactive spatial arrangements with complex sound textures investigating how personal binaural headphone space is changed by atmospheres, moods, spatial cognition and constraints of the medium.

It came as a surprise that some of us would not take off our headphones even when not working on the App, using the transparency mode for communication but even more the noise-cancelling function for blocking out the rest of the noisy world to find concentration and quiet during the day. And for incoming calls, you just tap a button to blend from one space into the next and back again. Considering the fact that homes will become smaller and smaller in cities, hyper-realistic spatial audio solutions for “enlarging” the feeling of your own walls and blocking out unwanted sounds from the neighbours will become as important as heating, interior design, and lighting. You can, and you will live in your 3D audio bubble. My prognosis for the coming years is that NOT wearing headphones will sound strange to our ears. We are about to enter a completely mediatized sound world technically produced inside our ears. In other words: The world will sound unrealistic without headphones. Thus, there will be moments where it is hard to tell whether this is “Metaverse” - or not.

© spæs Lab Berlin

(fundamental for the degree of communication)
What do I know about the perceptive abilities of my audience?

(fundamental in all the arts today) 
Why create while there is so plenty?

(fundamental when it comes to 3D Audio)
Why not Stereo?

(fundamental for knowing who is talking)
How many immersive pieces have you composed with Mono?

(fundamental for alternative future spatial aesthetics)
Can you do this with less reverb?

(fundamental for knowledge transfer, dissemination and archiving)
How will we record spatial audio pieces?

There have been a lot of publications, especially in the past 15 years, in this field. Most of these publications deal with technical descriptions and information on setups. Therefore, I would suggest a combination of readings from different backgrounds.

Absolutely fundamental for the question of how to understand space as a sounding phenomenon better: 

• Blesser, B., Salter, L., (2006) „Spaces Speak, Are You Listening?“ Experiencing Aural Architecture. Cambridge, Massachusetts: MIT Press.

For a quick overview of space in music and spatialisation, just read: 

• Zvonar, R. (2006). “A History Of Spatial Music”. In: eContact  7.4.

For understanding phenomenology and perception in 3D audio from technical and artistic foci better (must read).

• Kendall, G. S. (2010). “Spatial Perception and Cognition in Multichannel Audio for Electroacoustic Music”. In: Organised Sound 15.3, pp. 228–238.
• Kendall, G.; Cabrera, A. (2011). “Why Things Don’t Work: What You Need To Know About Spatial Audio”. In: Proc ICMC.

If you (really) want to understand Ambisonics better, I must recommend:

• Zotter, Franz; Frank, Matthias (2019). Ambisonics, A Practical 3D Audio Theory for Recording, Studio Production, Sound Reinforcement, and Virtual Reality, Springer.

For all who want to dive into binaural completely:

• Blauert, Jens, Ed. (2013). The Technology of Binaural Listening, Springer, ASA Press.
• Still, an interesting concept of formalised sonic space, written from the perspective of the listener/composer (old school classic):
• Smalley, D. (2007). “Space-form and the acousmatic image”. In: Organised Sound 12.1, pp. 35– 38.

Great for thinking about composing sound as spatial textures: 

• Nyström, E. (2013). “Topology of spatial texture in the acoustic medium”. PhD thesis. City University London.

If you are interested in composing sound AS space considering contemporary spatial theory and practice:

• Sharma, Gerriet K., (2019). Aural Sculpturality. Spatio-temporal Phenomena within Auditive Media Techniques, ZKM/Hertz-Labor (Hg.), ZKM.