The quest for instrumental control

Composers in control

During the first half of the 20th century, composers became increasingly aware of the potential scope of electronic sounds. The idea of ‘infinite possibilities’ was coined by Luigi Russolo in 1913 in L'arte dei Rumori:

“We must replace the limited variety of timbres of orchestral instruments by the infinite variety of timbres of noises obtained through special mechanisms...” and “The variety of noises is infinite. We certainly possess nowadays over a thousand different machines; among whose thousand different noises we can distinguish. With the endless multiplication of machinery, one day we will be able to distinguish among ten, twenty or thirty thousand different noises. We will not have to imitate these noises but rather to combine them according to our artistic fantasy.”

© Seuphor, 1930

Luigi Russolo behind his Russolophono.

In the following decades, the notion about an “infinite variety” sparked the imagination. However, in 1958 Stockhausen noted in a lecture: 

“... it is also clear that the diversity of sounds that can be produced electronically is not unlimited. Electronic Music as a genre has - in defiance of all our initial notions of abolishing "genres" in the realm of music and of including all possible sound processes – its own phenomenology of sound, which is conditioned not least by loudspeaker playback.” (published 1961 in Die Reihe 5, transl. Jerome Kohl)

Today, composers of electronic music seem to have become more realistic about the future and are well versed - if not content - with the tools at their disposal for manipulating sound. They have been able to develop and use these tools ‘off-line’ in the relatively comfortable environment of the studio they work in.

© DallE 2023-11-14, 11.39.10.

Different types of synthesis have evolved that offer a wide range of sounds and elaborate temporal transitions between different aural stages. Especially the temporal manipulation of sound is key to advanced sound design: it allows for composers to manipulate their source material into captivating sonic experiences.

Control on stage

Quite different in this respect are the current circumstances for many instrumentalists who work with live electronics on stage and who try to avoid the limitations of electronic versions of traditional acoustical instruments – more about that later. Even with the latest and fastest laptops they will still have limited access to extensive and smooth transitions from one timbre to another. On highly developed traditional acoustic instruments, the available accuracy and efficiency to vary sound parameters in real-time is still superior to that found on both analog and digital electronic musical instruments. This is mainly because with acoustic instruments the parameter control points - excited by human input - and the type of sound production coincide, i.e., there is a direct physical relationship between the applied human force (breathing, pressure, etc.) and the resulting sound characteristics. Needless to say that our human brain, with its sophisticated aural and sensory feedback, seems to be perfectly designed for just that.

© thunai.org

Hunt and Kirk put it as follows:

“In stark contrast to the commonly accepted choice-based nature of many computer interfaces are the control interfaces for musical instruments and vehicles, where the human operator is totally in charge of the action. Many parameters are controlled simultaneously and the human operator has an overall view of what the system is doing. Feedback is gained not by on-screen prompts, but by experiencing the moment-by-moment effect of each action with the whole body.” Andy Hunt and Ross Kirk, 'Mapping Strategies for Musical Performance', in Trends in Gestural Control of Music, ed. Marcelo M. Wanderley and Marc Battier (Paris: IRCAM, 2000), 384.

The development of controllers

Fundamentally, the challenge for designers of electronic instruments lies in bridging the gap between the voltage control of the electrical power source and the actual type of synthesis or processing.

© Clara Rockmore

Clara Rockmore's Theremin schematic, hand-drawn by Robert Moog.

Since the advent of electronic music, many more or less successful attempts at designing suitable controllers have been made. Although a complete list of these attempts is beyond the scope of this article, it is clear that notable progress over the last three decades has been made using digital technology. Institutions around the world, such as the now demised STEIM in Amsterdam, ZKM in Stuttgart, and MIT in Boston, to name just a few, have developed so-called ‘alternative controllers’, often using MIDI or OSC as the communication protocol.1

These controllers are more than mere translations from the ‘acoustic world’ (such as MIDI keyboards): often they are exotic-looking gadgets with unusual configurations of knobs, sliders, or other control points. Sometimes they are designed for use as extensions of already existing acoustic or electric musical instruments, sometimes the designs follow completely new directions.2

© Roger Linn Design

Left, the Buchla Thunder (1998), right the Linnsrument (2014)

Constraints

Unfortunately, we still see large limitations if we compare these controllers to traditional acoustic instruments. Despite the results of research that has been done on the subject of relationships between body gesture and electronic sound, such as expressed in the publications of Claude Cadoz or the substantial studies on gestural acquisition and mapping by Marcelo M. Wanderley, there seems to be little change in the day-to-day experiences of performers who use electronics on stage.

As the original Theremin shows, a limited number of controllable parameters offers logical control at the cost of timbral variety. If we want to achieve seamless control over different types of sounds from a single system, then we will inevitably need more parameters and consequently more control points. Although it is certainly possible to create captivating sounds via the simple acts of layering (additive synthesis) or sequential processing (such as FM- and subtractive synthesis), even these techniques require a substantial amount of control points, often already far exceeding the available physical human input possibilities of the performer, who is further handicapped by a limited overview of each individual control parameter’s current state. And there are more challenges. For example: multifunctional individual control points, i.e., switchable from one parameter to another, are problematic as they will leave previously controlled parameters in mid-air. Moreover, switching between different timbres with different parameter demands will break up the continuity of the sound.

The current limitations mentioned above are partly imposed by the commercially available hardware controllers and to a lesser extent, software. However, I believe that a particular custom system design can overcome these restrictions.

A possible alternative

One possible solution is a logical physical input scheme that keeps track of a multitude of parameters by assigning them to different areas within a multidimensional matrix that can be accessed via a ‘control continuum’, a virtual 3D space.

In 2014, I started with the Leap Motion gesture controller, combined with custom-designed software to achieve continuity in aural changes.

Over the last decade, I re-wrote the software to be able to process both sound and image in real-time. This led to the Trilogy in 2020. 

The last substantial software update was for the Leap Motion V2, which I developed during my artist residency at IEM in Graz in the summer of 2023. This new version has improved skeletal tracking.

More detailed information about the HyperTheremin and its history can be found on this page of my website: https://jbbollen.net/hypertheremin/ 

Watch that space for my latest works for the instrument, such as: 

With HyperTheremin, limited horizons, and fixed sound-character domains can facilitate the internalization of a ‘3D control space’. The characteristics at particular points or areas can be explored and memorized. Alternative routes can be outlined. Jumps can instantly take the performer from one location in that space to another, with or without interpolation. The possibilities that are opened up by this design are not only beneficial to performers. It touches the very core of the system, audio and visual design. It means that composers, too, can start to leave behind their off-line techniques and make the studio their stage by designing their own multidimensional sound continuum.

The solution described above requires continuous research. Fundamental ergonomic points of view are taken into account when designing a system that does not compromise sound design and its real-time control. Moreover, the choreography of body movements with regard to audio results and visual aesthetics also demands attention.

Further exploration of the relationships between instrumentalists and their tools is needed if we want electronic systems to match the advanced controllability that acoustic instruments offer.

• 1

  The resolution and bandwidth of MIDI are now considered a serious limitation for real-time use. Hence the tendency among live performers to prefer custom build UI’s (created in Max/Msp, Supercollider, C, or other) that connect directly to sound sources, thereby skipping the MIDI bottlenecks. OSC offers a substantially higher resolution and both LAN and Bluetooth have opened new corridors to interconnectivity and the use of new controllers (such as the Leap Motion).

• 2

   I highly recommend reading Seppo Gründler’s article The interface is the message on this platform for more on existing interfaces.