Notes for a Future self by Oded Ben-Tal utilising deep learning music models as composition tools

Machine learning has periodically grabbed headlines in recent years by seemingly allowing computers to perform tasks previously deemed the sole preserve of humans. But do the techniques applied and the research they rest on justify the headlines? Equally, Big Data techniques (the previous trend setter in upending the old order) have been applied to music and other domains of humanities, arts (and social science) with questionable results. In January 2022 we will be launching the AHRC-funded research network Datasounds, datasets and datasense: Unboxing the hidden layers between musical data, knowledge and creativity. Our aim is to examine questions of music, data and the gaps between music and data, and identify core questions that will drive forward the next phase in data-rich music research, focused on creative music making. 

Much recent advances in data-rich technologies as applied to creative tasks have been driven primarily by technological capabilities rather than by artistic sensibilities. The predominant paradigm can be summed as: what can a new machine learning technique achieve when applied to painting, poetry, music, or some other creative domain? This technologically deterministic approach, while perhaps appropriate in the early stages of development, runs the risk that as the technology matures, it begins to define what should be considered possible rather than being challenged by what is actually required.  As such, we must depart from these established technologically-oriented working patterns to create step-change developments, accounting for, and approaching, problems from the perspectives of multiple disciplines and methods and synthesising these into novel approaches.

Questions about creativity have been at the forefront of research in several fields in the last decades. The proliferation of data opens new opportunities to study creative music making – such as capturing performer’s gestures or analysing recordings – but also poses serious challenges to our notions of creativity. Creativity is an ill defined concept, but notions of originality, skill, and value are among the common attributes we often refer to. Involving computers in the idea generation stage of the process challenges many of our intuitions. How do you judge the quality of  100,000 tunes? Does it matter that some of them are good and enjoyable? The fact that some of the outputs are duds does not prevent us from appreciating human composers as towering geniuses of creativity. This highlights another challenging question – we often associate creativity with the effort and personal involvement of the creator. Obviously, that a computer worked for five full days to train a computational model using the most expensive chip does not resonate in the same way as a story about the artist struggling for a year to compose a piece. But at the same time we do not only value creative outputs that are the fruits of long struggles. The fact that Mozart seems to be able to conjure wonderful music quickly is celebrated as a mark of his creative genius (and skill). In other words, injecting computational methods into creative work requires us to re-evaluate our thinking and possibly abandon some cherished notions.

One driver in recent developments is the significant growth in the availability of data. Many hours of audio recordings, digitised scores and lead-sheet notation, MIDI renditions of classical and popular music, and curated data sets developed by the Music Information Retrieval community can be found and accessed online. However, data only captures some aspects of the complex phenomena that is music. As Reybouck (Reybrouck 2020) argues, music entails listeners structuring experience from sound-time phenomena through a process of sense-making. Sense-making is an active and adaptive process, not mere processing of information. It is an interplay between sound and a dynamic listening stance within a physical and cultural environment. Sense-making is also a key requirement for musicians creating meaningful and engaging musical experiences. Current musical data only captures some of these elements and filling the gaps between data and music as a cognitive, affective and embodied experience is a complex challenge. Practice-based researchers have been dealing with parallel problems with music and its representations: performers and composers often fill gaps between music as a communicative act and its manifestation as a notated score. They engage in the process of sense-making of these representations through a combination of embodied knowledge, contextual information, creative exploration and through practice and rehearsal often relying heavily on intuition. The research network outlined at the outset will integrate this untapped resource by focusing on music making activities and bringing practice into the proposed activities.

The network is a partnership between composers, performers, and researchers with expertise in music psychology, empirical musicology, computational creativity and mathematical modelling with a shared passion to understand the complex interrelation between music and data. Our interest lies in understanding how data-rich technologies, including AI, can support human creativity, in contrast to the human-vs-machine narrative prevalent in public discussions of this new technology. After all, technology – from the bone flute, to the piano, and the record player – gave rise to new forms of music and new modes of engagement with music. But making the current new technology musically ‘smarter’ and more attuned to human needs requires a deeper understanding of the complex network of interactions – between composers, performers, and listeners, between notated scores and performers, between soundwaves and active perception – that underpin the phenomena of music. Just like any other music technology, the knowledge and knowhow of musicians is crucial for its development. The person who carved the bone flute was, in all likelihood, an instrument maker, performer, improvisor and composer rolled in one. Both performers and composers shaped the development of the early piano into the modern instrument in current use. DJs discovered creative ways of using the turntable which were eventually incorporated into the technology turning it into a performance gear. 

Monthly virtual seminars will be the first forum for exploring these questions with experts each bringing their own unique perspective. These talks will be held on the last Monday of the month (afternoon UK time) with the first set running from January to June. Our first in-person event (Covid permitting) will be jointly hosted by PRiSM the Centre for Practice & Research in Science & Music at the Royal Northern College of Music and the Department of Music at the University of York where we will be focusing on improvisation as a particularly interesting and challenging topic for the music-data nexus. Improvisation brings together many of the activities and behaviours that constitute music: it includes actively listening, performing, inventing and imagining. It draws on cultural knowledge and accumulated experience. And, as George Lewis argues, improvisation is what we do every day as we interact with others. Further events will include a symposium and concerts in Stockholm hosted at the MUSAiC lab and the Royal Conservatory of Music.