by X-System, partner in MuseIT
One of our key goals in MuseIT is to upgrade online spaces so that they are capable of hosting a wider array of human co-creativity for everyone, regardless of disabilities. We are doing this both by bringing together existing technology, and developing new software. A particular focus is musical co-creativity: how can people write and play music together online, in a satisfying way? And how can online spaces designed around the needs of musicians with disabilities help them feel empowered and included?
One of the many legacies of the pandemic has been an increasing reliance on online spaces in professional as well as non-professional settings. Whether you Zoom, Teams, Facetime or Whatsapp, we are increasingly finding ourselves navigating both our social and work lives from behind a screen.
However practical, this technology clearly has limitations in its capacity to replace face-to-face interaction. Barely a year after ‘to Zoom’ became a verb, ‘Zoom fatigue’ (1) emerged as its own distinct form of work burnout: through the lens of neuroscience, the reasons could not be clearer. A 2023 study by Joy Hirsch in Yale compared the brain’s response to real and online faces and found that real faces instigated longer eye contact, as well as generating higher autonomic arousal (2). Brain scans are even more revelatory: fNIRS (functional near infrared spectroscopy, which measure blood flow to the brain) shows higher cross-brain synchrony and EEG (electro-encephalogram, measuring electrical brain activity) shows that interacting with real faces boosts theta waves, which reflect a state of learning, memory and spatial awareness in our brains. In other words, engaging with people in real life makes us more alert and engaged both mentally and physically. Our mirror neurons, which mimic the thoughts, feelings and actions of others and facilitate empathy, also struggle with online communication. Although the technology helps us to address social needs when physical interactions are impossible or too daunting, research suggests that there are consequences for our mental health if relied upon too heavily.
The limitations of the technology itself have helped shape what kind of activities tend to flourish in these online spaces. Business meetings with a slow pace of conversation and talking in turns are a more natural setting for online than a call with a loved one with its more freeflow cadences and interruptions. Other more playful and creative activities have likewise seen slower adoption in mainstream spaces, with the marked exception of gaming, which has long supported co-creation in virtual spaces with real-time multi-user collaboration in Unreal Engine, creative games such as Minecraft and VR tools like Sculptr.
One of our key goals in MuseIT is to upgrade online spaces so that they are capable of hosting a wider array of human co-creativity. We are doing this both by bringing together existing technology, and developing new software. A particular focus is musical co-creativity: how can people write and play music together online, in a satisfying way?
In musical performance, one of the biggest issues with existing video meeting software is latency: the delays that occur when data travels over large distances. We can adapt to the odd pause in a conversation, but in a jam session or some other time-sensitive activity, it can be fatal. In order for musicians to play together with a common sense of rhythm, studies show that delay must be less than 25 milliseconds between transmission and reception of the signal (3). The major conferencing applications, like Zoom, have a latency closer to 100 milliseconds. This problem was solved in 2010 by Chris Chafe and colleagues at Stanford (4) with the introduction of Jacktrip, which uses a different internet protocol (User Datagram Protocol or UDP instead of the more common, and newer Transmission Control Protocol or TCP) to streamline data transfer and reduce latency.
While Jacktrip offers the foundation for a new online co-creation platform, audio latency in itself is only part of the problem. In a world where most of our understanding in human interactions is communicate non-verbally, how can we enhance online communication to reveal other aspects of how we are thinking and feeling? Non-invasive sensors such as heartrate (ECG) and brainwaves (EEG) can be used to construct multimodal representations of our emotional state. Can representations of this data help mimic the more rich sensory environment of physical spaces and keep our brains healthily engaged?
Designing a platform from the ground upwards offers us an exciting opportunity to build around inclusivity. While physical co-creation spaces offer rich experiences, they also present multiple barriers to entry, especially to those who live with disabilities. Something as simple as travelling to and accessing the rehearsal space itself can be an insurmountable obstacle for musicians with disabilities. Can online spaces designed around their needs help these musicians feel empowered and included?
Countless studies (5) have shown that we are all deeply musical, even if we have not had any formal training. For those of us who want to join in but lack experience with an instrument, we are investigating new ways in which we can harness everyone’s innate music creativity. Can AI help turn a hummed phrase into a full melody? Can brain waves and bodily rhythms help to create music that is even more in sync with other performers?
The curb-cut effect (6) teaches us that placing the needs of those with disabilities at the core of decision making leads to outcomes that benefit everyone, so we are developing our technology in the context of participatory workshops, where we will receive feedback and direction from musicians with disabilities.
Through MuseIT, we hope to help enrich our digital lives by broadening the scope of digital communication. If we can re-introduce some of the more subtle and emotional communication channels that we rely on in real life, our mental health online need not suffer so much, and when access for all is a fundamental part of our approach, we hope to enable many more people to experience the joy of collective music-making. Let’s play!
© Picture by ShareMusic and Performing Arts
References
Lee, J. A Neuropsychological Exploration of Zoom Fatigue. Psychiatric Times. 2020
Zhao N., Zhang X., Noah J.A., Tiede M., Hirsch J. Separable processes for live “in-person” and live “zoom-like” faces. Imaging Neuroscience; 1 1–17, 2023
Bartlette C., Bocko M. Effect of Network Latency on Interactive Musical Performance. Music Perception - MUSIC PERCEPT. 24. 49-62. 10.1525/mp.2006.24.1.49, 2006
Cáceres, J-P and Chafe, C. JackTrip: Under the Hood of an Engine for Network Audio, Journal of New Music Research, 39: 3, 183 — 187, 2010
eg. Mehr S. A. et al. Universality and diversity in human song, .Science 366, eaax0868, 2019
Blackwell, A. G., The Curb-Cut Effect. Stanford Social Innovation Review, 15(1), 28–33, 2016