Author: Dereck De Abreu Coelho

“If you are there and what appears to be happening is really happening, then this is happening to you!”

This quote speaks on the idea that virtual reality is real. That it’s part of our experience that we are feeling right now, with our physical body. And the experience is truthful and transparent and makes us feel that it’s actually happening.

XR as new media.

In the 90s we had a rise in personal computers. Companies started investing and virtual reality became more viable. It did not become a thing at that time potentially because of economic reasons and technology needs financial support to develop. As well as the fact that technology wasn’t there, it does take time to develop these sorts of hardware. Hardware at the time was clanky and heavy, it would reach high temperatures and it wasn’t viable to create such products.

The rest of the lecture spoke on immersion and the theory behind how one can induce a participant into an immersive experience. As well as how to guide presence. The sound was spoken on but not a considerable amount, I will be taking the theory into my own interpretation. What it did help with was the larger ideas around virtual reality and what it takes to develop content for it.

Leslie Deere

Exploratory nature is key to knowing Leslie’s work. Leslie always says that her work is in progress and the approach of always being in process and this allows her to be always reaching further. She is also a stranger to complacency as this allows her to be always moving forward and see the possibilities of technology. It comes from her work being interdisciplinary and this allows Leslie to see the gaps between these disciplines and combine them.

We then watch a video of Leslie’s work called Array Infinite, Leslie Deere, is into gestural visual performance art. And found herself using the Kinect for such pieces. The next logical step for her was using VR to take her work forward.

Array infinitive. It is a piece that requires gestures to create audiovisuals. The hands control the sound particle trails in the VR world live as It happens. She’s designed parameters of reverb and delays to the XYZ Gestures. This allows the performer to explore and create new soundscapes using gestures. Her own group practice of online meditation influenced this project, She’s part of VR groups that meditate online together, which are speech led and she’s doing the opposite in this project she’s doing group meditation through audiovisual experience. They don’t walk around in the virtual world, instead, they’re on a journey through colour and sound.

The performer becomes the guide and the direct connection between the world-changing and the audio being manipulated for themselves and the other participants sitting in the room.

The sound is amplified into the space instead of headphones to add an anchor into reality into the group mentality and this is part of the research she is currently doing. She created seven scenes that related to the chakra system and related the scenes towards the relation to the colours. Green is associated with air, orange with water, and blue is about voice. She uses a three-part vocal female harmony here. Purple is ether and the sound is more into a distant realm.

Leslie used unity to develop the program and local data to connect the headsets to each other. She’s worked with a team to create this. The piece uses Ableton, MAXMSP and unity to combine everything together. 

She also recorded multiple musicians for this work and used their pieces for every colour and the audio experience associated with it. 

She beings by speaking about how big of a project it is, and predominantly her work is collaboration. She finds it an enriching way of working and you can find out so many different things, it can lead to discovering speciality industries and a niche way of working and she enjoys a collaborative practice.

She is making a project about audiovisuals and gestural work. During her PhD annual review sessions, her examiner eventually started saying, what is the context of how she got to the project?

Her background is in dance, she knew about John cage, and she knew about experimental sound but through the context of dance. She didn’t know much about it. She knew of it but not about it too much so this was her way into the practice. During her last few years in New York, she got into tap dancing where the performer would trade 16 counts, it was very musical, the tap dancers would improvise with musicians, bands, and drummers. Dance that makes sound is what she got into.

She moved to London to do a course on the sonic arts at Middlesex, she didn’t come from a DJ or band background so she had no studio knowledge, or on signal flow. She wanted to make sounds similar to music concrète and collage them in an artistic way. There were two pathways she could take, one more performative way being laptop performances and an installation route. She went the installation route and felt it has a similarity with dance, working with space, the body.

So she created this series titled The Amplified Series, its sculptures that don’t make a sound. They pick sound up. There’s a microphone on a yellow box with a headphone output, so when you listened you could hear the pendulum swinging and the room as well. And she was interested in the idea of the room and the environment being part of the piece. Deep listening to what’s around you became part of her work.

When you have headphones on, and record someone you can hear subtle details that you wouldn’t hear otherwise, and she made a piece with two sets of headphones and it’s supposed to bring the room and a conversation into the piece. She also made a similar idea but with a hydrophone in a tank of water and the participant would listen and adjust the mixer to their liking.

This gave her an interest in how sound affects different spaces and the relationship between them.

She modified the mixers on her earlier pieces, she would paint them and this was to demystify the technology, for an average person this would be intimidating and the idea was to make it more playful. The idea of her work is to filter our environment, what we hear and what we don’t hear. What do we hear in an amplified room?

So at this point, she still hasn’t performed with sound. And her journey into sound performance became an interest. Live sound-making was what she wanted to do next. She received a residency in London and she developed a project that used a Kinect camera to pick up her movements and play sounds. Her motion would trigger sounds. It was called Modern Conjuring for Amateurs. This became the thematic backbone of what she does. The basic idea was that when she moved it would create sound and the silhouette would echo her movement. 

The title of her piece came from a book with the same title. She was interested in the turn of the 20 century and the technology surrounding it, electricity, radio, and the phonograph. A lot of technology was available and spiritualism was the theme of this era. Clara Rockmore and her theremin would have been futuristic for the time and considered weird and she was into the idea of this. Her project tried to embody these ideas that had sprouted inside her head.

In her performance with the Kinect when she closes her hands and it loads in a new bank of sounds. It went through different sonic landscapes. The idea was to create Long-form drones with her performance and make the audience zone out and sink into the immersion she was creating. She performed her piece in numerously different places, she acknowledged that Kinect in live situations is not perfect and she picked a difficult technology to use during performances. But agrees it was a good learning curve. 

After this project, she felt she wanted to take this into VR. Her PhD project was called array infinitive. She was thinking of Infinitive as a form of speech 

To make one feel, to activate something. An infinitive. A Similar concept to gestural audio-visuals but she wanted to concentrate on a group experience and the zoning out. Is it possible to gain an altered state experience in a group VR performance?

Erika Fischer-Lichte speaks of a performance as an art event, and through the visceral experience of performance, the audience is transformed. Jonathan Weinel’s work also speaks about the developments of electronic music and art and the different opportunities for creating different consciousness. Can art that uses this as an idea affect these mind states?

Mark Grimshaw speaks on how the design of sound contributes to player immersion in games. The perception of sound and the real vs the virtual world and the rise of immersion. Are we comparing and contrasting sounds? Are they supposed to sound real or fake in the game world? Does this add or take away from the game world? What were the intentions of the composition or the design, was it to promote a state of immersion? Listening across disciplines, listening for science or politics. What are the ideas behind combining listening? Lastly Altered States by Dr David Luke, doing trials of psychedelics. This all helped her with a pilot test on the first function of her piece.

So what were her inspirations? 

Catherine Yass Works with films and photography. She has a piece called the lighthouse and it doesn’t have sound but it was projected on the whole wall of a gallery. The shots were sweeping swooping shots that play with perspective and it was mesmerising and she could tell it had an effect on An audience. And this was one of the first times she saw a piece of art affect an audience. She works with colours and the psychology of space as well.

Ann Veronica Janessens,

Minimal work and she considers light to be her primary medium. She perceives her work as experiential and very hard to document. Her work is about the experience and also uses colours, movement and sound. To heighten our awareness of colours and sound and movement is a large theme of her work and this sensory experience of reality and to play with our senses.

James Turrell,

Again light and perceptional phenomena and there’s a reflective meditative theme in his work. He talks about the emotional effects of luminosity. Seeing ourselves seeing and understanding ourselves as an entity of being.

Hilma af Klint,

Considered one of the first abstract painters, she was into channelling. Born in 1862, again the time period of technology, radio, phonograph, spiritualism. She was into mediumship and every painting she did she was channelling from somewhere else. Also abstract and uses colours.

Agnes Pelton

Similar themes, born in 1862, same time period. Born in Germany and relocated to the US and has spiritual themes including, communicating, and translating. She depicted the spiritual reality in meditation.

Jeff Cornelis.

She found his work through her research and he made this film called Colours of the Mind. He’s a dutch filmmaker. His film Colours of the Mind explores the social and spiritual role of self-induced trances. Rave culture as well and does well to link everything together. He connects raves with the transcendental acts that we engage in when we enter these spaces.

For sound, the inspirations begin with Eliana Radigue.

Her work is minimal which is a similar theme to the other artists she is interested in. Her tape loops and the idea of playing loops that get out of synch and that when doing so new elements start to emerge. Duration, Longform and seeing where the journey takes you. Unending music. She became a Buddhist later in life and this became an important part of her practice.

William Basiniki.

The disintegration loops, she likes that he has a cosmic idea towards things. Taking a sound from out of this earth. Taking something cold and transforming it.

Maryanne Armacher,

Sound objects and the idea of the third ear. The idea is that we have a third ear and when we listen other elements pop out. As well as discussing the feeling of sound and ritualistic notions of sound. As well as listeners’ position in the room, how close are you to the sound source, what if you move into a different place. How does this alter your perspective and listening experience? 

For her PhD, she recorded musicians in Glasgow, Processed bowls, Cello, and a choir. To make the work was a huge undertaking, a big thing to try to pull off. Ross flight designed the interface that she used for the Kinect camera, he’s a sound engineer and works a lot with theatre, so they worked together again on her PhD project and modified the earlier version. Stuart Cupitt kindly sponsored her work in the development of the project. Chris Speed is an amazing DJ who makes music and also helped out in the Art.

Last summer she had an official pilot test and had 25 participants, the ethical clearance for that during covid was insane but it was a great test. She had an age range of 17-80 years old. A mixed audience and quite a lot of people afterwards would sit down and chat, some people would state their neurodivergent and would like to see where it goes. It was also interesting to see people experience it that know nothing about technology. She also noticed that gamers want to do things and want to get up and move around.  

The idea of the piece is that there are four people sitting down and one performing and the group experience was key to her as they travel through the colour spectrum. Unfortunately in the pilot stage none of them noticed the group element, most likely due to the overload of technology. She noticed the piece was interpreted in different ways. The sound is also amplified into the room and she didn’t use headphones intentionally. She wanted the sound to be experienced similar to a gig. She was unsure if this would ruin people’s experience as you could hear her moving around and this would take them out of the immersion but she found out people enjoyed it. And found that people felt more grounded.

After the pilot test, she did a public Beta and for the first time she had a mixed audience, in and out of VR and she found it super interesting. It creates a different dynamic and the people outside of VR can see what she’s doing. And there exists dynamic feedback between her in VR and the audience observing.

Post Lecture Reflection

I found Leslies’ work to be very forward-thinking in terms of the cross-disciplinary aspect of it. It made me reflect on our current process in collaboration with the games design MA students and how that process has been, which is fruitful. We have in turn explained how we do sound design to the MA students and the process behind it and also how they explained the process of making a game. As well as using each other’s strengths. Alongside this in a smaller context within just my own group how we have managed to learn from each other within this collaborative project and understand each other’s workflows and ways of thinking has been refreshing and thoroughly insightful.

I also found Leslies’ process of research and how it actively reflects her own work and the process of creating different versions to finally reach an endpoint perhaps to be an ongoing process. From her first piece of Kinect and the original sound installations she did which eventually led to her creating her latest project. One can look back and easily understand the process of it. I also enjoyed her use of others in this project, she openly admitted she knew very little about the sound world but got the right people around her to help. I think it’s important to understand that sometimes even the people with the skills might not have the ideas to create projects such as these. That it might take an outsider of the discipline to enter with a fresh thought process, not under influence of the norm.

To look at my group’s work from a sound arts perspective I can say we’ve definitely done some things correctly others not. I do believe we have given ourselves enough time working together and the ideas have flourished and everyone has done their part. But the active reflective process that comes with collaboration wasn’t entirely there. The constant feedback was very little and did make it difficult to really go through the same stages that Leslie for example did in her work. I purpose that going on forth I’m unsure if the mix is all individually or as a group? But if it is an individual or group I want to ground our sounds in theory as they have some at the moment. But I feel there could be a greater theoretical influence on our decisions, as well as this understanding of how sound can be used outside of the MA games design understanding. They only asked for sounds that they needed and now we have an opportunity to completely decorate this game world. For students such as ourselves where sound is our everything we can enhance and work alongside the visuals and present immersion in ways that visuals cant do alone.

Since the crit we had as a group we have been working towards implementing the feedback provided.

One thing that’s working well was that the music and sound effects were really good and fit the idea of the game really well. We thanked them for their feedback, they said the music as well had a lot of layers which felt nice and didn’t clash too much with the game. As sometimes overly complex music can make a video game distracting to play as your brain is listening to the music rather than concentrating on the current action.

One thing to add or change. One student said a variation of sound effects, they enjoyed the music and the sound effects Jingya had made, but they offered the feedback that perhaps we could change them in between levels. Or create more variations of the sound effects we have created already, to allow an element of immersion that comes from variation in sound.

One positive overall. A student noted that what we did was a good job and they were very impressed with the music compositionally and creatively. They enjoyed our presentation and our take on what the MA students wanted us to do. They thought our ideas fit the game’s aesthetic perfectly and were excited to actually see the finished product.

Tutor Feedback. We were told similar to the student’s feedback, that our presentation was professional and excellently communicated as well as our slides were valuable to showcase our work. Our music fit the work correctly and even though we only showed one level, we were told that it was excellent and that the development should continue more and expand based on our ideas. As well to this we were congratulated on our work together as a group and that we should be very proud of our work ethic and attempts of working with the MA students as it took them a while to deliver anything for us to base the sound work for. An important part of the feedback which no one had said except the tutor was the idea of our perspective as sound art students, typically game design students will only think about the most obvious sounds. For example footsteps or breathing, shooting. But as sound artists and sound designers, it is our responsibility to think about everything that sound can encapsulate and how we can use sound to enhance motifs. Fear, strength, comfortability, a sense of place, all these things can be displayed with sound.

We took this all on board and did some final touches on our submission for the MA students, Jingya did some variations of the sound effects and typing keyboard noise. I and Will created some varied tracks for them to use across levels in the game. I also created a soundscape to go along with the levels as I felt even though the music was playing it needed a base layer of machinery noises.