Immersive Design 6: Audio Implementation
To implement audio into my immersive piece I used the DearVR MICRO plugin mentioned before in Adobe Audition to adjust the positions of all my audio files to where they should be. When this was done I exported the combined binaural audio file into Premiere Pro and rendered out the final 360 video.
I had to cut some of the sounds I intended to use as there wasn’t space within the animation for them to be implemented acceptably. To improve upon this I could’ve implemented test audio in Unreal to test the timings of the audio. This would’ve allowed me to adjust the animation times to allow for all the audio to be implemented.
References
BlondPanda (2022) Short Elevator Music Loop [Sound]. Available online: https://freesound.org/people/BlondPanda/sounds/659889/ [Accessed 24/5/2023].
Bradovic (2012) Eerie Ambience [Sound]. Available online: https://freesound.org/people/Bradovic/sounds/171992/ [Accessed 24/5/2023].
deleted_user_228014 (2018) failing lightbulb.wav [Sound]. Available online: https://freesound.org/people/deleted_user_228014/sounds/420651/ [Accessed 24/5/2023].
ForbiddenTheater (2022) Elevator Noise – Attempt 2.m4a [Sound]. Available online: https://freesound.org/people/ForbiddenTheater/sounds/658490/ [Accessed 24/5/2023].
icyjim (2019) cloverfield roar.wav [Sound]. Available online: https://freesound.org/people/icyjim/sounds/476083/ [Accessed 24/5/2023].
InspectorJ (2015) Tapping, Window, Single, A.wav [Sound]. Available online: https://freesound.org/people/InspectorJ/sounds/318929/ [Accessed 24/5/2023].
Joao_Janz (2019) City Ambience at Night 1.wav [Sound]. Available online: https://freesound.org/people/Joao_Janz/sounds/473693/ [Accessed 24/5/2023].
jorickhoofd (2013) Kicking in a glass door [Sound]. Available online: https://freesound.org/people/jorickhoofd/sounds/178582/ [Accessed 24/5/2023].
LukasEG98 (2018) 36-Pop_Edificio.wav [Sound]. Available online: https://freesound.org/people/LukasEG98/sounds/430246/ [Accessed 24/5/2023].
MATRIXXX_ (2022) Space Atmosphere 04 Remastered.wav [Sound]. Available online: https://freesound.org/people/MATRIXXX_/sounds/660108/ [Accessed 24/5/2023].
newlocknew (2022) ROCKImpt_Impact Of A Heavy Boulder.Designed x2_EM.wav [Sound]. Available online: https://freesound.org/people/newlocknew/sounds/667240/ [Accessed 24/5/2023].
reidmangan (2008) spotlight-stereo.aif [Sound]. Available online: https://freesound.org/people/reidmangan/sounds/49023/ [Accessed 24/5/2023].
Robinhood76 (2010) 01543 flying dragon.wav [Sound]. Available online: https://freesound.org/people/Robinhood76/sounds/93570/ [Accessed 24/5/2023].
TheHiddenVoice (2014) Predator Noise.mp3 [Sound]. Available online: https://freesound.org/people/TheHiddenVoice/sounds/214564/ [Accessed 24/5/2023].
tim.kahn (2010) ding.wav [Sound]. Available online: https://freesound.org/people/tim.kahn/sounds/91926/ [Accessed 24/5/2023].
vrodge (2011) Office_elevator_door open_ding.aif [Sound]. Available online: https://freesound.org/people/vrodge/sounds/119554/ [Accessed 24/5/2023].
Immersive Design 5: Development
My VR 360 experience is intended to be watched while standing up as the player is standing up within the animation. However, the experience should still be watchable while sitting down but there will be a disconnect between the height of the player and the user. As with some other VR experiences, it is up to the discretion of the user whether they want to sit or stand up.
360 Video Project
To create the animations for my immersive design piece, I used the level sequencer within Unreal to animate all the characters using the animations I sourced online. I didn’t implement any sound in Unreal because as stated already I am going to use Adobe Audition to simulate spatial audio. To make sure all elevator scenes lined up correctly I added a camera to my elevator blueprint so between sequences the camera stayed in the same place. This was especially necessary for the otherworld sequence as the scales were adjusted which caused an unconnected camera to be displaced from where it should be. However, the otherworld scene was a bit off center when I put all the scenes together in Adobe Premiere Pro so I adjusted the position of it in Adobe After Effects by duplicating the video of the otherworld sequence and moving it along to the left so it would match perfectly.
Office
The office sequence is very simple as it mainly consists of the player being able to look around before the elevator button is pressed and then the elevator opens before the screen fades to black. This sequence used to include the player being in the elevator but it worked better being part of the void sequence. This sequence was quite simple to animate as I just has to change a material and adjust the transforms of the elevator doors and the player hand.
Dark Void
As the majority of this area is dark, this sequence is going to rely on audio and used specific lighting to keep the player captivated. The intention was for the player to be on edge. This sequence starts with the player in the elevator as it is traversing to a different floor before it starts to strangely malfunction as the walls fade into transparency. The light then goes out as a spotlight turns on leading the player to see a glimpse of some mysterious entity. The entity also appears behind the player and then somehow sends the elevator to another strange place. This sequence will have audio of the void spirit talking to the player and audio of the spotlight to lead the player as to where they are supposed to look. To animate this sequence I used three animations that came with the Paragon: Sevarog asset pack to animate the spirit how I wanted as well as keyframing the transform of the spirit character. The rest of the animation involved toggling the visibilities of lights within the scene as well as changing materials. To change the materials of the elevator parts that become transparent I had to add a dither temporal antialiasing part to the material blueprint of the metal connected to opacity mask. This is because metallic materials can’t have their opacity manipulated normally. This is so the opacity of the metal goes down before it switches out with glass so the transition is smoother. This part will fade into the next scene to add to the magical teleportation as the two scenes will overlap momentarily.
Otherworld
This sequence starts with the player being accosted by inhabitants of the otherworld. One of them is cautious and taps the elevator, another is inquisitive and inspects the elevator and the final one is aggressive and attacks the elevator. The giant appears after a little while and scares them away. The giant then sees the player and roars before attempting to crush the player. The animations used in this section were all sourced from Mixamo. The positions of the characters were adjusted by manipulating the transforms as the animations intentionally didn’t move them.
Elevator
The elevator scene is a short scene in which the player seems to be back in the real world and is recovering from the events that just happened to them. The animation is simple as it is just manipulation of the transforms of both the player hands.
References
mixamo (N/A) Character Animations [Asset Resource]. Available online: https://www.mixamo.com/#/ [Accessed 5/1/2023].
Animations:
Banging Fist on Table
Climbing (Pulling Up To A Ledge)
Climbing Up A Wall
Goofy Running
Kicking (Male Thrust Kick With The Lead Foot)
Kneeling Inspecting (Kneeling and Inspecting Element With Hands)
Looking Around (Idle Stand Looking Around)
Looking Down (Looking Down And Away)
Look Over Shoulder (Quick Nervous Look Over Right Shoulder)
Mutant Roaring
Nervously Look Around (Nervously Looking Around Left To Right – Loop)
Petting (Petting A Large Animal)
Strut Walking (Male Strut Walk)
Reacting (Being Surprised And Looking Right)
Run Look Back (Running Looking Back)
Running (Running Forward Quickly)
Running (Silly Run)
Walking (Careful Walk)
Walking (Male Brutal Walk)
Immersive Design 4: Test Renders
This first test render shows an early version of the environment when the full area was lit up. There was an error with this render as the camera is faced the wrong way. This is due to how the 360 render works in Unreal as the start point will always be the start rotation of the camera when it is imported into the scene. I couldn’t figure out a direct fix so I instead rotated the entire environment so that it would face the right way. Flickering can also be seen on parts of the environment in the back. I couldn’t figure out why this was happening either which is one of the reasons I darkened this area as well as having no purpose for the animation.
My second test render shows experimentation with implementing the simulated embodiment. This test allowed me to figure out that the camera was a bit too far away from the elevator button as it isn’t realistic for the player to be that far from their hand. I also decided to render in 8K resolution for this test. Compared to the first test there was a noticeable increase in visual fidelity without the render times changing that much. From these results, I decided to render my final project in 8K resolution as higher visual fidelity would better immersive the users.
The final test render shows the final iteration of this office environment in which the important parts are the only parts lit up. It shows how I adjusted the position of the camera to better match the player hand as well as showing how I implemented the “navigation”. The elevator that the player fades into is actually a different one as I wanted to match the rotation of the camera so I duplicated the elevator facing the other way for when the player moves into the elevator.
Immersive Design 3: Planning and Asset Gathering
Idea Adjustments
Before starting to develop my final project, I decided to reorganise my story structure to make it a bit more cohesive and to allow for the most complex part to conclude the project. Originally, the idea was for the giant part to be the middle section and for the dark void area to end the piece. I decided to move the dark void area to the middle as it will end with the mysterious entity sending the player to the environment in which the giant resides. I also developed the giant part further. While searching for assets to use, I was inspired to change the setting from a fantasy village to an alien-like world. I personally thought this environment would be more unique and potentially more interesting to the player.
Environments
My approach to environment design was to make use of existing maps from asset packs and edit them to suit the needs of my animation or create them in Unreal.
Elevator
The basis of the elevator was a simple elevator asset from SketchFab but I recreated parts of the model using Unreal Engine’s unbuilt modelling tools as certain parts of the model had to be changed for what I wanted. This was because I wanted to change the material of the walls of the elevator to make them transparent but with how the original model was set up this changed too much of the elevator.
Office
The office utilised the “Loft Office (Modular)” asset pack. This perfectly captured the type of office I was looking for as it’s a nice looking environment that wasn’t overly complicated. I edited parts of the environment to better suit my project which wasn’t too hard due to the modular nature of the pack. I wanted to have a poster behind the player and the wall that was originally there didn’t work too well for that so I replaced it with another wall from the pack. I adjusted the lighting of the environment as I wanted the narrative to be that the player was leaving work late but the map was in the daytime. To match the later time setting, I closed the windows and replaced the open blinds as well as removing a clock with the wrong time. The original map had a meeting room which I removed as I instead placed the elevator where the doorway was. The poster which I added is a reference to the Alice in Wonderland brief as it is a picture of a white rabbit.
Dark Void
The void environment didn’t involve any asset packs as it was mainly just the elevator in a pitch black environment. To create it I duplicated the office map and removed the office assets leaving the elevator in darkness. However, as I wished to use light to direct the player for part of the void section I created a floor for the void using the Unreal tools.
Otherworld
For the otherworld I decided to look through an asset pack known as “Elite Landscapes: Bundle Pack” to see if there was a landscape that I would like to use. I decided to change my idea because of this pack as I was drawn to the alien landscapes included in the pack. The landscapes were quite big and intensive though so I had to delete parts of the landscape that weren’t going to be visible in my final render to make it easier to render. Due to the scale of the landscape, I had to adjust the size of my other models to better fit in with the landscape.
Characters
Player
As I wanted to show the player’s hands as a representation of VR controllers, I needed an asset for this. I decided to use the Remy character model from Mixamo which I imported into Maya to separate one of the hands. I also used the Mixamo skeleton to pose both a pointed hand and open hand to be used in the animations.
Void Spirit
The void spirit was intended to be spooky and mysterious so I decided on using the Paragon: Sevarog character asset pack which perfectly met these attributes. I did have to adjust the model slightly in Unreal as the character had a hammer which I didn’t want to use. To get rid of it, I used a material with 0% opacity as it was an easier solution than editing the model. This asset pack came ready with animations that would be suitable for my idea also so I didn’t need to find or create any animations for this character.
Citizens of the Otherworld
As I developed the otherworld section of the project, I decided that I wanted more characters than just the giant to add more narrative to this strange world. To meet the new aesthetic I decided on the Alien Soldier model from Mixamo. When testing this model in the world, it didn’t stand out in the environment as much as I wanted. To achieve this effect, I adjusted the material of the model to have a higher emissive value on it’s emissive texture which added a considerable glow to the characters removing any visibility issues.
Otherworld Giant
The final character of my piece was the giant. I wanted to get a menacing character model that would also fit in with the set aesthetic of the otherworld. I found this with the Paragon: Grux asset pack as it included a character model of an imposing figure wearing advanced technology. I chose a version of this model that had a magma effect on it’s materials to really make this character stand out when he arrives in the animation. This character pack did come with animations but they weren’t applicable for what I wanted so I brought the model into Maya to remove the weaponry as I wanted the giant to favour physical brute strength rather than weaponry. I also removed some accessories that I didn’t really like from the bottom half of the model.
References
Epic Games (2018) Paragon: Grux [Asset Pack]. Available online: https://www.unrealengine.com/marketplace/en-US/product/paragon-grux [Accessed 18/5/2023].
Epic Games (2018) Paragon: Sevarog [Asset Pack]. Available online: https://www.unrealengine.com/marketplace/en-US/product/paragon-sevarog [Accessed 18/5/2023].
H. Omar (2019) white bunny close-up photography [Image]. Available online: https://unsplash.com/photos/QUsiAcC90ZI [Accessed 9/5/2023].
Ihaegy.fu (2023) Simple Elevator with Animation [3D Model]. Available online: https://sketchfab.com/3d-models/simple-elevator-with-animation-770488a2f94a453fb90a7445bda63cd0 [Accessed 9/5/2023].
mixamo (N/A) Characters and Rigging [Asset Resource]. Available online: https://www.mixamo.com/#/ [Accessed 9/5/2023].
Characters:
Remy (Player Hands)
Alien Soldier (Otherworld Citizens)
Next Level 3D (2021) Loft Office (Modular) [Asset Pack]. Available online: https://www.unrealengine.com/marketplace/en-US/product/loft-office-modular [Accessed 9/5/2023].
Velarion (2019) Elite Landscapes: Bundle Pack [Asset Pack]. Available online: https://www.unrealengine.com/marketplace/en-US/product/elite-landscapes-bundle-pack [Accessed 11/5/2023].
Immersive Design 2: Digital Affordance and Considerations
Player Direction
As the project is going to have a 360 degree view for the player, I am going to have to use techniques to lead the player’s view to what I want them to see. The way I plan to do this throughout my experience is through the use of 3D audio and light. I will use these in tandem to get a better result in getting the player to look the right way. The audio will cause the player to look in the right direction and the use of light will keep them focused on where the next thing will happen so they manage to follow along with what is happening. However, I will allow for segments in which the player can get their bearings with the different environments so they are able to see most of them mitigating any fear of missing out on parts of the animation.
Navigation
My project idea makes the issue of navigation within VR not a big problem as the player is mainly going to be stationary within the elevator. The segment in which the player enters the elevator will make use of a fade to black transition as moving the player in a 360 experience like this could cause motion sickness as it is very easy to cause a disconnect with what the player is seeing and feeling. Especially because of the lack of interaction the player has with this prototype project as the movement will be completely out of their control and unnatural. Additionally, the 360 view will not take into account the movement because of the project being pre-rendered and not real time which could be very jarring for the player.
Audio
To implement audio how I mentioned before I am going to make use of Adobe Audition to simulate 3D audio for my video. I aim to do this by creating binaural audio from different audio clips timed correctly with my animation. To make this process easier I intend to make use of the dearVR MICRO plugin for Audition which gives greater control over the simulated 3D position of the audio clips.
Embodiment
I want this prototype to simulate a playable VR experience. To achieve this I intend to have some segments in which the player’s “hands” show up as if the player is controlling them with VR controllers to interact with the environment or react to the events of my 360 experience. This will be quite limited due to how the prototype functions but I will implement the audio techniques to attempt to make the player see the prototype embodiment. I am only using disembodied hands because attempting to simulate a full body in this prototype render would not work very well with the 360 degree view as the body would only face in one direction. This would make the experience less believable and therefore reduce the immersion.
References
Dear Reality (n.d.) dearVR MICRO [Application Plugin]. Available online: https://www.dear-reality.com/products/dearvr-micro [Accessed 22/5/2023].
Immersive Design 1: Idea Explanation
Premise
This immersive project is very loosely based on Alice in Wonderland as the idea of a figurative “rabbit hole” is the main focus instead. The base idea is that the player/user is transported to other dimensions or worlds by a seemingly normal elevator as they leave their place of work. The elevator takes the place of the “rabbit hole” as it takes the player deeper and deeper into fantastical locations. However, this will probably only involve a couple of locations with the limited time I have for the final video. Besides the obvious Alice in Wonderland inspiration, I am also partly inspired by the Roald Dahl story “Charlie and the Great Glass Elevator” as the idea of a fantastical adventure in a transparent elevator is part of my base idea.
Story
As there is a limit of about a minute for the final immersive prototype, the story for the idea is very fast paced. The initial idea is for the player to be in a normal office building before being transported to a fantasy village type location via the elevator. The elevator will seemingly start to break down before the walls peel off to show the village. After the player has had adequate time to take in the sights of the village, a giant will climb over some nearby mountains and approach the elevator. The giant will pick up the elevator and then swallow it and the player whole. This transitions to the next area which makes use of darkness to hide some kind of sinister presence. The aim in this part is to try and spook the player a little with use of audio and limited light. After this section, the player appears back in the world with everything seemingly fine. The question is whether what just happened was real or not.
Design
The software this project will be created in is Unreal Engine with the use of an optional Movie Render plugin that allows an Unreal render to be captured in a full 360 degree view. However, as it will be a render there won’t be any actual interaction for the player but this is fine for this as it is only a prototype of what this idea could be if ever developed further. Unreal Engine was chosen as I have much more experience in it than Maya and I can make use of the level sequence cinematics system to animate my scenes. The render times should also be more manageable for my personal computer allowing me to test my project semi-regularly to work out any issues.
References
yannick (2007) Glass Elevator Boulogne Paris [Image]. Available online: https://www.flickr.com/photos/21624989@N07/2120607541/ [Accessed 18/5/2023].