Virtual Embodiment

The most praised ability of Virtual Reality is its capability to immerse the user in a Virtual Environment — to the degree that the subject feels present in it. The magic is to be fooled by the system so that one feels present where one actually does not physically reside. This effect can, however, turn even more magical. A deeper step into the effects of technological immersion is found in the concept of Virtual Embodiment. If a subject is embodied virtually,  not only is the virtual environment accepted as such; the subject also identifies with a virtual body or avatar inside the virtual environment. This differs from realizing which character you control in a game — within Virtual Embodiment it is the same processes that make you identify with your real body that makes you identify with a virtual one. This is a key point, as it is why research into virtual embodiment is important.

Peeling layers of the onion: VR can be a tool to discover who we are, through investigation of what and how we identify with our bodies. Illustration: “Mask of Day by Day” by Paulo Zerbato.

Hacking and Experimenting with Consciousness

What is fascinating about both of these possibilities of illusion, then — is how, and that, they are possible at all. Knowledge on how to achieve such immersion is obviously relevant for all VR developers, but the knowledge that can be obtained by researching these phenomena goes far beyond knowing how to apply it in VR technology. By creating experiments in VR, we can generate, and investigate, phenomenas of the mind under various experimental conditions. Exploring Virtual Embodiment, for instance, can enable us with a better understanding of our self-consciousness and the relationship between body and mind. Because of this wider span, research on Virtual Embodiment attracts neuroscience researchers, psychologists, information scientists and philosopher’s alike.

The Rubber Hand Illusion

The Rubber Hand Illusion (RHI) is an excellent example of the kind of ‘brain hacks’ that can be achieved by sensory manipulation. The illusion, as illustrated below, is a perfectly simple experiment that does not even require the use of VR technology to perform.  The RHI was introduced by Ehrson, Spence & Passingham (2004) and has been an ingenious way to illustrate how we identify with our bodies. More importantly for this entry, the results of the experiment has inspired further research on Virtual Embodiment.

Illustration from Thomas Metzinger’s book “The Ego Tunnel: The Science of The Mind and The Myth of the Self”

In the RHI, the hand of the subject is replaced by a rubber hand, while the normal hand is blocked from sight by a separating wall. When the subject is sitting as such, a researcher will stroke each hand, both the rubber and the physical hand, simultaneously. Now, the question is what happens when experiencing the sensory impression of stroking, all the while seeing a corresponding stroke on the rubber hand?

Put very simply, the brain does a ‘reasonable guess’ that this hand is indeed the correct physical hand attached to your body.  You feel that the rubber hand is yours, with nerve-endings and all — and you couple your physical feelings to the vision of the hand. This means that in your subjective experience, the rubber hand is the hand that has the sensation. Ehrson et. al write that their results suggested that “multisensory integration in the premotor cortex provides a mechanism for bodily self-attribution”. When our brains receive sensory information from two differing sensory inputs (sight+feel), these are coupled: the brain is coupling the stroking-sensation with imagery of a nearby-hand being stroked, and this is enough for the brain to attribute its self with the hand, to acknowledge it as its own.

This simple experiment share a lot of principles with the concept of Virtual Embodiment, and has inspired research in the field that we will present in this entry.

Some experience out of body experiences (OBEs) on the onset of sleep or waking up. Often they may feel that they are floating over their bodies. VR may help to study such states of consciousness by systematically inducing them.

Virtual Body Illusion

In a later experiment by Lenggenhager et.al (2007), not only the hands of the subjects — but their whole bodies were replaced with virtual representations. Moreover, in the experiment they present, the bodies are seen from behind. In effect, they were simulating out-of-body experiences, with very interesting results.

The experiment was conducted as such: the subjects wore a Head-Mounted Display which projected imagery from a camera located behind the subjects. As such, the subjects could see a representation of their bodies “live”, but from behind. Of course, this is deviating slightly from how we normally experience life. Although the subjects saw their body responding and performing actions in real time as under normal conditions — there is a logical dissonance due to the mismatch between the location of the subjects’ eyes in the virtual environment, and what these eyes see. Effectively, the user is seeing inside a pair of “portal” binoculars (HMD), which display the light from, if not another dimension, then at least a few feet away. And this will be a part of the point.

What is interesting about this experiment is not necessarily simply that the users feel present where they do not reside physically, but how the distance is only a few feet off. The users feel present right outside of their bodies. The situation is similar, the body and the environment is there, but everything is a bit off. What is interesting to investigate then, is how the body adapts to this. Will it accept that it now controls its body from a third person perspective, similarly to how Stratton’s subjects got used to seeing the world upside down?

What they studied was basically whether this change of perspective had an impact on where the users felt embodied. To investigate this, the researchers stroked the subjects as they did in the Rubber Hand Illusion, except at their backs — so that it was perceivable by them. The question is then where this physical feeling will be attributed to — how will the phenomena of the subjective experience present themselves to the subjects?

Out of Body experiences can be achieved virtually by using sensory impressions from other locations, for instance five meters behind you as in the experiment by Ehrson (2007). You can then effectively look at yourself from the outside.

First of all, to be clear on this — the sensory data of being stroked will initially be provided by the nerves in the physical shoulder of the user. The problem of the brain, however, is that the shoulder is out of sight — blocked by the Head-Mounted Display. There is, however, the visual impression of a shoulder on a person standing in front — being scratched in exactly the same way. Although the nerve-endings definitely feel the stroking, the problem is that where this feeling will be placed in our subjective experience is not the responsibility of the shoulder, but rather the brain. And, as the placement of the physical feeling in the bodily self-consciousness is largely dependent on vision for coordinates, what will happen? How will the brain fix this sensory discord?

In this beautifully written article by The New Yorker, its author Rothman describes one of the co-authors of the research paper, Thomas Metzinger’s, own experience undergoing the experimental conditions:

Metzinger could feel the stroking, but the body to which it was happening seemed to be situated in front of him. He felt a strange sensation, as though he were drifting in space, or being stretched between the two bodies. He wanted to jump entirely into the body before him, but couldn’t. He seemed marooned outside of himself. It wasn’t quite an out-of-body experience, but it was proof that, using computer technology, the self-model could easily be manipulated. A new area of research had been created: virtual embodiment.”

Are We Already Living in Virtual Reality?” — The New Yorker has a brilliant, long, read on Virtual Embodiment that features interviews with VR and Consciousness researchers Prof. Mel Slater and Prof. Thomas Metzinger.

Phantom Pain

Another curious potential effect of Virtual Embodiment, is the possibility of phantom sensory impressions as well. Handling virtual objects while being embodied, for instance, may convince your body to expect pain or touch — and so this is, somehow, actively generated. Because of this, VR may be a way to study how phantom pain is created, and further how it can be alleviated. For instance, several studies show how VR can embody a subject missing a leg in a body with two legs, similarly to traditional mirror therapy treatment, which is effective in reducing phantom pain. Again — what may be most interesting here is the possibility of systematically creating the phenomena and studying it afterwards. For instance, as Metzinger is quoted on in The New Yorker’s article, it may be supposed that phantom pain is created by a body model not corresponding to the physical reality. This will be the case for phantom pain in VR: it is not based on the physical reality, you are only relating to a virtual reality instead. Similarly, those those with real phantom pain may also be relating to a certain kind of “virtual reality”, but rather one in the format of their skewed narratives — maintained by their minds instead of a computer.

That the narrative, worldview and consciousness that our brain’s experience and generate is often not the best match with reality is not something new. As for Matrise, these concepts reminds us of the conclusion from our three-series entry towards a metaphysical standpoint on VR, in which we discussed VR as rather examplifying of our abstracting tendencies of mind. These entries can be read at Matrise, and were called: 1) On Mediums of Abstraction and Transparency, 2) Heidegger’s Virtual Reality, and 3) The Mind as Medium.

Virtual Embodiment for Social Good

Now that we have discussed the concept of Virtual Embodiment, it may be natural to discuss what this knowledge can be used for. As discussed already, generating experiments in VR that hacks our self models, may provide useful knowledge on the structure of our self-consciousness. Apart from this general knowledge, some may also have practical utilisation in applied VR for specific scenarios.

Racial Bias

A very exciting paper that describes work utilizing virtual embodiment, is one by Banakou, Hanumanthu and Slater. In the project, they embodied White people in Black bodies, and found that this significantly reduced their implicit racial bias! The article can be found and read in its entirety here (abstract available for all).

Domestic Violence

Another interesting project by Seinfeld et. al, is one in which male offenders of domestic violence became embodied in the role of a female victim in a virtual scenario. At first in the experiment, the male subject is familiarized with his new, female, virtual body and the new virtual environment. When the body ownership illusion, or virtual embodiment, has been achieved, a virtual male enters the room and becomes verbally abusive. All this time, the subject can see his own female body reflected in a mirror, with all his actions corresponding to his. After a while, the virtual male starts to physically throw around things and start to appear violent. Eventually it escalates and he gets closer into what feels like the subjects personal space, and appear threatening.

They write:

Our results revealed that offenders have a significantly lower ability to recognize fear in female faces compared to controls, with a bias towards classifying fearful faces as happy. After being embodied in a female victim, offenders improved their ability to recognize fearful female faces and reduced their bias towards recognizing fearful faces as happy”

The article can be read in its entirety at ResearchGate.

Staying Updated in the field of Virtual Embodiment

Research on Virtual Embodiment is happening continuously. To stay updated on this area of VR research, I enjoy following Mel Slater, Mavi Sanches-Vives and Thomas Metzinger on Twitter. Last but not least, I would stay updated on Virtual Bodyworks at Twitter, of which both Sanchez-Vives and Slater are co-founders of.


N.B: This entry lies at the centre of Matrise’s interests, and we are planning on writing several entries on this topic further in philosophical directions. Have any ideas or want to contribute? Please contact us.

Literature list

 

 

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSaveSaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

Inner as Outer: Projecting Mental States as External Reality.

Introduction to Mysticism

Within Mysticism, the merging of Self and World — Inner and Outer — is seen as the utmost aim. Mysticism can be found within most of the world religions, such as Buddhism, Christianity, Hinduism and Islam — and its aim is often formulated as union with God. Depending on the religion, however, the degree to which Mysticism is the common way of practicing the religion varies. Although many religions have such contemplative practices, they are not always adopted by the religion’s followers at large.

When discussing «Union with God», it should be noted that the term «God» varies in its meaning between these religions. The contemplative practices often have significantly varying metaphysics, for instance Monotheistic (Christianity), Polytheistic (Hinduism), and relatively Atheistic or Agnostic (Buddhism). Be this as it may, their descriptions of the experience of this merging of Self and God is often strikingly similar. These states of enlightenment are often described as ecstatic, in which the conscious experience can not be placed within our normal frames of language or understanding.

What also unites the different traditions, is that such states of consciousness is usually  worked towards through contemplative practice such as yoga, meditation or other disciplines of focus or conscious attention. Other techniques for achieving these ecstasies have have been ascetic ones, such as fasting, waking, isolation — or other ways of stirring the Self to war.

The experience of seeing the Inner as Outer, and the Outer as Inner, is often described as the feeling that living itself is an experience of seeing and perceiving Oneself and/or God. Within this worldview, there is no Self relating to anything external.

Non-duality: synchronization of Inner & Outer

The concept of merging Inner and Outer, or Self and God, can each be viewed either in very material or spiritual terms. Although materiality and spirituality do not have to differ metaphysically, separating these gives us some communicative benefits — and Mysticism may be explained and spoken of from both these perspectives. Discussing the Inner as Outer purely «scientifically», if you will, makes sense in that all our perceptions of the Outer world is indeed created Inner, and as such — Reality will always be a synergy of Inner and Outer. We know that we do not see, or have ever seen, anything which we ourselves do not actively generate. As neuroscientist and consciousness researcher Anil Seth put it, “our brains are actively hallunicating our conscious reality”.

States where a subject experiences the Inner and Outer as ‘one’, is often referred to as «non-dual».  Often while speaking of Inner and Outer, we tend to implicitly reinforce the Self and the World as a duality (when pitching a solution we often have to pitch the problem first). By using the word «non-dual» instead of ‘one’, we may pinpoint the nuance that it is not a duality in separation, but neither completely “same-same”. Although it is non-dual, neither is it all same or flat — least of all static!

Although we classify and divide our reality, fundamentally what we perceive is a stream of experience, which in every sense is simply “reality” before divided, and, again, actively created by us. This is not to say that there are no external reality or world — but it definitely is to say that all which is external is perceived first and foremost, solely, internally. Experientially — externality has never been perceived, except as a subcomponent of internality.

A vase, or two faces? Each defines the other, and neither exist without the other.

Experiencing and Sensing the Non-Dual

This causal explanation, however, leaves out the experiential aspects of the non-duality. Although it may make sense on paper, it matters little to us as we absolutely perceive the world as dual — as subjects relating to a World. Within Philosophy, this traditional way of adhering to and speaking of the world,  is referred to as the subject-object dichotomy. Although, between different cultures and continents, the degree to which we adhere to this way of thinking vary in its intensity, it is nevertheless definitely an essential part of the human experience which we share.

How the material explanation can be said to be different from the spiritual in this sense, is that the spiritual concern is to experience the Inner as Outer, not to understand it cognitively. As such, and towards that, meditation practices such as Mindfulness and Yoga have existed, to increase wellbeing by increasing the degree to which one feels in union with God, or for those who do not fancy the term; to the degree which one has peace with oneself and the world.

Contemplative practices such as yoga and meditation, has the last fifty years become more popular in western societies. Although they have been subject to a certain degree of metaphysical raffination the last years, these methods are nevertheless largely old and traditional. The most common of these contemplative practices we see today is adopted from the Vippassana practice, commonly known as Mindfulness. These methods are now commonly used in psychological treatment of anxiety and depression, and research has the latest years started to uncover the benefits of learning to be able to sit quietly with your mind and, well, deal with shit, or seeing it for what it is.

In the next section, we will discuss an approach utilizing Virtual Reality to aid in Mindfulness meditation — which can help to perceive the Inner as Outer.

A common belief is that the aim of contemplative practices is to empty the mind. In a sense, it can be said to be correct, in that meditation practices often seek to eradicate, dilute or cancel the self-referential narratives.

The effects of Mindfulness meditation

The essence of Mindfulness or similar contemplative practices, lie in their manipulation of identity. We stated “the problem” of Mysticism as the gap between Self and Other — and for this separation to be there, we must necessarily have a relatively thoroughly defined sense of self. For most of us, this tend to be limited to the cognitive processes that constitute our mental narrative (the personalized voice in our heads, our formulated will, and how it appears to direct our actions and plans our lives). It is actually to a far lesser extent our bodies, although this also attributes to our self-consciousness.

Mindfulness is about being present attentively in each moment to one’s state of mind. When doing such focus excercises directed at the mind, and observing these mental processes closely, the idea or view of them as solid things starts to unravel. When rather seeing them as thoughts from a distance, they appear untangled to us, and we perceive our own existence as distinct from those thoughts.

Virtual Reality Biofeedback as Meditation aid

One of the great benefits of VR is its ability to project and represent data in the format of the reality encompassing us. Within the context of this entry, we could say therefore that VR can simulate what we perceive as the Outer. The question may then be asked: how can we project our Inner in to this medium of Outer?

Although I believe we will see more work on VR biofeedback within this domain in the future, in this entry we will focus only on one research paper in particular to examplify our case. At last years CHI conference, the world leading conference on Human Computer Interaction, Joan Sol Roo and his colleagues presented their work on Inner Garden: a mixed reality sandbox for mindfulness. The artifact is a physical sandbox, which the user can shape to a given terrain. The sandbox is given generally visually augmented by a projector with colors and shapes — and physical changes to the sandbox will also alter the output of the projector, which deliver terrain information such as sea levels and green growth.

The sandbox is just not physical, however; by placing a physical avatar in the physical sandbox, you can enter into the land you created in Immersive VR. A 3D-model of the land you created physically can be seen virtually, from the viewpoint of your placed avatar.

The Sandbox, which heights of the sand have been turned into an island by the projector.

Attached, to measure your inner states, is both breathing- and heart rate sensors — which are coupled to provide visual and auditive feedback. In this way, you can synchronize your breath to control the environment and rythm and breaking of the waves. The Inner Garden represents your inner state, and. by practicising breathing techniques, the flora of your world will get greener and more animals will appear.

In this way, Inner Garden works as a great example of representing Inner phenomena as External Reality. Very conceptually interesting, and hopefully one day we will also see empirical studies on similar artifacts.

You can read more about Inner Garden, which received an honorable mention at CHI’17, here.


What do you think? Do you have any ideas for VR applications using biofeedback?  Please comment below.

Literature list

 

 

 

 

 

SaveSave

SaveSave

 

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

The History of Virtual Reality

In recent years, Virtual Reality (VR) technology has finally reached the masses. 2016 was called “The Year of VR” as several actors released their Head-Mounted Displays (HMDs) on the consumer market. While HTC, Oculus and Playstation delivered high quality HMDs that require external computers to run, the year also opened up for high quality mobile VR. Both Google with their Daydream View and Samsung/Oculus with their GEAR VR have provided an easier step for consumers to enter the world of VR. These mobile VR solutions  offer better internal measurement units than the simpler Cardboard devices, and also feature simple controllers for interaction. We now see the market spreading out both  in quality and accessibility: in 2018 we have both seen the coming of the HTC Vive Pro, a more expensive high-end HMD with increased resolution, and the Oculus Go, which is a reasonably-priced ($200) stand-alone 3DOF (3 Degrees of Freedom) HMD for the starters.

It is natural to wonder how all of this started. Why did we for instance not see much VR before 2016? When it now seems to be relatively easy for commercial actors to push out HMDs down to $200, why did it not happen sooner? Of course, we have had Oculus’ development kits since 2013 — but even this is very recent.  When Google released their Cardboard (a simple HMD made out of cardboard  and some lenses), it seemed incredilous that VR could be attainable for the smartphone for only 50 cents. This, however, only points us toward how fascinatingly simple the underlying pinciples of VR technology actually are.

In this entry, we will trace the VR tech we see today back to its roots. We will go back about two hundred years, and work ourselves jumpingly forward to the very recent innovative technologies.

Stereoscope

A drawing of the Lothian Stereoscope, released in 1895; one of many different models.

In 1838, Sir Charles Wheatstone developed what would be the first Stereoscope. Even before the camera was invented, people were seeing (drawn) images with 3D effect through stereoscopes.  Stereoscopy, that is, perceptory illusion of depth, is achieved by displaying a slightly different segment of an image to each eye. Wheatstone achieved this by separating the two images by a piece of wood, and providing a lens directing the light, between each eye and the corresponding image. While looking through the stereoscope, our brains perceive the two images as one image, with the added 3D effect due to the varying segments of the images. This effect is simply caused by an utilization of how our eyes and brain work, by combining the sensory data from each eye into one. We may, for instance, most likely be able to recall  sometimes «seeing double», when our brains have yet not our varying visual impressions.

Since Wheatstone, different stereoscopes have been produced all the way up to the Google Cardboard or other HMDs; which instead of drawn images, or later photographs,  utilizes a screen to deliver the imagery to the eyes. Actually, in the early 1900s, Stereoscopes functioned as home entertainment devices, and «stereo cards» such as the image seen below could be purchased from photography companies.

Stereo card of St. Peters Church in the Vatican. Such cards, picturing tourist attractions all over the world, could be purchased and viewed at home in a Stereoscope.

Stereoscopes and modern day Virtual Reality HMDs share the essential feature of stereoscopic depth illusion (3D). Apart from that, however,  a lot has obviously happened since 1838, which we now regard as essential for the feeling of presence and realism, and which makes the technology capable of simulating realities. The most important of these have been moving images, 3D environments, interaction, and 360 degrees of orientation. With the stereoscope, images very static in every sense.

Sensorama

In the mid 1950s, however, some people saw the opportunity to spice up their stereoscopes a bit. A bold attempt at enrichening this, was the Sensorama. In addition to providing a stereoscope with motion pictures in 3D and color,  all quite revelutionary, the device had fans for simulating wind, odor transmitters for smell of the environment, stereo sound, and even a moving chair!

The Sensorama, or «Experience Theather». Illustration from Morton Heilig’s 1962 US Patent.

Pygmalion’s Spectacles

The idea of the Sensorama, or VR in general, can as many other innovative future-defining ideas, be found in science fiction literature. Before its conception, in the 1930s, the science fiction writer Stanley G. Weinbaum introduced the idea of «Pygmalion’s Spectacles». By wearing these, the user could experience a fictional, or virtual world, with holographs, smell, taste and touch, and make the virtual come alive. Pygmalion, which «Pygmalions Spectacles» were named after, were a Greek sculptor who fell in love with his sculpture, and so begged Venus that it would come alive. The Myth sheds an interesting light on VR as an ultimate dream of humanity, to create realities for ourselves to inhibit, or to create images in the format of reality.

Pygmalion, which «Pygmalions Spectacles» were named after, were a Greek sculptor who fell in love with his sculpture. He begged Venus that it would come alive. Painting by Jean-Baptiste Regnault.

Information Technology

To take a leap towards another paradigm shift in VR tech, we must enter the land of 1s and 0s.  The Stereoscope slowly moved from drawn images, to photographs, and further to moving images with the Sensorama. None of these, however, supported spherical environments that could be perceived in all their 360˚. To achieve this,  certain sensors and further computation based on their sensory input has been necessary.  The most important and interesting of these sensors, has been the Gyroscope.

The Focault Gyroscope, created by physicist Jean Bernard León Focault.

The Gyroscope was given its name by Phycisist Jean Bernard León Focault in 1852 who used the device as a means to prove the rotation of the Earth.  The gyroscope is a device consisting of a spinning top with a pair of gimbals. Its origin can not be traced to a single invention or inventor, as tops have originated in many ancient civilizations — however, unlike the «complete» Gyroscope, these were not necessarily used as instruments.  Although Focault’s gyro were not the first that were used as a measuring instrument, its affordances work well to examplify the usefulness of gyroscopes in VR HMDs; the important feature it affords is the measure of rotation, which key lies in the Gyroscope’s tops’ possibility for free rotation.

Gyroscopes are fun artifacts to play with as they seem to defy gravity. While spinning, they can remain stable in most positions. If placed on a platform, that unlike the gyro remain stable, the position in terms of rotation can be measured relatively to the platform, and as such we can also measure the rotation of a HMD. It should be noted, however, that the gyroscopes of today are not pretty mechanical objects of brass anymore, which, although they do no longer satisfy our aesthetic appetite, at least have the benefit of fitting into our smartphones and HMDs. Today, gyroscopes have heights, widths and lengths of only millimeters, which opens the possibility for placing them inside smartphones and HMDs.

The Sword of Damocles

The Sword of Damocles, an old greek cultural symbol of Mortality — ever close to those in power. We see the sword hanging from a single horse hair over the head of Damocles.

Fifty years ago, in 1968, Ivan Sutherland and his student Bob Sproull created the first computer-driven stereoscopic (3D) Head-Mounted Graphical Display with 360˚ head-tracking. The HMD was not exactly lightweight, and was named after the «Sword of Damocles» because of the heavy stand hovering over its users head. As can be seen in the illustration below, the head-tracking was mechanical, and did not in fact use a Gyroscope. Later, however, this became a more fruitful approach, so as to avoid the massive device rotating over the users head.

The field of view and graphical fidelity of the Sword of Damocles were obviously quite low, yet the Sword of Damocles is the first widely known HMD, and has since its dawn inspired and launched further decades of VR research.

The first Virtual Reality Head-Mounted Display, named after the Sword of Damocles, because of its great weight hanging over the user’s head.

Towards the modern HMD

Since the invention by Sutherland and Sproull,  creation and use of HMDs was seen more and more within research. As computational power became faster and cheaper, the HMDs decreased in size, and increased in field of view, graphical fidelity and refresh rates. Yet — even back in the 1990s for instance, the technology was still expensive, and poor in terms of graphical realism. It often caused cybersickness due to low refresh rates, and high motion to photon latency. Of this reason, as with any really powerful computer from that time, VR was reserved for research universities that could invest into the technologies, or businesses with resources to experiment with the technology. There were some attempts at commercializing VR for gaming purposes, such as the SEGA Genesis and Nintendo Virtual Boy — however, both of these remained largely as prototypes and were later discontinued. To this day, none of these companies has since experimented with the technology, although Nintendo in 2010  released the Nintendo 3DS which utilizes a stereoscopic display that does not require any glasses.

 

Image of a 3D model of the HTC Vive Pro.

Conclusion

Since the Sword of Damocles, VR technology has undergone small incremental changes leading to where we are today, mainly as a result of general computer and graphics research, and the natural progression of Moore’s Law; today our processors are smaller and more powerful, and our screens of higher resolution.

In addition to this, however, there are certain very recent technologies that have impacted the VR as we know it today as well. In Matrise’s glossary, we briefly present and define some of these technologies. Some that can be read about is Foveated Rendering and  Low Pixel Persistence Modes.


Did we miss anything? Any thoughts are welcome in the comments section.

SaveSave

SaveSave

Virtual Reality and Exposure Therapy

The most essential feature of VR is its ability to simulate what is not real. This is its core concept, and what causes its radical exclusivity and novelty. The benefits of ‘avoiding reality’ in this sense, is most often that virtuality is more cost-effective than reality. For instance, corporations worldwide train their employees in VR as it saves money to avoid renting a physical location and hiring physical trainers. ‘Money’ in this case, is of course just a measure of effectivity: it takes less resources to achieve certain objectives virtually than physically. The cost is not the only benefit, however; the virtual may also be safer. We see this especially within surgery, where a failed operation on a virtual patient is much preferred than on a real one.

“Scream”, by the expressionist painter Edvard Munch. Want to experience Munch in VR?  Read our entry on Art in Virtual Reality.

Exposure Therapy
Another scenario where virtuality may be preffered  is psychological treatment of anxiety disorders. Anxiety is a terrible disorder in the way it is eating away the lives of the sufferers, and is hard to treat to by non-addictive pharmaceutical medicine. Psychological treatment, however, is in general very successful towards certain anxiety disorders.  Agoraphobia, arachnophobia, glossophobia, etc., can be treated by what we call “exposure therapy”.

Under exposure therapy, the patient usually get together with a psychologist, and is asked to express their fears of the situation of exposure. Here they answer what they think will happen, and how they think they will react. Their fears are pinpointed, and their catastrophic thinking is outlined. In these cases, it is not uncommon that patients believe they will literally stop breathing, or die, etc.: the narrative which operates is something they buy heavily in to, and the key of exposure therapy is to challenge their acceptance of this narrative. To a certain extent, this is a central problem of anxiety disorders: patients very seldom challenge these fears, of obvious reasons, and so their map of how the world works is not challenged and updated by reality. This is, through exposure therapy, systematized.

When the patient has been exposed to their fear scenario — the psychologist confront the patient with their initial fears that were written down prior to the exposure. The patient is then encouraged to reflect on the gap between their fears and what actually happened, something which we refer to as inhibitory learning. This kind of treatment falls under what is depicted as Cognitive Behavioral Therapy (CBT); by actively challenging the patients’ mental model of the world by reflection on facts.

Long story short — exposure therapy works. The largest problem with exposure therapy is, as usual, the cost. Having highly educated psychologists dedicated to the task is expensive enough in itself — but arranging the exposure to a fear scenario is an often greater challenge, practically and economically. It is not really convenient to summon spiders into the psychologist’s office, for instance. Arranging complicated fear scenarios and executing them is not convenient, and at high cost, which is a hinder for an otherwise effective treatment.

Virtual Reality Exposure Therapy can be used to treat, for instance, arachnophobia (fear of spiders). If you are not afraid of spiders, you might need to increase their size. I can recommend trying Farpoint for PSVR, which features giant space spiders similar to those in this illustration by Alphonse de Neuville.

Virtual Reality Exposure Therapy
This is where the concept of VR enters our story, as we start talking of Virtual Reality Exposure Therapy (VREP). By using virtual environments instead of actual physical locations, effective exposure therapy can be offered to more people at lower cost. At the University of Bergen, through the research project INTROMAT, we develop and do research on VR Exposure Therapy for adolescents with fear of public speaking. The INTROMAT project aims to introduce personalized treatment of mental health problems using adaptive technology. Currently, we are working on a very exciting approach to this, which Matrise will cover in more detail later.

The question that often raises itself when we discuss the concept of VRET, is whether we can fear what we know is not real. Although we know what it is like to be nervous before talks, it is perhaps hard to imagine being afraid of speaking in front of virtual subjects in which ‘nobody’s home’. On this point, however, the research is very clear. As Lindner et. al (2017) writes, “decades of research and more than 20 randomized controlled trials show that [VRET] is effective in reducing fear and anxiety”. The reason why VRET is interesting now, today, is then not necessarily because VR is finally good enough to deliver realistic virtual scenarios. VRET has been shown to be effective with VR technologies far inferior to those setups we have commercially available today.  The reason for its relevance as a research subject now, is because the technologies are finally cheap enough to successfully be used in large scale treatment. It is therefore time to revisit the previous research, and look at how this treatment can be improved further.

This is why INTROMAT now is looking at new ways of improving and investigating ‘state of the art VRET’.

Thoughts are welcome below!

Literature list

SaveSave

Virtual Reality Memory Palaces

The ancient mnemonic “method of Loci” has been in use for thousands of years. From memory competitors to students, the method helps to memorise items by combining visual and spatial cues. Many have perhaps heard of the technique through popular TV shows such as Sherlock or Hannibal, where their “Mind Palaces” or “Memory Palaces” are portrayed as a genius trait.

Illustration based on “Mind Attic” by wonderful artist Tisserande. Original version without Head-Mounted Display can be found here.

The real method, however, is perfectly simple. You start by finding a location to act as your Memory Palace. It should be a place you know well, for instance your apartment or your university. You further isolate several rooms, and sub-parts of those rooms, that amount to the number of things you want to recall. Next, you may want to have a certain route which you mentally walk through your Memory Palace. This is where the method comes in: you visualise what you want to recall at the given places in your Mind Palace. Need to go shopping? Imagine coffee beans poured out on the floor, orange juice cartons in the sofa, and bananas hanging on the TV. If you need to recall something more abstract, you can get creative and use the visualisation as an association to what you want to recall instead.

This method is believed to be powerful because of the combination of information with visual cues (the visualisation of the memory items), and with spatial cues (their given place in the environment). Curiously, however, this exploitation of the visual and spatial cues in the brain, does not involve our vision, or true perception of spatial three-dimensional spaces at all — only through our “inner eye” and our already-established memory of spatial environments.

“Get out — I need to go to my Mind Palace”. Benedict Cumberbatch in BBC’s Sherlock  — Quite busy in his Memory Palace. Illustration of by edthatch.

But what if we could perform this method with raw, fresh vision that gives us both the visual and spatial cues? Such a task is perfect for the medium of Immersive Virtual Reality. In VR, we are provided with highly optimisible a visuo-spatial environment made of software, which gives us the opportunity  to tailor the Memory Palaces to our own needs. By exploiting the visuo-spatial elements of the method of Loci in a much more explicit sense,  the method may perhaps work even better, or be more fun and easy to use.

Addressing this, I created an a VR memory palace app as part of my master’s thesis in Information Science. The application is called “Mnemosyne”,  after the Greek goddess of memory, of which country the Method of Loci originated. Developed with A-Frame, the application presents you with a text field where you can enter the things you want to recall. You then put on your VR headset, and are presented with an apartment with five rooms where you can place these memory items you entered. The algorithm returns images based on your text preferences from a search engine, and places these in the virtual environment. When this is done, you just walk around and watch to memorise the items.

Mnemosyne is still just a prototype, and we can imagine it will have room for many improvements in the future – after more research has been done. Currently, we at the Centre for the Science of Learning & Technology (SLATE) have performed a pilot study with 18 participants. The short paper discussing the findings was published in June 2018 in the Springer’s series «Lecture Notes on Computer Science», in the «Augmented Reality, Virtual Reality and Computer Graphics» edition. The conference paper was presented in the beautiful village of Otranto at the Salento AVR conference in July.

Our first paper on VR Memory Palaces was posted in the LNCS on Augmented Reality, Virtual Reality and Computer Graphics.

Our paper, ‘Mnemosyne: Adapting the Method of Loci to Immersive Virtual Reality’, can be accessed at Springer. Stay put at Matrise for further updates on our work regarding Virtual Reality Memory Palaces.

Literature list

Virtual Reality Journalism

Journalism is largely defined by which medium it uses to convey its message. The last hundred years, it has moved from medium to medium: from text to radio, and further from photographs to video. With each new medium the fidelity of the message it is providing is steadily increasing. This is perhaps especially clear now with the use of Virtual Reality (VR) technologies for journalistic purposes. By using 360° stereoscopic (3D) cameras, we are getting very close to capturing subjective realities at given points in spacetime.

The Ultimate Empathy Machine

In his Ted Talk, Chris Milk describes the potential of VR for creating “the ultimate empathy machine”, which later has been the subject for extensive debate. The message is that we may be more empathetic towards others if we can “literally”, or at least virtually, view the situation through their eyes. Many journalistic projects have focused on refugees or war zones, such as the stories “Fight for Fallujah” and “The Displaced“. In these stories, the camera works as the eyes of the observer, as the screen in the VR goggles is mounted directly on to the eyes. The user is presented to the story through a first person perspective, and may feel present in the story as if he or she is actually being transferred to the environment.  This presence to the stories, and the following perceived realism, is what is believed to be able to increase empathy in the viewers. Of this reason also, Google News Lab’s Ethnographic Study on Immersive Journalism  describes VR news as more fit for the term “storyliving” rather than “storytelling”, indicating that the user feels a part of the story that is being conveyed.

‘See for yourself’: Is the NYT VR pitching their VR production by indicating that you can see the situation yourself, instead of adhering to the mediated version by a journalist who’ve been there?

Critics of VR as the “ultimate empathy machine”, or as capable of delivering “storyliving”, say that you can not possibly know how it is to be in a refugee camp while lying on the couch a friday evening with your VR goggles and a glass of wine. And, of course, you can not. It may, however, prove to be a more empathetic instrument than a regular video as it may seem more real and thus affect us differently.  It is no wonder that it may seem more ‘real’, when it is presented in the ‘format of reality’. Some do agree, however, and believe that it may have the power to make viewers more empathetic – nevertheless, they think that it may be unethical to use it as such. Should journalists have this power, to distribute realities to news consumers, and by new technologies we are not used to, affect people this strongly? These critics may be afraid of the “brainwashing” potential of the technology. It is easier to distance one self from a 21 inch screen than it is to distance yourself from an encompassing, immersive virtual reality you inhibit. Perhaps the most crucial question is whether it is brainwashing if what you are showed in fact is real?

Fake News

This brings us to another point of VR Journalism: how hard it is to manipulate the content. The journalist can not hide behind the camera. There is no artificial lighting or too narrow segment of the shot. The camera shoots in 360° horizontally and vertically — it is a totally observant witness at that given space and time.  In a sense, news in 360° is far less directed than in traditional flat videos: the viewer chooses which part of the video he or she wants to see. Naturally, the journalist is still an active role which chooses where to shoot, and does the final edit, however, it is a great shift from traditonal video footage. As we all know, during the last few years the concept of ‘fake news’ and mistrust in the media has arisen. The transparency that shooting with 360° 3D offers may help combat this. Perhaps also Journalism has to adapt to these changes, and rather deliver (immersive) content on which its readers themselves can decide and conclude upon. Euronews, one of the largest European news agencies, argues that this is why they have produced several 360° videos each week for now two years. In the VR session of the Digital World Conference in 2016, Editor-in-Chief of Digital Platforms, Duncan Hooper, stated they “want[ed] to let [the users] make their own decisions, not tell them what they should be watching, not to tell them what they should be thinking”.

VR Journalism at the University of Bergen

During the spring of 2018, I taught 20 undergraduate students in VR Programming, 360° video shooting- and editing and photogrammetry. The aim was that the students should be able to create their own prototype delivering Immersive Journalism. As the rules and practices within the new concept is not very well established, we did not teach the students exactly to solve their tasks, but rather how to experiment with the novelties of the medium and try to innovate and create new genres. This is often called ‘Innovation Pedagogy’. The end result have been four brilliant prototypes, that was presented at the Norwegian Centre of Excellence (NCE) Media’s media lab in Media City Bergen. We discuss two of these here. Interested in the other two? These are mentioned in an entry where we go more in depth, philosophically, on the concept of Experience Machines.

Drug addict
The first of the VR experiences is called “Narkomani” which from Norwegian can roughly be translated to “Drug addict”. The aim of the production is to see the world from the point of view of a drug addict, perhaps living on the streets in Bergen. How is it to be frowned upon, walking around the streets, uneasy to get the next shot of dope? As my colleague Nyre stated in the introduction the projects, this VR project features “not a first person shooter from Los Angeles, but first person social realism from Bergen”.

Schizophrenia
The second of the VR experiences, attempts to create understanding on how it is to live with schizophrenia. In the experience, the user perceives visual hallucinations, and audio of up to five different personalities. The concept is brilliantly illustrated by the poster, and the experience tries to portrait a subjective reality falling apart.

Conclusion

Journalism through the medium/technology of VR has great potential. Immersive Journalism is still in its infancy, but the projects done so far shows promising. Much will depend on VR goggles entering into people’s homes, as with any other technology. For insights into where we are in the terrain of VR technology in 2018 — take a look at our entry discussing the History of Virtual Reality.

Literature list


Are you interested in more reading on this subject?

This executive summary on “Virtual Reality Journalism” by Owen & Pitt at the Tow Center for Digital Journalism is one of the first reports on VR Journalism.

Further, the report by Doyle, Gelman & Gill at the Knight Foundation is a good background read.

Finally, the Reuters report by Zillah Watson is more recent and sheds light on the more current situation of the medium for journalistic purposes. This report  illuminates a change we have seen recently, with the use of consumer/”prosumer” cameras for easier production by newsroom. This will definitely turn out, as with traditional cameras, to be a prerequisite for the adoption of this medium across Journalism as whole. When it is easier to produce content in 360° video, more newsrooms will do it.

As the reports by The Knight Foundation, The Tow Center, and further Sirkunnen et. al indicate, Immersive Journalism has not been so prevalent in less-affluent media houses. We may know of VR stories such as 6×9 by The Guardian, but have not necessarily heard of any from our local newspaper. This may change in the near future due to better consumer products.