From Thought to Reality

This particular entry discusses a perspective that can be had towards Virtual Reality technology, and in so describes a certain way of understanding or relating to it. The aim is to provide, if not a theory, at least a perspective that resonates with the experience of creating and experiencing a virtual world.  This perspective was first worked upon in preparation for the 2019 Christie Conference at The University of Bergen — and is here developed further in text. The perspective tries to approach an understanding of what gives the virtual its magical qualities — and how this comes to be.

The Hobgoblin, a powerful wizard that travels through realms on his black panther. Illustration by Tove Jansson.

The Experience of Virtual Reality

VR has the capability of enchanting us. It has the power to introduce us to virtual worlds — and represent our designs in ‘the format of reality’. The content appears non-mediated, something Metzinger relates to the «phenomenal transparency» of the mind — we see «through the medium» —  and so only the content of the representation is available for introspective access. As with our mind, the underlying processes are hidden to us, and to experience VR is to experience the content it presents, not what lies beneath it.

There is a certain kind of experience that is exclusive to VR.  Finding oneself in a virtual world — orienting, navigating and interacting with it — produces an experience of a certain distinct character.  There is something intriguing, stimulating and marvelously weird in experiencing virtual realities. The experiential quality is affected by the sheer virtuality, or unreality, of it — and this, in turn, may make the illusion unexpected and beautiful. The experience has a certain quality to it: the disassociation between its unreality on the one hand and feeling of reality on the other. We know that VR is a synthesized, not-naturally-occurring experience, and further that we react to these stimuli as if they were real. Due to its unique character of offering convincing illusions, and our unique quality of, on the one hand seeing through them, and on the other being totally helpless in responding to it as if it weren’t real — we get the weird, thrilling experience of VR. The clear illusion; the transparent veil — a weaving of smoke: beautiful, but not substantial.

How did this oxymoron of technology — VR — come to be? What does it represent in us as humans? And last but not least, how should we view, relate to, and approach the emerging virtual worlds that the technology will enable for us?

Origins of Virtual Reality

A central aspect we have to address as part of this investigation is the origins of VR. Why did it arise, and what does it mean to us? This will ultimately affect the way we relate to and understand the medium. To do this, we can look into what desire of ours that the technology has fulfilled. Although we have previously discussed the History of VR this does not really account for the underlying motivations or dreams, but rather their outward results in terms of the resulting technology. Thus, when we instead want to discuss the origins of the idea of VR, we are attempting to approach technology “in its essence” — through its origin. For some readers, this may not be unfamiliar, as we have discussed this somewhat lengthy and theoretically in our three-series entry on Heidegger and VR. Here we will limit his technology criticism to a brief summary of two sentences:  Heidegger’s definition of technology as “not in itself something technological” means that the origins of the technology we use, and what it is and means to us in its essence, leans more towards an underlying ideal and thought than what it does towards different physical artifacts. Technology is, essentially, a way of viewing the world.

Fingolfin, Elf and High King of the Noldor, in a duel against Morgoth Bauglir, fallen Melkor of the Valar. Fingolfin and Morgoth can each represent different ways of embracing technology (read below)

Virtual Reality as Thought

If we shall try to understand the earliest origin of VR, it is appropriate to consider VR as an idea. Essentially a thought, or even a dream. The dream of being able to define reality, creating a re-presentation: the same dream that inspired cave paintings several ten thousands of years ago. VR is a product of the creative element in humans, for good and for worse. The dream of absolute control over matter, but also, the dream of a creative medium without limitations.

Similarly to how Heidegger imagined art as the potential savior of the way technology enframes the world and ourselves, J.R.R Tolkien’s distinction of magic in the universe of The Lord of The Rings can serve as a metaphor. The Elves use their magic only for artistic purposes and are consciously aware of the difference between reality and deception — the enemy, however, uses it to deceive and control. Heidegger’s point of technology is similar — technology (techne) may separate us from a more original revealing of truth by enframing the world in a certain narrative or story, while art (poiesis) may open up reality towards new interpretations. In Tolkien’s magic for purposes of domination, there is a will that is opposed to nature and thus will have to veil nature in its bringing-forth of its ‘truth’ or end. Similarly, Heidegger’s technology, as a way of revealing-concealing, will, to achieve its success, have to enframe nature and man with it — it reduces us to mere means to ends, not ends in ourselves. Tolkien’s elves use their magic in harmony with the real world; and similarly, Heidegger’s more preferred technologies let nature be as it is, instead of enframing it as a means to an end.

How are we to think of VR according to these technology criticisms? Whether we view the potentiality of the virtual as the dream of absolute control or domination (as Morgoth would have), or rather its potential for creative revealing and enhancement of the world (Fingolfin) — we can view VR as a Technology according to Heidegger, or as Magic according to Tolkien. Readers who are interested in what or how a Heidegger or Tolkien-inspired VR-application could be can go on to read the authors position paper in one of our latest blog entries. There we discuss the concept of an existentialist design — a “controlled accident” —which does not seek to dominate the user experience, but rather open the world up to new interpretations.

Virtual Reality as Reality

The exploration of VR as thought, or essentially as an idea, have taken us thus far. This idea or thought of VR is, however, now more actualized than ever — and what was once primarily an idea, is now more than ever a reality that we can relate to. We are able to step in, and immerse ourselves, in worlds after our own design. We can actualize, externalize, and instantiate our designs.

Memory Palaces are systems of thought that utilizes visual and spatial cues to aid memorization.

An example of this that can illuminate the perspective of VR as thought, is the Virtual Reality Memory Palace. This ancient mnemonic (it is mentioned by Cicero in De Oratore as early as 55 BC) is based on thought: we are to close our eyes and visualize what we want to recall in the memory palace.

The memory palace is then, as the origins of VR, an idea or thought. It is internal and subjective. VR, however, allows the externalization of thought. In the same way that the idea of VR is now actualized, it allows the externalization of other ideas. We can use technology to immerse ourselves in the instantiated ideas of our mind. Is then akin to the works of Morgoth Bauglir, or Fingolfin son of Fëanor? This will depend on which thoughts are actualized: it depends on which levels of our inner being we want to realize as our outer world.

Conclusion

This entry has discussed a perspective on VR that compares it to the magic: through VR, we can define our external reality based on our inner thought. VR can be perceived as the materializer of form; the instantiator of the abstract. We described Heidegger’s explicit technology criticism and paralleled it to Tolkien’s implicit one. We also linked this to the authors position paper on a Heidegger-inspired VR technology.

This entry is at the core of Matrise’s interests, and if you want further reading, these previous entries are related:

1: Inner as Outer: Projecting Mental States as External Reality

2: Sensory Deprivation — Floating in Virtual Reality

3: On Mediums of Abstraction and Transparency

4: Heidegger’s Virtual Reality

5: The Mind as Medium

 

 

Existentialist Design

The term “Philosophy” can refer to a method or approach to investigate how we should relate to and understand ourselves and the world. Moreover, how we understand ourselves and the world is dependent on technology — especially in the case of mediums that present and abstract information to us. We have previously at Matrise discussed the philosophy of technology as a subfield of Philosophy. In this entry, however, we will discuss how philosophy can, concretely and directly, inform information technologies — especially within the field of Human-Computer Interaction.

Martin Heidegger by Barry Bruner.

That how we understand ourselves and the world is dependent on technology is being recognized this year at the CHI 2019 conference in May on Human Factors in Computing Systems — the premier international conference on Human-Computer Interaction (HCI). In December last year, there was issued a call for position papers to a workshop called “Standing on the Shoulders of Giants: exploring the intersection of Philosophy and HCI“, led by experienced researchers with experience in philosophically informed design or research. In the call they write:

“Philosophy has provided a vital perspective for HCI on how we navigate, experience, understand and judge the world around us and its artifacts. Lately, HCI scholars have also sought to use philosophy’s program of answering what it means to live a “good” life to investigate the ethical and moral implications of the technologies we design. As philosophy in its many forms continues to open up new influences and our relations with technology broaden, we believe it is timely to have a meta-discussion about what links philosophy and HCI. As we understand it, philosophy’s strength lies in its diversity, depth, and interpretive flexibility.”

I’m attending CHI as I am an author of a paper there and was naturally interested in submitting a position paper. In the position paper,  I present a view of design inspired by philosophical thinkers such as Kierkegaard and Heidegger, where the aim of the design is a controlled accident in which we do not want to dominate the user experience, but rather open up for new experiences that can be interpreted by the user.

The rest of the entry is best explained by the position paper itself, with its abstract, introduction, body and conclusion. The paper can also be download and read at ResearchGate.

Title: A Controlled Accident: Imagining VR as a Catalysator for Self-exploration

Abstract
In this position paper, I discuss an existentialist design approach towards Human-Computer Interaction (HCI). The aim of the existentialist designer is to not dominate the user experience, but rather to design for a controlled accident in which the medium itself is explored, and one’s self through the medium, where the overarching purpose is the exploration itself. This line of thinking is inspired by existential philosophers such as Kierkegaard and Heidegger, and its aim can be illustrated in how Kierkegaard discusses life not as a ‘problem to be solved, but a reality to be experienced.’ The hope is that such an approach towards technology escapes the somewhat limited view of technology as simply a tool to get from A to B, and that technology may be seen as a lens through which reality can be presented free from an otherwise culturally enframing narrative. The aim is, therefore, to design technology as a catalysator for a more original revealing of truth. The paper illustrates this with an example of the employment of Virtual Reality (VR) technology in sensory deprivation tanks.


Introduction
According to Martin Heidegger’s technology criticism, the essence of technology is not itself something technological. Put way too short, technology is rather a way that we understand the world, or in Heidegger’s words, ‘a way of revealing’ (Heidegger, 1977). What we reveal in this technological framework, that is, what we deem as true, is not necessarily the truth, but only ‘correct’ relative to the framework itself. Thus, the common correct definition of technology – that technology is a human activity and a means to an end – although correct relative to the technological narrative, is not necessarily true. For Heidegger, these two definitions must not only be combined, as in that it is a human activity to think of means to ends, we must also recursively look on how this activity impacts the way we look at the statement itself.

It is the mindset of thinking in terms of means and ends, of interpreting and enframing things within this technological framework, which is the essence of technology according to Heidegger. When Heidegger speaks of ‘revealing’, he means what is presented as true and brought forth into that way of revealing. In the case of the technological framework, what is revealed is within the narratology of ‘man versus nature’, which is a fundamental view that reveals the world as such. For Heidegger, the danger is that man himself cannot escape his enframing, and that it ‘may be denied for him to enter into a more original revealing’. Heidegger’s brief comment towards a solution to this problem – although he explicitly states that modifying technology can never be the answer – is a different kind of technology, or techne, poiesis; the Greek word for both art and technology. Here he refers to art, as art is not fixed in terms of interpretation or the straight rules of means to ends, and A’s to B’s, and may, therefore, bring a more original revealing – or at least another narrative that provides another revealing.

Martin Heidegger contributed to the existential and phenomenological tradition of Philosophy in the mid-1900s. Although he never lived to see Information Technology, his philosophy on technology is not necessarily concerned with the details of different technologies, but rather what technology is in its essence. In this way, his works may still influence the design of artifacts in Human-Computer Interaction (HCI) in the 2020s. But how can existential philosophy benefit the research of the relationship between humans and machines? How can such a critical view of technology in its essence benefit the design of technology?

Existentialism in HCI
Arguably, methods and approaches appropriate for creating usable, enjoyable, and practically useful products and services, cannot be assumed to be also appropriate for addressing the issue of how technology is related to the most fundamental aspects of human existence” (Kaptelinin, 2018).

Kaptelinin (2018) presents a broad overview of previous existential approaches to the field of HCI. The paper is further contributive and practical in that it approaches a framework compatible with where HCI is today. Kaptelinin (2018) writes that previous attempts at employing existentialism in HCI research has not been very popular, and argues this is because it is ‘too distant from traditional HCI problems and concerns, and too abstract to provide concrete support for analysis and design’ (p. 1). This is a danger for any approach inspired by an underlying ideal and something which this position paper also is subject to. For instance, Kaptelinin (2018) discusses Karlstrøm’s (2006) paper as criticizing the ‘problem-solving attitude of HCI’. As this paper will present a similar approach, it should, therefore, immediately comment on how problem-solving itself can be undesirable.

We may, therefore, begin by restating the points of Heidegger and Kierkegaard: the problem-solving can itself be a problem. In relating to the givens of existence, the solution is not necessarily to define them as problems and find strategies to eliminate them. An approach can, however, be to explore them and see them for what they are, and thus enter into a more authentic relationship with them. Problem-solving as an attitude may provide the illusion that a fix is possible by pushing through. Thus, existentialism may claim that it is not the givens of existence that is the problem, but rather how we relate to them, either as problems or something else. The standpoint, therefore, is that problem-solving may be the real problem, as it enframes the world as something that can be solved. This is correct relative to its own framework, but in cases of how we relate to the givens of existence, this need not necessarily be true. This is further coherent with the way Kaptelinin (2018) discuss existential psychotherapy: there is not one solution to it, and that may mean that we require technology that is far more open, adaptive and exploring. It may be that in these situations technology need not provide a solution, but perhaps even on the contrary be an important tool to reinstate the problem for a more clear inspection, and by means of this lay grounds for establishing a different relationship towards it.

The aim of such a technology will rather be to open up the world for new possible interpretations, than aiming for one specific function. The question that would be explored by interaction with such technologies is whether technology can help us break a certain narrative, or put in Heideggerian terms, whether technology provides us a more original revealing. In the next section, an example of a technology that can be used this way is presented.

Existentialism in HCI
In VR, presence is often defined as the degree to which the subject feels present in the virtual world. What is interesting to note, is that this naturally has to be viewed relatively to the degree that the subject feels present in the physical world as we usually receive information from both our physical and our virtual environments. There can thus be two separated approaches to designing for presence in virtual reality environments: one is to provide the sensory stimulus of the virtual environment, and the other is to block sensory stimulus from the physical environment. Both approaches work towards the same goal of immersion – the encapsulation of the user in the Virtual environment (VE).

Obviously, the principle of adding and removing sensory experience go hand in hand; by equipping a Head-Mounted Display you are blocking the physical impressions and replacing them with virtual impressions, all the while shielding for incoming light from the surroundings. Blocking light, however, is not the only way to deprive the senses of information from the physical environment. The inclusiveness of the immersion can also be achieved by sensory deprivation through floatation tanks.

Alone With Your Thoughts”, Illustration by Cole Ott

Floating in Virtual Reality
Floatation chambers, or sensory deprivation tanks, are pools of water with copious amounts of Epsom salt. The tanks are sealed for any incoming light and sound, and the air- and water temperature is equal to that of your body. When you lie down, you will feel how the salt makes you float even though the pool is very shallow. As you lie there, you notice how the ripples you created when lying down start to slowly subside as you sink down into weightlessness. After a while, because of the air- and water temperatures are the same as that of your body, you can no longer pinpoint where the water ends and the air around you begins. In fact, it gets hard to distinguish anything from anything else, including your body from the air and water. There is really nothing that is easy to grasp as isolated, save perhaps your breath. And as the minutes go, with total physical relaxation and lack of much sensory impression at all, things may start to change.

The most significant, explicit change one may notice in the tank is that after a while your bodily self-consciousness is not what it used to be. Your mental model of where your body is in relation to the world around you starts to become blurred. Normally reinforced by tactile stimuli of air and water (of varying temperatures), and visual and auditive stimuli from the environment, your body model is now lacking information on which to create it. Your sense of spaciousness has also changed – that is the feeling of your position as defined relatively to say, the walls, mountains, and the sky have disappeared. You now really experience nothing around you, but neither any edges to this lack of information about your surroundings. You may get the feeling of floating in empty space, but where are you in all of this? What, in this stream of conscious experience is matter and what is mind?

Example Experiment
To exemplify the ideas discussed in this paper, I imagine the following experiment. A user employs a VR HMD that is connected to biometric sensors, e.g. EEG, GSR, heart rate, breathing, etc. A connected computer visualizes the feedback through abstract imagery in a 3D visualization. The direct effect is that an abstraction of the user’s state is projected externally, but the application does not do a hard classification to moods in the form of emoticons. Rather, the user can meditate and explore the visualization as the floating continues and can establish a way of exploring the technology through relating to both the medium and through it themselves. It would further be interesting to use eye-tracking technology as a way of navigation in the vast, abstract visualizations. If one traveled towards where one saw, one could even be interactive while lying still in the floatation tank. This could also possibly have curious effects on which parts (perhaps the eyes), we identify with our selves âĂŤ perhaps the placement of our self could be altered by changing the agency for transportation. My interest in such a prototype or such a future experiment would be to which extent it could open us up to the direct here-and-now experience, and attempt to have experiences beyond the traditional subject-object hierarchy. It is existential in the sense that it seeks to delete the traditional narrative. 

Literature list

The Virtual Freud

We are not few who would save a coin or two by being able to be our own psychologist. In a very concrete sense, this is the topic of today’s entry — how Virtual Reality (VR) can allow us another perspective on ourselves, and how this may better our mental health. At Matrise, we have previously discussed how VR can benefit anxiety sufferers through virtual reality exposure therapy. We have also discussed how the medium can facilitate Mindfulness meditation. In this entry, however, we will discuss a VR application that lets you have a conversation with Dr. Sigmund Freud. Oh, but there’s a twist!

Sigmund Freud, oil on linen. Mathieu Laca (2015).

In 2015, Sofia Adelaide Osimo, Rodrigo Pizarro, Bernhard Spanlag and Mel Slater published a paper called “Conversations between self and self as Sigmund Freud — A virtual body ownership paradigm for self-counseling“. The paper discusses an application where you sit in a chair facing Dr. Sigmund Freud. Upon entering the virtual environment, you do not float in empty space as one often does in VR — rather, you notice you have a virtual body that responds to your movements. This may lead you to identify the virtual body as your own, a magical feature commonly referred to as Virtual Embodiment. We have written extensively on this subject in a previous entry — but put shortly, the effect, apart from being very interesting in itself, has many practical applications. Self-identification with a virtual body can be exploited to, for instance, reduce implicit racial bias and make offenders of domestic violence get better in noticing the fear in victims.

Self as Other

Seeing outside into the inner

When you sit in your new virtual body, facing Sigmund Freud, you are asked to tell him about a problem. Sometime after you have emptied your heart, the virtual environment fades to black, before you once again are placed in a body, but this on the other side of the room. You are now Dr. Sigmund Freud and your patient, who looks remarkably like you, starts talking. You hear a recording of what you just said minutes ago, but you get to view your statement in a ‘new dress’: a 3D model of yourself is saying it, while you are virtually embodied elsewhere.

As humans, we know ourselves inside-out (or at least we believe we do). This may lead us to be more critical towards ourselves than others, as we compare our worst to the others best, our shame to their facade. We know all our terrible, dirty secrets, and talking to ourselves we do not have to adhere to any sort of social norms or even any general courtesy for that matter. This may lead to our inner voice becoming quite … crude. If we could focus on our own problems, in the form of the problems of others, it may be easier to be more loving towards ourselves, by utilizing the love we usually give to others. The technology can have remarkable results in affecting our selves.

In their paper abstract, Osimo et. al. write:

…this form of embodied perspective taking can lead to sufficient detachment from habitual ways of thinking about personal problems, so as to improve the outcome, and demonstrates the power of virtual body ownership to affect cognitive changes”

Internal as External

This detachment from the habitual may be very beneficial, perhaps especially in terms of Self and Identity. We have discussed this previously in our entry called “Inner as Outer: Projecting Mental States as Immersive Virtual Reality“. Apart from the philosophical buildup of the entry, the article discusses an application that, to a certain extent, allows you to view your inner states (measured through pulse and breath), as your encompassing external reality. In our entry on the use of VR in floatation tanks, we also discuss the extreme potential of this — the possibility to be stimulated by only sensory deprivation, of which can be based on your inner phenomena, thus resulting in an experience where there is no separation between the inner and the outer, thus refuting the subject-object dualism that affects our everyday living experience.

Do you have any ideas to this? Feel free to comment below.

Philosophy & Science

On our ‘About’ page, we describe Matrise as a blog that focuses on the topics of VR and Consciousness, and the related Science & Philosophy. In this entry, we will discuss why we find both Science & Philosophy essential and how the disciplines compliment each other. The entry is apologetic of Philosophy as a discourse, and especially critiques the philosophy of Scientism and other extremes of logical positivism — of which the former can be understood as an ignorance of the role of philosophy in general. The discussion is highly relevant as this is a worldview increasingly adopted by young people and promoted by popular science figures. The danger is a spread of anti-intellectualism, which ironically is going on under the flag of science, and further even tries to adopt its authority.

It should be noted that although the entry critiques some ignorant science communicators, it is not the aim of this entry to argue philosophy in any way triumphant over science.  The aim is rather to replace “versus” with “and”, and illustrate their mutual dependency. Like my philosopher friend and colleague Deborah G. Johnson once said of her technology criticism: “I prefer to think of my role as similar to that of an art critic” — an art critic loves art, and similarly, in this entry, my critique is out of love for the scientific discourse. It is, by far, not directed at the conceptual core of science, but rather towards the bad tendency that may be had towards it.

In my occupation as a Ph.D. fellow, I ought to, and do, value both science & philosophy, and the main argument of this entry will be the inseparability of the two disciplines and the beauty of their interplay.

Philosophy is Dead, Long Live Philosophy

Illustrating excerpt from “The Philosophy Force Five vs the Scientismists” by Existential Comics. The left guy seems to be Richard Dawkins. Original can be found here.

It may benefit us to personify the attitude towards science and philosophy that we are criticizing with an example. In 2010, the late brilliant Professor Stephen Hawking declared that ‘Philosophy is Dead’, and that the torch guiding the quest for knowledge had been passed to scientists rather than the philosophers. The strong irony present is obviously how Hawking argues the death of philosophy in terms of philosophical arguments, thus illustrating the futility of his statement simultaneously. In his book, “The Grand Design”, Hawking writes that metaphysical questions such as the meaning of life and the nature of reality traditionally were questions for philosophers — but that now, alas, philosophy is dead. Luckily, however, Hawking himself comes to the rescue: further in the book he writes that the “purpose of this book is to give the answers that are suggested by recent discoveries and theoretical advances” — thus pronouncing ever more clearly that his book is one of philosophy rather than science — but which reasons on asserted scientific facts. Revolutionary, indeed. Why did not Philosophy think of just looking at the answers that were suggested by the facts? The whole field of Philosophy is dead, but Hawking has taken it up as a side project and written a philosophical book, so we need not worry.

A critical point that illustrates Hawking’s misunderstanding here, is how he announces that his book discusses “the answers that are suggested by science”. His attitude is therefore that the physics always imply its metaphysics; that descriptive facts of the world also tell of their own meaning, that the solution is always innate in the problem. The situation is that Hawking, of course, has a certain philosophy, he is simply not critical towards his acceptance of it. His assertion is easy to critique, however,  by arguing that science does not suggest anything — Hawking, however, suggest several things, and these may be great things and great ideas! It is easy to agree with his philosophy as broad as it is: through observation, we agree on facts, and with logic, we reason on the meaning of these facts. Hawking is philosophizing on scientific discoveries and theories, and that would be a great thing had he only been aware that was his approach. There is not (necessarily) anything wrong with his philosophical stance, except in exactly how it claims to not be a philosophical stance. The saying “Philosophy is Dead” instantly revives Philosophy as the utterance itself is a philosophical statement. The sentence achieves the same futility as a person grabbing a megaphone to announce that megaphones don’t really work.

Illustrating excerpt from “The Philosophy Force Five vs the Scientismists” by Existential Comics. Original can be found here.

The Science Gang

Unfortunately, Hawking as a popular science communicator is not alone in his ignorance. For instance, both Neil Degrasse Tyson and Bill Nye (the Science guy), has also publicly addressed their concerns over Philosophy. Neil deGrasse Tyson has called Philosophy ‘useless‘, advising bright students to stay away from it, and stated that Philosophy is not “a productive contributor to our understanding of the natural world”. He has also said that he is concerned that “philosophers believe they are actually asking deep questions about nature”, and described the venture of philosophy as being “distracted by questions”, and therefore not able to contribute anything to our understanding.

Our best example of the problem that lies at the core of these poor critiques, however,  is given by Bill Nye in a YouTube video where he is faced with a question from a Philosophy major on what his take on Philosophy is. Nye addresses the topic of Philosophy by introducing several great philosophical problems such as the nature of consciousness and reality, only to dismiss them based on the argument, or, rather saying, that he is quite sure that reality is real. Deep dive. The best illustrative quote would perhaps be his rejection, and total misunderstanding, of Descartes’ famous argument ‘Cogito ergo sum — I think, therefore I am.’ Nye comments on this: «Well, what if you don’t think, do you not exist anymore? You probably still exist even if you don’t think about existence”. I hope I don’t have to explain why this is ridiculous.

At the end of the video, Nye goes further into depravity when he presents a thought experiment to validate his lack of philosophical skepticism.  Nye simply asks the questioner to drop a hammer on his foot. This kind of argumentation is classic and has long before been termed ‘argument ad lapidem‘: a logical fallacy that consists of dismissing a statement as absurd without giving proof of its absurdity. What is even more unfortunate, however, is that this was a brilliant opportunity for Nye to illustrate how science actually benefit philosophy, as there is much evidence from scientific experiments that show we may not always be able to trust our senses, and even our own reasoning (read for instance about the fascinating split-brain experiments, or even here on Matrise on how we can fool our senses to achieve Virtual Embodiment or our entry on The World of Illusions). For those who can emotionally afford to experience a few cringy minutes, Bill Nye’s YouTube video can be watched here.

A Philosophical Argument

Illustration borrowed from Existential Comics’ “Philosophers And Physicists” available here.

The problem with a defiant attitude towards Philosophy is that it promotes an ignorance of thorough, critical thinking. Our sciences are infused with philosophy and we need philosophy to reason on our discoveries and their meaning, and further to discuss what we want to research. The best way to understand this is that our scientific methods are an accomplishment of philosophy with the epistemology, methodology, and rationality underlying it: the validity of the scientific discourse is argued for in philosophical terms. Science is not given, it is a construct: our scientific methods have been developed, and are still being discussed and developed to this day. Science & Philosophy does not compete with each other; philosophers compete with philosophers; the logical positivists compete with the idealists, and the rationalists compete with the empiricists. Science is conceptually within Philosophy, and one of its greatest accomplishments. They do not exist separately.

The Uninvestigated Myth

What Philosophy can teach us, and which perhaps especially the field of Existentialism stresses, is to critically reinvestigate our beliefs. It stresses the possibility of each individual to realize their own potential, and also implicitly, the possibility of failing to do so. Heidegger, for instance, a philosopher which we have discussed in several entries earlier is concerned with authentic existence. Allowing an uninvestigated myth such as Scientism to guide one’s life could be a good illustration of Heidegger’s “Das man”; which could be said to be his impersonated term for the norms and culture we blindly accept. It is hard to translate “Das man”, but Heidegger defines it as a possible state of Dasein’s Being. A common way to explain or translate “Das Man” is to the “They” or the “One”, as in “One should always get up early”. Heidegger explains that “‘Das man’ prescribes one’s state-of-mind, and determines what and how one ‘sees'”. This can be read as that culture, or uninvestigated norms that provide a narrative to our existence determines how we fundamentally interpret reality. In the words of crazy Terence McKenna: «Culture is not your friend!»  Rather than to accept the cultural and societal narratives, we should critically investigate them, and not let anyone dictate a narrative for us that we should just subscribe to. What narratives in our society do we subscribe to, in relation to who we are and what we want?  What did we adopt from our parents and our society in relation to how we understand the basic principles of the world?

Illustration borrowed from Existential Comics’ “Philosophy News Network: Philosophy Solved” available here.

 Conclusion

It is a very naïve view, indeed, to believe that the solutions to the problems of mankind may be solved if all the young, bright minds go into the natural sciences. We need not only to find out how nature works, but also how we ought to work with it and further with each other. Science alone does not tell us what we should do in our lives. No scientific fact can alone provide us meaning, or say what we should value and pursue. Dismissing philosophy is a dismissal of thorough criticism, it is being attached and dependent on one’s settled, foundational narrative of how the world works.

Concluding this entry is an accompanying text to one of the comics in this entry on Scientism by Corey Mohler. As this entry has not necessarily been very explicit on what Scientism is, it may be informative to read his very clear presentation on this.


“‘Scientism’ is the position that Science can solve all problems, or that all problems are empirical. Philosophically, it is mostly associated with the strongest statements made by the logical positivism movement, which mostly died out in the mid 20th century. Culturally, however, it is stronger than ever, and is closely tied to movements like the so-called “New Atheists”. These newer, more naive forms of Scientism, also have a strong tendency to call philosophy “a big waste of time”, “pointless arguing”, “nothing but semantics”, etc. Rhetorically, they tend to say that non-empirical ideas have no way to guarantee they are true, so are pointless to talk about. This is a rather ridiculous point to make, since their entire movement is based around spreading a certain set of non-empirical, philosophical norms, which they apparently don’t feel it necessary to open up to criticism. What they mostly seem to mean is, assuming everyone agrees with us on the important philosophic questions, such as atheism, utilitarianism, capitalism, eliminative materialism, etc., then we don’t need anything but science. Well, this is maybe true in a strange way, insofar that if everyone agreed on every philosophical position, i.e. if philosophy was solved, then we probably wouldn’t need philosophy. Philosophy, however, has not been solved. Furthermore, if it is going to be solved, it certainly won’t be solved by a bunch of people who don’t even read or engage in philosophy. The real goal is often just to draw a border around what we should or shouldn’t question, because they don’t want any of the fundamental aspects of society to change. And, well, people who don’t want society to change often also find themselves not wanting people to even think about changing society.” — Corey Mohler behind ExistentialComics.com

The Capture of Reality

When creating experiences for Immersive Virtual Reality, there are essentially two approaches. The first one of these is manual construction through Computer-Generated Imagery (CGI), and is how most games and VR experiences are made.  The second approach is far more automatic and attempts to ‘capture reality’ instead of actively generating it. It is this approach that we will discuss in this entry. In addition to presenting the technicalities of the methods of capture, we will also discuss its limitations, and provide an innovative example of how these can be solved in the future, drawn from a student project at the University of Bergen.

An early 360° camera — horisontally at least — probably the first with a synchronised shutter.

360° Video

In a previous article on Virtual Reality Journalism, we discussed how 360° 3D cameras can be used to present a user to an immersive experience. This approach has several unique benefits. First of all, it is far less time-consuming to capture and re-use already existing physical environments, instead of spending time creating it through 3D modelling. The same is perhaps especially the case when the environment involves any human actors, as it easier to avoid the uncanny valley effect and simultaneously maintain high standards of realism when using image capture equipment, than it is to create it with 3D animation.

How does it work?

360° cameras usually comprise two or more (ultra-)wide angle lenses. In the case of cameras with just two lenses, such as the GEAR 360 or Ricoh Theta V, each of these lenses then have to be able to capture 180° degrees horizontally and vertically. The recordings from these lenses, when raw straight from the camera, are separate — and need to be stitched together with software (for instance) an equirectangular view to compose a spherical view of 360° (See Illustration 2). Illustration 1 illustrates how the equirectangular format works, in the format of a world map, perhaps our most relatable example of spherical / global shapes presented in the format of rectangles.

Illustration 1: A relatable example of the equirectangular format. The furthest point west is close to the furthest point east, and as such we deal with a ‘sphere’, or more rightly globe, that is stretched out to a rectangle. The closer we get to the poles, such as Antarctica, the more the image is stretched, as the circumference of the earth is lesser at the poles.
Illustration 2: In this equirectangular photo, captured with a Ricoh Theta V, we see the same effect as in Illustration 1. My hands, which enclose the bottom of the camera, are given the same effects as Antarctica in the map. The stairs, however, which appear to be circular are straight, but it’s bending by the lenses are especially clear when viewing it ‘equirectangularly’.

When an equirectangular image is viewed through an HMD or a smartphone, the software selects only about 110° of 360° of the image, relying on the sensors in the HMD or phone on which degrees of the image to present.

3D Images

Although regular 360˚ cameras (GEAR VR; Ricoh Theta V) to a large extent cover the world as we see it in all it’s 360°, their images are still monoscopic. Essentially, this means that the same image  is presented to each eye when viewed in a HMD, and this is not the way we ordinarily see reality. As our eyes are distanced by a centimeter or two,  the visual feed slightly varies in its capture of reality. It is this which enables us to perceive the depth of the world, that is, when our eyes are not fooled by illusions exploiting this effect, such as VR itself. We discuss this in more detail in our entry on the History of VR, in which we discuss the invention of the Stereoscope, but a small introduction will also be given in this entry. Essentially, 3D 360˚ cameras utilise the same feature as human beings to capture depth, by separating the cameras similarly to that of the human eye. Such cameras are, however, more cumbersome and costly to produce, and to capture stereoscopic images one needs to double the minimum of lenses — leading to a minimum of four lenses —two for each eye for each 180˚ of capture. Unlike the  4K 360˚ monoscopic cameras available rather cheaply at the commercial market (from $200 and up), stereoscopic cameras have not entered the market at very reasonable prices yet. There is hope, however, and I can personally recommend Vuze+, a 360˚ 3D camera that deliver 4K resolution per eye, and comes with a well-designed acommpanying stitching- and editing software. The price is still a bit stiff for most non-professional use, at $1200, but it brings hope for future technology that these can soon be more affordable. We have used the Vuze+ camera in a research project at the University of Bergen, with good results. It is comparable to the quality of a Ricoh Theta V — except that it delivers the stereoscopic images rather than monoscopic ones.

Regarding Resolution

Unfortunately, a resolution of 4K per eye sounds great — and many are dissapointed when they view the recordings of a camera such as GEAR VR, Ricoh Theta, or the Vuze+. They may recall their images on their 4K TV as incredibly sharp, and yet, their recorded videos appear somewhat blurry and pixelated. The answer to why this is the case is quite simple. The 360˚ images do indeed have a 4K resolution, however, we are unable to view all the pixels at a time as they are stretched out on a sphere.  To keep matters simple, let’s say that your Head-Mounted Display has a Field of View of 90˚ (although most have 110˚). In this case, just  1/4 of the 4K image is being seen at any given time. Thus, we will have to divide the pixel count by four. This is somewhat simplified because of stretching, but it should be enough to get the point. To get an effective resolution of 4K, or something akin to 3K such as the HTC Vive Pro and Samsung Oddysey(+) can afford, one would need a far higher resolution of the cameras.

Another Step in Fidelity: Volumetric Video

At first thought, it may perhaps be hard to imagine how we can proceed to more details in immersive  360˚ 3D recordings except by increasing the resolution. As we briefly commented, however, stereoscopics in 3D movies at the cinema, or in 360˚ 3D recordings merely provide an illusion of depth — not actual depth. The same goes for our eyes, although they mostly perceive it correctly,  they are easily fooled. 360˚ 3D cameras is an example of this, they merely fool our eyes: although it seems that there is depth, we can not really move in the image — as there is no actual depth to it. Here, volumetric video acts differently, and affords positional interaction. Volumetric video attributes the recorded images in a 3D (x,y,z) space, in addition to delivering stereoscopy so that we can perceive it. Volumetric video is unfortunately very hard to create while still retaining high quality, and plug-and-play solutions still seem far off. To get an idea of how volumetric video works, we recommend to look into the concepts of photogrammetry — and perhaps even to create a 3D model yourself, using images captured with your smartphone. This YouTube tutorial shows you how to do this in Agisoft Photoscan Pro, which has a free trial available.

Limitations

Developed in an undergraduate course at the University of Bergen, the short 360 movie “Schizophrenia“, experimented with interactive 360 video.

Despite these great innovations in the capture of reality, CGI has some benefits that neither 360˚ 3D or Volumetric videos can really achieve. The most important of these is that of interactivity . As 360° videos are linear (that is, they have a predetermined beginning and end), the user can not really affect what happens in the video — except by choosing which degrees of the video to see.

In our course in VR Journalism at the University of Bergen where I taught students VR programming, 360° video and photogrammetry — we faced this exact limitation. A group that worked on providing an experience of the reality-shattering disorder of Schizophrenia, wanted hallucinations to occur when the user viewed at certain areas. The students solved this by placing transparent gifs over the video in A-Frame, edited based on the real footage, and put gaze event listeners to activate the playing of the gif. The results were extraordinary, and could well provide a new way to provide a means of simpler interaction on top of 360° videos. The experience, which voices are in Norwegian, can be viewed here (WebVR browser such as Chrome is necessary).

Sensory Deprivation — Floating in Virtual Reality

If we look to our glossary, we see Presence within Virtual Reality (VR) defined as the degree to which the subject feels present in the virtual world. What is interesting to note, is that this naturally has to be viewed relatively to the degree that the subject feels present in the physical world — as we usually receive information from both our physical and our virtual environments.

There can thus be two separated approaches to designing for presence in virtual reality environments: one is to provide sensory stimulus of the virtual environment, and the other is to block sensory stimulus from the physical environment. Both approaches work towards the same goal of immersion — the encapsulation of the user in the VE. Slater and Wilbur (1997) recognise this in their definition of Immersion, which is closely related to the notion of Presence. They define immersion in terms of four qualities the system can afford, the first one of which is called inclusiveness. Inclusiveness they define as the extent to which physical reality is shut out.

Obviously, the principle of adding and removing sensory experience go hand in hand; by equipping a Head-Mounted Display you are blocking the physical impressions and replacing them with virtual impressions, all the while shielding for incoming light from the surroundings. Blocking light, however, is not the only way to deprive the senses of information from the physical environment. In this entry, we will discuss how we can maximize the inclusiveness of the immersion by achieving sensory deprivation in floatation tanks. Floating in Virtual Reality!

Floatation Chambers

Floatation chambers, or sensory deprivation tanks — are pools of water with copious amounts of epsom salt (≈600kg). The tanks are sealed for any incoming light and sound, and the air- and water temperature is equal to that of your body. When you lie down, you will feel how the salt makes you float even though the pool is very shallow. As you lie there, you notice how the ripples you created when lying down start to slowly subside as you sink down into weightlessness. After a while, because of the air- and water temperatures are the same as that of your body, you can no longer pinpoint where the water ends and the air around you begins. In fact, it gets hard to distinguish anything from anything else, including your body from the air and water. There is really nothing that is easy to grasp as isolated, save perhaps your breath. And as the minutes go, with total physical relaxation and lack of much sensory impression at all, things start to change.

“Alone With Your Thoughts”, Illustration by Cole Ott

The most significant, explicit change one may notice in the tank  is that after a while your bodily self-consciousness is not what it used to be. Your mental model of where your body is in relation to the world around you starts to become blurred. Normally reinforced by tactile stimuli of air and water (of varying temperatures), and visual and auditive stimuli from the environment, your body model is now lacking information on which to create it. Your sense of spaciousness has also changed, that is the feeling of your position as defined relatively to say, the walls, mountains and sky has disappeared. You now really experience nothing around you, but neither any edges to this lack of information in your surroundings. You may get the feeling of floating in empty space — but where are you in all of this? What, in this stream of conscious experience is matter and what is mind?

Inner vs Outer

In our entry — ‘Inner as Outer: Projecting Mental States as External Reality‘ — we discussed the potential of using VR for meditation purposes in experimental ways. In the introduction to the entry, we discussed our feeling of Self as a duality of Inner and Outer, of which our everyday experiences usually comprise. We discussed how technology may have the power to transform our consciousness away from this traditional subject-object hierarchy and into a non-dual one, where the Inner is seen as the Outer, and the Outer as Inner. In this entry we are building further on these ideas. Similarly to visualising inner states in VR through biometrics, using VR in floatation tanks might provide illusory experiences where the conscious experience is significantly altered.

One other entry relevant to our experiments with VR in floatation tanks should be mentioned before we go on: the entry on Virtual Embodiment. In the entry, we discuss the great potential of VR to hack our consciousness; why it is possible, and what it can be used for. The research is highly relevant for floatation in VR, as both floatation tanks and VR alter our self-model, as both alter the sensory impressions necessary to maintain it.

Research on Virtual Embodiment in Floatation Tanks

Matrise partnered up with Bergen Flyt, a local company offering floatation therapy in the heart of Bergen city. We used a Samsung Gear VR with a Samsung S8 phone. We did not use a HTC Vive (Pro) as it would be more risky exposing the cable to water. Also, no room tracking or even much head orientation was needed, and in terms of resolution the HMD is quite high in ppi. We chose to first try out some abstract visualisations through the application “Fractal Lounge”, that shows varying psychedelic visuals and floating through space.

My Experience

“After I had showered, I put on the GEAR VR headset, started the application, and slowly entered the floatation pool. I held my hands towards the wall, as I did not see anything else than the visuals in the headset. When I was inside, I closed the glass door, and slowly lowered myself into the water — back first. It took a few seconds before I dared to lower my head all the way down, but very soon I was totally relaxed. As expected, the electronics in the display was kept well above water, due to the intense amount of salt in the water …”

The kind of visualisations provided from Fractal Lounge, the application that was tried in the floatation tank.

“The visualisation pulsated, floated, drifted along — and often totally changed in colours and shapes. It took probably about ten minutes before my feeling of body totally vanished, to the degree that it was a larger gap between wanting to move the body and actually being able to move it than usual. I felt like perceiving a great drama and scene, and I got engaged in the forms and ways of the visualisations, sometimes quite invested in it, as it felt close and reality-defining for me. After about twenty minutes in, I felt as if I was drifting along in space at high speed, because of the steady movement of stars away from me. At the same time, there was no sound, which made the quick travel feel peaceful and smooth. As with normal floating, about every ten minutes there is a sort of reality-check moment where you remember you are in the tank and contemplate how weird it is. This also happened in VR, and was … equally as weird”

Reflections and Future Work

My first experiment with floatation in VR lasted for about 45 minutes. Sometimes, unfortunately, the VR headset glided slightly off my face, and I had to reposition it with my wet, salty fingers. After this happened about three times, I had to leave the tank in order to save the equipment.

Thank God that we have floatation pools instead of this creepy stuff.

My first experience of floating in virtual reality was very promising. The largest surprise was the feeling of movement through space at high speed. The largest frustration was the lack of any sort of interaction with ones surroundings at all, except the possibility to open and close one’s eyes. A great experiment would be to use eye tracking technology as a way of navigation in the vast, abstract psychedelic spaces. If one travelled towards where one saw, one could even be interactive while lying still in the floatation tank. This could also possibly have curious effects on which parts (perhaps the eyes), we identify with our selves. Perhaps the placement of our self could be altered by changing the agency for transportation.

Matrise will continue the cooperation with Bergen Flyt, and both try and develop different applications. Our plan is to measure the feeling of presence and self-identification and consciousness while in the tank.

 

Literature list

Augmenting our Reality

Although Matrise usually cover the more encapsulating technologies on the Reality-Virtuality-Continuum, we are also very interested in innovative uses of all Extended Reality (XR) technologies.  In this entry we will illustrate the utility of Augmented Reality (AR) technologies with an exciting project we are presenting at this years IBC conference in Amsterdam.

Short History of AR

AR has seen a similar hype as VR have with products as Microsoft Hololens, Magic Leap and the Google Glass. The technology concept has, similarly to VR, the power to change our orientation towards reality — however, AR technology lets you see your surroundings in addition to the augmented virtual phenomena. We discussed its historically conceptual origins previously in our entry on the Camera Lucida, but apart from this case — the technology is somewhat younger than VR technology.

The first Virtual Reality Head-Mounted Display, named after the Sword of Damocles, because of its great weight hanging over the user’s head. Named after an old greek cultural symbol of Mortality — the anecdote is that the sword hung from a single horse hair over the head of Damocles after he rose to wealth.

In our History of VR we mentioned ‘The Sword of Damocles’ as the first thorough Head-Mounted Display. It should be noted, however, that this essentially also was an AR display, not just VR. The glasses, as can be seen in the illustration, were somewhat see-through, and could therefore be used to augment the physical surroundings of its wearer. Today, however, we are far more privileged, and can experience sophisticated AR that let us view virtual phenomena within the environment itself hyperlocally, with extremely room tracking with six degrees of freedom in the interaction. This enables far more sophisticated usage of the technology.

AR has yet to have a commercial launch similar to that of VR. Although mobile AR with smartphones has gained some popularity, we will have to wait some more for the non-invasive affordable Head-Mounted Displays to hit the market (the Magic Leap currently costs $2295  and the Microsoft Hololens is at about $3000 for developers). That being said, it is fun to play and develop with these technologies and be part of creating new solutions with immersive technologies. Although they are not yet fit for mass-adoption, the Microsoft Hololens is a great project that illustrates the potential of these technologies.

 

The Microsoft Hololens: an AR Head-Mounted Display released for developers. It is believed we will see the next iteration of the HoloLens in 2019.

HoloSuite: A Mobile AR Video Editing Suite

Saturday and Sunday at IBC 2018, Joakim from Matrise is joining four masters students from media- and interaction design in presenting an AR application for the broadcasting industry. The application was developed as a student project for the Bergen-based company Vimond, and is presented at Media City Bergen’s stand at IBC’18 — to represent the fruitful collaboration of the University of Bergen with the companies in the NCE Media cluster Media City Bergen.

The masters students involved; Audun Klyve Gulbrandsen; Johanne Ågotnes and Fredrik Jenssen also has lots of other interesting projects that can be read about at their website UiB MixMaster.

The presentation takes place at 12:30 at Sat&Sun, in hall 8, at booth D10 (MCB Village).

Abstract of Presentation

«Professional video editing suites of today are resource-demanding. A video editor needs great machine power in addition to multiple screens to tackle the varying formats of today’s media landscape. Effectively, this results in reduced freedom of mobility for video editors; they are dependent on their stationary office space to work. In addition to reducing flexibility, this lack of freedom may slow down the turnaround process for news agencies.

In our presentation, we describe and demonstrate an Augmented Reality (AR) cloud-based video editing suite, where up to five virtual screens are presented to the user through a Microsoft Hololens Head-Mounted Display. By employing cloud computing, the prototype can access machine power remotely through the cloud, which has benefits in terms of mobility. Effectively, the AR application is a prototype of an office for video editors that can be carried in a backpack, and utilized wherever there is network connectivity»

 

Figure illustrating the timeline and the preview screens (resolution is higher, and FOV lower in the HoloLens display itself).
Same content from a different angle, illustrating the fixed environmental position (resolution is higher, and FOV lower in the HoloLens display itself).

Do you have any great AR ideas? Matrise will soon publish a new entry on an AR product development process we are involved in — so stay tuned.

 

Virtual Embodiment

The most praised ability of Virtual Reality is its capability to immerse the user in a Virtual Environment — to the degree that the subject feels present in it. The magic is to be fooled by the system so that one feels present where one actually does not physically reside. This effect can, however, turn even more magical. A deeper step into the effects of technological immersion is found in the concept of Virtual Embodiment. If a subject is embodied virtually,  not only is the virtual environment accepted as such; the subject also identifies with a virtual body or avatar inside the virtual environment. This differs from realizing which character you control in a game — within Virtual Embodiment it is the same processes that make you identify with your real body that makes you identify with a virtual one. This is a key point, as it is why research into virtual embodiment is important.

Peeling layers of the onion: VR can be a tool to discover who we are, through investigation of what and how we identify with our bodies. Illustration: “Mask of Day by Day” by Paulo Zerbato.

Hacking and Experimenting with Consciousness

What is fascinating about both of these possibilities of illusion, then — is how, and that, they are possible at all. Knowledge on how to achieve such immersion is obviously relevant for all VR developers, but the knowledge that can be obtained by researching these phenomena goes far beyond knowing how to apply it in VR technology. By creating experiments in VR, we can generate, and investigate, phenomenas of the mind under various experimental conditions. Exploring Virtual Embodiment, for instance, can enable us with a better understanding of our self-consciousness and the relationship between body and mind. Because of this wider span, research on Virtual Embodiment attracts neuroscience researchers, psychologists, information scientists and philosopher’s alike.

The Rubber Hand Illusion

The Rubber Hand Illusion (RHI) is an excellent example of the kind of ‘brain hacks’ that can be achieved by sensory manipulation. The illusion, as illustrated below, is a perfectly simple experiment that does not even require the use of VR technology to perform.  The RHI was introduced by Ehrson, Spence & Passingham (2004) and has been an ingenious way to illustrate how we identify with our bodies. More importantly for this entry, the results of the experiment has inspired further research on Virtual Embodiment.

Illustration from Thomas Metzinger’s book “The Ego Tunnel: The Science of The Mind and The Myth of the Self”

In the RHI, the hand of the subject is replaced by a rubber hand, while the normal hand is blocked from sight by a separating wall. When the subject is sitting as such, a researcher will stroke each hand, both the rubber and the physical hand, simultaneously. Now, the question is what happens when experiencing the sensory impression of stroking, all the while seeing a corresponding stroke on the rubber hand?

Put very simply, the brain does a ‘reasonable guess’ that this hand is indeed the correct physical hand attached to your body.  You feel that the rubber hand is yours, with nerve-endings and all — and you couple your physical feelings to the vision of the hand. This means that in your subjective experience, the rubber hand is the hand that has the sensation. Ehrson et. al write that their results suggested that “multisensory integration in the premotor cortex provides a mechanism for bodily self-attribution”. When our brains receive sensory information from two differing sensory inputs (sight+feel), these are coupled: the brain is coupling the stroking-sensation with imagery of a nearby-hand being stroked, and this is enough for the brain to attribute its self with the hand, to acknowledge it as its own.

This simple experiment share a lot of principles with the concept of Virtual Embodiment, and has inspired research in the field that we will present in this entry.

Some experience out of body experiences (OBEs) on the onset of sleep or waking up. Often they may feel that they are floating over their bodies. VR may help to study such states of consciousness by systematically inducing them.

Virtual Body Illusion

In a later experiment by Lenggenhager et.al (2007), not only the hands of the subjects — but their whole bodies were replaced with virtual representations. Moreover, in the experiment they present, the bodies are seen from behind. In effect, they were simulating out-of-body experiences, with very interesting results.

The experiment was conducted as such: the subjects wore a Head-Mounted Display which projected imagery from a camera located behind the subjects. As such, the subjects could see a representation of their bodies “live”, but from behind. Of course, this is deviating slightly from how we normally experience life. Although the subjects saw their body responding and performing actions in real time as under normal conditions — there is a logical dissonance due to the mismatch between the location of the subjects’ eyes in the virtual environment, and what these eyes see. Effectively, the user is seeing inside a pair of “portal” binoculars (HMD), which display the light from, if not another dimension, then at least a few feet away. And this will be a part of the point.

What is interesting about this experiment is not necessarily simply that the users feel present where they do not reside physically, but how the distance is only a few feet off. The users feel present right outside of their bodies. The situation is similar, the body and the environment is there, but everything is a bit off. What is interesting to investigate then, is how the body adapts to this. Will it accept that it now controls its body from a third person perspective, similarly to how Stratton’s subjects got used to seeing the world upside down?

What they studied was basically whether this change of perspective had an impact on where the users felt embodied. To investigate this, the researchers stroked the subjects as they did in the Rubber Hand Illusion, except at their backs — so that it was perceivable by them. The question is then where this physical feeling will be attributed to — how will the phenomena of the subjective experience present themselves to the subjects?

Out of Body experiences can be achieved virtually by using sensory impressions from other locations, for instance five meters behind you as in the experiment by Ehrson (2007). You can then effectively look at yourself from the outside.

First of all, to be clear on this — the sensory data of being stroked will initially be provided by the nerves in the physical shoulder of the user. The problem of the brain, however, is that the shoulder is out of sight — blocked by the Head-Mounted Display. There is, however, the visual impression of a shoulder on a person standing in front — being scratched in exactly the same way. Although the nerve-endings definitely feel the stroking, the problem is that where this feeling will be placed in our subjective experience is not the responsibility of the shoulder, but rather the brain. And, as the placement of the physical feeling in the bodily self-consciousness is largely dependent on vision for coordinates, what will happen? How will the brain fix this sensory discord?

In this beautifully written article by The New Yorker, its author Rothman describes one of the co-authors of the research paper, Thomas Metzinger’s, own experience undergoing the experimental conditions:

Metzinger could feel the stroking, but the body to which it was happening seemed to be situated in front of him. He felt a strange sensation, as though he were drifting in space, or being stretched between the two bodies. He wanted to jump entirely into the body before him, but couldn’t. He seemed marooned outside of himself. It wasn’t quite an out-of-body experience, but it was proof that, using computer technology, the self-model could easily be manipulated. A new area of research had been created: virtual embodiment.”

Are We Already Living in Virtual Reality?” — The New Yorker has a brilliant, long, read on Virtual Embodiment that features interviews with VR and Consciousness researchers Prof. Mel Slater and Prof. Thomas Metzinger.

Phantom Pain

Another curious potential effect of Virtual Embodiment, is the possibility of phantom sensory impressions as well. Handling virtual objects while being embodied, for instance, may convince your body to expect pain or touch — and so this is, somehow, actively generated. Because of this, VR may be a way to study how phantom pain is created, and further how it can be alleviated. For instance, several studies show how VR can embody a subject missing a leg in a body with two legs, similarly to traditional mirror therapy treatment, which is effective in reducing phantom pain. Again — what may be most interesting here is the possibility of systematically creating the phenomena and studying it afterwards. For instance, as Metzinger is quoted on in The New Yorker’s article, it may be supposed that phantom pain is created by a body model not corresponding to the physical reality. This will be the case for phantom pain in VR: it is not based on the physical reality, you are only relating to a virtual reality instead. Similarly, those those with real phantom pain may also be relating to a certain kind of “virtual reality”, but rather one in the format of their skewed narratives — maintained by their minds instead of a computer.

That the narrative, worldview and consciousness that our brain’s experience and generate is often not the best match with reality is not something new. As for Matrise, these concepts reminds us of the conclusion from our three-series entry towards a metaphysical standpoint on VR, in which we discussed VR as rather examplifying of our abstracting tendencies of mind. These entries can be read at Matrise, and were called: 1) On Mediums of Abstraction and Transparency, 2) Heidegger’s Virtual Reality, and 3) The Mind as Medium.

Virtual Embodiment for Social Good

Now that we have discussed the concept of Virtual Embodiment, it may be natural to discuss what this knowledge can be used for. As discussed already, generating experiments in VR that hacks our self models, may provide useful knowledge on the structure of our self-consciousness. Apart from this general knowledge, some may also have practical utilisation in applied VR for specific scenarios.

Racial Bias

A very exciting paper that describes work utilizing virtual embodiment, is one by Banakou, Hanumanthu and Slater. In the project, they embodied White people in Black bodies, and found that this significantly reduced their implicit racial bias! The article can be found and read in its entirety here (abstract available for all).

Domestic Violence

Another interesting project by Seinfeld et. al, is one in which male offenders of domestic violence became embodied in the role of a female victim in a virtual scenario. At first in the experiment, the male subject is familiarized with his new, female, virtual body and the new virtual environment. When the body ownership illusion, or virtual embodiment, has been achieved, a virtual male enters the room and becomes verbally abusive. All this time, the subject can see his own female body reflected in a mirror, with all his actions corresponding to his. After a while, the virtual male starts to physically throw around things and start to appear violent. Eventually it escalates and he gets closer into what feels like the subjects personal space, and appear threatening.

They write:

Our results revealed that offenders have a significantly lower ability to recognize fear in female faces compared to controls, with a bias towards classifying fearful faces as happy. After being embodied in a female victim, offenders improved their ability to recognize fearful female faces and reduced their bias towards recognizing fearful faces as happy”

The article can be read in its entirety at ResearchGate.

Staying Updated in the field of Virtual Embodiment

Research on Virtual Embodiment is happening continuously. To stay updated on this area of VR research, I enjoy following Mel Slater, Mavi Sanches-Vives and Thomas Metzinger on Twitter. Last but not least, I would stay updated on Virtual Bodyworks at Twitter, of which both Sanchez-Vives and Slater are co-founders of.


N.B: This entry lies at the centre of Matrise’s interests, and we are planning on writing several entries on this topic further in philosophical directions. Have any ideas or want to contribute? Please contact us.

Literature list

 

 

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSaveSaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

Inner as Outer: Projecting Mental States as External Reality.

Introduction to Mysticism

Within Mysticism, the merging of Self and World — Inner and Outer — is seen as the utmost aim. Mysticism can be found within most of the world religions, such as Buddhism, Christianity, Hinduism and Islam — and its aim is often formulated as union with God. Depending on the religion, however, the degree to which Mysticism is the common way of practicing the religion varies. Although many religions have such contemplative practices, they are not always adopted by the religion’s followers at large.

When discussing «Union with God», it should be noted that the term «God» varies in its meaning between these religions. The contemplative practices often have significantly varying metaphysics, for instance Monotheistic (Christianity), Polytheistic (Hinduism), and relatively Atheistic or Agnostic (Buddhism). Be this as it may, their descriptions of the experience of this merging of Self and God is often strikingly similar. These states of enlightenment are often described as ecstatic, in which the conscious experience can not be placed within our normal frames of language or understanding.

What also unites the different traditions, is that such states of consciousness is usually  worked towards through contemplative practice such as yoga, meditation or other disciplines of focus or conscious attention. Other techniques for achieving these ecstasies have have been ascetic ones, such as fasting, waking, isolation — or other ways of stirring the Self to war.

The experience of seeing the Inner as Outer, and the Outer as Inner, is often described as the feeling that living itself is an experience of seeing and perceiving Oneself and/or God. Within this worldview, there is no Self relating to anything external.

Non-duality: synchronization of Inner & Outer

The concept of merging Inner and Outer, or Self and God, can each be viewed either in very material or spiritual terms. Although materiality and spirituality do not have to differ metaphysically, separating these gives us some communicative benefits — and Mysticism may be explained and spoken of from both these perspectives. Discussing the Inner as Outer purely «scientifically», if you will, makes sense in that all our perceptions of the Outer world is indeed created Inner, and as such — Reality will always be a synergy of Inner and Outer. We know that we do not see, or have ever seen, anything which we ourselves do not actively generate. As neuroscientist and consciousness researcher Anil Seth put it, “our brains are actively hallunicating our conscious reality”.

States where a subject experiences the Inner and Outer as ‘one’, is often referred to as «non-dual».  Often while speaking of Inner and Outer, we tend to implicitly reinforce the Self and the World as a duality (when pitching a solution we often have to pitch the problem first). By using the word «non-dual» instead of ‘one’, we may pinpoint the nuance that it is not a duality in separation, but neither completely “same-same”. Although it is non-dual, neither is it all same or flat — least of all static!

Although we classify and divide our reality, fundamentally what we perceive is a stream of experience, which in every sense is simply “reality” before divided, and, again, actively created by us. This is not to say that there are no external reality or world — but it definitely is to say that all which is external is perceived first and foremost, solely, internally. Experientially — externality has never been perceived, except as a subcomponent of internality.

A vase, or two faces? Each defines the other, and neither exist without the other.

Experiencing and Sensing the Non-Dual

This causal explanation, however, leaves out the experiential aspects of the non-duality. Although it may make sense on paper, it matters little to us as we absolutely perceive the world as dual — as subjects relating to a World. Within Philosophy, this traditional way of adhering to and speaking of the world,  is referred to as the subject-object dichotomy. Although, between different cultures and continents, the degree to which we adhere to this way of thinking vary in its intensity, it is nevertheless definitely an essential part of the human experience which we share.

How the material explanation can be said to be different from the spiritual in this sense, is that the spiritual concern is to experience the Inner as Outer, not to understand it cognitively. As such, and towards that, meditation practices such as Mindfulness and Yoga have existed, to increase wellbeing by increasing the degree to which one feels in union with God, or for those who do not fancy the term; to the degree which one has peace with oneself and the world.

Contemplative practices such as yoga and meditation, has the last fifty years become more popular in western societies. Although they have been subject to a certain degree of metaphysical raffination the last years, these methods are nevertheless largely old and traditional. The most common of these contemplative practices we see today is adopted from the Vippassana practice, commonly known as Mindfulness. These methods are now commonly used in psychological treatment of anxiety and depression, and research has the latest years started to uncover the benefits of learning to be able to sit quietly with your mind and, well, deal with shit, or seeing it for what it is.

In the next section, we will discuss an approach utilizing Virtual Reality to aid in Mindfulness meditation — which can help to perceive the Inner as Outer.

A common belief is that the aim of contemplative practices is to empty the mind. In a sense, it can be said to be correct, in that meditation practices often seek to eradicate, dilute or cancel the self-referential narratives.

The effects of Mindfulness meditation

The essence of Mindfulness or similar contemplative practices, lie in their manipulation of identity. We stated “the problem” of Mysticism as the gap between Self and Other — and for this separation to be there, we must necessarily have a relatively thoroughly defined sense of self. For most of us, this tend to be limited to the cognitive processes that constitute our mental narrative (the personalized voice in our heads, our formulated will, and how it appears to direct our actions and plans our lives). It is actually to a far lesser extent our bodies, although this also attributes to our self-consciousness.

Mindfulness is about being present attentively in each moment to one’s state of mind. When doing such focus excercises directed at the mind, and observing these mental processes closely, the idea or view of them as solid things starts to unravel. When rather seeing them as thoughts from a distance, they appear untangled to us, and we perceive our own existence as distinct from those thoughts.

Virtual Reality Biofeedback as Meditation aid

One of the great benefits of VR is its ability to project and represent data in the format of the reality encompassing us. Within the context of this entry, we could say therefore that VR can simulate what we perceive as the Outer. The question may then be asked: how can we project our Inner in to this medium of Outer?

Although I believe we will see more work on VR biofeedback within this domain in the future, in this entry we will focus only on one research paper in particular to examplify our case. At last years CHI conference, the world leading conference on Human Computer Interaction, Joan Sol Roo and his colleagues presented their work on Inner Garden: a mixed reality sandbox for mindfulness. The artifact is a physical sandbox, which the user can shape to a given terrain. The sandbox is given generally visually augmented by a projector with colors and shapes — and physical changes to the sandbox will also alter the output of the projector, which deliver terrain information such as sea levels and green growth.

The sandbox is just not physical, however; by placing a physical avatar in the physical sandbox, you can enter into the land you created in Immersive VR. A 3D-model of the land you created physically can be seen virtually, from the viewpoint of your placed avatar.

The Sandbox, which heights of the sand have been turned into an island by the projector.

Attached, to measure your inner states, is both breathing- and heart rate sensors — which are coupled to provide visual and auditive feedback. In this way, you can synchronize your breath to control the environment and rythm and breaking of the waves. The Inner Garden represents your inner state, and. by practicising breathing techniques, the flora of your world will get greener and more animals will appear.

In this way, Inner Garden works as a great example of representing Inner phenomena as External Reality. Very conceptually interesting, and hopefully one day we will also see empirical studies on similar artifacts.

You can read more about Inner Garden, which received an honorable mention at CHI’17, here.


What do you think? Do you have any ideas for VR applications using biofeedback?  Please comment below.

Literature list

 

 

 

 

 

SaveSave

SaveSave

 

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

SaveSave

Camera Lucida

We have previously discussed several interesting optical technologies of relevance to VR. For instance we discussed the fascinating 17th century Camera Obscura and in our entry on a History of VR, we discussed the 19th century Stereoscope, which technology still is used in modern day VR Head-Mounted Displays.

In this entry we will discuss yet another,  similarly old, optical technology, which in category leans more towards that of Augmented Reality than Virtual Reality; the Camera Lucida.

Invented by phycisist William Hyde Wollaston in 1807, the Lucida was a device praised by artists and illustrators for its aid in their art. Similarly to the earlier dated Obscura, the optical artifact could project and redirect images from the external world, making it easier to recreate them in ink. While the Obscura required a dark room to project its images on a surface, the Lucida had the benefit of redirecting the light directly to its users eyes, and was thus more appropriate to use in a lit office, or even while travelling.

Apart from the underlying technical difference, the practice of use was relatively similar; the user would perceive the redirected light representing the object of projection on the surface that should be drawn, and by following the lines with a pen, the image could be reproduced in ink. To draw objects far off, the light could also be captured by a telescope, or for very small details, even a microscope, as seen in the illustration below.

Camera Lucida and Modern day AR technology
The camera lucida share many conceptual and experiential similarities with Augmented Reality (the concept of augmenting our real world with virtual phenomena). When a user is looking through the Camera Lucida,  a ‘virtual’ representation of what the Lucida is directed at, is added to and combined with the user’s normal vision. In AR goggles such as the Microsoft Hololens, this concepts remain the same, only the HoloLens’ holographic images originate from software and not redirected light from the external world.

 

The Microsoft HoloLens, an AR Head-Mounted Display by Microsoft.

Obviously, this is not the only difference between the two — compared to modern AR tech, the functionality and applicability of the Lucida is bleak. The HoloLens is  capable of stereo pictures, and features placement and projection of almost any virtually conceivable object in to the environment. Yet, the beautiful Camera Lucida artifact does share the essential underpinnings of augmenting the environment with re-presentations.

A curious example of the similarity between the two, is how the Lucida these days are being recreated with (mobile) AR. Using for instance an iPad with its camera, the canvas and your hand drawing is displayed to you on the screen, with a see-through image of that which you want to draw. Even better, a similar application has also been developed for the Microsoft HoloLens, called SketchAR HoloLensEDU — and is currently being employed teaching young artists.


Do you know of any good, useful applications within the AR domain? Please comment below!

 

SaveSave