Sensory Deprivation — Floating in Virtual Reality

If we look to our glossary, we see Presence within Virtual Reality (VR) defined as the degree to which the subject feels present in the virtual world. What is interesting to note, is that this naturally has to be viewed relatively to the degree that the subject feels present in the physical world — as we usually receive information from both our physical and our virtual environments.

There can thus be two separated approaches to designing for presence in virtual reality environments: one is to provide sensory stimulus of the virtual environment, and the other is to block sensory stimulus from the physical environment. Both approaches work towards the same goal of immersion — the encapsulation of the user in the VE. Slater and Wilbur (1997) recognise this in their definition of Immersion, which is closely related to the notion of Presence. They define immersion in terms of four qualities the system can afford, the first one of which is called inclusiveness. Inclusiveness they define as the extent to which physical reality is shut out.

Obviously, the principle of adding and removing sensory experience go hand in hand; by equipping a Head-Mounted Display you are blocking the physical impressions and replacing them with virtual impressions, all the while shielding for incoming light from the surroundings. Blocking light, however, is not the only way to deprive the senses of information from the physical environment. In this entry, we will discuss how we can maximize the inclusiveness of the immersion by achieving sensory deprivation in floatation tanks. Floating in Virtual Reality!

Floatation Chambers

Floatation chambers, or sensory deprivation tanks — are pools of water with copious amounts of epsom salt (≈600kg). The tanks are sealed for any incoming light and sound, and the air- and water temperature is equal to that of your body. When you lie down, you will feel how the salt makes you float even though the pool is very shallow. As you lie there, you notice how the ripples you created when lying down start to slowly subside as you sink down into weightlessness. After a while, because of the air- and water temperatures are the same as that of your body, you can no longer pinpoint where the water ends and the air around you begins. In fact, it gets hard to distinguish anything from anything else, including your body from the air and water. There is really nothing that is easy to grasp as isolated, save perhaps your breath. And as the minutes go, with total physical relaxation and lack of much sensory impression at all, things start to change.

“Alone With Your Thoughts”, Illustration by Cole Ott

The most significant, explicit change one may notice in the tank  is that after a while your bodily self-consciousness is not what it used to be. Your mental model of where your body is in relation to the world around you starts to become blurred. Normally reinforced by tactile stimuli of air and water (of varying temperatures), and visual and auditive stimuli from the environment, your body model is now lacking information on which to create it. Your sense of spaciousness has also changed, that is the feeling of your position as defined relatively to say, the walls, mountains and sky has disappeared. You now really experience nothing around you, but neither any edges to this lack of information in your surroundings. You may get the feeling of floating in empty space — but where are you in all of this? What, in this stream of conscious experience is matter and what is mind?

Inner vs Outer

In our entry — ‘Inner as Outer: Projecting Mental States as External Reality‘ — we discussed the potential of using VR for meditation purposes in experimental ways. In the introduction to the entry, we discussed our feeling of Self as a duality of Inner and Outer, of which our everyday experiences usually comprise. We discussed how technology may have the power to transform our consciousness away from this traditional subject-object hierarchy and into a non-dual one, where the Inner is seen as the Outer, and the Outer as Inner. In this entry we are building further on these ideas. Similarly to visualising inner states in VR through biometrics, using VR in floatation tanks might provide illusory experiences where the conscious experience is significantly altered.

One other entry relevant to our experiments with VR in floatation tanks should be mentioned before we go on: the entry on Virtual Embodiment. In the entry, we discuss the great potential of VR to hack our consciousness; why it is possible, and what it can be used for. The research is highly relevant for floatation in VR, as both floatation tanks and VR alter our self-model, as both alter the sensory impressions necessary to maintain it.

Research on Virtual Embodiment in Floatation Tanks

Matrise partnered up with Bergen Flyt, a local company offering floatation therapy in the heart of Bergen city. We used a Samsung Gear VR with a Samsung S8 phone. We did not use a HTC Vive (Pro) as it would be more risky exposing the cable to water. Also, no room tracking or even much head orientation was needed, and in terms of resolution the HMD is quite high in ppi. We chose to first try out some abstract visualisations through the application “Fractal Lounge”, that shows varying psychedelic visuals and floating through space.

My Experience

“After I had showered, I put on the GEAR VR headset, started the application, and slowly entered the floatation pool. I held my hands towards the wall, as I did not see anything else than the visuals in the headset. When I was inside, I closed the glass door, and slowly lowered myself into the water — back first. It took a few seconds before I dared to lower my head all the way down, but very soon I was totally relaxed. As expected, the electronics in the display was kept well above water, due to the intense amount of salt in the water …”

The kind of visualisations provided from Fractal Lounge, the application that was tried in the floatation tank.

“The visualisation pulsated, floated, drifted along — and often totally changed in colours and shapes. It took probably about ten minutes before my feeling of body totally vanished, to the degree that it was a larger gap between wanting to move the body and actually being able to move it than usual. I felt like perceiving a great drama and scene, and I got engaged in the forms and ways of the visualisations, sometimes quite invested in it, as it felt close and reality-defining for me. After about twenty minutes in, I felt as if I was drifting along in space at high speed, because of the steady movement of stars away from me. At the same time, there was no sound, which made the quick travel feel peaceful and smooth. As with normal floating, about every ten minutes there is a sort of reality-check moment where you remember you are in the tank and contemplate how weird it is. This also happened in VR, and was … equally as weird”

Reflections and Future Work

My first experiment with floatation in VR lasted for about 45 minutes. Sometimes, unfortunately, the VR headset glided slightly off my face, and I had to reposition it with my wet, salty fingers. After this happened about three times, I had to leave the tank in order to save the equipment.

Thank God that we have floatation pools instead of this creepy stuff.

My first experience of floating in virtual reality was very promising. The largest surprise was the feeling of movement through space at high speed. The largest frustration was the lack of any sort of interaction with ones surroundings at all, except the possibility to open and close one’s eyes. A great experiment would be to use eye tracking technology as a way of navigation in the vast, abstract psychedelic spaces. If one travelled towards where one saw, one could even be interactive while lying still in the floatation tank. This could also possibly have curious effects on which parts (perhaps the eyes), we identify with our selves. Perhaps the placement of our self could be altered by changing the agency for transportation.

Matrise will continue the cooperation with Bergen Flyt, and both try and develop different applications. Our plan is to measure the feeling of presence and self-identification and consciousness while in the tank.

 

Literature list

Camera Lucida

We have previously discussed several interesting optical technologies of relevance to VR. For instance we discussed the fascinating 17th century Camera Obscura and in our entry on a History of VR, we discussed the 19th century Stereoscope, which technology still is used in modern day VR Head-Mounted Displays.

In this entry we will discuss yet another,  similarly old, optical technology, which in category leans more towards that of Augmented Reality than Virtual Reality; the Camera Lucida.

Invented by phycisist William Hyde Wollaston in 1807, the Lucida was a device praised by artists and illustrators for its aid in their art. Similarly to the earlier dated Obscura, the optical artifact could project and redirect images from the external world, making it easier to recreate them in ink. While the Obscura required a dark room to project its images on a surface, the Lucida had the benefit of redirecting the light directly to its users eyes, and was thus more appropriate to use in a lit office, or even while travelling.

Apart from the underlying technical difference, the practice of use was relatively similar; the user would perceive the redirected light representing the object of projection on the surface that should be drawn, and by following the lines with a pen, the image could be reproduced in ink. To draw objects far off, the light could also be captured by a telescope, or for very small details, even a microscope, as seen in the illustration below.

Camera Lucida and Modern day AR technology
The camera lucida share many conceptual and experiential similarities with Augmented Reality (the concept of augmenting our real world with virtual phenomena). When a user is looking through the Camera Lucida,  a ‘virtual’ representation of what the Lucida is directed at, is added to and combined with the user’s normal vision. In AR goggles such as the Microsoft Hololens, this concepts remain the same, only the HoloLens’ holographic images originate from software and not redirected light from the external world.

 

The Microsoft HoloLens, an AR Head-Mounted Display by Microsoft.

Obviously, this is not the only difference between the two — compared to modern AR tech, the functionality and applicability of the Lucida is bleak. The HoloLens is  capable of stereo pictures, and features placement and projection of almost any virtually conceivable object in to the environment. Yet, the beautiful Camera Lucida artifact does share the essential underpinnings of augmenting the environment with re-presentations.

A curious example of the similarity between the two, is how the Lucida these days are being recreated with (mobile) AR. Using for instance an iPad with its camera, the canvas and your hand drawing is displayed to you on the screen, with a see-through image of that which you want to draw. Even better, a similar application has also been developed for the Microsoft HoloLens, called SketchAR HoloLensEDU — and is currently being employed teaching young artists.


Do you know of any good, useful applications within the AR domain? Please comment below!

 

SaveSave

Camera Obscura and the World of Illusions

A few years ago I visited the beautiful scottish city of Edinburgh. Apart from the old pubs, the whisky and its mighty castles, the city also have attractions for those interested in the art of illusion.  In a castle on one of the heights of the city, we can find an example of an ingenious yet simple optical technology, called the Camera Obscura. We have previously published an entry on the History of VR, where we discussed the invention of the Stereoscope as the first technology underpinning the VR of today. With a broader definition of VR, we could say that the Camera Obscura is an even earlier VR technology than the Stereoscope; in the mid 1600s,  by using the Camera Obscura, one could live stream a photographic segment of reality at much higher refresh rates than what we can do with information technology today.

 

Four people using a Camera Obscura, all the while remaining unseen behind closed doors.

The drawing above illustrates the workings of the Camera Obscura: In a dark room, the light from the world outside is directed by a mirror through a lens, which focuses the light on to a leveled surface. Often made of white stone, the surface functions as a canvas for the photographic reflection. As this is light straight from its source, the responsiveness is immediate and as the lens is continuously open, the pictures are moving. It is a very interesting experience to stay in the Camera Obscura of Edinburgh, and wholly undetected watch and perceive the actions of the masses of people walking the streets outside.

We should note, however, that even the mirrors and lenses are not necessary to create this effect. The camera obscura is in essence an extremely simple concept, and the simplest version of it is called a pinhole camera, which is as simple as a dark room with a hole for which the light to enter through. The light that enters through it represents what reflects it —which of course is the environment outside. As such, all light contains information, and pinhole cameras utilize this by letting the light enter through a small hole in a wall into a dark room, so the visual information can stand alone and be perceived relative to the dark background. In more complicated camera obscuras, lenses are used to strengthen and focus the light, and mirrors to redirect it.

Illustration of a Pinhole Camera, displaying an image upside down on a wall in a dark room.

As some may know, when light hits our eyes, the retina actually perceives the world upside down. Our brains, however, flips this back again — resulting in the world as you see it today. Traditional pinhole cameras or simple camera obscuras also suffer this effect, and so often the image is seen as upside down, as in the illustration above. In the Edinburgh Camera Obscura, they use lenses to maintain the normal orientation. Effectively, the image is inverted twice — once by the aperture, and further back using the lenses. For those who want to try to achieve this at home, we recommend this experiment, which highlights the workings of the lenses.

The Camera Obscura used for the art of drawing.

Another interesting use of the camera obscuras, and a source of their popularity, was for the art of drawing. By projecting directly to the canvas one is drawing on, the lines of the environment can be outlined more easily. What is becoming increasingly clear here, is the role of the camera obscura in the creation of the modern photographic camera. The technology is quite simply the same, only instead of a continuous stream of light to a canvas — we have a limited, controlled exposure to a surface that adapts to the light. It is related to this exposure where photography features make sense, such as aperture (how much light we let in); ISO (the sensitivity of the image sensor), and shutter speed (the amount of time that light should be let in). We are still playing with light and lenses.

The World of Illusions

If you visit Edinburgh to look at their old Camera Obscura from the 1850s, you will find in the same castle what they call «The World of Illusions»; five floors containing over 150 different optical illusions. Caleidoscope rooms; 3D stereoscopic mirrors; mazes of mirrors and much more. We will discuss and explain a few of these in more detail, the first being “The Ames Room”.

The Ames Room

The Ames Room, showing three men of similar size.
An overview of the Ames Room, dissolving the illusion. The illusion illustrates our lacking capability to perceive actual depth (3D).

For the illsion of the Ames Room to work, you have to see it from a certain perspective, which in the above illustration is referred to as the viewing peephole. The Ames Room in Edinburgh, unlike in our illustration, also use floor tiles as in a chessboard to further improve the illusion, which from the viewing hole appears to be of similar size. The illusion is a funny one, and an obvious photo-opportunity.

The Vortex Tunnel

Phtograph from the World of Illusion in Edinburgh.

Another illusion, which is more bodily, is their Vortex Tunnel. You are in a room, where a bridge connects the two ends. The task is to walk over the bridge (a fully stable, stationary bridge). Now, this shouldn’t necessarily be a problem, if it weren’t for the fact that the cylindric vortex walls are spinning around you. It doesn’t matter how hard you try, you simply can not walk a straight line: it is as if the gravity draws you toward the rails of the bridge. If you close your eyes, however, everything is fine.


Do you know of any other fun illusions or old optical technologies? Please comment below!

SaveSave

SaveSave

SaveSave

The Web of VR Will Change Everything

The World Wide Web (WWW) need no further introduction. The greatest innovation of the Information Age is now essential to the world like no other technology. Before the WWW – computers, programs and information were not linked. The computers were lonely, and users could not browse the millions of interconnected computers the way we do today.

The Web has been changing ever since its dawn in the 90s, and has seen its distinct phases. What we call “Web 1.0”, for instance, was a static web. Websites could be visited and navigated, but they were static in the sense of not affording any user interaction. Web 2.0 opened up for more dynamic web applications that could be altered by user input. These did not just allow download, they could also be uploaded to – a feature that is now an essential underpinning of social media and web-based applications. Companies like Facebook, Snapchat, Instagram, etc., do not provide content to be downloaded, but rather a computer service to be used where the users provide the content.

Illustration by Lancelot Speed from Andrew Lang’s “The Red Fairy Book”.

It should be noted that “Web 1.0” and “Web 2.0” are just terms: there are constantly being added changes to the Web. The terms does, however, signify when these changes are inducers of a paradigm shift in the use of the Web. What is to be classified as Web 3.0 is therefore also discussed. Although not necessarily a feature of the Web, the Internet of Things (IoT) is a candidate for what has become a paradigm shift within web technologies, as more and more devices and artifacts are connected, allowing for ubiquitous computing. Others discuss the personalised Web we see today, as it is influenced by social media, while some are joining the AI hype, and claim that the Web has now become smart. The latter is far from a paradigm shift as of yet.

The Virtual Web
In this entry we will discuss WebVR, or Virtual Reality through, and on, the Web. The title of this entry, which claim that “The Web of VR Will Change Everything”, may indicate a stand towards the debate that I have introduced, of what will be the “Web 3.0” –  but that is not the point of this entry. I do not wish to make a claim of WebVR as a paradigm shift of the way the Web operates, but it most significantly will be a paradigm shift in how we experience the web technology as it is.

The concept and role of the Web, nevertheless, is the same: we have a dynamic web which features download and upload of web documents. In the case of VR the difference is that what we download and upload, are perceived as realities for us: the web is the mediator of realities, and this new way of using the web changes VR more than VR necessarily changes the Web. What characterizes the Web, is its simplicity, its openness and the innate element of surprise. Anything can be found, and the exploration as such is an important part of it. These features are the same that will be valuable in VR as well: to discover open virtual worlds, created by anyone.

Mozilla’s A-Frame is now ready for Link Traversal through hyperportals

Creating VR for the Web
WebVR, a framework for browsers to support Three-Dimensional Stereoscopic Virtual Environments, is already developed and supported by many browsers. As HTML, CSS and JavaScript already have the powers to create and render graphics through frameworks such as WebGL, the web languages have increasing power to support such scenarios. Lately, frameworks combining these different frameworks to make implementation of WebVR even easier. Mozilla’s A-Frame, lets the user set up a Virtual Environment only with less than 20 lines of code (see their Hello WebVR example), using ‘normal’ HTML tags, which they call primitives, to create 3D objects in 3D space. A-Frame utilizes Three.JS to do this, and Three.JS uses the WebGL.

It is now easy to create Virtual Environments on the Web, even arguably easier than creating them through Unity. The great benefit of this is that they can be connected to each other, by a standard hypertext reference, instead of uploading to Steam or Oculus Store, etc. A-Frame introduced hypertext support, which they call “Link Traversal”, in July of 2017, but the browsers are only just catching up. As of now, it is only supported by Firefox and Supermedium on PC, however, as of February 2018 Oculus Browser has supported it on GEAR VR, and most likely also on Oculus Go.

A-Frame’s Diego Marcos called this a great achievement, as A-Frame finally achieved their ‘Web badge’. For this they deserve congratulations, A-Frame has now completed an essential step towards the Web of Realities. In their introductory blogpost, they introduce a “hyperportal” example, which provides you with a preview of the VR world you are about to enter, and which redirects you to the page when you virtually walk through it. This is a piece of very fun code to play around with. A neat feature is that the portal itself is “transparent”, and so provides a preview of the virtual environment to which you are travelling.

The future of WebVR
As with anything within VR, we are still a few years behind its potential. WebVR has had a solid boost the latest few years, but before a Head-Mounted Display is commonplace, we probably wont find a VR search engine or enough websites for exploration to be truly amazing. This is not bad news, however, it means that this is just the right time for creative ideas. We see the inevitable emergence of the VR Web, and can help shape it. For instance, at Matrise, we have previously discussed Virtual Reality Memory Palaces. This would be great to incorporate for sharing on the Web, so each memory palace could be interconnected, creating vast banks of knowledge for memorization.

Do you have any good ideas for any WebVR apps?
Feel free to comment below.

The ancient Greeks created “Memory Palaces” to recall important information.

Virtual Reality Memory Palaces

The ancient mnemonic “method of Loci” has been in use for thousands of years. From memory competitors to students, the method helps to memorise items by combining visual and spatial cues. Many have perhaps heard of the technique through popular TV shows such as Sherlock or Hannibal, where their “Mind Palaces” or “Memory Palaces” are portrayed as a genius trait.

Illustration based on “Mind Attic” by wonderful artist Tisserande. Original version without Head-Mounted Display can be found here.

The real method, however, is perfectly simple. You start by finding a location to act as your Memory Palace. It should be a place you know well, for instance your apartment or your university. You further isolate several rooms, and sub-parts of those rooms, that amount to the number of things you want to recall. Next, you may want to have a certain route which you mentally walk through your Memory Palace. This is where the method comes in: you visualise what you want to recall at the given places in your Mind Palace. Need to go shopping? Imagine coffee beans poured out on the floor, orange juice cartons in the sofa, and bananas hanging on the TV. If you need to recall something more abstract, you can get creative and use the visualisation as an association to what you want to recall instead.

This method is believed to be powerful because of the combination of information with visual cues (the visualisation of the memory items), and with spatial cues (their given place in the environment). Curiously, however, this exploitation of the visual and spatial cues in the brain, does not involve our vision, or true perception of spatial three-dimensional spaces at all — only through our “inner eye” and our already-established memory of spatial environments.

“Get out — I need to go to my Mind Palace”. Benedict Cumberbatch in BBC’s Sherlock  — Quite busy in his Memory Palace. Illustration of by edthatch.

But what if we could perform this method with raw, fresh vision that gives us both the visual and spatial cues? Such a task is perfect for the medium of Immersive Virtual Reality. In VR, we are provided with highly optimisible a visuo-spatial environment made of software, which gives us the opportunity  to tailor the Memory Palaces to our own needs. By exploiting the visuo-spatial elements of the method of Loci in a much more explicit sense,  the method may perhaps work even better, or be more fun and easy to use.

Addressing this, I created an a VR memory palace app as part of my master’s thesis in Information Science. The application is called “Mnemosyne”,  after the Greek goddess of memory, of which country the Method of Loci originated. Developed with A-Frame, the application presents you with a text field where you can enter the things you want to recall. You then put on your VR headset, and are presented with an apartment with five rooms where you can place these memory items you entered. The algorithm returns images based on your text preferences from a search engine, and places these in the virtual environment. When this is done, you just walk around and watch to memorise the items.

Mnemosyne is still just a prototype, and we can imagine it will have room for many improvements in the future – after more research has been done. Currently, we at the Centre for the Science of Learning & Technology (SLATE) have performed a pilot study with 18 participants. The short paper discussing the findings was published in June 2018 in the Springer’s series «Lecture Notes on Computer Science», in the «Augmented Reality, Virtual Reality and Computer Graphics» edition. The conference paper was presented in the beautiful village of Otranto at the Salento AVR conference in July.

Our first paper on VR Memory Palaces was posted in the LNCS on Augmented Reality, Virtual Reality and Computer Graphics.

Our paper, ‘Mnemosyne: Adapting the Method of Loci to Immersive Virtual Reality’, can be accessed at Springer. Stay put at Matrise for further updates on our work regarding Virtual Reality Memory Palaces.

Literature list