Leap Motion Controller Pursues Virtual Self-Awareness

This is exciting to me because as far back as last year I remember thinking that Oculus VR should integrate Microsoft Kinect technology directly into their headset so the system would have a native form of 3D input.

The Leap Motion controller uses technology similar to the Microsoft Kinect, but is much smaller and claims higher resolution.

Originally the promise of the Leap Motion device was to deliver a Minority Report style gesture control interface with a small and affordable USB peripheral.

Today Leap Motion published this video demo and announced to their developer community that they are releasing the demo hardware & software and also working on future VR tech they call Dragonfly. From the announcement:

… codenamed “Dragonfly.” It possesses greater-than-HD image resolution, color and infrared imagery, and a significantly larger field of view. With next-generation “mega-sensors” like this, a Leap Motion device can literally become your eyes into the digital and physical realms — allowing you to seamlessly mix and mash, fade and blend, between virtual environments and the sharpness of the real world.

While this quote from the announcement seems rousing I think it greatly understates the role personal 3D input tracking like this plays in crossing the all important threshold of prescience in virtual reality.

Why 3D Input is So Important to VR

The literal definition of prescience in the dictionary is: “the fact of knowing something before it takes place.”

In VR developer circles prescience is currently THE hottest buzzword. It drives the demanding performance specs of VR input / output devices and the dogged determination of developers to deliver on those specs without compromise.

In the context of VR prescience occurs when the hardware and software deliver such a convincing experience that the human brain “believes” on a deeply biological level that the virtual experience is “real.”

It is the holy grail for developers because when prescience is achieved, physiological responses are observed in a person interacting with a virtual world that are normally only present in the real world.

For example, once a VR system achieves a certain level of prescience one can be honestly startled by virtual crocodile suddenly emerging from the water to attack and gets genuinely nervous and fearful standing on a virtual ledge high atop a virtual skyscraper.

In these scenarios the heart rate and breathing quickens, hands sweat, and blood vessels dilate or constrict depending on the stimulus even if that stimulus isn’t real.

Prescience is like the suspension of disbelief experienced when watching a movie except it’s completely involuntary.

Sounds really cool, right? How does a VR system create prescience though? It is achieved by successfully duplicating a number of subtle cues to the brain.

For example, when one turns their head or moves it forward or backward or side-to-side, they expect the images received by their eyes to change in 3 dimensions precisely synchronized with those movements.

Similarly, when one looks down and lifts up their hands, they expect to see their arms, hands and fingers precisely oriented in front of them and their body and feet beyond that!

Prescience as it relates to VR is really about expectations. People expect the virtual world to behave the same as the real world before they turn their head or look at their virtual hand.

This obvious little fact begins to explain why native 3D input is so important for virtual reality to truly achieve prescience and gain traction in the world.

But, there is another even deeper psychological reason that drives the importance of personal 3D input to VR even more. That reason is self awareness.

The Rouge Test

A scientific experiment known as the “rouge test” tells us that somewhere between the ages of 15-24 months old children become self aware.

Take a look at this video that demonstrates and talks about the rouge test:

The video states the rouge test is, “a mirror self-recognition test.” Starting at about 18 months old, “[the child] notices the red dot and the link between the image in the mirror.” Scientists believe this realization in children marks the beginning of self awareness that develops more over time.

The video states it best and goes on to say:

… Over the next few years children go on to fulfill their human birthright: constructing an increasingly sophisticated sense of self. They will have a rich and complex life of the mind and a sense of being the central characters in their own dramas…

I admit the concept is not quite the same when applied to VR, but I want to argue here that the level of personal 3D input tracking Leap Motion is promising with their announcement is fundamentally important to virtual reality because of this self awareness that begins developing at a very early age in people.

And, input is at least 50% of the whole user experience with output being the other 50%.

Without good personal 3D input tracking in virtual reality the connection between one’s self and their virtual image is severely limited and such a user is just a bodiless head floating in space always somewhat removed and disconnected from the virtual world.

With excellent 3D input in VR users not only interact with a virtual world in a physically natural way, but can also begin to have profoundly emotional responses and connections when they look down to see themselves with a virtual wound or wearing a suit of armor, uniform, or even nothing at all.

Virtual reality is more engaging and psychologically fulfilling when one realizes that everyone in that virtual world is also self aware and everyone sees the same thing.

Personal 3D input tracking is the key to our virtual selves truly becoming digital extension of our real selves.

Prediction

Oculus VR (now owned by Facebook) and any other entry into the VR hardware space will either integrate personal 3D input tracking similar to a Leap Motion / Kinect sensor directly into their virtual reality systems at some point OR fail.

They will do this either by purchasing a company and all their intellectual property or licensing the technology.

I make this prediction for all the aforementioned reasons and from an engineering standpoint because this type of tech performs better when tightly coupled with the rest of the system. Remember, performance is key to achieving prescience so a peripheral isn’t going to cut it.

Oculus VR has already hired former Leap Motion employees so I don’t think it’s a crazy prediction to make.

Virtual reality has been one of the biggest tech zeitgeists for the past 3 decades and it will continue to elude success unless it solves both the input and output problems simultaneously.

By the way, if you found this article useful or interesting to you in any way please follow my twitter, google+ or RSS feeds to be notified every time I post something new.