Hands on Virtual Reality

With the advance in the technology of VR headsets and graphic engines, the experience of a virtual world is getting more vivid and realistic every day.
Virtual Reality however, goes far beyond the simple viewing experience. Interaction with the virtual world is the key to make the technology useful, and to do that we need some way to provide inputs.
One of the key input tool we have are hands. We use them every day to do a multitude of tasks, from moving objects to gestures, and they are one of the most characterizing features of humans.

VR developers realized this very soon, and created the controller. The controller represents the medium by which the hand input is translated into VR. The hand acts on the controller and the controller acts on the virtual environment. The idea itself is very good, and in fact it is the predominant technology now, but there are two important issues. First, the controller has a very limited set of inputs that it can receive from the hand, usually one trackpad and around 7-8 buttons. This means that all the possible behaviors of one hand must be constrained to a few dozen button combinations, effectively limiting their expressing power. Moreover, the user will have to learn again most of the gesture he has used every day since its birth. To point at something, for instance, the user would need to press a button that shoots a laser in VR, whilst usually it would just raise his index in the direction of interest.

These two issues have forced an interest towards hand motion recognition and identification. Between the many companies that have started investing in this research, FINEXUS is one of the most successful ones. They are currently studying a revolutionary system that can track multiple fingers in real-time, effectively changing the way we can interact with the virtual world.

There are multiple approaches to finger tracking. The most common is based on computer vision. A camera placed in front of the user registers the user movement and using advanced CV algorithms can recognize the fingers and compute their positions. There already are good implementations of this techniques, but the system suffers one great limitation. The camera needs to see the finger to work. This means that the user cannot move around too much and is seriously limited in terms of rotation. Another solution that has tried to solve this problem used IMUs. Inertial Measurement Units measure the movement of an object by means of accelerometer and gyroscopes. They do not require any line of sight, but they need very accurate calibration to work properly. Moreover, gyroscopes and accelerometers suffer from drift that increases the imprecision of the measurement as time goes by.

Given the limitations of these two methods, FINEXUS has decided to follow a third way. FINEXUS tracking system works by mean of magnetic fields. Four magnetometers are placed at fixed and known location around the space, while each finger is equipped with an electromagnet that continuously emits a magnetic field.
The system works somehow like an inverted beacon system.
Each magnetometer senses the magnetic field emitted by a source and extracts is circular distance. The distances of the four magnets are then interpolated to obtain a unique point representing the position of the finger. That is the opposite of the GPS, where the fixed beacon emits a signal and the moving items perform the computations, but the underlining idea is the same.
To solve the issue of tracking multiple fingers, each finger works at a different frequency. The magnetometers are then capable of splitting the magnetic fields based on their frequency of origin and consequently distinguish between each finger.

The advantages of this system are clear. Because the magnetic field does not require line-of-sight the user can rotate and move around without any limitation. However, with respect to the computer vision technique, FINEXUS requires some hardware to be placed on the fingers which is quite bulky and can be very annoying at times.

Overall then FINEXUS proposes a novel approach that can be very interesting for improving the VR experience. There are indeed some issues, but we can expect that further investments and research will reduce the size and invasiveness of the finger-mounted hardware.


Nov. 24th, 2017


This article was inspired by the research of Debojit Kaushik which can be found here.

Note: This article was written as an homework for course CS 491 – Virtual and Augmented Reality, held by professor Johnson at UIC.