Detecting and Leveraging Finger Orientation for Interaction with Direct-Touch Surfaces

Related comments:
On Zach's blog


Summary:
As direct-touch interaction becomes more prominent in personal computing, there will be an increasing need to break from traditional methods of contact detection so as to make the most of this new technology.  Currently most touch-sensitive surfaces process contact data based on x-y coordinate information, the same technique that has been utilized for mice and stylus input for decades.  This technique determines cursor position by averaging center coordinates of a human finger's contact region, which can lead to ambiguous input and a less natural interaction.  A team of researchers from Kochi University Japan, Microsoft Research Cambridge, and Univ. of Manitoba Canada (Feng Wang, et al) have proposed a different technique -- calculate user's finger orientation based on the dynamics of the finger-landing process.  Orientation is important as it implies direction and angle from a point of reference.  The system works by analyzing the finger contact patch on the surface when the user makes an oblique touch (at an angle where most of the fingertip contacts the surface).  The contact patch is fitted into an elliptical shape by the computer and put through an algorithm to determine its orientation:

The system was tested on a group of eight volunteers, who were asked to perform four different tasks six times, resulting in 384 total trials.  Of those trials, only 13 errors in the disambiguation algorithm were generated, a 96.7% success rate.  The four tasks showcased user control and algorithm response to different finger positions, all with favorable results.  The latter part of the paper is concerned with proposing new user interface designs that take advantage of orientation-sensitive touch technology, such as aim-and-grab selection, orientation dials, estimation of and correction for image occlusion by the hand, inference of relationships between detected fingers, and multi-finger mouse emulation.

Discussion:

I think this is some pretty astounding research that could shape the way we interact with multi-touch surfaces in the years to come.  Such surfaces are not much better than a regular display if they don't take advantage of the dynamic, natural input that human hands provide.  By treating fingertips like mouse cursors with static x-y coordinates, multi-touch interaction has been limited far below its potential.  By taking into account things like user position and finger direction, this research has opened up vast possibilities for natural-feeling direct-contact interfaces.  The most important contribution, in my opinion, is the use of this technology to create occlusion-sensitive menus that adapt to the user rather than vice versa.