Detecting and Leveraging Finger Orientation for Interaction with Direct-Touch Surfaces

Related comments:
On Zach's blog


Summary:
As direct-touch interaction becomes more prominent in personal computing, there will be an increasing need to break from traditional methods of contact detection so as to make the most of this new technology.  Currently most touch-sensitive surfaces process contact data based on x-y coordinate information, the same technique that has been utilized for mice and stylus input for decades.  This technique determines cursor position by averaging center coordinates of a human finger's contact region, which can lead to ambiguous input and a less natural interaction.  A team of researchers from Kochi University Japan, Microsoft Research Cambridge, and Univ. of Manitoba Canada (Feng Wang, et al) have proposed a different technique -- calculate user's finger orientation based on the dynamics of the finger-landing process.  Orientation is important as it implies direction and angle from a point of reference.  The system works by analyzing the finger contact patch on the surface when the user makes an oblique touch (at an angle where most of the fingertip contacts the surface).  The contact patch is fitted into an elliptical shape by the computer and put through an algorithm to determine its orientation:

The system was tested on a group of eight volunteers, who were asked to perform four different tasks six times, resulting in 384 total trials.  Of those trials, only 13 errors in the disambiguation algorithm were generated, a 96.7% success rate.  The four tasks showcased user control and algorithm response to different finger positions, all with favorable results.  The latter part of the paper is concerned with proposing new user interface designs that take advantage of orientation-sensitive touch technology, such as aim-and-grab selection, orientation dials, estimation of and correction for image occlusion by the hand, inference of relationships between detected fingers, and multi-finger mouse emulation.

Discussion:

I think this is some pretty astounding research that could shape the way we interact with multi-touch surfaces in the years to come.  Such surfaces are not much better than a regular display if they don't take advantage of the dynamic, natural input that human hands provide.  By treating fingertips like mouse cursors with static x-y coordinates, multi-touch interaction has been limited far below its potential.  By taking into account things like user position and finger direction, this research has opened up vast possibilities for natural-feeling direct-contact interfaces.  The most important contribution, in my opinion, is the use of this technology to create occlusion-sensitive menus that adapt to the user rather than vice versa.  

Mouse 2.0: Multi-touch Meets the Mouse

Related comments:
On Nate's blog


Summary:
In this paper from a team at Microsoft Research (Nicholas Viller, et al), five ways of incorporating multi-touch (MT) with traditional computer mice were implemented and tested, with some pretty interesting results.  Multi-touch technology has become more prevalent as an interface tool in recent years, but it is primarily used in mobile devices or larger form factors like kiosks and tabletops and hasn't made its way to the desktop environment -- which is still the dominant environment for most users.
The first MT mouse discussed in the paper utilized Frustrated Internal Total Reflection (FTIR) as its sensing technique.  FTIR works by edge-illuminating a sheet of acrylic with infrared (IR) light; when fingers are pressed on the surface they scatter IR light which is then detected by an IR camera.  The team at Microsoft Research created a form factor that was similar to traditional mice but still enabled FTIR to be used successfully.
The next attempt, the Orb Mouse, used IR technology similar to the FTIR mouse, except that the IR light is generated inside the device and reflected back into the camera by user's fingers that contact the hemispherical surface of the mouse.  However, the IR picture captured by the camera is heavily distorted and must go through several processing steps before it can be used for input.
The third design switched technologies, this time using a flexible matrix of capacitive-sensing electrodes.  The Cap Mouse (short for capacitive) works by detecting disturbances in the sensors' mutual capacitance caused by finger contact and mapping the position of the disturbance as input.  This design is relatively accurate while consuming low amounts of power.
In a break from the previous three designs, the Side Mouse actually dispenses with touch-sensitive mouse surfaces altogether and functions by reading breaks in a projected IR beam caused by fingers placed in front of the mouse, upon which the user's palm rests.  This design gives the user the freedom touch the table surface and perform a wide range of gestures.
The final design presented in the paper was the Arty Mouse (short for articulated).  Having perhaps the most unconventional design of the five mice, the Arty Mouse features two prong-like extensions for the user's thumb and index finger, each of which contains a separate optical mouse sensor to track its movement across the surface.  Each extension and the base part also have conductive metal rings around them to sense when any piece is touching the other.

From L-R: FTIR mouse, Cap mouse, Arty mouse, Orb mouse, Side mouse

The latter part of the paper was concerned with the integration of multi-touch input with the software of a desktop computer, primarily through augmenting the traditional cursor with a "cloud" of dots representing multi-touch contacts.  Simple adaptations of existing software produced successful results and hinted at the possibility of 3D interaction in the near future.  Each mouse was also tested with a group of 6 users in a pilot study.  Of the five designs, Arty proved the most comfortable and popular, while the Orb seemed to allow users to make more natural gestures to control the computer.  The Cap was the easiest for users to learn, while the Side and FTIR presented some problems with ergonomics.  The researchers found that ergonomics are crucial when designing a mouse form factor, as comfort affects a user's experience greatly.  Compromises between ergonomics and multi-touch capability must be struck to find the most balanced mouse.


Discussion:
I chuckled a little when I started reading this paper -- having been published in early October, the paper omits the obvious achievements made by Apple in this front with the release of their Magic Mouse in November of last year; when compared to the Magic Mouse, most of these designs look either ridiculous or horribly uncomfortable.  Some of them, like the Arty Mouse and the Orb Mouse, present unique changes to the traditional mouse design that could foreseeably improve the user experience.  Others, however, are either too similar (Cap Mouse) or too uncomfortable (Side Mouse) to ever have any kind of impact on the multi-touch interface market.  The trouble with MT mice is that they must be different enough to actually be able to utilize MT powerfully, but similar enough for users to adjust easily.  Apple did this very well with the Magic Mouse, which despite minor ergonomic issues is an effective combination of MT gestures and traditional mousing.

Sorry, Microsoft.  Apple gets the win on this one.

EverybodyLovesSketch: 3D Sketching for a Broader Audience

Related comments:
On Justin's CHI blog


Summary:
EverybodyLovesSketch is a gesture-based program that allows users with little or no artistic training to create 3D curve sketches comparable to those one might see in professional drafting work.  The program applies some of the fundamentals of perspective drawing (planes, vanishing points, grids) in an intuitive way that provides a variety of predictable sketch surfaces for users.  The interface enables the user to define a sketch plane quickly and easily by making a "tick" gesture; one a sketch plane is established, the user draws a 2D shape on the plane as they would on paper.  The user can then select, rotate, and extrude these planes with a variety of gesture-based actions.

The EverybodyLovesSketch program was evaluated by a group of 49 high school students over a period of 11 days, where their performance using the tool was charted; most of the students created meaningful 3D models within the first two days of use and reported that their 3D spatial understanding increased over the entire testing period.  Students found the program to be (for the most part) easy to use and created some very impressive-looking work in just a few days.  It is clear from these results that EverybodyLovesSketch has enormous potential as a teaching tool for art and design.

Discussion:

Thinking in 3D space is probably one of the most difficult aspects of art and industrial design.  Getting over the idea that in order to draw something that looks like a square in 3D space, you may have to draw something that is not a square can be challenging.  EverybodyLovesSketch makes the concepts of perspective drawing much more intuitive and provides a set of tools that seem to enable effective creation and manipulation of 3D sketches.  The only flaw I found with the tool was its lack of support for solid-model definition; it would be nice to be able to do something with a solid surface (like putting "skin" over a 3D sketch or defining color and texture) other than use it as a guide for drawing (page 6).

Abracadabra: Wireless, High-Precision, and Unpowered Finger Input for Very Small Mobile Devices

Related comments:
Couldn't locate anyone else's post on this article, so I commented on Jill's instead.


Summary:  
Abracadabra was developed as a solution for interacting with very small mobile devices.  Utilizing a multi-axis magnetometer as the primary sensor, Abracadabra takes advantage of the relatively larger space around a small mobile display by using a magnet as the input device.  The field produced by the magnet (attached to a finger with velcro or other means) is large enough that the effective input area is extended to several times the size of a mobile device's screen -- this translates to fine motion control on a much smaller screen without requiring tiny, precise movements.  More complex interfaces are possible this way, and may also allow users with diminished fine motor control to use such devices.  Since the input device is simply a magnet, it requires no additional power or recharging.

The device reads input in two primary ways: measuring the field strength of the magnet relative to one of the sensors' axes (for cursor-like motion i.e. an eastward-weakening magnetic field indicates a motion to the right) and a binary-type "click" operation which is triggered when the magnet is dipped below the plane of the sensor, reversing the magnet's polarity.  This instantaneous change in polarity registers as a "click."  Early tests of other input gestures and styles such as circular "scrolling," "swipe," and rotation have produced favorable results.

Several tests were conducted to study the effectiveness of the Abracadabra system, with very good results.  A group of 15 participants each navigated a pointer (using the magnet input device) over pie pieces of decreasing angular width, and were found to navigate accurately 92% of the time for angular widths of 16 degrees.  Additionally, the users' targeting time decreased dramatically when using the Abracadabra system and error was reduced by half.

Discussion:

Being the (I wouldn't say proud is really accurate) owner of a Windows Mobile device with a touchscreen, I know for a fact that relying on my fat fingers to correctly type keys or select the right fields is a bit of a gamble.  An accurate way to interact with a mobile screen that didn't require me to possess dexterity equal to one of the Shoe Elves is a welcome sight.  The magnetic sensing technology for interface design also reminds me of the part in Minority Report where Tom Cruise dons gloves (similar to the magnet-tipped "ring" in the Abracadabra system) that allow him to manipulate the displays in front of him.  Abracadabra could be scaled up to achieve something like this in the near future.

There are only two problems I immediately perceive: that a magnet-ring is just as easy to lose as a stylus, and that the likelihood of mobile displays being reduced further in size (to, say, a watch-face) is low.  Abracadabra therefore wouldn't be applied fully to its intended purpose.

hello world!

Two blogs at the same time?  I doubt I can pull it off.  I suppose writing about academic papers pertaining to computer science is gonna have to take priority over telling you what kind of music I like.  Come to think of it, that's probably a good thing.  You're welcome.