Re-thinking the Surface

Although all the rave right now is around multi-touch, electronics are getting more portable, which means decreasing surface size. But surfaces can offer a lot more than a glossy glassy screen on an iPhone. Several researchers have begun exploring the use of widely available surfaces, such as walls or tables, or even your arms and fingers, to replace static, often awkward, input surfaces.

Skinput

By measuring body acoustics (ultralow frequency sounds resulting from touching different points on the skin), researchers Chris Harrison, Dan Morris and Desney Tan have developed Skinput: Appropriating the Body as an Input Surface. Basically, a complicated looking armband with lots of wires detects these vibrations, then a pico-projector projects an interface of buttons onto your arms. You tap the buttons directly on your skin to initiate an action, thus interacting with the interface. Or you can map certain parts of your hand and fingers to gameplay actions in a Tetris game, allowing you to play Tetris in the palm of your hand.

To imagine that this could become a commercially available product in a few years is not so far-fetched. Go to http://www.chrisharrison.net/projects/skinput/ for more detailed information on this project. (Images and video from Chris Harrison’s site)

Muscle-Computer Interfaces

Muscle-Computer Interfaces allow you to “interact with a computer using hand and finger based input without holding physical input devices.” By using EMG technology “to decode muscle signals from the surface of the skin” from electrodes and wires connected to the outer surface of your arm. This detects gestures from hands and fingers, which are then mapped to actions on a computer.

Thus, they have completely done away with the notion of a [graphical] interface, and with that the need for a separate input surface is also gone. The video demo shows a person playing Guitar Hero by “strumming” and “fretting” in the air. See their site for more information.

Sixth Sense

The pioneers at MIT Media Lab have re-defined portable computing with SixthSense: A Wearable, Gestural Interface to Augment Our World. By hanging a small box around your neck (like a long necklace), any surface becomes a multi-touch gestural surface. In short, this box contains a projector, a mirror, and a camera, and is connected to a computer in your pocket, along with some caps that look like thimbles for your fingers. With this combination, gestures from finger movements (like pinching, rotating, grabbing) and hand movements (swiping, pushing, pulling) are mapped to actions on the interface, at best in an intuitive manner. So for example, grabbing the corners of an image with the index and thumb of both hands and pushing both hands together will shrink the image.

So far, it sounds like an iSomething with multi-touch. However, with Sixth Sense these gestures are in the air, and the surface can be any wall, table, torso, hand, etc. Or no surface. If you position both hands to form a square frame, it captures a photograph. It also interacts with objects, scanning the cover of a book to give you more information, or dynamically updates the weather map in a newspaper. You can watch movies or play on iTouch-like racing game on a piece of paper (and for those skeptics who thought paper would become obsolete…). Pinch a paragraph off of a textbook, then pinch it back onto your projected paper surface, then pinch it back to your desktop!

Check out this amazing project here: http://www.media.mit.edu/research/highlights/sixthsense-wearable-gestural-interface-augment-our-world and here: http://www.pranavmistry.com/projects/sixthsense/.

The Challenge Awaits

It is exciting to think of all the challenges we face that come along with these ideas; not just the technological challenges of hardware/software, but the design challenges as well. When anything can be a surface, it becomes quite difficult to predict how a user will interact with the given interface. The usual paradigm of menu-button-navigation-input-etc might need to be completely reinvented for this new medium. Equally exciting will be the applications of this technology – the inventors have only conjured up a minimally small number of possible uses. I can’t wait to see what comes next.

“Imagination is the only limit of what you can think with this kind of technology merging with real life” says Pranav Mistry in his TED presentation. As evidenced by these three awe-inspiring ideas, and the myriads more in development, this is not fiction anymore.

Tags: input surface, muscle-computer interfaces, sixth sense, skinput, wacky wednesday

Leave a Reply