Busting Through the Physical Barrier in Digital Design

An emerging 3D design medium will close the gap between what we imagine and what we can create

Interview with Jinha Lee

The interface through which most people have dabbled with technology for decades — a computer screen and keyboard and newer variations with mobile devices — is long overdue for some improvements. Jinha Lee, an interface designer and engineer working at the boundary of the physical and the digital environments, has developed some breathtaking alternatives. Lee explores ways to employ our innate kinesthetic and sensory skills to interact with the world of data by seamlessly weaving digital information into physical space and material. His ideas include a pen that penetrates into a screen to draw 3D models and SpaceTop, a computer desktop prototype that lets you reach through the screen to manipulate digital objects.

Why is this work so important? Why is our current interface with computers so limiting?

The power and the potential benefits of our digital technologies are undeniably huge. We are reminded of this every day in almost all aspects of our lives. But we make use of only a portion of the tremendous power of this data.

Here’s the problem: The communication link between humans and the digital tools that surround us is often very small and narrow. It’s typically just the tip of your finger. Touch a screen to enter data. On the other hand, we have all developed great skills to sense and interact with the physical space that surrounds us. Unfortunately, we cannot use most of these innately human skills to work within the digital world. The two-dimensional, flat touch screen, less than a millimeter thick, now divides the two worlds, both physically and metaphorically.

Lee's See-Through 3D Desktop allows users to reach into the projected 3D output space with their hands to directly manipulate windows.

Photo: Jinha Lee

What got you interested in exploring this sort of technology?

I have been inspired in my work by my mother, who teaches origami professionally. It has always been amazing to observe how she transforms her complex ideas into the physical shape. In her work there is a beautiful intersection between abstract information and physical expression. This often makes me think of what it would be like to physically interact with (digital) information.

What becomes possible when these two worlds — physical and digital  begin to merge?

It’s time to think about interfaces that blur the boundaries between the digital and physical space, that engage digital information in physical and spatial forms, drawing on our innate senses to interact with digital information using the full dexterity of our hands, creating a more intuitive, effective and enjoyable experience. One of my research projects, SpaceTop, which I started at Microsoft Applied Sciences Group and MIT Media Lab, looked at this approach. We turned a little space above the keyboard into a 3D digital workspace wherein users can reach in with their hands to manipulate digital objects. Instead of staying away from traditional user interfaces like touch-pads, SpaceTop combines design solutions with this 3D direct interaction, taking advantage of multiple interaction modalities. It is a combination of a transparent display, depth cameras and a new design of desktop hardware that makes it possible for users to manipulate 3D pixels in space with their bare hands. I believe the applications of this will be huge.

How soon will these concepts evolve into real products?

Researchers have been exploring this field for decades, and it looks like we are finally seeing real-world use cases and thinking about products that integrate these designs. But I think we are still about 10 years away from when they will become commonplace. Delays in graphical processing capabilities, as well as a lack of affordable display technologies and better precision sensing tools remain obstacles. But as these devices come out of the laboratory, I believe there will be a high popular demand for these types of products.

Who will benefit the most from these innovations?

The ultimate aim in creating all these sorts of new interfaces is that we lower the barrier for people to access the power of computation. This means that a wider range of people can benefit from it in a bigger context — if we can use a big part of our innate senses and skills to interact with the digital information. Ultimately, such technologies will be simply invisible and become a part of our ordinary lives. The late Mark Weiser, a chief scientist at Xerox PARC, once said: “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”

Jinha Lee, Head of Samsung Electronics' Interactive Visualization Lab, was selected as a TED Fellow in 2013. In addition to inventing SpaceTop, he co-founded Eone, a tactile watch brand that can be used by the visually impaired.


Interview with James Daly, a veteran journalist and media entrepreneur. Most recently he was an executive producer at Byliner, and the founding editor of TED Books.