Apple patents glasses-free holographic 3D display

Mid May the U.S. Patent and Trade Office reveals a new patent application from Apple, original submitted Jan. 14th of 2010. Under the title Three-Dimensional Display System it describes an "angularly responsive reflective surface".

A projector generates an image onto a special screen, which reflects the images into the right and left eye of the observer.

Furthermore the position of an observer is tracked by an advanced camera system / light sensor syncs the projector system to ensuring that the light beam for the left and right images would correctly reach the observer's respective left and right eyes:

"The positions of one or more observers are also tracked in real time so that the images that are being projected to the observers can be continually customized to each observer individually. [...] The real time positional tracking of the observer(s) also enables 3D images having a realistic vertical as well as horizontal parallax."

"In addition, each 3D image can be adjusted according to the observers' individually changing viewing positions, thereby enabling personally customized and individuated 3D images to be viewed in a dynamic and changeable environment. Further, the positional tracking and positionally responsive image adjustment enable synthetization of true holographic viewing experiences."

The patent also describes "unobtrusive 3D virtual desktop" allowing users to "manipulate objects within the desktop by reaching into the virtual display" and "grasping" and "pushing" the objects.

"The manipulation of the virtual objects occurs because the feedback mechanism recognizes observer movements, such as finger movements, at the locations of the virtual objects and reconfigures the display of the virtual objects in response."

The idea of accomplishing multi-touch with projected objects in a 3D space isn't all new. In February entry Hand tracking introduces Johnny Chung Lees similar work.

Project Natal

Johnny Chung Lee is currently working with Microsoft on Project Natal - a new technology of game controlling without a gamepad. It's a continuation and advancement of Lee's work on the Wii Remote hand tracking system.

It works quite in the same way as his Wii Remote project does. With one significant difference. The video game is just controlled by hands, feet, head and voice without any other kind of device, except a sensor, which is placed near the TV.

The sensor is able to recognize moves of several people in front of the TV. Besides a number of built-in microphones - to ensure communication by voice from any point in the room with the console - it has got two attached cameras. An ordinary Red-Blue-Green colour camera, and 3D camera, which allows to locate the user's position an recognize his moves.

Project Natal Xbox 360 announcement

The new device is expected to be available by autumn/winter 2010 under a name that is not know yet.

In June 2009 Shinoda Lab presents on SIGGRAPH Touchable Holography using Johnny Lee's Wii Remote hand tracking system. Next SIGGRAPH conferences are going to take place in Los Angeles, 25-29 July and Seoul, 15-18 December. Right now there are no announcements, that Shinoda Lab is going to present a further developed version of its Touchable Holography.

Nevertheless it would be very interesting to see, what it would look like.

Hand tracking

Giving a tactile sensation to holography, besides of course a projection of a floating three dimensional image, among others a tracking system is needed, which works as an interface between hologram and human. As it is shortly shown in the attached video in entry Touchable holography a Wii Remote can be used as a simple tracking system.

Johnny Chung Lee gives a little more detailed view on how the Wii Remote tracking system works. Even he applies it in another context than holography, that ought to be shown below, because the mechanics behind it is of the same kind.

Wii Remote tracking system

"While camera-based and marker-less hand tracking systems are demonstrated these days, we use Wiimote (Nintendo) which has an infrared (IR) camera for simplicity. A retroreflective marker is attached on the tip of user's middle finger. IR LEDs illuminate the marker and two Wiimotes sense the 3D position of the finger. Owing to this hand-tracking system, the user can handle the floating virtual image with their hands."

To give a more complete understanding of how touchable holography - as Shinoda Lab from University of Tokyo presents it at SIGGRAPH 2009 - works, another entry about airborne ultrasound tactile display is added.

The snippet attached below wants to illustrate what Johnny Chung Lee is talking above about Minority report.

Minority report (2002)

This movie forestalls the technology of hand tracking several years prior it finds application in combination with holography. So to say the technology catches the phantasm of user interface including hand tracking.