Visually appealing and intelligent projection with natural interaction

Projectors satisfy our natural urge to interact with the virtual world with large, human-sized surfaces. We develop novel techniques to create realistic and appealing projection, using a single projector, on ad hoc but ubiquitous Lambertian dual-planar surfaces, and demonstrate support for a range of meaningful interactions with natural interaction. Deploying projectors in such environments creates several image artifacts resulting in degradation of observed imagery by end-users. Several artifacts due to global illumination, defocus blur, geometry distortion, and ambient light can occur in such environments. To correct these artifacts the projector is coupled with a camera that senses the environment thereby forming a closed loop system. In these environments interreflection of light results in global illumination effects. Our method to compensate for these effects is based on the systematic adaptation and interpretation of the classical radiosity equation in the image domain. Our method does not assume prior knowledge of 3D scene geometry. Our algorithm achieves compensation in real time. The output of our method has better contrast and is thus more appealing to the viewer. Projectors can create bright and crisp images on a single planar surface. The large aperture lenses used in projectors to create displays restricts their depth of field, thereby resulting in defocus blur artifacts when projectors are used in ad hoc environments. We advance the state of the art by demonstrating defocus correction in a non-parametric setting. Our method differs from prior methods in that (a) the luminance and chrominance channels are holistically considered, and (b) a sparse sampling of the surface is used to discover the spatially varying defocus kernel. Certain area of the large display surface could be non-projectable due to the presence of undesirable characteristics (like running wires, saturated color patches, etc.). This causes the projected content to distort and/or occlude making it unreadable. This leads us to the problem of projection in limited area on a single planar surface that we call as the re-targeting problem. We explore some of the research challenges involved in solving this problem and develop a content based re-targeting (CBR) solution for re-targeting content of presentation slides. CBR along with natural user interaction has been used to develop an intelligent application called SmartPro. It allows dynamic control of projection area and intensity. This allows us to support a range of interactions like dynamic annotation of the projected content on the display surface, holistic movement of the projected content to unoccluded parts, etc. Traditionally one or more external sensors observe the projection environment and user interaction is extracted and interpreted from the observed data. This requires appropriate instrumentation and configuration of the environment. Further a lot of data (in addition to the interactions) are captured. We propose a different approach wherein the sensor is associated with the interaction, called sensor on activity. The task of interaction extraction is now simplified. This paradigm has been effectively demonstrated through a virtual shooting range application where the sensor (camera) is mounted on the weapon and is directly associated with the shooting activity. Tracking and inferring position and orientation of weapon, as in traditional setups, to determine if a fire has been successful is not required any more. This system is able to support firing at video frame rates. Thesis submitted in partial fulfillment of the requirements of the degree of Doctor of Philosophy of the Indian Institute of Technology Bombay, India and Monash University, Australia.