Dr. Shree K. Nayar of Columbia University recently gave the final lecture at the Faculty of Computer Science’s Distinguished Lecturer series. Nayar discussed his work on the computational camera, an innovation that combines the properties of a camera and a computer to improve the quality of photography.
Unlike the traditional camera, the computational camera captures various imaging dimensions, including temporal resolution, spectral resolution, spatial resolution, dynamic range, field of view, and depth. This enables every detail in a picture to be captured with high-resolution clarity. The computational camera captures light rays differently than a traditional camera, and is capable of assigning rays to different pixels by modifying colour and brightness before the ray reaches the detector. In a traditional camera, the ray passes directly through the detector to produce an image, without any modification. In a computational camera, the ray passes from the detector into the computational module, where it is stored and can be modified, allowing for the production of a myriad of unique images.
The computational camera and the traditional camera comprise different technologies including field of view. The traditional camera has a very narrow field of view that is unable to capture minute details. On the contrary, the computational camera has various mirror-lens combinations—an approach called catadioptrics—which allows it to obtain wide-angle images while maintaining a single viewpoint. This single viewpoint in the wide-angle camera produces an image that appears to have been taken by a rotating camera. The wide-angle camera, with a 220-degree vertical field of view and a 360-degree horizontal field of view, exhibits advantages over a traditional rotating camera. The scene does not have to be static in order to take the picture; one shot using the wide-angle camera is enough to clearly capture an entire dynamic scene. Video surveillance and video-conferencing are two applications of this wide-angle field of view.
Another improvement over the traditional camera is the computational camera’s dynamic range—the ratio between the maximum and minimum light intensities. Digital cameras are incapable of measuring a wide range of brightness values and cannot capture the nuances of colour in a photo. The computational camera resolves this issue by having an assortment of pixels with different light sensitivities (all of the pixels in a traditional digital camera have the same sensitivity). A high dynamic range is achieved after the data undergoes image reconstruction to produce the optimal image.
A 3D version of an image can be obtained using a computational camera. Using a mirrored cone placed in front of both the detector and the lens on the camera’s optical axis, three distinct perspectives are produced: the direct scene point, as well as two reflections that lie on the same plane as the optical axis and the direct scene point. A matching algorithm is able to pick out the similarities in these perspectives and compose a 3D image. This 3D imaging can be used to determine the texture and reflectance of materials in the image.
The computational camera has a programmable imaging system, which allows images to be altered and particular facets of the image highlighted. The digital micromirror device (DMD) is used in tandem with the programmable imaging system. It uses a micromirror array that can be switched between a maximum of +10 and a minimum -10 degrees. At +10, the detector is exposed to the scene point while at -10 the detector receives no light. The DMD can switch between the maximum and minimum within microseconds. This system enables exposure duration of individual pixels to be altered, changing the overall exposure pattern of the micromirror array. The programmable imaging system has applications in feature detection and object recognition.
In addition to the DMD, a 3D (volumetric) aperture can be placed in front of the detector in the programmable imaging system to capture more 4D light rays from the surrounding scene. These are modulated before they reach the 2D detector. With this volumetric aperture, it is possible to have a high resolution split field-of-view to more accurately focus on specific points in a scene.
A programmable flash is key to the success of a computational camera. While traditional cameras originally used flashes to take pictures of dimly lit scenes, the computational camera uses a 2D projector light, which illuminates all of the points visible to the camera, captures these points, computing what is in the scene. The programmable flash uses a technique called the temporal defocus method, which utilizes the projector’s narrow depth of field to obtain a different depth associated with each independent pixel. The depth map created as a result of the temporal defocus method allows the photographer to alter the depth of the image as per their personal preference.
The 2D projector light combines global and direct illumination to create a more accurate representation of a scene. Direct illumination refers to the light received directly from the source, whereas global illumination is the light from all points in the scene. While a photo taken with direct illumination only captures the light reflected off an object, global illumination captures the subsurface scattering of light in addition to the colours of objects in the scene. However, a photo taken only with global illumination often looks unrealistic and artificial. Combining the two types of illumination in the computational module can fix this problem and produce a more accurate image.
Many novel principles are at work in the computational camera. It is able to capture all of the details in a scene with the utmost precision. The camera’s applications are endless, although its success depends on individual advances in the areas of image detectors, digital projectors, and imaging optics. Regardless, it is clear that the computational camera will dramatically improve the way we see our world.
Images taken with a wide-angle catadioptric camera, which “has a 220-degree field of view in the vertical plane and a 360-degree field of view in the horizontal plane.”
