Leonardo observed that if you hold an object near your eye, and that object is smaller than your pupil (he used a needle), then it no longer obscures your vision. A dense array of cameras can be treated as a synthetic "eye" with an unusually large pupil. Such a system has a shallow depth of field, allowing us to "see through" partially occluding objects like foliage and crowds. We call this idea synthetic aperture photography (SAP). In the first part of this talk, I will describe two systems we have built to explore this idea: a camera aimed at an array of mirrors, and an array of 128 custom CMOS video cameras.

Similarly, a dense array of projectors can be treated as a single synthetic projector with an unusually large aperture. Such a system produces a real (aerial) image having such a shallow depth of field that the image ceases to exist a small distance away from the focal plane. We call this idea synthetic aperture illumination (SAI). In the second part of this talk, I will describe two systems we have built to explore this idea: a video projector aimed at an array of mirrors, and an array of miniature video projectors.

Finally, I will describe how these two ideas can be combined to implement a large-scale analogue of confocal microscopy - a family of techniques  that employ wide-aperture structured illumination and synchronized imaging to produce cross-sectional views of biological specimens. Replacing the optical apertures used in confocal microscopy with arrays of cameras and video projectors, we can selectively image any plane in a partially occluded environment, and we can see further through weakly scattering environments such as murky water than is otherwise possible. By thresholding these confocal images, we produce mattes that can be used to selectively illuminate any plane in a scene. These capabilities should find applications in scientific imaging, remote sensing, surveillance, shape measurement, and stage lighting.