Based on the author’s 1988 Ph.D. thesis, this book describes a physical model of optical reflectance for a class of materials, and applies it in an algorithm for color image segmentation, and two other algorithms for the decomposition of a scene into intrinsic images (body reflection, surface reflection, and noise). It also discusses the advantages of such a physical approach and the perspectives that it opens for computer vision.
The starting point is an analysis of the reflectance of dielectric (electrically nonconductive), nonuniform, opaque materials, such as wood, plastic, paper, and paint. The reflectance function decomposes itself into a body reflectance, which accounts for the object’s matte color, and a surface reflectance, which produces the object’s gloss. The main observation is that these two reflectance components have different chrominances. For example, in a varnished red wooden cylinder under white light, the glossy highlights look whiter than the underlying matte red. Under certain restrictions on the shape of objects and on the position of the illumination source and camera, the histogram of reflected colors of an object is shown to take a characteristic shape (the “skewed T,” shown on the cover of the book). A local-to-global analysis of the color histograms of small regions in an image allows a classification of the various image points (dark, matte, or highlight points, material changes, and so on), and their grouping into regions; this produces a segmentation of the scene into distinct objects, and the decomposition of reflected light on each object into three components (body and surface reflection, and noise). This approach is cleverer than traditional statistical classification algorithms, which rely on the implicit assumption that color changes in an image are due either to material changes in the scene or to noise, the problem being the elimination of noise; they invariably tend to split objects along their highlights, and sometimes they even decompose shaded matte areas into several objects.
The first part, consisting of chapters 2 and 3, contains the physical basis of the author’s approach, namely a detailed account of what happens to light when it is reflected by objects and when it enters the camera. Besides preparing for the remainder of the book, it is interesting as an introduction to photometric aspects of natural image generation.
Chapter 2 describes a physical model of reflectance. Materials can be classified as dielectric or nondielectric; nondielectric objects (such as metals) have only surface reflectance. Dielectric materials can be nonuniform or uniform; uniform dielectrics (such as glass) are usually light-transmitting (transparent or translucent) and have no body reflectance. Nonuniform dielectrics can be either opaque or transmitting; they have both surface and body reflectance. Lambertian and specular reflectance are idealizations of body and surface reflectance respectively; exact quantitative estimations of body and surface reflectance are usually more complicated. A qualitative observation, however, is that these two components of reflectance generally have different chrominances; thus the reflectance of a colored (dielectric nonuniform opaque) object spans two dimensions in color space. Under certain limiting assumptions on the scene geometry (the object is spherical, there is only one light source, the camera lies at the same distance from the object as the light source, and so on), one can derive certain constraints on the light reflection by the object; one of them is that surface reflection produces highlight lines where the reflected color is the addition of a varying amount of surface reflection color and a fixed body reflection color whose intensity is above median. This gives the characteristic skewed-T color histogram on the image of the object, and this observation is taken as a general heuristic for more general scene geometries, namely groups of curved objects illuminated by a single source, in a laboratory setting (not outdoors).
Chapter 3 recalls physical properties of optical sensors (mainly charge-coupled device cameras): spectral integration into three color channels; limited dynamic range; clipping and blooming of luminance values outside this range; different spectral responsiveness than the human eye; gamma-correction; and chromatic aberration. Each of these factors distorts the dichromatic reflection model described in chapter 2, and this distortion must be taken into account in the application of this model to image analysis.
The second part, consisting of chapters 4 and 5, applies the dichromatic reflection model to image understanding. The author has designed, implemented, and tested three algorithms for segmenting color images and separating such an image into three intrinsic images giving body reflection (matte color), surface reflection (highlights), and noise. The algorithms are described informally; neither code nor pseudocode is given. This is not a defect, because the main point of the book is not some particular detail of the algorithms (which can always be improved by taking into account other types of information), but an existence proof of the applicability of the model for image understanding in order to obtain good results.
Chapter 4 describes the algorithm for segmenting color images. The image is partitioned into small regions, and in each of them a color histogram is computed. Statistical analysis allows an estimation of the dimensionality of the space around which histogram values tend to cluster. By a combination of statistics and heuristics, the initial regions can be merged or split, and a segmentation of the image is progressively built by combining regions whose color histogram has a given dimension (first one, then two, and afterwards both are combined). The final segmentation is generally accurate, except in image regions where color is influenced by more complex optical phenomena not taken into account in the model (such as secondary illumination, in which one object is reflected in another).
Chapter 5 introduces two algorithms for extracting from the original image and its segmentation a set of three intrinsic images. The local algorithm uses the analysis of local histograms made in the segmentation algorithm in order to separate the three components of reflection (body, surface, and noise) for each region. The global algorithm does not use these previous data, but starts directly with an analysis of the color histogram in each object produced by the segmentation. The algorithms are comparable in efficiency and accuracy. Each is better in different situations, but the local algorithm is generally more robust, while the global one tends either to succeed completely or to fail utterly. Probably a combination of both would be better.
The third part, comprising chapters 6 to 8, discusses results and perspectives. Chapter6 gives results of the algorithms for several scenes under varying illumination colors. Explanations are given for some failures (mainly due to physical phenomena not taken into account by the model, such as secondary illumination, or to scene geometries departing from the simplifying assumptions made in chapter 2). Klinker discusses the values of the various heuristic control parameters used by the algorithms, and gives some justification for the choice of these values (which may be modified in difficult cases). Chapter7 summarizes the advantages of the author’s approach, in particular the use of a physical model (instead of an arbitrary statistical model of signal and noise), the direct access to unfiltered image data at all stages of the analysis, and the predictability of failures of the algorithms for scene characteristics exceeding the limitations of the model.
While the rest of the book is based on the author’s Ph.D. thesis, chapter 8 discusses recent work done on physics-based image understanding. The author’s opinion is that such an approach can either be applied to specific practical applications, where the scene’s optics are constrained, or combined with traditional image processing methods for improved general-purpose image understanding algorithms.
A striking fact is that the author’s algorithms use only one type of information, namely the general form and dimensionality of local color histograms, and that this suffices to produce accurate results whenever the scene fits the reflection model. In particular, this approach is inapplicable for gray-level images, since the gray-level space is invariably one-dimensional. This limitation proves the usefulness of chromatic analysis of light reflection for image analysis. In my opinion, the exact details of the algorithms do not matter (and they use too many heuristics and control parameters anyway), and the author’s approach should not be used in isolation (taking into account all optical phenomena renders the task complicated). It could rather be combined with other approaches, such as luminance-based photometric methods, including edge classification from luminance profile, shape from shading, texture, or highlights; and general-purpose image processing techniques, including geometric methods, mathematical morphology, and Fourier and wavelet analysis.
This book is well written and instructive, especially in its description of physical models of reflectance. Its 213-item bibliography, mainly on photometry and its applications to image understanding, is a good guide for the reader who would like to explore the subject further. The book can be used for research, to a certain extent for teaching (especially chapters 2 and 3), and for discussions on future approaches to image analysis.
