Input image `turtle' (1a.) and `panther' (1b.) with their ground truth separation (2.) and (3.) from `Ground truth dataset and baseline evaluations for intrinsic image algorithms' by Grosse et al. (ICCV09). Figures (4a.) and (5a.) show the results by our proposed method using cross validation to set the parameters. Figures (4b.) and (5b.) show the full flexibility of the proposed method by setting the parameters to the result that yields the lowest error.
The problem of decomposing an image into its different intrinsic layers, such as shading, reflectance, and shape components, is one of the fundamental problems of computer vision. We believe that many computer vision tasks will benefit from having access to such separations. As an example, object detection should benefit from factoring out shadow and lighting effects, or image segmentation of materials should be improved by working on reflectance properties alone. In this project we aim to both decompose images into their intrinsic layers, and also show that this results to empirical improvements on different tasks.
Our approach is to model the physical process of light interacting with materials. We model material properties, shape, and lighting conditions as separate sources that result in the observed pixel color and brightness. In "Recovering Intrinsic Images with a Global Sparsity Prior on Reflectance" we extend the Retinex theory by including models of material reflectances. Our conditional random field model encourages solutions in which the image is explained by a sparse set of reflectance types. This corresponds to the intuition that there are only few materials in an image. In the figure we show some example outputs of this model
Intrinsic image decomposition problems arise in many different contexts and we study several different variants that arise when some scene modalities may be known. For example when using multiple depth cameras for full body scanning, we are provided with a good 3D estimate of the scene, or given a video we know that light varies smoothly. Our approach is to model the physical process using techniques from computer graphics together with probabilistic inference to reason about the unknown sources that constituted the scene.
1 Microsoft Research Cambridge 2 MPI for Intelligent Systems
Recovering Intrinsic Images with a Global Sparsity Prior on Reflectance
Gehler, P., Rother, C., Kiefel, M., Zhang, L. and Schölkopf, B.
Advances in Neural Information Processing Systems (NIPS),
We address the challenging task of decoupling material properties from lighting properties given a single image. In the last two decades virtually all works have concentrated on exploiting edge information to address this problem. We take a different route by introducing a new prior on reflectance, that models reflectance values as being drawn from a sparse set of basis colors. This results in a Random Field model with global, latent variables (basis colors) and pixel-accurate output reflectance values. We show that without edge information high-quality results can be achieved, that are on par with methods exploiting this source of information. Finally, we are able to improve on state-of-the-art results by integrating edge information into our model. We believe that our new approach is an excellent starting point for future developments in this field.