COLOR MAPPING FOR CAMERA-BASED COLOR CALIBRATION AND COLOR TRANSFER NGUYEN HO MAN RANG NATIONAL UNIVERSITY OF SINGAPORE 2016 COLOR MAPPING FOR CAMERA-BASED COLOR CALIBRATION AND COLOR TRANSFER NGUYEN HO MAN RANG (B., Ho Chi Minh City University of Technology) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY SCHOOL OF COMPUTING NATIONAL UNIVERSITY OF SINGAPORE 2016 c 2016, NGUYEN Ho Man Rang Declaration I hereby declare that this thesis is my original work and it has been written by me in its entirety. I have duly acknowledged all the sources of information which have been used in the thesis. This thesis has also not been submitted for any degree in any university previously. Nguyen Ho Man Rang August 17, 2016 To my parents and wife Acknowledgements I would like to thank all the people who contributed in some way to the work presented in this thesis.
First and foremost, I would like to express my sincere appreciation and gratitude to my advisor Prof. Brown for his enthusiastic and patient guidance and extremely encour- aging advice during my research. His guidance helped me in all the time of research and writing of this thesis. Additionally, I would like to thank my committee Prof.
Mohan Kankan- halli, Prof. Ng Teck Khim, and Prof. Yasuyuki Matsushita for their insightful suggestions and feedbacks from my thesis proposal. Their comments and advices were critical in making my thesis more accurate, solid and widen my research from various perspectives.
I would also like to thank my co-authors Dr. Dilip Prasad, Dr. Seon Joo Kim for their great contribution to my research work. Besides, I would like to thank Dr.
Lin Haiting, Dr. Den Fanbo, Dr. Gao Junhong, Dr. Li Yu, Dr.
Cheng Dongliang, Russell Looi, Mahsa Paknezhad, Abdel- rahman Kamel, Hakki Can Karaimer, Hu Sixing and ther members in vision lab, who as both labmates and friends, were always willing to help and gave their best suggestions. Our friendship has made my life as a graduate student very colorful and enjoyable. Lastly, I would like to express my great gratitude to my family for their unflagging love and unconditional support throughout my life and my studies. I would like to thank my parents for their constant love and support.
I would also like to thank my wife who is always by my side. This thesis would not have been possible if without her love, understanding and support. iv List of Figures. vi List of Tables .2 Selective Literature Review.
9 2 Background and Related Work 10 2.4 Color image formation and camera pipeline .1 Color calibration between camera devices .2 RAW reconstruction from its corresponding sRGB image .3 Color transfer between a pair of images. 28 3 RAW-to-RAW: Mapping between Image Sensor Color Responses 29 3.3 Evaluating mapping approaches .2 Global versus illumination-specific .4 Proposed illumination-independent method .5 Experiments and results .1 Controlled image set .2 Outdoor image set .7 Discussion and Summary. 52 4 Raw to Photo-finished sRGB Output Mapping 54 4.1 In-Camera Imaging Model Estimation .2 Modified Octree Partitioning .1 White-Balance Correction .5 Discussion and Summary. 71 5 Color Transfer between a Pair of Images 75 5.1 Matching white points .2 Matching brightness channel .3 Aligning the color gamut .4 Undoing white-balance .4 Discussion and Summary.
92 6 Conclusion and Future Directions 94 6.1 Harmonizing a group of images .2 Two-way reconstruction between RAW and sRGB. 96 Bibliography 98 iii Abstract This thesis examines color mapping methods that aim to reduce color difference between images in three contexts. The first context is at the camera sensor level, where differences in spectral sensitivity functions of the sensors result in different RGB responses to the incoming light. This work attempts to produce an accurate color mapping between camera sensor-specific color spaces such that the imaged scenes appear the same.
The second context targets the camera processing pipeline where in-camera photo-finishing operations have heavily processed the original RAW image to produce the final sRGB output. This work aims to find a mapping to undo the in-camera processing to obtain the original sensor-specific colors. The third context targets color mapping between images from unknown sources (e. from the internet, photo-sharing sites, etc).
For these type of images, our work focuses on color transfer methods that attempts to manipulate a source image such that it shares a more similar “look and feel” of a specified target image. This thesis begins by motivating the need for color calibration and color transfer between images. This is followed by a brief introduction on how color is represented and related work in the literature focused on both color calibration and color transfer. Afterwards, we describe three contributions made as part of this thesis work.
In particular, we present a novel approach to estimate a mapping to an image of an arbitrary scene and illumination from one camera’s raw color space to another camera color space. This is achieved using an illumination-independent mapping approach that uses white-balancing to assist in reducing the number of required transformations. Our second contribution is a new method to encode the necessary metadata with a photo-finished sRGB iv image for reconstructing its corresponding unprocessed RAW image. Our third contribution is a new approach for color transfer between two given images that is unique in its consideration of the scene illumi- nation and the target image’s color gamut.
The thesis is concluded with summary of the existing contribution and potential future works. v List of Figures 1.1 This figure shows an example of RAW images of the same scene and illumination from different cameras. (a) and (b) show RAW images taken from Canon 1D, Nikon D40 respectively. The color map shown on the right explains how much error each color denotes for (e.
blue color denotes 0% error, while red color denotes up to 20% error). Note that RAW images shown in (a) and (b) are applied a gamma of 1/2.2 for better visualization purpose.2 This figure shows an example of RAW and sRGB images. The bottom show their corresponding sizes.3 This figure shows an example of color transfer problem.1 Normalized spectral sensitivities of short (S), medium (M), and long (L) wavelength cones. The image is reproduced from [Fairchild 2013].2 The figure shows the electromagnetic spectrum for different ranges and the close-up of the visible spectrum.
Note that the visible spec- trum is a rather narrow portion of the electromagnetic spectrum. The image is reproduced from [Fairchild 2013].3 The diagram shows how scene spectral reflectances are converted to the CIE XYZ color space. CIE XYZ proposed three spectral response functions that map real world spectral power distributions (SDPs) to the X/Y/Z basis. The Y value in the CIE XYZ standard is mapped to the standard observer’s luminosity function and is taken to represent the perceived brightness of the scene.4 The sRGB and NTSC color spaces primaries and white-points as defined in the CIE XYZ color space.
These establish the mapping between CIE XYZ and sRGB/NTSC and vice-versa.5 The figures from left to right show the color sensitivity functions of three different cameras: Canon 1D Mark III, Nikon D40, and Sony Nex5N respectively.6 This figure shows the pipeline to obtain sRGB image in consumer cameras. Note that the red circles denote for ’white’ point while the coordinate systems represent the corresponding color space.1 Top row shows three RAW images taken from Canon 1D, Nikon D40, and Sony α57. Bottom row shows the numerical difference as root mean square error (RMSE) between the RAW images. The color map shown on the right explains how much error each color denotes for (e.
blue color denotes 0% error, while red color denotes up to 20% error).2 This figure shows the RAW-to-RAW calibration setup. Images of color calibration charts are taken under several different lighting conditions by the source and target cameras. Mapping between these two cameras’ RAW-RGB colorspaces can be estimated using a global mapping (all illuminations combined) or via multiple illuminant- specific mappings.3 This figure shows the overview of our RAW-to-RAW calibration and conversion approach. (A) shows the steps of our calibration proce- dure.
A global mapping f G is computed using all of the color chart points. White-balancing is then applied to the color charts values from both cameras. Next a mapping on the canonical-illumination, f Lc is computed. (B) illustrates the conversion procedure.4 Example images of the controlled image set of paint chips and paper samples.
The cyan rectangles are regions used for extracting the RAW values.5 Comparison between all approaches. This figure shows the results on a Canon 1D and Nikon D40. Two lighting conditions: fluorescent and tungsten are shown with the camera setting given to the DNG software. Results show the mean RAW pixel errors (normalized) and the errors at the 25%, 50% (median) and 75% quartiles (Q1, Q2, Q3).6 Comparison between all approaches.
This figure shows the results on a Nikon D40 and a Sony α57. Two lighting conditions: fluorescent and tungsten are shown with the camera setting given to the DNG software. Results show the mean RAW pixel errors (normalized) and the errors at the 25%, 50% (median) and 75% quartiles (Q1, Q2, Q3).7 Comparison between all approaches. This figure shows the results on a Olympus E-PL6 and a Panasonic GX1.
Two lighting conditions: fluorescent and tungsten are shown with the camera setting given to the DNG software. Results show the mean raw pixel errors (nor- malized) and the errors at the 25%, 50% (median) and 75% quartiles (Q1, Q2, Q3).8 Comparison between all approaches. This figure shows the results on a Canon 600D and a Nikon D5200. Two lighting conditions: flu- orescent and tungsten are shown with the camera setting given to the DNG software.
Results show the mean RAW pixel errors (nor- malized) and the errors at the 25%, 50% (median) and 75% quartiles (Q1, Q2, Q3).9 The figure shows example images of outdoor image set.10 This figure shows an example of image mosaicing application. Three different cameras Nikon D40, Sony α57, and Canon 1D are used. This figure shows the comparison before and after conversion. All the images are converted to the RAW-RGB space of the Sony α57.
These mosaics have been adjusted by a gamma for better visualization. (b) Original RAW image is 25,947 KB. (c) Our reconstructed RAW im- age using the data in the self-contained JPEG. Overall reconstruction error is 0.2 This figure shows an overview of our approach.
The section of the detailing each component is shown.3 This figure shows an example of with/without using saturation threshold for estimating an inverse tone-curve f −1 .4 This figure shows an example of partition color space using uniform and octree approaches. The same number of bins 64 = 43 is used for both two approaches.5 This figure shows an example of our encoding method which avoids null characters.6 This figure shows comparisons between our approach and our im- plementation of the upsampling approach proposed by Yuan and Sun for various scenes and cameras (a Canon 1Ds Mark III, a Canon 600D, a Nikon D5200, and a Sony α57). The white points on the difference maps indicate overexposed pixels with a value of 255 in any of the channels. The RMSEs for the each method are shown in the bottom right of each error map.7 This figure shows an example of using different qualities of sRGB- JPEG images for reconstructing the RAW-RGB image.
Here, three different qualities: fine, normal, and basic (which supports in Nikon cameras) are examined. The RMSEs for the each quality are shown in the bottom right of each error map.8 This figure shows examples on correcting white-balance for different cameras: a Canon 1Ds Mark III, a Canon 600D, a Nikon D5200, a Nikon D7000, a Sony α200 and a Sony α57. The first column is the input images captured under the wrong white-balance settings; the second column shows the ground truth images captured under the proper settings.