Blind Facial Image Quality Enhancement
using Non-Rigid Semantic Patches
Ester Hait, Guy Gilboa
Electrical Engineering Department,
Technion – Israel Institute of Technology, Haifa, Israel
We propose to combine semantic data and registration algorithms to solve various image enhancement problems, such as denoising, super-resolution and color-correction, in the case of facial images.
We address the difficult real-world problem of blind image quality degradation, involving multiple flaws: noise following possible nonlinear processing, resolution reduction, a certain degree of motion blur and contrast and color changes. To handle it, we rely on today’s easily available photography devices, and assume that prior high-quality data of the person to be processed is available.
We use semantically-aware patches, with adaptive size and location regions of coherent structure and context, as our building blocks. We define a new affinity measure derived from the non-rigid Demon registration . We then construct data-driven measure-based affinity spaces, displaying various expression variations of facial features. The measure’s robustness to quality degradation allows to accurately match low-quality features to similar examples. Thus, we obtain a high-quality facial image, while preserving identity, pose and expression.
We demonstrate our method for the real-world problem of cellular photography enhancement of dark facial images. We show how our method significantly enhances image quality, both visually and quantitatively. Relying on only tens of personal priors, we enhance image quality and embed new image details. We compare our method to state-of-the-art denoising, deblurring and super-resolution methods.
We compare our results visually and quantitatively (using the NIQE score ) to the following methods:
- State-of-the-art BM3D denoising .
- State-of-the-art blind deblurring using a coupled adaptive sparse prior .
- State-of-the-art blind deblurring using a dark channel prior .
- State-of-the-art super-resolution using sparse representation , using different dictionaries.
Quality Enhancement Example (1020X768 pixels): identity #2, right pose:
Quality Enhancement Example: identity #2, right pose, close up:
Comparison to Super-resolution (1020X768 pixels): identity #2, right pose:
Quality Enhancement Example (950X640 pixels): identity #3, frontal pose:
Quality Enhancement Example: identity #3, frontal pose, close up:
Comparison to Super-resolution (950X640 pixels): identity #3, frontal pose:
Quality Enhancement Example (1050X658 pixels): identity #1, frontal pose:
Comparison to Super-resolution (1050X658 pixels): identity #1, frontal pose:
Quality Enhancement Example (1130X710 pixels): identity #2, frontal pose:
Comparison to Super-resolution (1130X710 pixels): identity #2, frontal pose:
Quantitative results using the NIQE score :
We normalize the NIQE scores of different methods to the NIQE score of the example HQ image. Therefore the closer the normalized score is to 1 – the better the visual quality.
Normalized NIQE Quality Assessment for 17 examples: Our method compared to Denoising and Deblurring methods:
Normalized NIQE Quality Assessment for 17 examples: Our method compared to Super-resolution methods:
Hait, Ester, and Guy Gilboa. “Blind Facial Image Quality Enhancement using Non-Rigid Semantic Patches.” IEEE Transactions on Image Processing 26.6 (2017): 2705.
(see older version at arXiv:1609.08475)
Matlab Code, including HQ image dataset
Contact us at:
- J-P Thirion. Image matching as a di_usion process: an analogy with maxwell’s demons. Medical image analysis, 2(3):243-260, 1998.
- Anish Mittal, Ravi Soundararajan, and Alan C Bovik. Making a “completely blind” image quality analyzer. Signal Processing Letters, IEEE, 20(3):209-212, 2013.
- Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space. In Image Processing, 2007. ICIP 2007. IEEE International Conference on, volume 1, pages I-313. IEEE, 2007.
- Haichao Zhang, David Wipf, and Yanning Zhang. Multi-image blind deblurring using a coupled adaptive sparse prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1051-1058, 2013.
- Jinshan Pan, Deqing Sun, Ming-Hsuan Yang, and Hanspeter Pfister. Blind image deblurring using dark channel prior. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, 2016.
- Jianchao Yang, JohnWright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE transactions on image processing, 19(11):2861-2873, 2010.