Ester Hait

Blind Facial Image Quality Enhancement

using Non-Rigid Semantic Patches

Ester Hait, Guy Gilboa

Electrical Engineering Department,

Technion – Israel Institute of Technology, Haifa, Israel

 

Picture2

Abstract:

We propose to combine semantic data and registration algorithms to solve various image enhancement problems, such as denoising, super-resolution and color-correction, in the case of facial images.

We address the difficult real-world problem of blind image quality degradation, involving multiple flaws: noise following possible nonlinear processing, resolution reduction, a certain degree of motion blur and contrast and color changes. To handle it, we rely on today’s easily available photography devices, and assume that prior high-quality data of the person to be processed is available.

We use semantically-aware patches, with adaptive size and location regions of coherent structure and context, as our building blocks. We define a new affinity measure derived from the non-rigid Demon registration [1]. We then construct data-driven measure-based affinity spaces, displaying various expression variations of facial features. The measure’s robustness to quality degradation allows to accurately match low-quality features to similar examples. Thus, we obtain a high-quality facial image, while preserving identity, pose and expression.

We demonstrate our method for the real-world problem of cellular photography enhancement of dark facial images. We show how our method significantly enhances image quality, both visually and quantitatively. Relying on only tens of personal priors, we enhance image quality and embed new image details. We compare our method to state-of-the-art denoising, deblurring and super-resolution methods.

 

Problem and assumptions of model: blind enhancement of LQ facial images using prior data

Problem and assumptions of model: blind enhancement of LQ facial images using prior data

 

Algorithm's flowchart: facial image quality enhancement

Algorithm’s flowchart: facial image quality enhancement

Experimental Results:

We compare our results visually and quantitatively (using the NIQE score [2]) to the following methods:

  1. State-of-the-art BM3D denoising [3].
  2. State-of-the-art blind deblurring using a coupled adaptive sparse prior [4].
  3. State-of-the-art blind deblurring using a dark channel prior [5].
  4. State-of-the-art super-resolution using sparse representation [6], using different dictionaries.

Quality Enhancement Example (1020X768 pixels): identity #2, right pose:

Quality Enhancement Example (1020X768 pixels): identity #2, right pose.

Picture3

Picture4

Quality Enhancement Example: identity #2, right pose, close up:

Capture

Comparison to Super-resolution (1020X768 pixels): identity #2, right pose:

Capture3

Capture2

Capture

 

Quality Enhancement Example (950X640 pixels): identity #3, frontal pose:

Capture

Capture2

Quality Enhancement Example: identity #3, frontal pose, close up:

Capture

Comparison to Super-resolution (950X640 pixels): identity #3, frontal pose:

Capture3

Capture4

Capture5

 

Quality Enhancement Example (1050X658 pixels): identity #1, frontal pose:

Capture

Capture2

Capture3

Comparison to Super-resolution (1050X658 pixels): identity #1, frontal pose:

Capture3

Capture2

Capture

Quality Enhancement Example (1130X710 pixels): identity #2, frontal pose:

Capture

Capture2

Comparison to Super-resolution (1130X710 pixels): identity #2, frontal pose:

Capture3

Capture2

Capture

 

Quantitative results using the NIQE score [2]:

We normalize the NIQE scores of different methods to the NIQE score of the example HQ image. Therefore the closer the normalized score is to 1 – the better the visual quality.

Normalized NIQE Quality Assessment for 17 examples: Our method compared to Denoising and Deblurring methods:

table niqe

NIQE_graph_17_2

Normalized NIQE Quality Assessment for 17 examples: Our method compared to Super-resolution methods:

table niqe SRNIQE_graph_SR2

Supplementary

Downloads:

Reference:

 

Hait, Ester, and Guy Gilboa. “Blind Facial Image Quality Enhancement using Non-Rigid Semantic Patches.” IEEE Transactions on Image Processing 26.6 (2017): 2705.

(see older version at arXiv:1609.08475)

 

Matlab Code, including HQ image dataset

 

Contact us at:

etyhait@campus.technion.ac.il

 

References:

  1. J-P Thirion. Image matching as a di_usion process: an analogy with maxwell’s demons. Medical image analysis, 2(3):243-260, 1998.
  2. Anish Mittal, Ravi Soundararajan, and Alan C Bovik. Making a “completely blind” image quality analyzer. Signal Processing Letters, IEEE, 20(3):209-212, 2013.
  3. Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space. In Image Processing, 2007. ICIP 2007. IEEE International Conference on, volume 1, pages I-313. IEEE, 2007.
  4. Haichao Zhang, David Wipf, and Yanning Zhang. Multi-image blind deblurring using a coupled adaptive sparse prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1051-1058, 2013.
  5. Jinshan Pan, Deqing Sun, Ming-Hsuan Yang, and Hanspeter Pfister. Blind image deblurring using dark channel prior. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, 2016.
  6. Jianchao Yang, JohnWright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE transactions on image processing, 19(11):2861-2873, 2010.