Image enhancement based on neuro-fuzzy gradient profile clustering

This paper proposes a technique for image enhancement using Neuro-fuzzy based gradient profile generation to reconstruct the high resolution image from a single low resolution one. The natural gradient priors are collected and their statistics are analyzed and learned through Neuro-fuzzy model. The model adopts powerful data adaptation from neural network and combines with fuzzy system to enhance the ability in knowledge interpretation and explanation in terms of natural language. The triplet gradient profile is then generated as a result. The gradient profile results are used to regulate the Gaussian weighted sum filter in enhancement process. Then, all the weights were appropriately adapted according to gradient priors. From the experimental results, it can be seen that the proposed algorithm can greatly compensate the contrast and noise distortion in the low resolution image and demonstrate successful recovery of the high resolution image with quantitatively and perceptually performance improvement.


INTRODUCTION
Image enhancement is a fundamental process for recovery and reconstruction of high resolution (HR) image from low resolution (LR) image that is effected by down sampling matrix, blurring operation and Gaussian noise.However, the performance of high resolution is direct variation from hardware cost.With the limitation of the image capture hardware, image enhancement software is preferred (Komatsu et al., 1993).Image enhancement software technique can be divided into three categories such as image enhancement based on multi-frame reconstruction (Hou and Andrews, 1978;Li and Orchard, 2001;Farsiu et al., 2004;Ahmad and Qureshi, 2012;Faramarzi et al., 2013), single image interpolation technique (Kim et al., 1990;Kim and Su, 1993;Grover and Kasana, 2015;Yan et al., 2015;Bajić et al., 2016;Gohshi, 2016;Ngocho and Mwangi, 2016), and example based learning (Freeman et al., 2002;Yang et al., 2010;Tai et al., 2010;Kim and Kwon, 2008;Lu et al., 2011;Suo et al., 2017;Shi et al., 2016).
The hypothesis of multi-frame reconstruction is that the high detail information lost in one low resolution scene can be found in another low resolution scene.Therefore, the process is to combine multi-sequence low resolution images to produce a high quality image (Farsiu et al., 2004) using registration and regularization techniques.Several researchers have been proposed multi-frame image reconstruction algorithms (Ahmad and Qureshi, 2012;Faramarzi et al., 2013).Ahmad and Qureshi (2012) demonstrated multi image super resolution technique using grid analysis with the Projection Onto Convex Sets (POCS).Even though POCS tried to smooth the image, the algorithm was not tolerant to noise.Faramarzi et al. (2013) presented Unified Blind Method for the single image reconstruction from multiple images based on alternating minimization function with the regularization prior from smooth low frequency area and edgeemphasizing operation estimating blurring with enhancement in the high frequency area.However, its difficulty is in compensating the blurring effects with only estimated parameters obtained from multiple LR images.The additional enhancement filter may be needed in order to improve the HR results.The limitation of multiframe reconstruction is that there might not be sufficient number of input images containing enough information available for reconstruction.
In single image enhancement, only a single low resolution image is required and used to extract parameters to reconstruct HR image based on assumption of the priors.Interpolation technique is adopted based on polynomial approximation (Zhang and Wu, 2006;Li and Nguyen, 2008).This technique estimates mathematic model and apply it to reconstruct HR image.However, the interpolation technique provides good quality in low frequency (LF) area but poor in high frequency (HF) area.Alternative technique is to emphasize or to boost edges in HF area while smoothing the LF area to obtain the improvement of the HR image.In spatial domain, Grover and Kasana (2015) presented Fast Edge-Adaptive Interpolation technique based on approximated nonlinear parameters.This technique attempted to find edge area in difference resolution and group up edge area to connect and sharpen HF area.Yan et al. (2015) used Gradient Profile Sharpness in single image super-resolution.The triangle and Gaussian gradient description model were calculated to construct edge sharpness matrix.This technique can improve the quality in HF area.Bajić et al. (2016) estimated blur area and noise in LR image to reconstruction HR image based on Poisson Gaussian distribution.In frequency domain, Gohshi (2016) analyzed single image super resolution using two dimensional fast Fourier transform with nonlinear processing for generating HF characteristic element in reconstruction process.Ngocho and Mwangi (2016) presented iterative Back-Projection and Log Sharpening to LR image with bilateral filter and guided filter to approximate and improve HF in reconstruction process.It used the smoothing filter to remove the effect of noise while adopted guided filter based on Back-Projection to enhance edge area.However, the reconstruction performance is comparable to bicubic interpolation in terms of PSNR and SSIM where LF area Ngernplubpla and Chitsobhuk 43 can be much improved.Nevertheless, reconstruction in HF area is still a challenging issue for image superresolution.Image enhancement based on example based learning from example data patches is another approach to improve the HR reconstruction.The technique encodes the relationship information between HR and LR images in the database to estimate prior knowledge (Tai et al., 2010).Each LR patch is compared to the example LR patches in database and matched to the nearest LR patch.Then, the corresponding HR patch will be retrieved as the image output (Kim and Kwon, 2008).Lu et al. (2011) presented image super resolution based on a new dictionary with Homotopy method for learning strategy in order to choose parameter that correct patches.Suo et al. (2017) proposed feature enhancement mapping for single image super resolution based on learning feature matrix between LR and HR to improve the feature extraction and to learn the detail mapping matrix for image reconstruction.Shi et al. (2016)  This paper proposes single image enhancement to resolve obscure data, discover hidden information, and enhance the LR image to obtain the HR image.High frequency reconstruction is the general problem in image enhancement that sharpens edge in high frequency area and smooth in low frequency area.In Zhang and Wu (2006), Adaptive Edge Enhancement Algorithm (AEEA) was used to reconstruct HR image.This algorithm presented new edge model called Edge Sketch Image (ESI) based on maximum likelihood estimation (MLE) to approximate edge using statistical analysis in high frequency area.This technique can supply great edge information in high frequency area.However, it also permitted some amount of noise in low frequency area.To improve the quality of HR image, AEEA combined the low frequency filtering using Wiener filter with high frequency emphasis through weights generated from the ESI and local variance value.The results of AEEA showed smoothness in low frequency area and great recovery in high frequency area.However, the weights were learned from limited number of samples thus led to unsuccessful results in some cases.Therefore, in Ngernplubpla and Chitsobhuk (2015), image enhancement was proposed based on edge boosting algorithm (EBA) using Priority Map Generation.The Priority Map was generated as regularization on how the gradient priors should influence the enhancement of the LR image.The results demonstrated high quality in both low frequency and high frequency area.However, there will still be degradation in some areas especially in high detail area with medium frequency, that is, the detail of Lena's hat.For further improvement, this paper proposed image enhancement technique based on Neuro-fuzzy clustering.Neuro-fuzzy model adopts ability to handle the uncertainty from fuzzy system together with the ability of the neural network to learn the appropriate structure and rules from data.Thus, clustering based Neuro-fuzzy framework can offer a great opportunity to distinguish ambiguous groups of gradient information.The clustering result is used as gradient profile to regulate the weight selection.Finally, edge emphasis is performed according to weight regulation and neighbor intensity relation through Gaussian weighted sum filtering.

METHODOLOGY Image reconstruction model
The relationship between LR image and HR image is presented in Equation 1, where X is the HR image, Y is the LR image.
H is blurring operation, D is down sampling matrix and  is modelled as Gaussian noise.This models how the HR image is affected by degradation and noise.Therefore, it is necessary to approximate appropriate parameters and efficiently create image enhancement filter for high resolution reconstruction (Zhang et al., 2015).

Edge sketch image
Edge sketch image (ESI) was the new edge emphasizing algorithm proposed in Ngernplubpla and Chitsobhuk (2015).This technique used Maximum Likelihood Estimation (MLE) (Montgomery and Runger, 2010) for estimating a weight (WML) from LR image and enhanced the edges in the high frequency area as shown in Equations 2 and 3.
x is normal distribution related to unknown parameter  .The edge sketch image can be computed with the Maximum Likelihood Estimation from Equation 4.
where ESI f is high frequency emphasized image based on ESI technique, g is LR input image, and LM is local mean value calculated from g.The example result using the ESI based technique is as shown in Figure 1.

Gradient statistics
Good image prior is essential to HR image reconstruction.In this paper, the statistics of the separable second-order gradient are considered as gradient prior, which captures the intensity changes in x and y directions as denoted in Equation 5.
Even though the velocity of intensity change provides reasonable difference between low and high frequency area, the acceleration of intensity change offers even larger difference and better representation of the high frequency and especially texture area.

Neuro-fuzzy clustering
In regular image enhancement algorithm, an LR image is partitioned into low and high frequency area in order to create the enhancement regularization or profile.Each group has its own enhancement parameter.There are several clustering techniques e.g.quantization (Rigau et al., 2004), fuzzy system (Lazli and Boukadoum, 2017), neural network (Caramihale et al., 2016) and neuro-fuzzy (Yeung and Wang, 2000).An example of simple clustering is quantization (Q(x,y)) technique used to classify information based on thresholding as shown in Equation 6.
Fuzzy system can be used to describe its encoded knowledge but it cannot learn the knowledge from training examples.However,  (Roohi and Phil, 2013).With the integration of both systems, the neuro-fuzzy model can offer great strengths in data adaptation and knowledge interpretation.The neuro-fuzzy model is as shown in Figure 2 (Boskovitz and Guterman, 2002).TSK (Osowski et al., 2002), referred to Takaki-Sugeno-Kang Neuro-fuzzy network, is used to model multiple inputs ( x ) and single output ( y ) as  7where u k denotes the firing strength of kth rule.The parameter p is the learning process.
Adaptive network based fuzzy inference system (ANFIS architecture) is one of neuro-fuzzy system model based on Takagi Sugeno's method (Panella and Gallo, 2005).This architecture composes of five layers as shown in Figure 3.The first layer performs membership function mapping according to input relationship.The second layer uses T-norm operation to estimate the parameters and rules while rules are later normalized in the third layer.The fourth layer returns results from rule evaluation and then all the results are combined to select the best answer in the fifth layer (Aru et al., 2016).In learning process, the model is initialized with the fuzzy model and tuned by means of a hybrid technique combining gradient descent back propagation and mean least-squares optimization algorithms.The learning process will be terminated once the system reaches either the predefined epoch number or error rate is obtained.

Proposed algorithm
In this paper, neuro-fuzzy based gradient profile generation is proposed for reconstructing high resolution image in high, medium, and low frequency.This technique attempts to solve the problems that cause blur in low frequency area, low edge detail in high frequency area, and low contrast in texture area.Consequently, frequency classification plays an important role since its results affect how each frequency area should be enhanced.Image priors can be used as a guideline to compensate the degradation in each area.The prior profile can be generated from simple classification techniques (Zhang et al., 2012).Without learning, the classification results are limited.Therefore, a neuro-fuzzy clustering was proposed as the frequency classification, which offers a great opportunity to distinguish ambiguous groups of gradient information.
The classification results are then used to generate the gradient profile and used to model weights in image enhancement process.
The overview of the proposed method is as shown in Figure 4.

Feature extraction
Feature extraction is the first process for image enhancement.Before extraction process, color image input will be transformed to grayscale image.The goal of feature extraction is to generate five features: 1st and 2nd order gradient in vertical and horizontal directions and edge sketch image data.Then, the velocity and acceleration gradient in each direction are classified into three Gradient Level Values (GLV) (L0-L2) as illustrated in Equations 7 and 8 based on thresholds ( T ).After that, the Relational Gradient Direction (RGD) is computed in Equation 9.

Gradient profile generation
The gradient prior is usually grouped into low and high frequency areas.However, the mid-low frequency area such as texture area is also influential on human perception.Without considering the texture area, it is difficult to successfully obtain the high quality of HR image [24].In this paper, gradient prior classification based on neuro-fuzzy algorithm is proposed for gradient profile generation.The gradient priors are grouped into three groups: low frequency (G0), medium frequency (G1), and high frequency areas (G2).The training based on neuro-fuzzy system is shown in Figure 5, where training inputs are RGD, 2nd order gradient in vertical and horizontal directions and ESI data (2000 data for training: 1000 from original HR data and 1000 for LR input with noise) accompanied with ground truth.After training process, the result of ANFIS structure was obtained as shown in Figure 7.The gradient profile results are used to regularize the Gaussian weighted sum filter (GWSF) in enhancement process.

Image enhancement
In enhancement process, GWSF is applied to adjust the LR input image (g(x,y)) as described in Equations 10 to 13 where G W is the GWSF weight, C W is the weights derived from the gradient profile (GP) result,  is mean, 2  is variance, 0, ESI(x, y) 50 (x, y) 0.02, ESI(x, y) 50 From Equations 10 to 13, the enhancement results are regulated by the GWSF, the Gradient Profile obtained from ANFIS, the ESI, and local variant.All the weights are appropriately adapted according to frequency priors, which can help to improve the HR reconstruction.This results in great enhancement in smooth detail, finer texture, and sharpen edges with low noise.

Experimental setup
In the experiments, Lena color image with resolution 512×512 pixel is transformed to grayscale image to create the LR input image using bicubic interpolation with scaling factors 2, 3 and 4.This process is used to integrate disturbance to original image.
The region clustering is one of crucial problem in image enhancement since it identifies how to group the optimal area and select the best weights for image enhancement.Generally, regions are grouped into low frequency and high frequency areas.The texture area with medium frequency is buried in either group.This makes the quality of the enhancement rather low.In this paper, the input LR image is clustered into 3 groups (low, medium, and high frequency areas) based on neuro-fuzzy clustering.The parameter of neuro-fuzzy system is discussed previously.The clustering performance is illustrated earlier.Finally, the performance of the proposed image enhancement compared to the other techniques was discussed.
Image enhancement performance is measured in terms of the Weighted Peak Signal to Noise Ratio (WPSNR), Structural Similarity (SSIM) and Second-order Gradient Magnitude Difference (SGMD).WPSNR (Navas et al., 2011) is one of the image fidelity measurement techniques based on PSNR (Huynh-Thu and Ghanbari, 2008) with taking into account the Human Vision Model (HVM).The principle of redundancy of the human eye toward high frequency components in images is significantly concerned.The WPSNR is calculated as shown in Equation 14, where I MAX is the maximum range of intensity, MSE is the mean square error and NVF is Noise Visibility Function.SSIM (Wang et al., (2004) is the structure based measurement that is a combination of Luminance, Contrast and Structure comparison as shown in Equation 15where  is mean, 2  is variance and C is constant.Gradient based quality measurement is an alternative approach (Wang et al., (2004).The image gradient magnitude is sensitive to degradation introduced from compression, blur, or additive noise.It can be used to reflect the image quality.SGMD presented in [24] is used for comparing acceleration gradient between test image and reference image as shown in Equations 16 and 17 where nd 2 order G is acceleration gradient, SG is sum of total gradient.

Neuro-fuzzy parameter
In neuro-fuzzy system, the appropriate types of input and output membership functions are necessary to be examined.In this experiment, the suitable parameters are evaluated from 2000 data samples with total 50,000 epoches and selected error for each loop equal to 0.001.6 and Table 1.
From Table 1, the best performance is obtained from the selection of gauss2mf input and linear output membership functions with minimum error 0.19107.In this paper, these parameters will be used for gradient profile clustering based on neuro-fuzzy system.

Gradient profile clustering performance
This topic presents accuracy measurement for gradient profile clustering process.The gradient profile is categorized into three groups: low, medium, and high frequency.Figure 7 shows the example of each frequency group.The clustering performance comparison is conducted from three clustering models: thresholding technique (Ngernplubpla and Chitsobhuk, 2013), neurofuzzy system, Support Vector Machine (SVM) using a combination of RGD, 2nd order gradient in vertical and horizontal directions, and ESI descriptors.The results are shown in Table 2.
From the experimental results, neuro-fuzzy system using four input descriptors of RGD, 2nd order gradient in vertical and horizontal directions, and ESI provides the highest accuracy of 72.83%, which is about 26.58% as compared to simple thresholding technique and 9.60% compared to SVM.

Image enhancement performance
In this experiment, we will evaluate the performance of the enhancement process in two steps.The first experiment is to examine how to integrate the different gradient profile clustering to recover the distorted pixels in the LR image.Three techniques: Low-High (LH)   In the second experiment, the enhancement techniques are performed to the LR image with 2x, 3x, and 4x scaling factors.The examples of the LMH frequency triplet profile are the areas of low frequency at the skin area, medium frequency at the hat area, and high frequency at object boundary as shown in Figure 8. From Table 4, three objective measurements are conducted and it can be seen that the proposed algorithm outperforms other techniques in all three measurements.
In the proposed algorithm, suitable parameters are chosen, such as for Neuro-fuzzy Gradient Profile Clustering, 4 features of RGD, 2nd order gradient in vertical and horizontal directions and ESI are used as image gradient prior input with gauss2 input membership function and constant output membership function, and for image enhancement, the LMH frequency triplet profile with GWSF is adopted.As a result, the average performance in three measurements: SSIM, WPSNR and SGMD are 0.9484, 42.4437 and 35.33, respectively.The performance comparison can obviously illustrated the performance improvement of the proposed algorithm as compared to the other techniques: WPSNR and SGMD with 11.91 and 57.37%, respectively, higher than those of Wiener filter, 7.10 and 24.57%, greater compared to AEEA, and 4.28 and 5.12% better than EBA.When using SSIM, the proposed algorithm performs 4.71, 1.72, and 1.27% superior to Wiener filter, AEEA, and EBA, respectively.Figure 8 shows the original image, LR input image and the results of reference and proposed image enhancement techniques.From the results, it can be seen that the larger the scaling factor, the lower the contrast and more noise added to the LR image.Wiener filter can be used to restore the HR image with great performance in the low frequency area but tends to loss quality noticeably in high frequency area.For the AEEA, the algorithm enhances the LR image based on enhancement weights estimated according to the maximum likelihood estimation (MLE).Even though it shows performance improvement in high frequency area, the quality of the reconstructed HR is still not as strong as the EBA.Wiener filter and the AEEA poorly perform in the medium frequency area as shown in Figures 9 and 10.Gradient fields of the HR results from each enhancement techniques are presented in Figure 10b.Even though the EBA provides greater performance compared to Wiener filter and the AEEA, its result still contains lower contrast compared to that of the proposed algorithm.Therefore, it can be proven that the proposed algorithm can greatly compensate the contrast and noise distortion in the LR image in all frequencies.

Conclusion
This literature proposes a new algorithm to reconstruct the high resolution image from a single low resolution one based on neuro-fuzzy gradient profile clustering.This algorithm takes advantages of neuro-fuzzy clustering of several gradient fields: second-order gradient in vertical and horizontal directions, relational gradient direction, ESI, and local variance to generate the gradient profile.
Once the triplet profile is obtained, it is used as regularization and combined with Gaussian weighted sum filtering to incorporate the neighbor information into the proposed enhancement process.With the proposed gradient priors, all the weights are appropriately adapted.
From experimental results, it can be seen that the proposed algorithm can greatly compensate the contrast and noise distortion in the LR image and demonstrates successful recovery of the HR with superior performance measurement as compared to the other techniques.

Figure 1 .
Figure 1.The result of Edge Sketch Image.

Figure 2 .
Figure 2. The example of gradient.

Figure 4 .
Figure 4. overview of the proposed algorithm.
(x, y) f is the estimated HR output image reconstructed from GWSF of LR input image ( (x, y) g ) with the weighted ( W ESI ) ESI, and the weighted (

Figure 6 .
Figure 6.The example result from gauss2mf with linear parameter.

Figure 7 .
Figure 7.The example of all frequency area.

Figure 8 .
Figure 8.The results of image enhancement techniques.

Table 1 .
The errors from different types of functions used for Neuro-fuzzy system Eight types of input membership functions are evaluated e.g.trimf, trapmf, gbellmf, gaussmf, gauss2mf (the combination between guassian and sigmoid), pimf, dsigmf and psigmf.Additionally, two types of the output membership functions are examined e.g.constant and linear.The experimental results are as shown in Figure

Table 2 .
The clustering performance comparison with different combination of descriptors

Table 3 .
The result of different pixel recovery techniques.

Table 4 .
The image enhancement results.