A novel fusion algorithm for infrared image and visible light image based on non-subsampled contourlet transform

Aiming at the different characteristics of the infrared image and visible light image, the paper proposed a kind of fusion algorithm for the infrared image and visible light image based on non-subsampled contourlet transform. Firstly, the source images are made multiscale and multi-direction decomposition by using non-subsampled contourlet transform (NSCT). Secondly, to decomposed low frequency subband, a decision-making value with regional energy and variance is constructed and used in fusing the coefficients by choosing larger decision-making value. And for the decomposed high frequency subband, different fusion rules are employed for different levels. The fusion rule of selecting large absolute value of pixel is used for the highest level, and the fusion rule of selecting large regional variance based on regional energy matching degree is used to fuse the other levels. Finally, the final fused image is reconstructed by using the non-subsampled contourlet inverse transform. The experimental results have shown that the proposed algorithm can get more detail information and can exhibit better fusion performance.


INTRODUCTION
Image fusion refers to the information process of integrating the images or image sequence information of the particular scene which two or more than two sensors get at the same time or at the different time to generate a new interpretation about this scene (Lian et al., 2011).The fusion of the infrared image and visible light image is an image fusion method which has been widely used in military and security monitoring field.The infrared image is generally dark, low in signal-to-noise ratio and absent in the sense of hierarchy.It has no color information.But it often has obvious target information.The visible light image is abundant in the spectral information, large in the dynamic range, relatively high in the contrast ratio and abundant in the spectral information (Ye et al., 2008).It contains rich detailed information.But it is not easy to be observed on the dark background (Yang et al., 2012).To fuse infrared image and visible light image to improve the infrared target recognition and image clarity and to obtain more detailed and accurate information enable us to accurately determine the accuracy location of the heat source even in harsh environment, which has extensive practical value in the many fields of military operations, electronic products testing, resources exploration.
At present, the image fusion based on the multiscale decomposition is a research hotspot.Commonly used methods of multi scale decomposition of the image are mainly pyramid transform, wavelet transform, contourlet transform and the nonsubsampled contourlet transform (NSCT) and other methods.The different resolution details in the decomposition structure of the pyramid transform are related to each other, the stability of the algorithm is poor.Wavelet transform overcomes the instability of pyramid algorithm.However, although the traditional wavelet transform can efficiently handle the one-dimensional piecewise continuous signals, two dimensional wavelet base formed by the one-dimensional wavelet with tensor product can only obtain the information in the three directions of the horizontal direction, vertical direction and the diagonal direction, can not accurately describe the information of image edge direction, also can not represent two-dimensional images containing a line or surface singularity.Contourlet transform has the multi-resolution and time-frequency localization characteristic of the wavelet transform, also has a high directionality and anisotropy.It can well capture the geometric structure of two-dimensional image.But, because of requiring upsampling and downsampling operation in the transformation process, so it does not have the translation invariance.There is the spectrum aliasing phenomenon in the transformation process (Ye et al., 2012).NSCT not only has the wavelet multi-resolution and time-frequency localization characteristic of the wavelet transform, but also has a good directionality, anisotropy and translation invariance.The image information in any direction can be obtained.Therefore, this paper used the method based on NSCT to perform image fusion.

NON-SUBSAMPLED CONTOURLET TRANSFORM
Non-subsampled contourlet transform (NSCT) (Tang, 2012) is a multi-scale and multi-direction decomposition method of the discrete image.It is presented based on the theory of the contourlet transform.NSCT transform can be divided into two parts of non-subsampled pyramid filter bank (NSPFB) and nonsubsampled directional filter bank (NSDFB).It separately performs the multi-scale analysis and multi-direction analysis and cancels the direct upsampling and downsampling operation on the corresponding signal component.Firstly, NSPFB is used Liu and Liu 375 to perform the multi-scale decomposition of the image in the NSCT transform process.The original image produces a low pass subband image and a band-pass subband image through a level NSPFB decomposition.
After that, each level of NSPFB decomposition can be performed by the iteration of the low pass subband image.
Then NSDFB is used to perform the direction decomposition of the high frequency component obtained by the each level of NSPFB decomposition, thus the subband images (coefficients) with the different scales and different directions are obtained.Finally, the singular points distributed in the same direction are connected into a contour segment.Among them, NSPFB decomposition makes NSCT with the multiscale.And to use the à Trous algorithm to realize NSDFB makes NSCT with the multidirection.The organic combination of both makes the NSCT with multi-scale and multi direction of contourlet transform, but also with translation invariance (Figure 1).

IMAGE FUSION BASED ON NSCT
The low frequency parts and high frequency parts can be obtained by the multi-scale decomposition of the image.The low frequency parts represent the approximate components of the image.They mainly reflect the average characteristics of the source image and include spectral information of the source image and most of energy information.The high frequency parts represent the detail components of the image, such as edge, line, region boundary.They descript structure information of the image (Sun et al., 2012).

FUSION STEP
The image fusion specific steps based on NSCT are as follows: (1) To respectively perform the J level NSCT decomposition of the infrared image I and visible light image V to obtain respective NSCT coefficients for each of the decomposed image: , where (3) To perform the NSCT inverse transform for the fused low frequency subband coefficients and the fused high frequency direction subband coefficients under the each scale layer to obtain the fusion image F.

FUSION RULES
To select fusion rule is an important link in the image fusion, fusion rules will directly affect the quality of fusion results.The paper chooses the fusion rules based on the regional energy matching degree measure.And the high frequency sub-band and the low frequency sub-band are respectively fused.Regional energy is defined as shown in Equation (1).
In which, the size of M N  is 3 3  , 5 5  and so on (the paper take Regional energy matching degree is defined as shown in Equation (2).
Set  be the threshold value of the regional energy matching degree.Usually  =0.5～1.The paper takes 0.7.

Fusion rule for low frequency part
The low frequency part is the approximate description of the source image, containing most information of the source image.Its energy accounted for a larger proportion of total energy of image.And the regional variance of the image reflects the richness of the image information in the local region (Xue et al., 2011).Therefore, the fusion method of the harmony of the energy variance decision selection based on regional energy matching degree and the weighted average is used to select the fusion coefficients of the low frequency subband.Regional variance is defined as shown in Equation (3).
Energy variance decision value is defined as shown in Equation (4).
, , , x y   , then the weighted average is used to perform the fusion of low frequency coefficients.The fusion coefficient is shown in Equation ( 6).
In which, 1 p and 2 p are adaptive adjustment factor as

Fusion rule Lowpass subband
Highpass subband 1 Simple average Take big Absolute value 2 Simple average Take big regional variance 3 Energy weighted average Take big Absolute value 4 Energy weighted average Take big regional variance 5 The proposed method shown in Equation ( 7)

Fusion rule for high frequency part
High frequency part represents the detail components of the image, including the source image edge details.For the fusion of the high frequency part, a hybrid fusion rule is used.Because what the low layer reflects is the coarse information, so the fusion method of selecting large regional variance based on the regional energy matching degree is used to maintain the relationship between the pixel neighborhoods better so as to make the edge lines more natural.Because what the high layer reflects is the details and there are great deal of independence among various information, so the fusion method of selecting large absolute value of pixel is used (Ye et al., 2008).The specific fusion rules are as follows: (1) The high frequency coefficient on the highest layer whose decomposition scale is J (here take J=4) of the fusion image is as shown in Equation (8).
(2) The high frequency coefficient on the (J-l) (l≥1) layer whose decomposition scale is (J-l) of the fusion image is fused by the method of selecting larger region variance based on the regional energy matching degree.The fusion rules are as follows: if . In which, 1 p and 2 p are adaptive adjustment factor as shown in Equation ( 10)

FUSION EVALUATION INDEX AND EXPERIMENTAL RESULT ANALYSIS
To use MATLAB 7.0 to perform programming algorithm based on the above theory and rule can verify the effectiveness of the fusion algorithm for the infrared image and visible light image.The method proposed in this paper was compared with the following several fusion methods as shown in Table 1.The following are the results of the experiments: Figure 2(a) and (b) are respectively for infrared source image and visible light image; Figure 2(c) to (g) are respectively for the fusion image using fusion rule 1~ 5 in Table 1.Four layer NSCT was adopted to perform the image decomposition in the experiment (Figure 1).
To quantitatively evaluate the fusion performance of different fusion methods for infrared image and visible light image, this paper used the standard deviation, information entropy, average gradient and cross-entropy as indexes to assess the fusion method.The standard deviation of the image reflects the discrete degree of the image gray relative to the average gray value.If the standard deviation is large, the image gray level is scattered and image contrast is large.So much information can be seen.The information entropy reflects the richness of the image information.The bigger the information entropy is, the more abundant the information in the image and the better the quality of the image fusion.The average gradient reflects the clear degree of the  image.And it can reflect the tiny detail contrasts and texture variation characteristics.It is a physical quantity describing the image clarity.The larger the average gradient, the clearer the image, and the more the information being retained.The cross entropy directly reflects the difference between the corresponding pixels of the two images.That cross entropy is smaller illustrates that the difference between the images is smaller and the information which is extracted from the original images by the fused image is more, therefore, the fusion effect is better.The comparison results are shown in Table 2.
As can be seen from Table 2 that no matter from the information entropy, cross entropy, or from the average gradient and standard deviation, the fused image obtained by using the NSCT transform and the fusion rule in this paper to perform fusion of the images is better than one obtained by using other fusion rules.

CONCLUSIONS
By means of the complementary information and redundant information provided by the infrared and visible light images, the noise-signal ratio of the image can effectively be improved and more reliable image information can be obtained.The experiments showed that the fused image obtained by using the method proposed in this paper has more details and better fusion effect.
is the high frequency subband coefficients in the lth direction under the jth scale.(2)To use certain fusion rules to perform fusion processing on the different frequency components of each sub-class to obtain NSCT coefficients of the fusion image F:

Figure 2 .
Figure 2. Infrared image and visible light image and their fusion images in different fusion rules.

Table 1 .
Fusion rules used in the fusion experiment.

Table 2 .
Performance evaluation of image fusion.