Development of a biomedical virtual microscopy slide scanner using a line scan charge-coupled device ( CCD ) camera

This study developed a high speed biomedical tissue slide scanner using a line scan CCD camera to capture continuous images and complete a virtual slide of a panoramic image in real time. The virtual slide provides highly reliable data for pathology researchers in determining the pathogenic origin of biomedical tissues. The scanning system consisted of a motion control module, an image acquisition module, an image-processing module and a human-machine interface. During the image stitching process, many factors decrease the quality of the image, such as positioning errors and environmental effects. The purpose of this study was to speed scanning and obtain highly reliable images of biomedical tissues. This study applied pattern matching technique to image stitching and used a colorfilter to resolve problems of lack of back lighting uniformity. This study also proposes that using autofocus methods to overcome the lack of depth of field (DOF) problems results in rapid and high-quality scans of biomedical tissues.


INTRODUCTION
The rapid development of biotechnology in recent years has led to the digitalization of medical equipment to satisfy the demand for fast, convenient, sanitary, and safe medical operations.Factors related to pathological treatment have become increasingly important in the medical system, and demand for more accurate pathological analysis and diagnosis has increased.As a result, pathological observation using a traditional microscope no longer meets the requirements of modern medical science and must be improved to obtain digital data from tissue slides in real-time.Due to the difference in medical resources between urban and rural areas, and the fact that diagnosis of cancerous tissues requires the skills of experienced pathologists, medical facilities in remote areas are often incapable of immediate diagnoses and thus often miss the optimal window of time for treatment.If virtual slides could be produced (Darren, 2009) using a scanner, and transmitted over the Internet to various *Corresponding author.E-mail: D9703208@mail.ntust.edu.tw.Tel.:+886-2-27333141 ext.7348; fax: +886-2-27376460.locales, medical staff could obtain support for pathological diagnosis from distant locations.Figure 1 shows the potential improvement to safety resulting from the use of virtual slides.
Due to the limitations of CCD resolution and pixel size, capturing the entire tissue in one image is not possible.Therefore, multi-imaging stitching techniques are required to reveal the panoramic image.Numerous past studies have proposed image stitching methods.Some of these methods process images using mosaic (Zhao and Yang, 1999, Kirby et al., 2006, Kim et al., 2003) and eliminate stitching seams at image contact points after extensive calculations.However, these methods are limited to the integration of a small number of images and are not suitable for high pixels image processing or for use in systems, which capture images in real time.Additionally, cell integrity in the tissue slides is very important for pathological diagnosis.Excessive image processing may distort original image data in tissue slides, resulting in diagnostic errors in pathological analysis.As an example, Figure 2 shows the image stitching results of a tissue slide scanner.The image within the red square has been considerably blurred due  to excessive processing, causing pathology researchers to misinterpret the image.
The methods proposed in this study were based on the perspective of pathological analysis, which combines techniques of precision mechanics, motion control and image processing.Using pattern matching (National  Instrument Corp., 2007) reveals the relative positions of overlap in two images and decreases image defects caused by mechanical errors.This study also utilizes color-filter to resolve issues of chromatic aberration generated by unevenness in the color of each image during stitching (Gonzalez and Woods, 2002;Avanaki, 2009).By using pattern matching and color-filter, the original appearance, color, and size of the tissues can be retained.

Experimentation with development of a biomedical tissue slide scanning system
Figure 3 shows 75 × 25 mm tissue slides, which depict 30 × 24 mm tissue slices.The slides were 1 mm thick, and the tissue slices were 3 µm thick.The tissue cells measured approximately 8 to 20 µm, as shown in Figure 4. Currently, each pixel of a CCD camera is about 2 to 10 µm and requires magnification with microscopes to reach the standards required for pathological diagnosis.Because an objective lens magnifies a tissue image, the FOV (field of view) of the tissue image will be very small.Therefore, CCD camera technology cannot facilitate observation of the entire tissue slice.Using a 1/2 inch CCD camera with a magnification of 20 × as an example, the resulting FOV would be approximately 1/40 inch.However, tissue slices have a minimum size of 3 × 3 mm, indicating that a CCD camera is not capable of capturing the entire image in/of a tissue slide in one shot.Therefore, the completion of a panoramic image requires the superposition of many sub-images.

Experimental setup of the biomedical tissue slide scanning system
The system setup in this study included a microscope, a line scan CCD camera, a XY-axis positioning motion control system, and a Z-  axis automatic focusing system, as shown in Figure 5.

Image acquisition module of microscopic
The microscope used in this research was a Nikon TS100 inverted live-cell observation microscope.The magnification of the objective lenses was 20 × and the depth of field (DOF) was about 2 to 3 µm.
The CCD camera used a line scan CCD generated by Basler.The effective pixels were 2040.Pixel size measured 10 µm.A LED light bulb provided the backlight.The system resolution was 0.5 µm.
Figure 6 shows the framework of the image system module.

Module for precision positioning mechanism
The two major functions of the precision positioning mechanism module were horizontal dual-axis motion and automatic focusing, which allowed the slides to accurately move to the next image capturing region.The positioning system used in this study included a dual-axis micro-step positioning stage and Z-axis automatic focusing system, as shown in Figure 7.In the XY-axis positioning stage, the resolution of the step motor was 0.018 degrees and the linear scale was 0.1 µm.The pitch of the lead screw was 1 mm.The resolution of the Z-axis automatic focusing step motor was 0.18°.The linear scale resolution of the Z axis was 1 µm (Kuo et al., 2008;Nain and Trang, 2005).

Auto focus methods
This study conducted autofocus prior to the scanning of the slide by applying discrete cosine transform (DCT) and analyzing the spectral characteristics of the tissue slide with the 3-point approximation algorithm proposed by our research.The purpose of this algorithm was to ensure the quality of the scanned image.The basic framework for the autofocus process is shown in Figure 8 (Baina and Dublet, 1995;Chern et al., 2001).

Sharpness factor of AF measurement criteria: DCT algorithm
The results of one-dimensional DCT are used for one-dimensional digital signal processing, similar to discrete Fourier transform (DFT).However, with regard to computation, DCT is faster and less complex than DFT as it involves only real numbers (whereas the DFT involves both real and complex numbers).Two-dimensional DCT is an orthogonal transformation whose computation transforms an image from the spatial domain to the frequency domain for analysis.The resulting DCT coefficient matrix from such transformation depends on the spatial characteristics in the image.
Figure 9 shows a DCT diagram.f(i and j) is a 2D M × N image matrix.The equation for a 2D DCT is shown in Equation 1, with F(u and v) as the DCT output coefficients (Strang, 1999).

=
When u = 0, v = 0, DCT can be expressed as indicated in Equation 2:

Analysis of tissue slide spectral diagram
After understanding the properties/characteristics of the DCT diagram, this study separately used two tissue slide images, one in focus and the other out of focus, for DCT transform while analyzing their corresponding spectral diagrams.The case involving the out of focus image is shown in Figure 13, and the case of the in-focus image is illustrated in Figure 14.Most of the energy (the parts of the image where the color tended toward white or blue) in the two images was concentrated in low-frequency region(s) in the top left    After examining the spectral diagrams of the in-focus and out-offocus images of tissue slides, the absolute values of all DCT coefficients were added to SFall, as shown in Equation 3. Additionally, Figure 15 shows the absolute values of the DCT coefficients in image sections with relatively large changes in energy were added to SFsel, as shown in Equation 4. Lastly, SFall and SFsel were used independently as sharpness factors in focusing.

∑∑
Where ( ) was the Gaussian symbol, image size was indicated by M x N, and F(u and v) was the DCT coefficient.
Using the two sharpness factors (SF) from the spectral analysis, this study employed the function of global search at 5um intervals to acquire images and compute image sharpness.This method facilitates the determination of focus platform position and the focus curves of SFall and SFsel.The corresponding best point of SFall and SFsel was simultaneously revealed, as shown in Figure 16.The figure shows that the curve of SFall exhibited fluctuations at 2.005 mm on the focus platform, possibly due to its proximity to the DOF range.

Focus search method
The aforementioned discussion shows that when the focus point is approaching, if the distances above and below the focus are equal, the SF value would be very close to the focus point.Based on such a relationship, this study referred to the mountain climbing search method (He et al., 2003) and proposes that the 3-point approximation method is applicable to tissue slides.The principles of the 3point approximation method are such that when the SF difference changes its sign (positive or negative), the highest SF and its two adjacent points are arranged according to their values.For example, if the highest SF is SFmax, the intermediate is  2 This study utilized the best focus program, as illustrated in Figure 18.The program commenced execution by first carrying out zero return, after which CCD began capturing images while calculating the SF value.When the SF difference began to change its sign, researchers arranged the last three points, scanned as SFmax, SFmid, and SFmin, according to size.This study acquired the proportionality S and scanning increment X2 from motor reversal.In focus search, because the focus position is a variable, two types of exist for the relationship between the focus position and SF, as shown in Figure 17.In scenario 1, 2X2 from the reversal of the focus platform position corresponding to SFmid is used as the starting point for the search round.In scenario 2, 2X2 from the reversal of the focus platform position corresponding to SFmin is used as the starting point for the second search round.
In the aforementioned search, if S>T, then the focus platform position of SFmax is determined as the focus point.If S<T, the motor scans recursively until S>T.During recursion, should the course of X2 DOF, it stops when the SF difference changes its sign.The focus platform position corresponding to SFmax is the focus point.The threshold value T can be determined through the search intervals.When the search interval is 0.03 mm and S = 0.9, X2 is calculated as 0.0285 mm from Equation 6, a difference of 1.5 µm from the value of in the initial interval.In the DOF, when the focus point interval is 0.03 mm, the threshold value is set to 0.9, which is within the permissible range.

methods of virtual slide
The purpose of the panoramic seamless imaging method discussed in this chapter was to eliminate image stitching boundaries and provide better image continuity and color uniformity.This study applied the pattern matching technique (National Instrument Corp., 2007) to image stitching, and extract an image from the blank slide to use a color-filter, so as to eliminate chromatic aberration between two sub-image boundaries.When the images were captured by the developed system, images I2, I3…In resulted.Some overlaps existed in images I1I2, I2I3, and so on.The overlapping parts of two images were pasted together to produce a   pasting with pattern matching are illustrated in Figure 20.

Pattern matching technology
The pattern matching process in this study used pattern and input images to analyze and recognize the matching images.The positions and shifted angles of the recognized images were calculated and obtained from the process.Assume a feature image w(x and y) of size K × L, within an image f(x, and y) of size M × N, where K<M and L<N, as shown in Figure 21.The correlation between w(x, and y) and f(x, and y) at a point (i, and j) is calculated using Equation 7.

( ) ( ) ( )
Where i = 0, 1, … M-1, j = 0, 1, … N-1, and the summation encompasses the region in the image where w and f overlap.C is the correlation value.The maximum value of C indicates the position of optimal matching of w to f.
Pattern matching normally begins mathematical calculation at (0, and 0) and terminates at (M, and N).After calculation, statistical analysis revealed that the coordinate with the largest C value was the most desirable image position.After searching for the best matching position results, the correlation coefficient was computed using Equation 8.The value of this correlation coefficient revealed the correlation between the target image and original input image.The value of the best-matched result of the number was then determined as -1 < r (i, and J) <1.  ( ) Where w is the average intensity value of the pixels in the template w, and f is the average value of the region coincident located in w.

Image stitching in the scanning system
The camera used in this research was a line scan CCD camera, which is capable of capturing continuous images in a straight line.However, a complete virtual slide includes more than one image; therefore, image stitching is required.This study used pattern matching to compare two adjacent images.Researchers then obtained the relative positions of such images as the basis for image stitching.Figure 22 illustrates the tissue slide scanning diagram.Each time CCD completed a scan; this scan itself according to the starting point of the next scan, which formed the overlapping parts in two adjacent images.Figure 23 shows that overlapping the two images I1 and I2 to stitch for P × V units.ROI (region of interest) for the training pattern was selected from the downward side of I1 (K × L), and the ROI of the matching region was selected from the upmost side of I2 (M × N).The position errors and the linear guideway errors of the selected region must also be considered.In the X direction of the image, the image size was U>M>K.In the Y direction, the image size was N>P>L.
The precision of the position of the step motor and linear scale used in this experiment was 0.1 µm.However, errors resulting from mechanical and motor positioning still existed under 20 × magnifications.Figure 24 shows that when I2 was shifted upwards and overlapped with P, the two images were not completely stacked.Pattern matching used the two shifting quantities, xm and ym, to achieve complete matching of the overlapping region and decrease the stitching seams caused by mechanical error, compensated quantities of shifting ∆x and ∆y.Shift I2 using quantities, xm and ym:   in background color due to light scattering.As a result, discontinuity in coloring appeared at the parts where images joined.Figure 25 shows a stitched image before color correction.Tissue slides store a large amount of data.In the interest of reducing computational time and changes in tissue cells, the frequency domain algorithm known for its complexity was not considered.Arithmetic computation was carried out in the spatial domain.Under conditions, which did not affect changes in tissue cells, we conducted image correction while adequately enhancing appropriate parts of the tissue cells.Prior to scanning, this study used line scan CCD to scan a 2040 × 1 pixels image from a blank slide, as shown in Figure 26.This image was the color-filter, Ic(x).
Using the aforementioned color-filter, Ic(x), researchers uniformized the color of the original image through filter calculation.Researchers then inputted the image as Iinp(x, and y); the image enhancement constant was kT; the background color equalization was k = 255; and the output image after enhancement was Iout(x, and y), as indicated in Equation 9.
After subtracting Iinp(x, and y) from the color filter Ic(x), the remaining elements were the tissue characteristics, for all background colors had been filtered.This study added k to smooth out the background colors, and use kT to adjust the intensity of the slide illustrating tissue characteristics, the proportionality gain value to kT is indicated in Equation 10.As shown in Equation 9, Iinp(x,y) and the color filter Ic(x) can equalize background color but change the original color of the tissue, so that must be adjusted to its original color.In this equation, the numerator of kT is the average pixel value of tissue in Iinp(x,y), t I , after adding the average of Ic(x) subtracting k, the    ( )

Auto focus results from the 3-point approximation algorithm
Researchers placed the tissue slide onto the precision fixtures and applied auto focus to the characteristics at the four corners of the slide.The focus point of these characteristics is indicated in Figure 27.The system began with Z-axis zero return and the position was set to Z = 2.0 mm after commencement of focusing.The search intervals were X = 0.03 mm and the threshold was set to T = 0.9.The focus values, SF sel platform position, and sharp image were recorded, as indicated in Tables 1 to 4 and Figures 28 to 31.

Seamless image results from pattern matching and the proposed color correction algorithm
In this system, the CPU was Intel Core2 Dou E6750 2.66GHz, the RAM was DDR2 2GB, and the programming language was LabVIEW 8.5.A circular tissue approximately 3 mm in diameter required 10 images using a line scan CCD camera, as shown in Figure 32.The size of each image was 11500 × 2040 pixels.In motion control, the quantity of every movement of the X-axis stepping motor was set to 90% of the FOV Y , to create a 204-pixel overlapping region for two adjacent images.The training       region w(x, and y) was 800 × 150 pixels, the matching region w(x, and y) was 1000 × 250 pixels.According to the seamless image result, if pasting each image generates seams, some cells will be omitted.Thus, the pattern matching technique is necessary for image stitching processing of the entire tissue slide.The average of stitching time was 0.35 s for each seam between two adjacent images.Figure 33 illustrates the stitching results.
Figure 33 shows that although the images of the tissue cells were stitched seamlessly, phenomena of nonuniform color and discontinuous background color were still apparent.Using the color correction algorithm developed by this study, researchers used I c (x) and I inp (x, and y) to separately extract three planes: R, G, and B.

DISCUSSION
According to the auto focus result, each position indicates that SF value is 4.531 × 10 6 when the focus platform position is 2 mm; SF is 7.919 × 10 6 when the focus platform position is 2.03 mm; SF is 4.262 × 10 6 when the focus platform position is 2.06 mm.The proportionality exceeded the threshold value of 0.9 and the focus program determined the correct focus point to be at 2.03 mm.Sharp images are also observable at positions 1 to 4, indicating that a search interval of 0.03 mm is sufficient to achieve autofocus on every point in the slide.Show that the 3-point approximation algorithm has a large search range.The method compared with other search methods, it has more robustness.Using the color filter proposed by this study, a new color image can be obtained through utilizing conversion formula and the computational results from the three planes described earlier.Figure 35 shows the results of applying color equalization and image enhancement to an image after pattern matching, which comparison with Figure 33.The colors of the tissues were closer to their original colors.

Figure 1 .
Figure 1.Potential improvements to safety with virtual slides.

Figure 5 .
Figure 5. System for scanning slides of biomedical tissue.

Figure 6 .
Figure 6.Framework of the image system module.

Figure 8 .
Figure 8. Flow chart for the basic framework of the autofocus program.

Figure 10 .
Figure 10.Distribution characteristics in a spectral diagram.
DCT image data are comprised of several 2D cosine waves, in which u and v represent the vertical and horizontal cosine waves, respectively.The frequency variation of light intensity depends on the position of u and v, as shown in Figure10.In other words, the values of u and v increase as the frequency of light intensity in the original image increases.Discrete cosine transform coefficients with relatively high values are concentrated in highfrequency regions.When light intensity is evenly distributed, however, DCT coefficients with higher values tend to concentrate in low-frequency regions.If many vertical edges are apparent in the original image, DCT coefficients with relatively high values are distributed in regions of vertical energy.Figure11shows an 8 x 8 black and white image as an example.The black and white image intervals and their corresponding gray values vary; light intensity shows alteration as well.The characteristics of the image included vertical, horizontal, and crossing edges.In the corresponding spectral diagram, all the DCT coefficients with relatively high values, apart from DC value, are distributed/clustered around high-frequency regions and regions where vertical and horizontal energies intersect, as shown in Figure12.In these regions, the values of the DCT coefficients are absolute.Lighter color in the spectral diagram implies a higher corresponding coefficient in the spectral graph.Darker color in the image implies a coefficient with a smaller value.

Figure 11 .
Figure 11.8 x 8 black and white image interval.

Figure 12 .
Figure 12.Spectral diagram of a 8 x 8 black and white square image.

Figure 17 .
Figure 17.Diagram of focus search scenarios.

Figure 19 .Figure 20 .
Figure 19.Two images overlapped by w.(a) shows input image I1; (b) shows input image I2; (c) shows the pasting of I1 and I2; b is the characteristic of I1 and I2..

Figure 21 .
Figure 21.Diagram of image pattern matching.

Figure 23 .
Figure 23.Digram of pattern matching of tissue cell image.

Figure 24 .
Figure 24.Compensated quantities of shifting after image stitching.

Figure 27 .
Figure 27.Focus points of tissue slides.

Figure 32 .
Figure 32.Results of image scanning.

Figure 33 .
Figure 33.Result of each image stitched with pattern matching.

Figure 34 .
Figure 34.Color correction process (a) R plane (b) G plane (c) B plane and (d) processing result.

Figure 35 .
Figure 35.Result of image stitched with color correction.

Table 1 .
Focus informations of point 1.

Table 2 .
Focus informations of point 2.
denominator of kT is the average pixel value of tissue in t I .

Table 3 .
Focus informations of point 3.

Table 4 .
Focus informations of point 4.