A multi-sensor system for texture based high-speed hardwood lumber

inspection

Alfred Rinnhofer, Gerhard Jakob, Edwin Deutschl, Wanda Benesova, Jean-Philippe Andreu, Giuseppe Parziale, Albert Niel

JOANNEUM RESEARCH, Institute for Digital Image Processing, 8010 Graz, Austria

ABSTRACT

A novel solution for automatic hardwood inspection is presented. A sophisticated multi sensor system is required for reliable results. Our system works on a data stream of more than 50 MByte/Sec in input and up to 100 MByte/Sec inside the processing queue. The algorithm is divided into multiple steps. Along a fixed grid the images are decomposed into small squares. 55 texture- and color features are computed for each square. A Maximum Likelihood classifier assigns each square to one out of 12 defect classes with a recognition rate better than 97 %. Depending on the defect type a dedicated threshold operation is performed for segmentation. Threshold levels and the selection of the input channel (RGB + filtered images) is the result of the former classification step. A fast algorithm computes bounding rectangles from blobs. Defect type dependent rules are used to combine rectangles. Two additional fast high resolution 3D measurement systems add board shape and 3D defect information. All defect rectangles are passing an additional plausibility check in the last data fusion process before they are delivered to the optimization computer. To guarantee a short response time, image acquisition and image processing are performed in parallel on parallel computing hardware.

Keywords: real-time image processing, lumber inspection, color grading, texture analysis, line scan, industrial inspection

1. INTRODUCTION A remarkable part of the costs for lumber production are caused by grading. In most sawmills grading is done by qualified personal - with all the unwanted disadvantages of humans doing a complicated, but monotonous job. To improve this, several automatic grading systems are in use for the inspection of softwood lumber today. For hardwood species, grading is an even more complicated task. Algorithms developed for softwood inspection will not achieve acceptable results for controlling an unsupervised production process to manufacture industrial products for international markets. [1 – 20] The typically used algorithms try to reduce the data rate at a very early level of the processing queue to allow scanning at high speed and processing on simple computing systems. The disadvantage of these systems is that information about neighborhood relations are not used. Without this essential information the achievable reliability is limited to a rather low level. From our experience in texture grading for parquet industry we know that typical hardwood quality aspects are very good represented in textural features. This approach implies that a high performance computing platform is required to allow the application of complex algorithms at high data rates.

2. SYSTEM OVERVIEW A system for the online grading of beech boards was developed for a German sawmill. The customer already had experiences with available grading systems which could not satisfy him during a test installation. The main requirements from our customer are summarized in table 1.

Table 1: User requirements

The scanner is located directly in the production line a few meters in front of a multi- blade circular saw which is used for edging. Additional adjustable saw blades are used to split wide boards up to three smaller boards to maximize the yield. The quality definitions were given by the sawmill and are not completely corresponding to any existing grading rules like NHLA grading [21]. To meet needs of the customer a high resolution color scanning on both sides is necessary. For thickness measurement, the detection of 3D defects with reliable wane detection and for extracting depth values for unplaned regions a high resolution 3D system is added for both board sides. [22-27] 2.1 Hardware layout and scanning:

Figure 1: General hardware concept

2.1.2 Color System:

The color images from top side and bottom side are recorded with two TVI Pricolor 2048 CL color linescan cameras. These cameras are equipped with beam splitter technology for color separation, 3 CCD lines with 2048 elements each, and deliver 3 x 10 Bit RGB signals. Only the topmost 8 Bits of this signal are used in our application.

User requirements:

Wood Species Beech, Fagus Sylvatika L Thickness accuracy better +/- 0.1 mm

Inspection width 600 mm, on top and bottom side

Minimum size of

3D defects

2 mm x 2mm x 1 mm (length x width x depth)

Maximum board width 500 mm Response time < 2 sec after board was leaving the scanner

Board length 1,8 to 3,4 m Board thickness 18 mm to 85 mm

Minimum gap between

boards

500 mm

Speed: 2 m/s Result classes: 14,( see table 2)

To optimization Feeding direction

Illuminatio

TVI Color linescan

IR Laser

IR Laser

IVP 3D

TVI Color linescan

IVP 3D

The resolution for these cameras is 0.333 mm/pixel along the scan line and 0.5 mm/line in feeding direction. This gives an input data rate of approximately 24 MByte/second for each of the two board faces. The boards are illuminated by four specially designed lamps, housing 5 fluorescent tubes and a mirror each. 2.1.3 3D Measurement System:

For 3D measurement a light section technique [28] which is based on the triangulation principle is applied. The laser plane is built up by two lasers (see Figure 1: General Hardware Concept). To avoid interferences (see Figure 2: Spectral Overview) between the light used for the color system infrared lasers with a wavelength of 780nm are selected.

Figure 2: Spectral Overview

The 3D measurement system uses two IVP Ranger M50 [29,30,31] cameras (1536 x 512 Pixels) to grab images from the intersection of the board with the laser plane. The longitudinal resolution (in feeding direction) is 2 mm and results in a scanning frequency of 1 kHz. The measurement field, covered by the cameras is 600 x 300mm. This yields in a horizontal (board width) resolution of 0.4 mm. As the laser impact position is computed with half pixel accuracy, the vertical (board thickness) resolution is 0.3 mm. Because of the immense input data rate of 750MB/sec (1000 images with 0.75MB each) for one camera, the IVP Ranger M50 CMOS camera with 1536 parallel A/D converters and processors were selected. These on chip processors compute the laser impact position for each column of the chip in parallel and in real time. Transfer of complete images from these cameras to the PC is therefore not necessary. This reduces the camera to PC data rate to 1.9MB/sec per camera. 2.1.4 PC Cluster:

For each side of the color system, a dedicated PC with double CPU ( 2 x 2.8 GHz) and a high speed frame grabber (Matrox Odyssey) is used. The frame grabber is equipped with an additional CPU (Power PC 1GHz) and a hardware based processing unit for fast neighborhood operations. In our algorithm this unit mainly is used to run convolutions. The third PC processes the data from the IVP cameras, works on data fusion, system control and communication to the optimization computer.

3. METHODS The principle idea for the proposed method is the assumption that texture analysis is better suited to describe all the different types of hardwood quality aspects than traditional solutions [REF]. These solutions try to avoid this step because of its high computational costs. Figure 2 shows the principle computation steps and the mapping to the computation hardware.

Figure 3: Algorithm layout and mapping to computation hardware 3.1 Texture feature generation:

The first step is the localization of the board in the image. Overlapping squares are placed over the entire board surface. A set of convolutions with user defined kernels in combination with a three level image pyramid are applied. From the original color layers and the filtered images on different pyramid levels, a set of 155 texture features is computed for each of the squares. Additional color information in the CIE L*a*b* color-space [32, 33, 34, 35, 36], describing the overall color of the board is added to the features. An automatic feature selection process, based on genetic algorithms, is used to select the best suited features for online classification. The optimum number of features to use can vary for different training runs between 45 – 75 features, depending on increased input data and different settings, for priorities to maximize hits for specific classes, or to minimize false positive classification results. This step even increases the data rate up to 50 MByte/Second for each side. To allow processing under real-time requirements, the data reduction for the image pyramid (a fast 2 x 2 mean operation) and the convolutions are computed on the neighborhood operation unit (OASIS) of the Matrox Odyssey frame grabber. 3.2 Training and Classification:

During a supervised training phase more than 29.000 squares were addressed to 19 internally used classes. The board images are displayed in a tool together with the squares as an overlay. Squares can be selected and added to the classes. When ever a new computation of the features is necessary, the features of each square used for training is added to the corresponding feature class. This training process is an interactive procedure. After defining several hundreds of samples, complete boards are graded and the results are checked by humans. If typical false classifications are found, the corresponding squares are added to the correct class and the next run starts until one reaches a state where further improvement seems not be possible.

A Maximum Likelihood classification gives as result the most likely class for each square. The classification can achieve an overall rate of correct classification of more than 97% for 12 different output classes computed on approximately 23000 sample squares. The number of internally used classification classes is higher than the number of output classes because this will allow a better adjustment in the following segmentation step for several inhomogeneous defect classes. A short example can demonstrate this fact: Although it is not necessary for human graders to discriminate between tiny, small cracks and open splits, the segmentation of small cracks can be significantly improved working on dedicated filtered images. But these filtered images will not give very good results for open splits. So both internal types can be treated differently to allow optimum processing and produce the best possible correlation of the automatic segmentation with human interpretation after object merging. So the classification not only provides information about the defect type, it also supports the segmentation step for different sub- types of a defect.

3.3 Segmentation:

The type of defect is already known from the classification. Depending on the defect type the texture features can be used to set thresholds for the segmentation on different image channels (RGB and filtered images). For the best channel selection and the parameter adjustment typical samples for each defect type are segmented by hand. A linear regression model is applied on all reasonable channel/feature combinations. Adequate definitions of the cost function for different defect types are necessary to find an optimum compromise for maximum detection rate and to suppress false positive objects. Special tools were developed to allow visible feedback of the achieved training results because rating of the solutions only based on the output of the cost function often leads to unwanted results as long as the cost function is not perfect. Logical and morphological operations fill the gaps and improve the quality of the segmentation. For several types of defects no segmentation is possible and the complete square from the texture classification is assigned to the corresponding output class.

Figure 4: Result of different processing steps:

Upper row: left: original image, center: result of texture analysis, right: segmentation result bottom row: left: bounding rectangles, center: result after merging, right: board samples

3.4 Merging:

A rule based merging algorithm is used to reduce the number of objects delivered to the optimization tool. Merging criteria are: Defect type Defects size in x and y direction Distance between objects in x and y direction Percentage of the new bounding rectangle area covered by segmented defects Parameters of the merging process can easily be changed by the system operator to adjust the level of details according to the needs of the optimization process. Defects are always described by bounding rectangles, as well as the shape of the board is described by a set of rectangles. Based on defect type, size, shape and position on the board small objects are rejected. 3.5 3D Measurement

The 3D system measures three dimensional properties of the board. 3.5.1 From Camera to World Coordinates:

To transform detected laser position data into real world coordinates a geometric transformation is applied to each found position. Parameters of this rectification operation are determined in a calibration step. This step requires a calibration target (see Figure 5) with known world coordinates. Therefore calibration must be performed offline, i.e. during production breaks – but only after changes in geometry or after replacement of mechanical components.

Figure 5: Calibration Target (blue lines), Field of View of top and bottom camera (black lines), laser rays (red lines).

3.5.2 Shape Properties:

Especially to optimize bundling of the final board stacks (no bended boards outside, minimal gaps between boards of a layer, …) at the end of the production line, following shape properties are computed:

Left and right edge of the board, Enveloping hull, Minimum (aligned) enveloping hull, Maximum curvature of left and right edge, Board width at predefined positions

3.5.3 Dimensional Defects:

For each scanned line and separated for top and bottom side of the board a polynomial of 2nd order is fitted through world coordinates. Dimensional defects are computed as local deviation to this polynomial. Combining dimensional defects (deviations from fitted polynomial) with shape properties allows indication of wane defects. Additionally a ruler is applied to the top and bottom sides of the board. The maximum distance between the supporting points of this ruler to the surface (ignoring already known dimensional defects) of the board is calculated as measurement value for the cup property. 3.5.4 Thickness Defects:

As dimensional defects not cover deviations in thickness, thickness defects are calculated as local deviation from the average thickness of a line. Whereas thickness values itself are simply computed as distance between top and bottom world coordinates. Board thickness is finally computed as median of thickness values from defect free regions for several predefined zones on the board. These thickness values are used to detect any deviations from the nominal board thickness to find possible wrong adjustments of production machineries at an early state. The user has the possibility to adjust limits for nearly all of the properties and can therefore decide by himself, which size of defects should be indicated to external optimization. 3.6 Data fusion

Further rule based merging operations are performed to combine the 3D results with the surface defects for the final result. This step again allows the validation of the surface classification with 3D data for defects which have textural features and additional 3D information (like wane). Data reduction can be achieved by merging the information sources and produce a single output object. Typical examples for this processing step are unplaned regions on the board. For the color cameras these regions are classified correctly regardless of the depth of the defect. The 3D system can deliver the depth for each point within the region, but there is no information about the defect type. The user has the ability to adjust a parameter to set the threshold for generating an “unplaned region defect” based on depth variation. The size of this defect is defined by the 3D system and the type information comes from the color systems. Both data are merged to one defect object which is transmitted to the optimization PC.

4. RESULTS AND DISCUSSION The developed system allows grading of beech boards with a width of up to 500 mm at full production speed of 2 m/sec. The extracted information about board shape and board quality corresponds very well to results of human graders. Since optimization and board grading was not within the scope of our company, no final results of the correlation between manually graded boards and computed board classes can be given. We will present results from several processing steps instead. The first step which can be evaluated is the performance of the texture classification. Nearly 23000 sample squares were used to train the classifier. Running a classification on the training data gives an overall percentage of 97.3 % of correct classifications. The meaning of the different classes and the equivalences of the classes are given in table 2. The learning rate up to 55 features and the corresponding confusion matrix is given in figure 6 and table 3 respectively.

CLASS NAMES AND EQUIVALENCES A B C D E F G H I J K L M N O P R S T

A Clear wood X X

B Curle Grain

X X

C Cracked knot X X X

D Black knot X X X X

E Wane X

F Pin knot X

G Mineral streak X

H Bark inclusion X X

I Small crack X X X

J Wide crack X X X

K Planing defect X

L Pith, open X X X

M Pith X X

N Red heart, stripe shape X X X

O Blue decay X

P Sound knot X X X

R Insect damage X

S Dark clear wood X

T Red heart, uniform color X X

Table 2: Classes for texture classification (X for equivalent classes)

% Correct classified, Training with 22941samples

75

80

85

90

95

100

1 6 11 16 21 26 31 36 41 46 51

Number of Features

% Correct classified

% Correct classified

Figure 6: Learning rate of the Maximum Likelihood classifier for texture classification

Table 3: Confusion matrix for texture classification (for class names see table 2)

Running the described algorithm on a huge number of boards brought up several questions where improvements could be achieved. Red heart: Red heart covering the whole board face is extremely hard to detect. If no borders of the red heart are visible, the texture is not different from clear wood. Therefore color seems to be the only chance for discrimination. Unfortunately measurements with a precise spectrometer gave exactly the same L*a*b values for decay regions and clear wood regions both Sound knots: Although sound knots can be detected with a limited certainty using textural information it becomes nearly impossible to define the border of the knot precisely. But this is nearly impossible for some samples also for human graders. Trying to improve the detection rate for light red decay makes color as a decision criterion more important. This leads to the unwanted effect that false positive sound knots appear in regions where color and shape of grain show similarities with light sound knots. Due to the overlapping placement of the square regions of interest for texture classification, more than one classification result will occur in these overlapping regions. The data merging process has to handle this ambiguous information correctly to avoid false positive objects. A typical example is the start point of a crack, which will be classified as pin knot, if there is just a short part of the crack inside of a square. The merging step has to merge the crack with the pin knot to produce one crack and delete the pin knot result.

REFERENCES [1] R.W. Conners, C.W. McMillin, K. Lin and R.E Vasquez-Espinosa, “Identifying and locating surface defects in wood: part

of an automated lumber processing system”, IEEE Transactions on Pattern Recognition and Machine Intelligence 5, pp. 573-583, 1983.

[2] S. Tatari and W. Hättich, “Automatic recognition of defects in wood”, SPIE 804, Advances in Image Processing, pp. 229-236, 1987.

[3] P. J. Sobey and E.C. Semple, “Detection and sizing visual features in wood using tonal measures and a classification algorithm” Pattern Recognition 22, pp. 367-380, 1989.

[4] T-H. Cho, R.W. Conners and P.A. Araman, “A computer vision system for analyzing images of rough hardwood lumber”,10th International Conference on Pattern Recognition, Atlantic City, pp. 726-728, 1990.

[5] W.Pölzleitner, G. Schwingshakl. Real-time classification of wooden boards. In SPIE vol. 1384, High Speed Inspection Architectures, Barcoding and Charakter Recognition, pp 38-49, 1990.

[6] M. Månson. Technical Achievements Offering New Grading Prinziples for Softwood Lumber. Proc. 4th Int. Conference on Scanning Technology in the Wood Industrie, San Francisco, Oct. 28-29, 1991.

[7] W. Pölzleitner. Real-time Syntactic Segmentation with Applications to Images of Wooden Boards, In Proc. Workshop on Digital Image Processing and Computer Graphics, Vienna, Academy of Science, Oct. 1991

[8] Araman, P. A., D. L. Schmoldt, T. H. Cho, D. Zhu, R. W. Conners, and D. E. Kline. 1992. Machine vision systems for processing hardwood lumber and logs. AI Applications 6(2):13-26.

[9] W. Pölzleitner, G. Schwingshakl. Real-time surface grading of profiled wooden boards, In Industrial Metrology, vol.2, no.3-4, pp195-218, 1992

[10] T. Ojala, M. Pietikäinen and O. Silvén, “Edge-based texture measures for surface inspection”, 11th International Conference on Pattern recognition, The Hague, The Netherlands, vol. 2, pp. 594-598, 1992.

[11] Conners, R. W., T. H. Cho, C. T. Ng, T. H. Drayer, P. A. Araman, and R. L. Brisbin. 1992, A machine vision system for automatically grading hardwood lumber. Industrial Metrology 2:317-342.

[12] Erik Åstrand and Mikael Rönnqvist. Cross Cut Optimization of Wooden Boards Based on Automatic Defect Detection Automatic Inspection of Sawn Wood. In proc. SPIE, vol.2063, Visions, Sensors, and Control for Automated Manufacturing Systems, Boston Sep. 7-10, 1993

[13] J.F. Soest, P.C. Matthews, B.G. Wilson: A Simple Optical Scanner for Grain Defects, Proceedings 5th International, Conf. on Scanning Techn. & Process Control for the Wood Industry, Atlanta USA 1993

[14] O. Silvén and H. Kauppinen H, “Color vision based methodology for grading lumber”, The 12th International Conference on Pattern Recognition, Jerusalem, Israel, vol. 1, pp. 787-790, 1994.

[15] E. Åstrand and A. Åström, “An intelligent single chip multisensor approach for wood defect detection”, 12th ICPR, International Conference on Pattern Recognition, Jerusalem, Israel, vol. 3, pp. 300-304, 1994.

[16] Rinnhofer A., Deutschl E., "A Multi Processor System for Timber Automatic Grading using Texture Features", In Proc. Workshop Machine Perception Applications, IAPR TC8, Graz, Austria, September 1996.

[17] E. Åstrand: Automatic Inspection of Sawn Wood, Departement of Electrical Engineering Linköping University Sweden, Dissertation No.: 424 1996, ISSN 0345-7524

[18] R. Massen: Verfahren und Vorrichtung zur optischen Prüfung eines Holzprüflings, Patentschrift Nr.: DE-19609045, 1997 [19] Rinnhofer A., Deutschl E., "An X-ray Real Time System for Timber Automatic Grading", In Proc. of "The 10th

Scandinavian Conference on Image Analysis", IAPR, Lappeenranta, Finland, 1997. [20] Rinnhofer A., Deutschl E., „Multi Processor based Image Processing Applications in Wood Industry“, VCF Machine

Vision Seminar, 26. – 27. 5. 1998, Espoo, Finnland [21] National Hardwood Lumber Association, “Regeln für die Vermessung und Sortierung von Nordamerikanischem

Laubschnittholz, P.O. Box 34518, Memphis, Tennessee [22] R. M. Haralick, K. Shanmugam and I. Dinstein. Textural Features for Image Classification. In IEEE Transactions on

System, Man, and Cybernetics, vol. SMC-3, no 6, pp. 610-621, Nov. 1973. [23] A. J. Koivo, C. W. Kim. Classification of Surface Defects on Wood Boards. In IEEE International Conference on System,

Man, and Cybernetics, pp.1431-1436, Oct. 1986. [24] A.J. Koivo and C.W. Kim, “Robust image modeling for classification of surface defects on wood boards” IEEE

Transactions on Systems, Man, and Cybernetics 19, pp. 1659-1666, 1989. [25] John Y. Hsiao, Alexander A. Sawchuck. Supervised Textured Image Segmentation Using Feature Smoothing and

Probabilistic Relaxation Techniques. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. II, NO.12, Dec. 1989

[26] Jeong Hwan Lee and Nam Il Lee. A fast and adaptive method to estimate texture statistics by the spatial gray level dependence matrix (SGLDM) for texture image segmentation. Pattern Recognition Letters , 13, Apr. 1992

[27] C.W. Kim and A.J. Koivo, “Hierarchical classification of surface defects on dusty wood boards”, Pattern Recognition Letters 15, pp. 713-721, 1994.

[28] A. Niel, R.Knorr: "Non-cotact dimensional measurement of hot and cold profiles and bars." Steel World 1997 Vol.2 No.1. [29] Integrated Vision Products, Home Page: http://www.ivp.se/html/ivpmain.html [30] IVP Ranger M50/M40 User Manual, V2.0.0, 2003 [31] IVP Ranger M50/M40 Hardware Manual, V2.0.0, 2003 [32] G. Wyszecki and W. S. Stiles, Color Science: A Wiley-Interscience Publication, 2000. [33] P. Geladi and H. Grahn, Multivariete Image Analysis: John Wiley & Sons Ltd, 1966. [34] H. Kauppinen and O Silvén, “A color vision approach for grading lumber”, Theory & Applications of Image Processing II -

Selected papers from the 9th Scandinavian Conference on Image Analysis, Borgefors G (ed.), World Scientific, pp. 367-379, 1995.

[35] H. Kauppinen, O. Silven: The Effect of Illumination Variations on Color-Based Wood Defect Classification, 13th International Conference on Pattern Recognition (13th ICPR) Vienna, Austria 1996, S. 828-832

[36] H. Kauppinen, H. Rautio, O. Silvén: Non-segmenting defect detection and SOM based classification for surface inspection using color vision;; Conference on Polarization and Color Techniques in Industrial Inspection, (SPIE 3826), June 17-18, Munich, Germany, 1999, pp. 270-280.

---------------------------------