adfa, p. 1, 2011. © Springer-Verlag Berlin Heidelberg 2011

Automatic Fish Segmentation on Vertical Slot Fishways Using SOM Neural Networks

Álvaro Rodriguez1 Juan R. Rabuñal2 María Bermúdez3 Alejandro Pazos1

1 Dept. of Information and Communications Technologies, Faculty of Informatics, 2 Center of Technological Innovation in Construction and Civil Engineering (CITEEC),

3 Dept. of Hydraulic Engineering, ETSECCP, University of A Coruña. Campus Elviña s/n 15071 A Coruña

arodriguezta@udc.es juanra@udc.es mbermudez@udc.es apazos@udc.es

Abstract. Vertical slot fishways are hydraulic structures which allow the upstream migration of fish through obstructions in rivers. The appropriate design of these should consider the behavior and biological variables of the target fish species and currently existing mechanisms to measure the behavior of the fish in these assays, such as direct observation or placement of sensors on the specimens, are impractical or unduly affect the animal behavior.

This paper studies the application of Artificial Neural Networks to the problem of automatic fish segmentation in vertical slot fishways. In particular, SOM Neural Net-works have been used to detect fishes using visual information sampled by an under-water camera system. A ground true dataset was designed with experts and different approaches were tested providing promising results.

Keywords: ANN, Fish-Detection, Segmentation, SOM.

1 Introduction

The construction of engineering works in rivers, such as dams or weirs, alters the ecosystem of rivers, causing changes in the fauna and flora. One of the most im-portant effects is the obstruction of fish migration.

Out of the various solutions employed to restore fish passage, some of the most versatile are known as vertical slot fishways. This type of fishway is basically a chan-nel divided into several pools separated by slots.

An effective vertical slot fishway must allow fish to enter, pass through, and exit safely with minimum cost to the fish in time and energy. Thus, biological require-ments should drive design and construction criteria for this type of structures. Howev-er, while some authors have characterized the flow in vertical slot fishways [1-4] and others have studied fish swimming performance [5, 6], the analysis of the real fish behavior in full-scale physical fishway models are scarce in the literature, so the actu-al behavior of the fish within a fishway is practically unknown.

The detection of fishes in images, is a problem widely studied in the literature.

Some examples are techniques based on color segmentation such as [7] where fluo-rescent marks are used for the identification of fishes in a tank or in [8] where color properties and background subtraction are used to recognize live fish in clear water.

Other used techniques are stereo vision [9], background models [10], shape priors [11], local thresholding [12], moving average algorithms [13] or pattern classifiers applied to the changes measured in the background [14].

The works previously described, use color features, background or fish models, and stereo vision systems. These techniques are carried out in calm and low turbulent water where information about textures and color may be enough to discriminate the fish. However, these conditions are not fulfilled in the turbulent conditions of a verti-cal slot fishway.

In this work, the use of SOM Neural Networks will be studied to detect living fish-es in vertical slot fishways from images obtained through a network of video cenital underwater video cameras (Fig. 1).

Fig. 1. Camera system. (a) Camera position in the fishway. (b) Recording scheme. (c), (d)

Water conditions during assays.

This work is an extension of the carried out in [15] where an image system is de-signed to record assays with real fishes and a methodology to study fish behavior is carried out, consisting in analysing fish trajectories with computer vision techniques and computing fish statistics mixing observed trajectories with hydraulic properties of the fishway.

As a part of the system proposed in [15] a SOM based segmentation technique was defined. However, the segmentation question was not in depth analysed, and this technique was not compared with other techniques or tested with ground true data.

Furthermore, SOM networks have not been analysed or tested before in the prob-lem of fish detection.

This paper is focused in studying the fish segmentation problem in a vertical slot fishway as a part of a tracking system with the objective of generate the trajectory of the fish over time by locating its position in every frame of the video.

The behavior of SOM Neural Networks is studied, analysing different approaches with a dataset recorded in experimental conditions and a true ground data created by experts. Additionally a representation and filtering shape-based technique has been defined and tested.

The efficiency and accuracy of SOM Neural Networks in the task of fish segmen-tation is properly analysed in this paper for the first time.

2 Fish Detection

In order to study the behavior of the fish from the sequence of images acquired, the area occupied by the fish should be separated from the rest of the image or back-ground. This process is known as image segmentation.

The aim of image segmentation algorithms is to partition the image into perceptu-ally similar regions. Every segmentation algorithm addresses two problems, the crite-ria for a good partition and the method for achieving efficient partitioning [16].

Therefore it is necessary to find a variable which allow a robust separation of the fish from the background. Then it is necessary to choose a technique to classify the image in different groups according to a criteria.

As discussed before, most common criteria to detect fishes in the image are the color features and a priori knowledge of the background.

Even analysing calm and low turbulent water in high quality images, simple image segmentation techniques such as thresholding and edge detection do not perform well due to the low levels of contrast typical of underwater images [14]. Additionally, in the images recorded in the high turbulent water of a vertical slot fishway, background subtraction techniques are usually ineffective alone. Furthermore, the images will be characterized by extreme luminosity changes, huge noise levels and poor contrast, being texture and color information useless (Fig. 2).

Fig. 2. Example of images recorded in the fishway during the assays.

In this paper, a type of Artificial Neural Network (ANN) called Self-organizing map (SOM) [10] has been used for image classification.

The SOM network model is aimed at establishing a correlation between the num-ber patterns supplied as an input and a two-dimensional output space (Topological map); thus, the input data with common features activate areas close to the map. This characteristic can be applied to image segmentation, also giving the following ad-vantages: Adaptive Learning based on a training phase using input examples, general-ization ability, error tolerance, highly operable in parallel and integrable into the ex-

isting technology [10]. Thus, works like those carried out by [17, 18] have successfully applied the SOM

networks to image segmentation, and their importance in this field was pointed out by in [19, 20].

Finally, the SOM networks have the advantage of not needing to define a set of pat-terns to supervise the learning process. They can be straightforward created by defin-ing a representative image or set of images.

However, their behavior in fish detecting problems was not properly studied before. To test the proposed approach, different tests were realized with different SOM ar-

chitectures, and using different combinations of features for classification, such as color and grey values, local average, local standard deviation or local entropy.

After preliminary assays, a three-layer topology with 3 processing elements in each layer was fixed for each network.

In all cases, input images were normalized and preprocessed to enhance the con-trast of the image by using a contrast-limited adaptive histogram equalization (CLAHE).

Different SOM networks were considered, being Ei,j the input data for the pixel (i, j) formed by one or more NxN vectors parameters defined in the neighborhood of the pixel defined. The following input values were considered:

• The RGB intensity values of the image I from the neighborhood of the pixel (i, j).

{ } ( ) ( ), ( , ) /2, /2 , /2, /2( , )i j x y i N j N i N j N

E I x y= − − + +

= (1)

• The local average of the RGB values in a window centered in the neighborhood of the pixel (i, j).

{ }( ) ( )( , ) /2, /2 , /2, /2

, ,x y i N j N i N j N

i j x yE µ= − − + +

= (2)

• The local average and local standard deviation of the RGB values in a window centered in the neighborhood of the pixel (i, j).

{ }( ) ( )( , ) /2, /2 , /2, /2

, , ,x y i N j N i N j N

i j x y x yE µ σ= − − + +

= (3)

Two versions of each network were implemented; one, using information of the processed image and another including background modeling information, where a selected frame of the background was used to normalize the current image values.

Additionally, the SOM network proposed in [15] using the global and local aver-age features from the current image I and a background reference image I’ was con-sidered. This technique will be tested for the first time with ground true data.

( ) ( )( , ) /2, /2 , /2, /2

, , ,,

''

x y i N j N i N j N

x y x y x yi j

I I I

Eµ µ µµ µ µ

= − − + +

= −

(4)

3 Filtering and Representation

Once the image segmentation is obtained with the SOM network, the objective of the segmentation system is to determine the position of the detected fish in the image.

Due to the characteristics of the image, where the fish is often partially hidden and where is expected the presence of shadows, bubbles and reflections, the algorithm should respond well to partial or abnormal detections.

A simple algorithm has been built, in order not to increase the computational bur-den of the process. The algorithm is based on obtaining the connected-body vector from the segmented image. Each body will be characterized by the vector of pixels that make it up and by a set of descriptive parameters: its area, its centroid and the minimum ellipse containing the body.

Then the detected object is classified as a potential fish or noise, according to the size of the object and the minimum and major axis of the ellipse. An iterative fusion operation has been defined to replace two close unconnected bodies for a new one, which will be formed by the points from the two previous ones while shape criteria are satisfied.

Thus, the algorithm determines the mass centers of the fishes and puts together those connected bodies whose characteristics can be matched to a fish or a part of it, and discards those bodies that, due to their size or shape, are regarded as noise. Sub-sequently, the mass center of each detected fish is obtained.

4 Experimental Results

To measure the performance of the SOM networks, a set of experiments were per-formed with living fishes of the salmo trutta specie in a 1:1 vertical slot fishway mod-el located at Center for Studies and Experimentation of Public Works CEDEX, in Madrid.

A data set of 1000 images from 10 different cameras selected from different pool and fishway regions was defined and the corresponding ground true data was manual-ly created by experts.

To evaluate the dependence of the results with the selected training patterns, all the networks were training using a single image, and results were analysed using 3 differ-ent trainings (Fig. 3).

The average obtained results are shown in Table 1 and can be observed in Fig. 3.

Table 1. Average Results

Avg. Pixel Pixel BackGnd. Avg. Avg.

BackGnd. Avg.- St.Dev. Avg.- St.Dev. BackGnd. Features

Detections 690 1094 542 876 949 767 894 True Pos. 328 754 554 809 446 694 808 False Pos. 362 340 624 67 265 73 85 True Neg. 35 42 30 50 36 49 49 False Neg. 753 331 579 187 564 300 189 Precission 0.48 0.69 0.47 0.92 0.63 0.90 0.90

Recall 0.30 0.69 0.49 0.81 0.44 0.70 0.81 F.P. Rate 0.52 0.31 0.53 0.08 0.37 0.10 0.10 F.N. Rate 0.70 0.31 0.51 0.19 0.56 0.30 0.19

Fig. 3. Segmentation results

Analyzing the results it may be observed that the best precision has been obtained with the local average classifier with a background modeling. Therefore, the inclusion of temporal information appears to be a critical factor in the results. Additionally, none of the networks without background information obtained good results.

The SOM network proposed in [15] has obtained the second best result. It can be seen that a very low false positive rate is achieved, so obtained results are

very reliable, and they will represent true positions of the fish with a high probability. Furthermore, precision is the most important factor in a detection system because a high rate of false positives would make the results useless.

However, the false negative rate is much higher. Showing that, in almost a 20% of the measurements, the system will not provide information about fish position. This is due to the fact that the fish will be frequently occluded by the turbulence of water. Nevertheless, it must be taken in account that the system can execute up to 25 meas-urements per second, so it will provide a correct position more than 15 times per se-cond in average, which is far beyond any detection system used at present in this field.

5 Conclusions

In this work, a solution to detect fishes in vertical slot fishways based on SOM net-works and an algorithm to filter anomalous detections is analysed. The accuracy and performance of different SOM networks have been tested, analys-ing and comparing for the first time, the behavior of these technique in the fish seg-mentation problem.

The results obtained with this system have been very promising, as they allowed us to obtain the fish position in the image with a low error rate.

In future stages of this work, a tracking algorithm will be defined to manage the detected positions of the fish or fishes obtaining the trajectories of the specimens.

Although further research is needed, the results obtained will be used to elaborate a new methodology to study fish behavior inside vertical slot fishways so it can con-tribute to develop robust guidelines for future fishway designs and to establish more realistic criteria for the evaluation of biological performance of the current designs.

Acknowledgments: This work was supported by the Dirección Xeral de Investi-gación, Desenvolvemento e Innovación (General Directorate of Research, Develop-ment and Innovation) de la Xunta de Galicia (Galicia regional government) (Ref. 10MDS014CT) and from the Ministerio de Economía y Competitividad (Spanish Ministry of Economy and Competitiveness) Ref. CGL2012-34688. The authors would also like to thank the Center for Studies and Experimentation on Public Works (CEDEX).

References:

[1] J. Puertas, L. Pena, and T. Teijeiro, "An Experimental Approach to the Hydraulics of Vertical Slot Fishways," Journal of Hydraulics Engineering, vol. 130, 2004.

[2] N. Rajaratnam, G. V. d. Vinne, and C. Katopodis, "Hydraulics of Vertical Slot Fishways," Journal of Hydraulic Engineering, vol. 112, pp. 909-927, 1986.

[3] L. Tarrade, A. Texier, and L. David, "Topologies and measurements of turbulent flow in vertical slot fishways," Hydrobiologia, vol. 609, pp. 177-188, 2008.

[4] S. Wu, N. Rajaratma, and C. Katopodis, "Structure of flow in vertical slot fishways," Journal of Hydraulic Engineering, vol. 125, pp. 351-360, 1999.

[5] H. Dewar and J. Graham, "Studies of tropical tuna swimming performance in a large water tunnel– Energetics," Journal of Experimental Biology, vol. 192, pp. 13-31, 1994.

[6] R. W. Blake, "Fish functional design and swimming performance," Journal of fish biology, vol. 65, pp. 1193-1222, 2004.

[7] S. Duarte, L. Reig, J. Oca, and R. flos, "Computerized imaging techniques for fish tracking in behavioral studies," presented at the European Aquaculture Society, 2004.

[8] M. Chambah, D. Semani, A. Renouf, P. Courtellemont, and A. Rizzi, "Underwater color constancy enhancement of automatic live fish recognition," presented at the IS&T Electronic Imaging (SPIE), California, USA, 2004.

[9] R. J. Petrell, X. Shi, R. K. Ward, A. Naiberg, and C. R. Savage, "Determining fish size and swimming speed in cages and tanks using simple video techniques " Aquacultural Engineering, vol. 16, pp. 63-84, 1997.

[10] E. F. Morais, M. F. M. Campos, F. L. C. Padua, and R. L. Carceroni, "Particle filter-based predictive tracking for robust fish count," presented at the Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI), 2005.

[11] S. Clausen, K. Greiner, O. Andersen, K.-A. Lie, H. Schulerud, and T. Kavli, "Automatic segmentation of overlapping fish using shape priors," presented at the Scandinavian conference on Image analysis 2007.

[12] M.-C. Chuang, J.-N. Hwang, K. Williams, and R. Towler, "Automatic fish segmentation via double local thresholding for trawl-based underwater camera systems " presented at the IEEE International Conference on Image Processing (ICIP), 2011.

[13] C. Spampinato, Y.-H. Chen-Burger, G. Nadarajan, and R. Fisher, "Detecting, Tracking and Counting Fish in Low Quality Unconstrained Underwater Videos," presented at the Int. Conf. on Computer Vision Theory and Applications (VISAPP), 2008.

[14] J. A. Lines, R. D. Tillett, L. G. Ross, D. Chan, S. Hockaday, and N. J. B. McFarlane, "An automatic image-based system for estimating the mass of free-swimming fish," Computers and Electronics in Agriculture, vol. 31, pp. 151-168, 2001.

[15] A. Rodriguez, M. bermudez, J. R. Rabuñal, J. Puertas, J. Dorado, and L. Balairon, "Optical Fish Trajectory Measurement in Fishways through Computer Vision and Artificial Neural Networks," Journal of Computing in Civil Engineering, vol. 25, pp. 291-301, 2011.

[16] A. Yilmaz, O. Javed, and M. Shah, "Object Tracking: A Survey," ACM Computing Surveys, vol. 38, 2006.

[17] A. Verikas, K. Malmqvist, and L. Bergman, "Color image segmentation by modular neural networks," Pattern Recognition Letters vol. 18, pp. 173-185, 1997.

[18] G. Dong and M. Xie, "Color clustering and learning for image segmentation based on neural networks," IEEE transactions on neural networks, vol. 16, pp. 925-936, 2005.

[19] M. Egmont-Petersen, D. Ridder, and H. Handels, "Image processing with neural networks-a review," Pattern Recognition, vol. 35, pp. 2279-2301, 2002.

[20] P. Cristea, "Application of Neural Networks In Image Processing and Visualization," in GeoSpatial Visual Analytics, R. Amicis, R. Stojanovic, and G. Conti, Eds., ed: Springer Netherlands, 2009, pp. 59-71.