A Performance Characterization Algorithmfor Symbol Localization

Mathieu Delalandre1,2, Jean-Yves Ramel1, Ernest Valveny1, andMuhammad Muzzamil Luqman1,2

1 Laboratoire d’Informatique (LI), 37200 Tours, Francemathieu.delalandre@univ-tours.frjean-yves.ramel@univ-tours.fr

muhammadmuzzamil.luqman@etu.univ-tours.fr2 Computer Vision Center (CVC), 08193 Bellaterra (Barcelona), Spain

ernest@cvc.uab.es

Abstract. In this paper we present an algorithm for performance characterizationof symbol localization systems. This algorithm is aimed to be “generic” and “ob-jective” to characterize the performance. It only exploits single points as the resultof localization comparing them with the groundtruth, using information aboutcontext. Probability scores are computed for each localization point, dependingon the spatial distribution of the regions in the groundtruth. Characterization re-sults are first given with detection rate/probability error plots, describing the setsof possible interpretations of the localization results. Global scores compare theseresults, combining in a single and unified error-measures about localization andprecision of system (in the sense of precision/recall). We present experiments andresults done with the symbol localization system of [1], using a synthetic datasetof architectural floorplans and electrical diagrams (200 images, 3861 symbols).

Keywords: symbol localization, groundtruth, performance characterization

1 Introduction

In recent years there has been a noticeable shift of attention, within the graphics recog-nition community, towards performance evaluation of symbol recognition systems. Thisinterest has led to the organization of several international contests and development ofperformance evaluation frameworks [2]. However, to date, this work has been focussedon recognition of isolated symbols. It didn’t take into account the localization of sym-bols in real documents. Indeed, symbol localization constitutes a hard research gap,both for recognition and performance evaluation tasks.

Different research works have been recently undertaken to fill this gap [3]. Groundtru-thing frameworks for complete documents and datasets have been proposed in [4,5], anddifferent systems working at localization level in [6,1,4]. The key problem is now todefine characterization algorithms working in a localization context. Indeed the charac-terization of localization in complete documents is harder, as comparison of results withgroundtruth needs to be done between sets of symbols. These sets could be of differ-ent size, and significant differences could appear between the localizations of symbols

2 M. Delalandre, J.Y. Ramel, E. Valveny and M.M. Luqman

provided by a given system, and the corresponding ones in the groundtruth. Character-ization metrics must then be reformulated to take these specificities into account.

In the rest of the paper, we will first introduce in section 2 related work on this topic.We will present next in section 3 the approach we propose. Section 4 will give exper-iments and results we have obtained with our algorithm. Conclusion and perspectivesarising from this work will be presented in section 5.

2 Related work

Performance characterization, in the specific context of localization, is a well knownproblem in some research topics such as computer vision [7], handwriting segmentation[8], layout analysis [9], text/graphics separation [10], etc. Concerning symbol localiza-tion, at the best of our knowledge only the work of [4] has been proposed to date. Theperformance characterization algorithms aim to detect possible matching cases betweengroundtruth and localization results, as detailed in Table 1. Exploiting these results,different criteria could be computed such as segmentation rates or retrieval measures(precision, recall, F-measure).

single an object in the results matches with a single object in the groundtruthmisses an object in the groundtruth doesn’t match with any object in the resultsfalse alarm an object in the results doesn’t match with any object in the groundtruthmultiple an object in the results matches with multiple objects in the groundtruth

(merge case) or an object in the groundtruth matches with multiple objectsin the results (split case)

Table 1. Matching cases between groundtruth and localization results

The key point when developing such a characterization algorithm, is to decide aboutrepresentations to be used, both in results and groundtruth. Two kinds of approach existin the literature [9], exploiting pixel-based and geometry-based representations.

In a pixel-based representation, results and groundtruth correspond to sets of pixels.For that reason, algorithms exploiting such a representation are very accurate. They areusually employed to evaluate segmentation tasks in computer vision [7] or handwritingrecognition [8]. However, groundtruth creation is more cumbersome and requires a lotmore storage. Comparison of groundtruth with the results is also time-consuming.

In a geometry-based representation, algorithms employ geometric shapes to de-scribe the regions (in results and groundtruth). The type of geometric shapes depend onthe application: bounding boxes - text/graphics separation [10], isothetic polygons - lay-out analysis [9], convex hulls - symbol spotting [4], etc. Comparison of the groundtruthwith the results is time-efficient, and the corresponding groundtruth straightforward toproduce. Such a representation is commonly used in the document analysis field, as itis more focused on semantic [10,9,4]. Because the main goal of the systems is recogni-tion, evaluation could be limited to detection aspects only (i.e. to decide about a wrongor a correct localization without evaluation of the segmentation accuracy).

A Performance Characterization Algorithm for Symbol Localization 3

In both cases, characterization algorithms exploit information about regions in lo-calization results, and compare them to groundtruth. This results in boolean decisionsabout positive/negative detections, which raises several open problems:

Homogeneity of results: Regions provided as localization results could present a hugevariability (set of pixels, bounding boxes, convex hulls, ellipsis, etc.). This variabil-ity disturbs the comparison of systems. A characterization algorithm should takethese differences into account, and put the results of all the systems at a same level.

Precision of localization: Large differences could appear between the size of regionsin the results and the groundtruth. These differences correspond to over or undersegmentation cases. This results in aberrant positive matching cases between thegroundtruth and the detection results, when large regions in the results intersectssmallest ones in groundtruth. Some thresholds should be considered to make theevaluation more objective.

Time complexity: Complex regions, such as symbols, must be represented by pixelsets or polygons to obtain a correct localization precision. However, their compar-ison is time-consuming both for geometry-based and pixel-based representations[9,8]. This involves to use specific approaches to limit the complexity of the algo-rithms [9].

In this paper we propose an alternative approach to region-based characterization,to solve these problems. We present this approach in the next section 3.

3 Our approach

3.1 Introduction

Our key objective of this work is to provide a more “generic” and “objective” way tocharacterize performance of symbol localization systems. To do it, our proposal is toexploit only single points as the results of localization. Indeed, points are a more ho-mogenous and reliable information to evaluate localization. A result could be easilyprovided as point using the gravity center of the detected region. In addition, becauseinformation about the regions in results are not taken into account, this avoids the aber-rant detection cases.

However, such an approach makes it impossible to fully detect the matching casesbetween localization results and groundtruth e.g. how to interpret a point slightly out-side of a groundtruth region, a point equally distant of two regions, etc. To solve thisproblem, in our approach we exploit the information about context in groundtruth (Fig.1). For each result point, probability scores are computed with each neighboring regionsin the groundtruth. These probability scores will depend on the spatial distribution of theregions in the groundtruth. They will change locally for each result point. Fig. 1 givesan example. p1, p2 and p3 are located at similar distances of symbols in groundtruth.However, p2 and p3 present highest probability scores, but not p1. Local distributionof symbols in groundtruth around p1 makes the decision about the detection ambigu-ous. Final characterization results are given with a detection rate/probability error plot,describing the sets of possible interpretations of the localization results.

4 M. Delalandre, J.Y. Ramel, E. Valveny and M.M. Luqman

Fig. 1. Our approach

In this section we describe our characterization algorithm. We exploit four mainsteps to perform the characterization: (1) in a first step we use a method to comparethe localization of a result point to a given symbol in groundtruth (2) exploiting thiscomparison method, we compute next for each result point its probability scores withall the symbols in groundtruth (3) then, we employ a matching algorithm to identify thecorrect detection cases, and draw the detection rate/probability error plots (4) at last, wepresent how to compute global characterization scores from these plots. We will detaileach of these steps in next subsections 3.2, 3.3, 3.4 and 3.5 respectively.

3.2 Localization comparison

In our approach, groundtruth is provided as regions (contours with corresponding grav-ity centers) and localization results as points (Fig. 1). This makes impossible to comparethem directly, due to the scaling variations. Indeed, symbols appear at different scalesin drawings. To address this problem, we compare the result points with groundtruthregions by computing scale factors (Fig. 2). In geometrical terms, a factor s specifieshow to scale a region in the groundtruth so that its contour fits with a given result point.Thus, result points inside and outside a symbol will have respectively scale factors ofs ≤ 1 and s > 1.

Fig. 2. Scale factor

The factor s is computed from the line L of direction θ, joining the gravity centerg defined in groundtruth to a result point r. On this line L, s corresponds to the ratio

A Performance Characterization Algorithm for Symbol Localization 5

of lengths lgr and lgi. lgr is the Euclidean distance between the gravity center g andthe result point r. lgi is computed in the same way, but with the intersection point i ofL with contours c of symbol. This intersection point i is detected using standard lineintersection methods [11]. If several intersections exist (i.e. cases of concave contoursor holes in symbol), the farthest one from g is selected.

3.3 Probability scores

In a second step, we compute probability scores between result points and groundtruth(Fig. 3). For each result point r, relations with groundtruth are expressed by a set of npoints S =

⋃ni=1 gi(θ, s) (Fig. 3 (a)), where θ and s represent respectively the direction

and scale factor between the result point and a symbol i. We define next the probabilityscores between the result point r and symbols

⋃ni=1 gi as detailed in Fig. 3 (b). When

all symbols⋃n

i=1 gi are equally distant (i.e s1 = s2 = ...si.. = sn), thus⋃n

i=1 pi = 0.In the case of a si = 0, thus the corresponding pi = 1. Otherwise, any other cases0 < si < sj will correspond to probability scores 0 < pj < pi < 1.

Fig. 3. (a) plot (θ, s) (b) probability scores

The equations (1) (2) below give the mathematical details we employ to computethe probability scores. For each gi, we compute the probability score pi to the resultpoint r as detailed in (1). To do it, we exploit the other symbols

⋃nj=1,j 6=i gj in the

groundtruth. We mean the values f(

si

sj

)corresponding to local probabilities gi to r,

regarding gj . The function f(

si

sj

)must respect the conditions given in Table 2. in fact,

there exists different mathematical ways to define such a function (based on inverse,linear, cosinus functions, etc.). We have employed here a gaussian function (2), as it isa common way to represent random distributions. This function is set using a varianceσ2 = 1, and normalization parameters ky and kx. These parameters are defined in order

6 M. Delalandre, J.Y. Ramel, E. Valveny and M.M. Luqman

to respect the key values f(x = 0) = 1 and f(x ≥ 1) = 0. To respect the secondcondition, we bound to 0 the gaussian values f(x ≥ 1), with a kx determined to respecta threshold error λ3.

pi =n∑

j=1,j 6=i

f(

si

sj

)n− 1

(1)

f

(x =

si

sj

)=

ky√2π× exp−

(kxx)2

2 (2)

ky =√

2π√

2π2−∫ 1

0

f(x) dx < λ→ kx

Table 2. Table of function f(x = sisj

)

3.4 Matching algorithm

Once probability scores are computed, we look for relevant matchings between ground-truth and results of a system. We propose here a “fuzzy” way to find these matchings,using distribution plots. We compute different sets of matching results, according to theranges of the probability scores. Final results are displayed into distribution plots (Fig.4 (a)), where the x axis corresponds to score error ε (i.e. inverse of probability scoreε = 1− p), and the y axis to performance rates.

We compute different performance rates, Ts, Tf and Tm corresponding respectivelyto single detections, false alarms and multiple detections (see Table 1.). The rate of“misses cases” corresponds to 1 − Ts. To do it, we build-up a correspondence list be-tween results and groundtruth as detailed in Fig. 4 (b). This correspondence list is bi-partite, composed of nodes corresponding to the groundtruth

⋃ni=1 gi and the results⋃q

j=1 rj . Our probability scores are given as undirected arcs⋃np

k=1 ak = (gi, rj , pij) ofnp size. We use these arcs to make the correspondences in the list in an incremental way,by shifting the ε value from 0 to 1. An arc ak is added to the list when its pij ≥ 1− ε.For each ε value, the Ts and Tf rates are computed by browsing the list, and checkingthe degrees of nodes

⋃ni=1 dgi and

⋃qj=1 drj , as detailed in (3), (4) and (5).

3 In our algorithm, λ is fixed at√

2π10−5 corresponding to kx = 3.9.

A Performance Characterization Algorithm for Symbol Localization 7

Fig. 4. (a) result plot (b) correspondence list

∀gi ↔ rj , dgi = drj = 1→ s = s+ 1 (3)

Ts =s

n∀rj , drj = 0→ f = f + 1 (4)

Tf =f

q

∀rj ↔ gi, drj > 1 ∨ dgi > 1→ m = m+ 1 (5)

Tm =m

q

3.5 Global sores

Final step in our approach aims to compare different characterization results together.These results could come, either from different symbol localization systems, or from asame system with different parameter-settings (e.g. thresholds, training datasets, etc.),computed from a same or different dataset(s). In order to achieve this comparison, wepropose to compute global scores4i from our characterization results. Our goal is:

1. To combine, in a single and unified measure, errors about localization and precision(in the sense of precision/recall) of systems.

2. To make the scores comparable i.e. independent from tests and normalized (4iε[0, 1]).

Our proposal is detailed in Fig. 5 (a). We exploit a curve si(ε) describing the singledetection rates Ts, as it constitutes the main results of the characterization. We com-pare it to a curve g(ε), corresponding to the single detection rates obtained of the self-matching of groundtruth. Thus, this g(ε) curve is the best result that can be obtained bya localization system on a given test. It is characterized by the key value g(ε = 0) = 1.At last, to compute the global score 4i, we divide si(ε) by g(ε) and sum the resultson [0, 1] interval. Then, best and worst values 4i = 1, 4i = 0, will be obtained re-spectively for si(ε) = g(ε) and si(ε) = 0. In mathematical terms, this global score4i

8 M. Delalandre, J.Y. Ramel, E. Valveny and M.M. Luqman

corresponds to the mean ratio of single detection rates Ts

Ts, with Ts the top rate that can

be obtained at a given ε. In addition, in order to compare locally the scores, we makea visual comparison using the global score function 4i(ε) Fig. 5 (b). This functiondescribes the variation of4i on the [0, 1] interval, with the global scores4i = 4i(1).

Fig. 5. Global scores(a) global score4i (b) function4i(x)

Fig. 6 presents variations of 4i in regard to the number of results q provided bysystems. The best score 4i = 1 is obtained in the case of q = n (with

⋃n=qi=1 gi =⋃q=n

j=1 rj). If a system generate few results (q � n) or a high number of results (q � n),these will decrease consequently the values of 4i. A high value of q correspond toa dirac delta curve si(ε) (if no dual points exist in results, rj 6= rk) and a 4i →0. Therefore, to obtain a good score 4i, a system must provide results with a goodlocalization precision, and avoid the false alarms, misses and multiple detection cases.

Fig. 6. relation between4i and q

A Performance Characterization Algorithm for Symbol Localization 9

4 Experiments and Results

In this section, we present experiments and results obtained using our algorithm. Wehave applied it to evaluate the symbol localization system of [1]. This system relies ona structural approach, using a two-step process.

First, it extracts topological and geometric features from a given image, and rep-resents it using an ARG4 (Fig. 7). The image is preliminary vectorized into a set ofquadrilateral primitives. These primitives become nodes in the ARG (labels 1, 2, 3, 4),and connections between become arcs. Nodes have, as attributes, relative lengths (nor-malized between 0 and 1) whereas arcs have connection-type (L junction,T junction, Xjunction, etc.) and relative angle (normalized between 0◦ and 90◦).

Fig. 7. Representation phase of [1]

In the second step, the system looks for potential regions of interest corresponding tosymbols. It detects parts of the ARG that may correspond to symbols i.e. symbol seeds.Scores, corresponding to probabilities of being part of a symbol, are computed for alledges and nodes of the ARG. They are based on features such as lengths of segments,perpendicular and parallel angular relations, degrees of nodes, etc. The symbol seedsare detected next during a score propagation process. This process seeks and analyzesthe different shortest paths and loops between nodes in the ARG. The scores of allthe nodes belonging to a detected path are homogenized (propagation of the maximumscore to all the nodes in the path) until convergence, to obtain the seeds.

To test this system, we have employed datasets coming from the SESYD database5.This database is composed of synthetic document images, with the corresponding ground-truth, produced using the system described in [5]. This system allows the generation ofsynthetic graphic documents, containing non-isolated symbols in a real context. It isbased on the definition of a set of constraints, that permit to place the symbols on a pre-defined background, according to the properties of a particular domain (architecture,electronics, etc.). The SESYD database is composed of different collections, includ-ing architectural floorplans, electrical diagrams, geographic maps, etc. In this work, wehave limited our experiments to subsets of this database, including documents from theelectrical and architectural domains. Table 3. gives details of our dataset.

4 Attributed Relational Graph5 http://mathieu.delalandre.free.fr/projects/sesyd/

10 M. Delalandre, J.Y. Ramel, E. Valveny and M.M. Luqman

Table 3. Dataset used for experiments

Fig. 8 presents the characterization results we have obtained on floorplans. Thisplot presents the variation of {Ts, Tf , Tm} rates, and in addition the Tg one (i.e. the selfmatching of groundtruth). The system presents inA a good confidence for the detectionresults Ts ≤ 0.50, with an score error ε ≤ 0.05 and nearly none multiple detectionsTm ≤ 0.03. The best detection results is obtained inB with a Ts = 0.57, correspondingto an score error of ε = 0.11. However, these detection results are joined to a Tf = 0.31,highlighting a bad precision of the system. In addition, at this state confusions appearin localization results with a Tm ≥ 0.20. This rate results in the merging of false alarmswith less confident results, as the false alarm rate goes down to Tf = 0.31. Up to thispoint, Ts dies down slowly, and for score error ε ≥ 0.21 in C the Ts and Tm curvesstart to be diametrically opposite.

Fig. 8. Characterization results on floorplans

A Performance Characterization Algorithm for Symbol Localization 11

Fig. 9 gives results concerning electrical diagrams. The system presents a goodconfidence in A, for ε ≤ 0.04 and Ts = Tf = 0.45. However, at this point the sys-tem already does multiple detections with a Tm = 0.10. The best localization score isobtained at B with ε = 0.13 and Ts = 0.62. Therefore, the best localization score isbetter for electrical diagrams than floorplans. In addition, the system doesn’t make alot of false detections with a Tf = 0.13. However, multiple detections stay higher withTm = 0.30. Up to C, Ts decreases in linear way for score errors ε ≥ 0.23.

Fig. 9. Characterization results on electrical diagrams

Fig. 10 (a) gives global scores values4i (i.e. 4i(1)) and functions 4i(ε) for bothdatasets, floorplans and electrical diagrams. The global score obtained on electrical di-agrams is the best 4i(1) = 0.529, comparing to the one of floorplan 4i(1) = 0.496.However, we could notice an inversion of performance for low values of score errorε < 0.1 Fig. 10 (b). This could be explained in two points. First, by the multiple detec-tion problems of the system on the electrical diagrams (Fig. 9). Certainly the systemswill increase in a significative way its detection results, by introducing split / mergeprocedures of detected regions of interest. The second point concerns the generatedfalse alarms on floorplans (Fig. 8). These false alarms have a few impact for the lowvalues of score error ε < 0.1. However up to this point they are taken into account inthe computation of global score, and therefore reduce the performances of the system.In order to reduce the problem of false alarms, a point to be improved in the system isto introduce a checking procedure for detected ROIs.

12 M. Delalandre, J.Y. Ramel, E. Valveny and M.M. Luqman

Fig. 10. Comparison of global scores(a) interval [0− 1] (b) interval [0− 0.1]

5 Conclusions and Perspectives

In this paper we have presented an algorithm for performance characterization of sym-bol localization systems. This work aims to propose a more “generic” and “objective”way to characterize performance. Our proposal is to exploit only single points as theresults of localization, as it is a more homogenous and reliable information to evalu-ate localization. To compare them with the groundtruth, we exploit information aboutcontext. For each result point, probability scores are computed with each neighboringregions in the groundtruth. These probability scores will depend on the spatial dis-tribution of the regions in the groundtruth. They will change locally for each resultpoint. Characterization results are first given with detection rate/probability error plots,describing the sets of possible interpretations of the localization results. From theseplots, we compute global scores to compare localization results, combining in a singleand unified error-measure about localization and precision of system (in the sense ofprecision/recall). We present experiments and results obtained using our algorithm, toevaluate the symbol localization system of [1]. These experiments have been done on adataset of synthetic images (with the corresponding groundtruth), composed of 200 ofdocument images and around 3861 symbols from electrical and architectural domains.We conclude about the performance of this system, in terms of localization accuracy,precision level (false alarms and multiple detections) on both datasets.

In future, we aim to take-forward our experimentations to evaluate the scalability(large number of symbol models) and robustness (noisy images [12]) of system. Wealso plan to perform experiments with real datasets [4], and to compare the obtainedresults with the synthetic ones. And, our final goal is the comparison of different symbollocalization systems. We plan to take benefit of the work done around the EPEIRES

A Performance Characterization Algorithm for Symbol Localization 13

project6, to look for potential participants interested in testing their systems with thischaracterization approach.

6 Acknowledgements

This work has been partially supported by the Spanish projects CONSOLIDER-INGENIO2010 (CSD2007-00018), TIN2008-04998 and TIN2009-14633-C03-03. The authorswish to thank Herve Locteau (LORIA institute, Nancy, France) for his comments andcorrections about the paper.

References

1. Qureshi, R., Ramel, J., Barret, D., Cardot, H.: Symbol spotting in graphical documentsusing graph representations. In: Workshop on Graphics Recognition (GREC). Volume 5046of Lecture Notes in Computer Science (LNCS). (2008) 91–103

2. Valveny, E., Tabbone, S., Ramos, O., Philippot, E.: Performance characterization of shapedescriptors for symbol representation. In: Workshop on Graphics Recognition (GREC). Vol-ume 5046 of Lecture Notes in Computer Science (LNCS). (2008) 278–287

3. Delalandre, M., Valveny, E., Llados, J.: Performance evaluation of symbol recognition andspotting systems: An overview. In: Workshop on Document Analysis Systems (DAS). (2008)497–505

4. Rusinol, M., Llados, J.: A performance evaluation protocol for symbol spotting systems interms of recognition and location indices. International Journal on Document Analysis andRecognition (IJDAR) 12(2) (2009) 83–96

5. Delalandre, M., Pridmore, T., Valveny, E., Trupin, E., Locteau, H.: Building synthetic graph-ical documents for performance evaluation. In: Workshop on Graphics Recognition (GREC).Volume 5046 of Lecture Note in Computer Science (LNCS). (2008) 288–298

6. Locteau, H., Adam, S., Trupin, E., Labiche, J., Heroux, P.: Symbol spotting using full visi-bility graph representation. In: Workshop on Graphics Recognition (GREC). (2007) 49–50

7. Unnikrishnan, R., Pantofaru, C., Hebert, M.: Toward objective evaluation of image segmen-tation algorithms. Pattern Analysis and Machine Intelligence (PAMI) 29(6) (2007) 929–944

8. Breuel, T.: Representations and metrics for off-line handwriting segmentation. In: Interna-tional Workshop on Frontiers in Handwriting Recognition (IWFHR). (2002) 428–433

9. Bridson, D., Antonacopoulos, A.: A geometric approach for accurate and efficient perfor-mance evaluation. In: International Conference on Pattern Recognition (ICPR). (2008) 1–4

10. Wenyin, L., Dori, D.: A proposed scheme for performance evaluation of graphics/text sepa-ration algorithms. In: Workshop on Graphics Recognition (GREC). Volume 1389 of LectureNotes in Computer Science (LNCS). (1997) 335–346

11. Balaban, I.: An optimal algorithm for finding segments intersections. In: Symposium onComputational Geometry (SGC). (1995) 211–219

12. Kanungo, T., Haralick, R.M., Phillips, I.: Non-linear local and global document degradationmodels. International Journal of Imaging Systems and Technology (IJIST) 5(3) (1994) 220–230

6 http://epeires.loria.fr/