Exploring the OFIm framework to Support the Test ofPrograms with GUIs

Rafael A. P. de Oliveira1, Marcio E. Delamaro1, Fatima L. S. Nunes2

1 Instituto de Ciencias Matematicas e de Computacao (ICMC)University of Sao Paulo - Sao Carlos/SP, Brazil

Computer Systems Department (USP)2Escola de Artes, Ciencias e Humanidades (EACH)

University of Sao Paulo - Sao Paulo/SP, Brazil

{rpaes,delamaro}@icmc.usp.br, fatima.nunes@usp.br

Abstract. Automation of the testing activity is an essential requirement to render itviable in the software development process. A complete automation depends on asolution for the oracle mechanism, used to determine whether the behavior of anexecution of a system under testing (SUT) is correct or not. Consequently, as thecomplexity of the outputs increases, more efforts are required in the automation ofthe oracle. This paper explores the OFIm (Oracle For Images) framework, that usesCBIR (Content-Based Image Retrieval) concepts to configure an environment thatsupports testing programs with complex output formats, making possible to devisetesting oracle that supports systems with graphical user interface (GUI). We provideresults of experiments which indicate that the framework can be useful to support thesoftware testing of systems with GUIs.

1. IntroductionDeveloping high quality software products is a constant goal for developers. Therefore, ac-tivities of verification, validation and testing have been receiving special attention. These aresome of the most expensive and yet more important activities in the software developmentprocess. In this context, the automation of testing activities is fundamental to increase pro-ductivity and decrease development costs. To this purpose many researchers have dedicateefforts on developing techniques and testing criteria that can be automated by tools.

A testing environment should have a mechanism to decide whether the output orthe execution behavior of a program under test is correct, i.e, an oracle [Hoffman 2001].Depending on the specification of the system under test (SUT), an oracle can be definedaccordingly. For example, when a specification of a SUT is an informal textual definitionof requirements, the tester can play the role of the oracle and decide about the correctnessof the SUT’s execution. If there is a formal model for the SUT from which one can extractits behavior, it is possible to automate the oracle by building up a comparator between theoutput actually produced and the behavior defined in the model. There are also cases wherethe oracle does not exist or is only approximately defined [Weyuker 1982].

A non-trivial output format, e.g., the screenshot of a graphical user interface (GUI)[Memon and Xie 2005], makes the implementation of oracles mechanisms complex. How-ever, GUIs play a fundamental role in many software systems. Moreover, complex GUIsare becoming increasingly popular. GUIs are faced as a complicating factor to test and canbecome a negative aspect for the application of software testing techniques. A possible so-lution to this is to set some conditions and specifications about the output presumed to becorrect. In this context, it is possible to list some conventional characteristics (text, numbersor patterns) computed from the expected outputs (image format). Thus, the output of thetested program can be analyzed and the same conventional characteristics be computed andcompared with the ones used as references. In fact, the tester, who has full knowledge about

the SUT, believes that the similarity of the testing output with the model output is sufficientfor a succeeding test.

The OFIm (Oracle For Images) framework, implemented in our previous research[Oliveira et al. 2009], can be explored to build testing oracles which are able to supportthe testing of applications with graphical outputs. For this, the framework uses a techniqueknown as CBIR (Content-Based Image Retrieval), which has been used to facilitate the re-trieval of images based on similarities [Datta et al. 2008]. OFIm has been used to implementoracles, named graphical oracles, for CAD (Computer-Aided Diagnosis) schemas and WebApplications [Delamaro et al. 2010]. OFIm allows testers to use a feature extractor imple-mented specifically for the SUT through textual descriptions (oracle descriptions), whichcontain the extractors and similarity functions that should be used in a particular test. Thispaper shows how the OFIm framework can be used to implement testing oracles that sup-ports the testing of applications with GUI. For this purpose we present a case study usingthe GUI of the calculator named Gcalctool 1.

The remainder of this paper is organized as follows: Section 2 presents concepts oftesting oracles and CBIR, particularly regarding its use within the context of oracle automa-tion. Section 3 describes the OFIm framework which architecture promotes the efficient useof CBIR concepts to automate the creation of graphical oracles. Section 4 presents a casestudy in which OFIm is used to support testing of a GUI application. Section 5 summarizesrelated research. Finally, Section 6 draws some concluding remarks and describes futureresearch.

2. BackgroundThe level of appropriate testing to a particular software depends on the potential conse-quences of undetected bugs. Often, claiming cost and infeasibility issues, increase testingtimes of a project. From an economic point of view, a mechanism that automates the testingprocess reduces costs and time spent on testing activities. In this context, a reliable testingprocess should have a good oracle mechanism that can judge the testing results. Thus, theautomation of oracles can be faced as a key item to promote the productivity of the testingactivity.

GUIs are typical examples of non-trivial output formats to automate oracles [Memonand Xie 2005]. Consequently, some researchers have dedicated their efforts to developapproaches that relate to the construction of oracles, and in particular to the automation oforacles for programs with GUIs. Some of them are discussed in section 5. In this context,the OFIm framework, which is explained in the next section, proposes the use of a CBIRtechnique for this purpose. CBIR is a technology that has been used to retrieve images froman image database following judgments of similarity with a model image [Datta et al. 2008].

CBIR systems consist of programs that are designed to retrieve images in a databaseconsidering similarities with a model image. Extracted by means of image processing (IP)techniques, the similarity between images is defined by comparing the features that are re-lated to aspects of color, texture and shape, for instance. These features can be obtainedthrough the implementation of a feature extractor. The comparison is implemented usingfunctions that measure the distance between sets of characteristics of two images, calledsimilarity functions (SF). In general, a SF consists of a mathematical formula, which facili-tates its implementation.

The type of information retrieval used by CBIR systems has been explored in severalfields of knowledge. For instance, it is largely explored in the medical image area. Accord-ing to Gudivada and Raghavan [1995], other applications in which CBIR can be used are

1See http://live.gnome.org/Gcalctool

art galleries, museum management, engineering and architecture, perception and remoteadministration of land resources and weather forecasting, among others.

For this paper, a CBIR system is composed of two parts: feature extractors andsimilarity functions. The architecture of a framework that uses CBIR concepts to implementtest oracles for programs with graphical output - the OFIm framework - is presented next.

3. The OFIm frameworkThe OFIm framework, implemented and deployed in our previous research [Oliveira et al.2009], is built using the Java programming language (PL) and available as free software2.It takes CBIR concepts to automate testing oracles for programs which output is presentedas an image. In addition, the framework allows the tester to use a set of specific features toa domain or software. The flexibility is one of the framework principles. OFIm users canchoose the features which will be used in their tests. The CBIR purpose is not retrievingimages that look exactly like the model image, which could be implemented with a sim-ple pixel-by-pixel comparison. Its truly objective is to enable, using CBIR concepts, thecomparison of images more similar to the model image.

Comparison of objects using similarity is fundamental to treat complex datum andcapture what is more representative, i.e., extract information that best represent it. OFImconsists of a flexible environment that allows the installation of feature extractors and sim-ilarity functions to generate testing oracles applicable to programs with graphical output.Therefore, the framework configures a flexible tool which allows its users to customize eachtest of similarity between images. It is done by adding feature extractors to similarity func-tions and changes in the parameters of the feature extractors, creating a graphical oracle. Toidentify and manage installed items, OFIm uses XML (eXtensible Markup Language) files.

An intuitive idea of the OFIm architecture is provided by the Figure 1a. Additionally,we present a component diagram aiming to show how OFIm components are wired together.This diagram is presented in Figure 1b. The core of the framework provides an applicationprogramming interface (API) that allows the testers to install plugins (feature extractorsand similarity functions). After this, testers can use these plugins to set theirs graphicaloracles, which may be specified in a higher abstraction level by a text file. Then, an oracledescription (OD), in a text file, can be read by the parser that instantiates the oracle. All ofthis can be done in an interactive way by a wizard which is able to install plugins and createODs. The plugins depicted in Figure 1a are the major contributions from the tester, in termsof code, to create the oracle.

In general, to use OFIm, testers has to implement their feature extractors and simi-larity functions. This is a simple task, since, in general, testers have complete knowledge ofthe SUT domain. One class of an OFIm plugin, which will be installed, has to implementa specific Java interface. For example, if a plugin implements a feature extractor, its mainclass has to implement the Java interface IExtractor, presented in Figure 2a. This interfaceprovides some basic methods for the full working of the plugin. The same happens with sim-ilarity functions, which have to implement the ISimilarity interface. The ISimilarity definesmethods to instantiate a similarity function object, retrieving the name of such object andassociate IExtractor objects to it. In addition, it defines methods to compute a feature vector- the set of values for each IExtractor object associated to the ISimilarity - and compute thevalue of the similarity function based on the feature vector.

The wizard tool is able to install and remove plugins. This wizard has a GUI whichallows an interactive way to manage plugins. Additionally, the OFIm wizard also provides

2See http://ccsl.icmc.usp.br/drupal5/pt-br/content/o-fim-oracle-images

(a)

(b)

Figure 1: a) Architecture for creating graphical oracles using CBIR; b) OFIm componentdiagram

(a)

(b)

Figure 2: a) IExtractor interface; b) An example of an Oracle Description (OD)

support for the creation of ODs, which will be used by the parser to build an oracle. Anoracle description structure is presented in Figure 2b. These descriptions are used to guidethe framework how a specific test oracle should be composed. Therefore, in Figure 2b, forexample, the oracle description is composed by a similarity function named Euclidean andtwo feature extractors named MyExtractor and OurExtractor. This OD makes the frame-

work instantiate these three plugins, which should already be installed in OFIm, in order totest can be performed. The wizard also provides methods for creating oracles able to com-pute the distance between two images and decides they are similar or different, consideringa threshold set by the tester. Adjusting a threshold is simple, since the tester just has toprovide a value to the property named Precision. In Figure 2b, the value 0.46 was set asthreshold. During the test, the oracle will judge as successful the distances that result in avalue lower than 0.46.

Systems under test with GUIs or graphical outputs can pose complications for theconduction of testing techniques. In this context, the framework presented in this sectionhas been explored to support the test of CAD Schemas and Web applications [Delamaroet al. 2010]. In a particular case study, OFIm was successfully used to support the test ofan application that was developed to segment the breast structure in a digitized mammogra-phy, highlighting it from the background of the image [Delamaro et al. 2010]. In the nextsection we show how the OFIm framework can be used to create an oracle geared towardsapplications with GUI.

4. A Case StudyThis section presents a case study which aims to demonstrate the applicability of the useof CBIR to support the test of applications with GUIs. It also emphasizes the feasibility ofusing the OFIm framework for the construction of testing oracles.

Our goal is to use the framework OFIm to verify the operability of the Gcalctool GUIat different resolutions and different sizes. Then, the Gcalctool GUI differences betweenplatforms were not explored in this case study. In this context, suppose, for example, thatthe tester developed and tested her/his application using a 1024x768 display resolution andthen wants to distribute it to users with different environments (on the same platform), withdifferent display resolutions.

Gcalctool has 5 different kind of GUIs. For each of the possible formats of the GUIscreens were captured at 5 different resolutions and 2 different sizes (normal and maxi-mized). Then, to summarize, we captured 5 different GUIs in 5 different screen resolutionsand with 2 different sizes. The screen resolutions used were 640x400, 720x400, 800x600,1024x768 and 1280x800. Then, 50 screenshots such as the ones showed in Figure 3 werecaptured3. The screenshots captured with a resolution of 1024x768 and default size, whichare all operable, were used as model images. The number of buttons computed by the ex-tractor to this model images was used as a reference to decide about the SUT correctness.

Figure 3 shows all possible formats that the Gcalctool GUI can take: “basic”, il-lustrated in Figure 3a, “advanced” represented by the Figure 3b, “financial” presented inFigure 3c, the “scientific” calculator that is displayed in Figure 3d, and, finally, in Figure 3eit is showed the “programmable” calculator.

The characteristic of the applications being executable (or easily recompiled) on dif-ferent platforms that of origin can be faced as a fundamental concept of high-level program-ming. This is the software codebase characteristic which allows the reuse of the existingcode instead of creating new code when moving software from an environment to another.On the other hand, if the application has a GUI, its execution on different platforms can bringproblems that the developer has to face. Such problems may occur, for instance, becausethe GUI look and feel (L&F), color systems and the screen resolutions are different fromplatform to platform. For example, if a program has a GUI that was programmed to be ini-tialized in a given screen size, it may not work correctly when runs on a different platform.The same can occur when the L&F or the color system is different from the original.

3The GUIs were captured from the Ubuntu OS (operating system), running on the Mac4Lin theme

(a) (b) (c) (d) (e)

Figure 3: Gcalctool- a) basic ; b) advanced; c) financial; d) scientific; e) programmable

Aiming to illustrate this problem, Figure 4 shows two screenshots from theGcalctool. A GUI is considered operable when its buttons are legible and enable. Forexample, the Figure 4a a Gcalctool GUI is showed with a resolution of 1024x768 pixels(1024 pixels wide by 768 pixels high), which is fully operable allowing the users to use theprogram. The same GUI executing on a resolution of 720x400 pixels is not fully operable,as is shown in Figure 4b. This resolution, despite the fact that leaves part of the interfacevisible, disables some of the buttons, making the program inoperable.

(a) (b)

Figure 4: a) Gcalctool with a resolution of 1024x768; b) A Gcalctool with a resolution of720x400

Dealing with this problem could be even more complicated when it comes to morecomplex GUIs which allow other methods of user’s interaction. For example, an applicationwhich has a GUI with many buttons, radiobuttons and checkboxs, is prone to errors when itis executed at different resolutions and platforms. Then, not only trying to solve this kindof problem, but also to measure the effectiveness of using CBIR concepts for the testingof applications with GUIs, we proposed this case study which evaluates different GcalctoolGUIs.

4.1. The similarity functionThe similarity function used by our oracle was the Euclidean Distance (ED), which is oneof the most popular measures. Equation 1 shows how the ED is defined. Consideringtwo vectors of characteristics x and y, of size n, where the elements are called xi and yi,respectively.

ED(x, y) =√

(x1 − y1)2 + (x2 − y2)2 + ...+ (xn − yn)2 (1)

Then, the instantiation of the oracle used in our case study could be done by creatingand associating the Euclidean similarity function to the oracle. It could be done by providingthe core a textual description of the oracle. This textual description uses the Euclideandistance associated with the CountButtons feature extractor. This OFIm plugin is presentednext.

4.2. The CountButtons feature extractor

We implemented a feature extractor to compose an oracle able to count the number of but-tons. The feature extractor, which was named CountButtons, was simple to implement andit intends to identify and count how many buttons are visible on the Gcalctool screenshot.For example, the counter buttons extractor applied to the interface shown in Figure 3a, willreturn a value of 20.

CountButtons has two Java classes named MainCountButton and CountButtons. Thefirst implements the ISimilarity Java interface, like is shown in Figure 2a. The second classimplements some methods which are responsible for the pre-processing needed to count thebuttons on the interface. In this particular case, the pre-processing is done in four steps:

1. First, to make the manipulation of the image pixels easier , it was necessary to con-vert the screenshot image from RGB color model to grayscale, considering a finalimage with 256 gray levels. A scientific Gcalctool GUI in grayscale is shown by theFigure 5a;

2. The second step consists of applying a thresholding algorithm that is intended todifferentiate the pixels with lighter intensity from pixels with darker intensity. Theapplication of the thresholding allows the identification of agglomerations of whitepixels that can represent a button. Thus, in places that agglomerations are identified,the value of the pixel is set to the white (255), otherwise the pixel is set to the blacklevel (0). Figure 5b shows a scientific Gcalctool in grayscale after applying thethresholding process;

3. The image obtained after applying the threshold may have some image noising. Thenoises are created by small details in the image of the interface such as, for example,radiobuttons and checkbuttons, which were disregarded in our case study. To thiscase study, image with noises can cause undesirable errors. Then, to solve this prob-lem, we applied to the image a simple median filter which removes small noises. Themedian filter is a known nonlinear digital filtering technique, often used to removenoise. Its main idea is to run through the signal entry by entry, replacing each entrywith the median of neighboring entries. A scientific Gcalctool in grayscale, afterapplying the thresholding and the median filter is shown in Figure 5c; and

4. Finally, it is necessary a mechanism to count the number of white regions of the im-age. This number should represent the enabled buttons on the GUI. For this purposewe used a technique known as area-point transformation [Nunes and Schiabel 2001].This technique consists of applying a mask that turns all the white areas into a singlewhite pixel. Thus, for example, each cluster of white pixels, shown in Figure 5c willbe transformed into a single white pixel. The number of white pixels represent thenumber of buttons.

4.3. Results

The case study results are reported in Table 1. This table contains a column that showsthe type of the GSalctool GUI and another column to show the number of buttons used todecide about the SUT correctness. This numbers were extracted from the model images.Furthermore, the table presents the test results of all screen resolutions and sizes.

(a) (b) (c)

Figure 5: a) An image of scientific calculator GUI in grayscale; b) Figure 5a after applyingthresholding; c) Figure 5b after applying the median filter

Table 1: Case study results

Resolution and SizeGUI type Buttons 640x400 720x400 800x600 1024x768 1280x800

norm. max. norm. max. norm. max. norm. max. norm. max.

Basic 20 Pass Pass Pass Pass Pass Pass Pass Pass Pass PassAdvanced 32 Pass Pass Pass Pass Pass Pass Pass Pass Pass PassFinancial 42 Fail Fail Fail Pass Pass Pass Pass Pass Pass PassScientific 45 Fail Fail Fail Fail Pass Pass Pass Pass Pass Pass

Programmable 43 Fail Fail Fail Fail Pass Pass Pass Pass Pass Pass

Aiming to compose the results, we used the following rule: when an GUI has thenumber of buttons equal its model image, the oracle judge the test as Pass, otherwise, itjudge the test as Fail. It is important to note that the oracle used was able to identify allcases in which the GUIs were inoperable or incomplete. Not only for reasons of disabledbuttons but also because of the size of the GUI at a specific screen resolution. The resultsemphasize that our technique is also worthwhile. Through a visual analysis, we note thatthe results are consistent with the operability of the GUIs. The results were satisfactory andshown that CBIR concepts can be used to build oracles that support the testing of SUT withinterfaces.

A set of 50 images to be tested was obtained with a few variations of resolution andsize. Applying test activities in these systems without some automated method could betedious and time consuming if executed repeatedly. In this context, using frameworks andtools like OFIm aims at reducing this cost and making oracle automation an easier task. Onthe other hand, minor adjusts can extends this case study to a different OS, for example, tochange the threshold value.

Another case in which the approach we proposed can be exploited is in regressiontesting or the application of mutation testing. Suppose, for example, that the expected outputis given by an image, as the result of a previous execution of the SUT or another programsuch as in regression testing or the application of mutation testing. There are several factorsthat may cause the graphical representation of two executions with the same result not lookexactly the same. For example, when both executions of a program - one that produces thereference image and one that produces the image to be analyzed - are conducted in differentenvironments, with different L&F. This becomes increasingly common, given the portabilityachieved through some programming languages or if we consider Web applications, whoseappearance for the final user also depends on the client (web browser) used.

The Section 5 shows some approaches like our proposal. This approaches currentlyused in the construction of graphical oracles are directed mostly to GUI applications. Theirprinciple is to use the GUI structure to compose the expected behavior in executions. Wecan highlight some positive points in our proposal comparing it with other existing ones.The first positive point is that the OFIm framework is not restricted to GUI applications.The other approaches are limited to handling GUI applications. Another advantage is, evenfor the case of GUIs, that there are cases where the tester cannot or does not want to extractthe structure from the GUI for automation of the oracle. In addition, we believe that in somecases the visual aspect of the result can be as or more important than the contents of a fieldor the state of an interface component.

The next section summarizes some of the works dealing with graphic applicationtest that we found in the literature. They are mostly directed to GUI applications, as alreadymentioned.

5. Related WorkResearch shows that, nowadays, 60% of application code are used to implement the GUI ofthe systems [Memon et al. 2003]. In this context, some researchers have worked to createand explore testing techniques and criteria useful to systems with GUIs. This techniquesand criteria do not explore graphical aspects. To understand some of this techniques andcriteria, it is necessary to know how a GUI is composed. In general, a GUI has differentcharacteristics compared to conventional applications. GUIs are composed by states whichcan be summarized by a triple S = { (Wi, Pj , Vk) }, in which Wi represents your currentwidget, Pj represents your properties and Vk represents the values of the properties of thecurrent widget [Memon and Xie 2005]. Users can interact with GUIs through events likekeyboard input, clicks, mouse positioning. All of these events can change the GUI state and,then, cause an error.

Some approaches explores exactly the changes of states to set the test oracles[Memon and Xie 2005]. In this context, definitions of event sequences of widgets, prop-erties, property values and GUI states are used. In general a test case can be a set of eventsand an expected state for the GUI after these events. Thereby, a test oracle like these cantake a test case, execute its events, compute the expected state, get the current state and,finally, compare expected and current state to compose a verdict about the correctness of theapplication. Obviously, to do this it is necessary that the oracle implements a mechanism toinspect the execution of the SUT. Consequently, this task can become difficult depending onthe domain of the SUT. In the literature, there are reports of some of these approaches, as inMemon et al. [2003] and Memon and Xie [2005]. Some of these approaches requires moretesting time and restricts the programming language (PL). Unlike our approach, no relatedwork presented a solution that is PL independent. The next section provides the conclusionsdrawn from experiments reported in this paper.

6. Conclusions and Final RemarksThe desire for automated software testing is justified by the increment in the productivityand quality of the software product that comes with it. The use of oracles is an essentialpart of the testing activity. There are difficulties involved in ascertain correctness of animplementation of the program under test. The implementation of an oracle mechanismbecomes a difficult activity when it comes to programs which outputs are expressed in othertypes of images, such as in graphical user interfaces.

The OFIm framework, developed in our previous work, uses CBIR concepts to sup-port the build of testing oracles for programs with complex outputs. In this paper is presented

a practical way to support the testing of programs with GUIs using the OFIm framework.Additionally, the stepwise procedure of how image processing concepts can be used in test-ing activities was presented. The feature extractor and similarity functions are easy to im-plement because it uses well-known algorithms of IP. A successfully case study done withGUIs was described. The case study results shows that CBIR concepts and IP can be used tosupport testing activities in software with complex outputs format. We emphasize that ourapproach is language independent, therefore allowing for its application beyond GUI-basedJava programs like some related work.

To conclude, this paper shows how one can use our approach to support softwaretesting activities. This already constitutes a contribution. Thus, we see this paper as a wayto collaborate with the automation of testing activities, through mechanisms for buildingoracles for graphical applications. We believe that this is an innovative and promising ap-proach as it allows different areas to benefit from its use. As future work, we plan to use theframework to implement oracles to text-to-speech (TTS) systems.

AcknowledgementsThis work is sponsored by CNPq, a Brazilian funding agency, under grant number551002/2007-7. Rafael Alves de Oliveira is sponsored by FAPESP under grant number2008/07605-7.

ReferencesDatta, R., Joshi, D., Li, J., and Wang, J. Z. (2008). Image retrieval: Ideas, influences, and

trends of the new age. ACM Computing Surveys, 40(2):1–60.Delamaro, M. E., Nunes, F. L. S., and de Oliveira, R. A. P. (2010). Using concepts of

content-based image retrieval to implement graphical testing oracles. To appear in Soft-ware Testing, Verification and Reliability.

Gudivada, V. N. and Raghavan, V. V. (1995). Content-based image retrieval systems. Com-puter, 28:18–22.

Hoffman, D. (2001). Using oracles in test automation. In Proceedings of the 19th PacificNorthwest Software Quality Conference (PNSQC 2001), pages 91 – 102, Oregon Con-vention Center Portland, Oregon.

Memon, A., Banerjee, I., Hashmi, N., and Nagarajan, A. (2003). Dart: A framework forregression testing “nightly/daily builds” of gui applications. In Proceedings of the 19th

International Conference on Software Maintenance (ICSM 03), page 410, Washington,DC, USA. IEEE Computer Society.

Memon, A. M. and Xie, Q. (2005). Studying the fault-detection effectiveness of GUItest cases for rapidly evolving software. IEEE Transactions on Software Engineering,31(10):884–896.

Nunes, F. L. S. and Schiabel, H. (2001). Deteccao de agrupamentos de microcalcificacoesmamarias atraves de processamento de imagens mamograficas com deteccao semi-automatica de regioes de interesse. Revista Brasileira de Engenharia Biomedica,16(13):109 –121.

Oliveira, R. A. P., Delamaro, M. E., and Nunes, F. L. S. (2009). O-FIm - Oracle for Im-ages. In Anais da XVI Sessao de Ferramentas do Simposio Brasileiro de Engenharia deSoftware (SBES 2009), pages 1 – 6, Fortaleza – CE – Brasil.

Weyuker, E. J. (1982). On testing non-testable programs. The Computer Journal, 25(4):465–470.