Automated Black-Box Model Generation for Smart TV ... - arXiv

1

EvoCreeper: Automated Black-Box ModelGeneration for Smart TV Applications

Bestoun S. Ahmed, and Miroslav Bures

Abstract—Smart TVs are coming to dominate the televisionmarket. This accompanied by an increase in the use of the smartTV applications (apps). Due to the increasing demand, developersneed modeling techniques to analyze these apps and assess theircomprehensiveness, completeness, and quality. In this paper, wepresent an automated strategy for generating models of smartTV apps based on a black-box reverse engineering. The strategycan be used to cumulatively construct a model for a given app byexploring the user interface in a manner consistent with the use ofa remote control device and extracting the runtime information.The strategy is based on capturing the states of the user interfaceto create a model during runtime without any knowledge ofthe internal structure of the app. We have implemented ourstrategy in a tool called EvoCreeper. The evaluation results showthat our strategy can automatically generate unique states anda comprehensive model that represents the real user interactionswith an app using a remote control device. The models thusgenerated can be used to assess the quality and completeness ofsmart TV apps in various contexts, such as the control of otherconsumer electronics in smart houses.

Index Terms—Model generation, Smart TV application, Ap-plication reverse engineering, Model-based testing.

I. INTRODUCTION

The smart TV is a modern technological device that is ahybrid of a computer and a traditional television. In additionto a conventional TV terminal, this device incorporates digitalcontent and an operating system (OS) with an Internet con-nection. Smart TVs usually provide access to broadcast media,games, digital services, various online interactive sessions,on-demand entertainment, Internet browsing, and many otherservices, and these devices are expected to become even moreintelligent, interactive, and useful in the future [1]. Invest-ments in related technological advancements by electronicscompanies and IT firms have recently been increasing. As aresult, new terminals and applications for smart TVs have beenlaunched. It is expected that these devices will soon become acommon feature of smart homes within an Internet of Things(IoT) context [2]. This explains why the smart TV market hadgrown to be worth $265 billion by 2016 [3].

Like all new smart devices, a smart TV is operated by an OSthat handles the necessary hardware interaction functionalityand a set of applications (apps) installed on the OS to providevarious services to the user. Despite the visual similarities

This study is conducted as a part of the project TACR TH02010296 “QualityAssurance for Internet of Things Technology”.

B. Ahmed is with the Department of Mathematics and Computer Science,Karlstad University, Sweden and the Department of Computer Science, CzechTechnical University, Karlovo nam. 13, Prague, Czech Republic, email:[email protected]

M. Bures are with the Department of Computer Science, Faculty ofElectrical Engineering, Czech Technical University, Karlovo nam. 13, Prague,Czech Republic

between smart TV apps and mobile apps, the mode of userinteraction with smart TV apps is different. For mobile apps,the user interacts with the touchscreen of the device (i.e., theapps) directly by hand, whereas for smart TVs, the user inter-acts with an app through another device, namely, the remotecontroller. Of course, some vendors also provide touchscreeninteractions to users. Additionally, some recent studies haveinvestigated gaze-based interactive interface design for smartTV apps [4]. However, the way in which a smart TV behaves isstill primarily based on a remote control device when it comesto the navigation of the user interface (UI) states. Moreover,the user of any TV (including smart TVs) is usually far awayfrom the screen and uses the remote controller to operate appsalmost all the time.

With the increasing number of smart TV apps, there isan urgent need for methods of modeling UIs based on userinteractions. As in the case of the UIs of desktop or mobileapps, such a model can be used for many purposes. It can beused to assess the completeness of an app during developmentor to assess the quality of the app during testing. Such a modelcan also be used to trigger requirement specifications andanalyze poorly documented legacy apps. It can also be used inthe smoke testing of apps. In fact, most developers currentlycreate mental models of UIs in order to better comprehend thesoftware.

Creating a model of the UI via the user interaction mode isa common first step in assessing the quality and completenessof a UI-based app. While model generation is a common step,the presentation of the model depends mainly on the type ofapp and the user interaction mode. For a smart TV app, if theapp is to be used on a touchscreen TV, the same state-of-the-artmodel generation methods used for mobile apps may also beuseful when deployed in the smart TV operation environment.However, there is still a need to design a model generationmethod for gaze interaction with an app, and our method maynot be useful for this purpose. Thus, in general, the model ofa smart TV app does not look like the model of a mobile appdue to the different mechanism of transitioning among the UIstates. In a mobile app, a user can easily transition among appstates by going directly from one state to another. However,for example, in a smart TV app, when the user wants to gofrom one state to another, he must pass through some set ofstates by means of the remote control device in order to reachthe destination state.

It is possible to create a model using the same state-of-the-art approaches used for the reverse engineering of mobile apps.In practice, however, that model will be useless for a smart TVapp. Normally, a model is created for some specific purpose.For example, we may create a model to generate several test

arX

iv:1

904.

0295

6v1

[cs

.SE

] 5

Apr

201

9

2

cases. However, it will be impossible to run those test cases ona real smart TV because there will be significant differencesin the transitions among the states of the UI.

In this paper, we propose a strategy called EvoCreeper1

for the automatic creation of comprehensible UI models forsmart TV apps. EvoCreeper is a generic open-source modelgenerator for smart TV apps that can generate directed graphsby reverse engineering an app without knowledge of itsinternal code structure. In this strategy, the UI states and thetransitions among them are examined in order to create themodel. The contributions of this paper can be summarized asfollows:• We present a new comprehensive automated black-box

strategy in which a smart TV app is executed on anemulator and information is extracted from the UI duringruntime.

• We present an efficient exploratory algorithm that can ex-plore the elements of a UI by simulating user interactionswith a smart TV app.

• We illustrate the implementation of our strategy withinour EvoCreeper tool, which can create directed graphmodels of smart TV apps. The models thus generated canbe used for various development and testing purposes.

• We have developed an algorithm to verify the correctnessof the generated models.

• We have empirically evaluated our strategy through fourreal case studies.

The rest of this paper is organized as follows. In Section II,we summarize the technological background of smart TV apps.This section also mentions the related challenges and the mostclosely related works in the literature. Section III summarizesthe details of our approach for generating models of smartTV apps. Section 4 presents a proof of concept for our modelgeneration approach. Section V reports the evaluation results,and Section VI summarizes the threats to the validity of ourevaluation experiment. Finally, Section VII gives concludingremarks and discusses our future research directions.

II. BACKGROUND AND RELATED WORK

Smart TV apps are developed using software developmentkits (SDKs). Each platform has its own SDK for the de-velopment of software for TV devices. For example, theAndroid and Tizen SDKs can be used for smart TV appdevelopment. Recently, a few SDKs have also begun to supportcross-platform development. For instance, the Mautilus [5]Smart TV SDK is an example development framework, butat present, the apps developed in this framework work ononly some versions of the supported devices. The Smart TVAlliance [6] was another project for supporting cross-platformdevelopment; however, this project has been inactive for sometime. In fact, the Tizen SDK is currently the SDK that is mostcommonly used since it provides a set of tools and frameworksfor the development of smart TV apps through Tizen Studiothat utilize the latest web technologies, such as JavaScript,CSS, HTML5, and W3C widget packaging, which are used by

1EvoCreeper can be downloaded here: https://bit.ly/2StQeG4

most smart TV apps. Additionally, JavaScript is used in mostapps as a standard programming language for programmingtheir behavior. The use of JavaScript endows an app withthe page jumping capability. It also enables the developer tocode complex expressions and calculation structures such asconditional branches and loops.

In general, a smart TV app can be of one of two types:installed or cloud-based. An installed TV app is a stand-aloneapp installed on a smart TV without the need for an Internetconnection, while a cloud-based TV app primarily acts as aninterface between the cloud and the TV and offers only shallowcontent (almost no additional functionality) when there is noInternet connection available.

Regardless of the visual appearance of these apps, mobileand smart TV apps are different in several significant respects.For example, the size of the screen can affect the layout of anapp. Smart TVs have wider screens than small mobile devicesdo. The background color of a smart TV app may also bedifferent from the corresponding color on mobile devices. Thesize of the icons could also be different. From the perspectiveuser interaction, smart TV apps typically involve less textentry because of the difficulty of entering text using a remotecontrol device. Most smart TV apps are designed to retrievecontent from the Internet, whereas this is not the case formobile apps, which can be standalone apps without Internetconnection interfaces [7]. The typical smart TV app is muchmore straightforward than the typical mobile app, especiallyin its design layout. We have explained these analogies anddifferences in detail in our previous study [8].

The way in which the user interacts with the app constitutesan essential difference between smart TV and mobile apps.The user of a mobile app interacts directly with the app withoutan intermediate device, while for a smart TV app, the userinteracts with the help of a remote controller. In fact, the UIsof smart TV apps are sometimes called 10-foot UIs since 10feet (3 m) is the standard distance between the user and the TV.Developers consider this distance when developing smart TVUIs [7], [9]. Using a remote control device at this distance isnot a user-friendly or responsive experience. Hence, the designof a smart TV UI must consider this significant difficulty.

Navigation in a smart TV app is achieved through a remotecontrol device. Although some new TV devices offer theability for the user to directly interact with the screen, the mostcommon form of interaction with a TV device is still througha remote control device. A remote control device includes fouressential navigation buttons: Right⇒, Left⇐, Up ⇑ and Down⇓. Additionally, a remote control device has an OK button tochoose any selected item in an app after navigating to it and aBack button←↩ to navigate back to the previous screen. Thesesix key buttons should work properly when using an app.

In addition to these six buttons, there are many other buttonson remote control devices that vary from one TV brand to an-other depending on the level of the functionalities they access.Some of them are related to the hardware functionalities of theTV itself. For example, the power button turns the TV on andoff. There are typically also ten number buttons (from 0 to9) for channel jumps and entering numbers in text fields ifnecessary.

3

The UI layout of any app plays a primary role in (black-box) model construction. A better understanding of this layoutcan lead to a more accurate model. Smart TV apps typicallyfollow one of a limited number of layout patterns. Figure 1shows the three main patterns followed by most smart TVapps. Of these, layout (b) is most commonly used since itpresents many items on a single screen.

The remote control device places constraints on the navi-gation from one state to another because it supports only onestep of navigation at a time. Hence, each move in the layout isa step. Accordingly, the transition from one state to another isnot smooth as in mobile or desktop apps. To move from onestate to another in a smart TV UI, the user may need to passthrough several other states before reaching the desired state.For nonadjacent states, more than one step is required to movefrom one to the other. As a simple example, we consider theUI of a puzzle game smart TV app, as shown in Figure 2.

Clearly, there are 12 items (each regarded as a state) inthe UI of the app in Figure 2. For clearer illustration, we havenumbered these states. For instance, state 1 is the starting state.From here, there are only two valid moves: Right to state 2 orDown to state 3. Note that the user cannot jump directly fromstate 1 to state 4; we consider this to be an invalid move (redarrow). Hence, to move from state 1 to state 4, the user mustpress Down twice, passing through state 3.

Generally, model generation for UI-based software apps isperformed frequently for many purposes during the develop-ment process, as described in the literature. Such models havebeen used in the literature for testing and quality assurancepurposes (e.g., [10]–[14]). Regardless of the app type, as longas the model is for the UI, it must simulate the users interactionwith the app in some form. Using this approach, Memonet al. [15], [16] proposed a reverse engineering techniquecalled GUITAR for modeling the UIs of desktop apps. Thetechnique starts from the main window and automaticallycaptures UI widgets to construct an event-flow graph model.That model is then used to generate test cases for programtesting. This technique has undergone considerable develop-ment in many subsequent research papers. Aho et al. [17]presented an extensive survey of those studies and techniques.This approach has also been used for model generation formobile apps. For example, Joorabchi and Mesbah proposed areverse engineering technique for generating models of mobileapps, and Amalfitano et al. [18] developed the MobiGUITARstrategy for creating models of mobile apps.

Mesbah et al. [19] also proposed a reverse engineeringtechnique called CRAWLJAX for generating models of webapps. However, this technique relies on a dynamic crawleracting on a web app and detecting the clickable states. Themodel thus generated can then be used to generate test suitesand analyze the app. In fact, this technique works like a gray-box technique rather than a black-box technique because italso scans the code of the app.

Gimblett et al. [20] tried to define a generic approachfor establishing models of interactive software to simulateuser actions. It is clear from the literature that most relatedtechniques follow the same basic principles. However, thetechnology, application type, and user interaction mechanism

introduce differences and thus pose challenges for modelgeneration.

As another class of smart devices, smart TVs are currentlybecoming increasingly popular due to the rapid developmentof apps that can be installed on these devices and the ability tocontrol other consumer electronics and connect with them inthe IoT context. Some examples of the possible applications ofsuch technology include controlling home appliances throughsmart TVs [21], [22], home sleep care with video analysisusing a smart TV app [23], controlling smart homes fromsmart TVs [24], healthcare applications for smart homes [25],smart lighting control [26], and smart security camera systems[27]. With the increasing prevalence of these applications,it is becoming increasingly difficult to ignore the uniquerequirements of quality assessment for smart TV apps andassume that the same quality assessment procedures can beused as for mobile apps. As with other smart devices, creatinga model of a smart TV app is the first step towards qualityassessment.

To create a model for a smart TV app, it is necessaryto detect the active states of the app and the transitionsamong them. While the same basic principles and concepts arefollowed as in previous approaches to detect new states andstate transitions, as mentioned previously, there are significantdifferences due to the different technologies and modes of userinteraction. In desktop GUIs and web apps, the combinationof a keyboard and mouse is still the standard mode of userinput for interacting with these apps. However, this is notthe case for mobile apps because the user interacts with thetouchscreen of the device by means of his or her fingers,and hence, different users will exhibit different interactionbehaviors. Although this issue has led to the developmentof new models for mobile apps, many of these strategiesstill benefit, wholly or partially, from the earlier methods andpractices established for the reverse engineering of desktop andweb apps. Nevertheless, the differences in interaction createmany obstacles and difficulties. For example, Nguyen et al.[16] used an event-flow graph (EFG) as a model of the UIof a desktop app, whereas Amalfitano et al. [18] used a statemachine as a model for a mobile app due to the differentnatures of their interactions. For smart TV apps, neither EFGnor state machine models are applicable. In an app of thiskind, each transition from one state to another is, in practice,just one step, while this is not the case in other apps. Forexample, in a mobile app, the distance between two icons(states) is irrelevant to the transition, whereas this is a criticalissue in a smart TV app, and this difference will lead to adifferent model.

A significant effort to formulate such a model has beenmade recently by Cui et al. [28]. In their study, a hierarchicalstate transition matrix (HSTM) was proposed as a model for anAndroid smart TV app. While this model is promising, thereis a need to further develop and formulate it for the complexstructures of different apps. However, this type of model isuseful when model optimization and reduction are needed. Infact, Cui et al. [28] used a white-box crawler approach toscan the code of an app to construct a preliminary model.This initial model contains many obsolete nodes because the

4

(a) (b) (c)

Fig. 1: Three main layout design patterns for smart TV apps [7]

1 2

3

4

Fig. 2: A puzzle game smart TV app to illustrate the valid and invalid move based on the remote device

crawler detects all views present in the UI code. Even whena view is not an active state in the UI, the crawler algorithmconsiders it as a node in the model. For example, a piece oftext in an Android app is a view; however, in practice, it isnot a clickable state. When this approach is adopted, there is apossibility of combinatorial explosion in the resulting model.Therefore, the authors proposed an algorithm for reducing andoptimizing the model by distinguishing obsolete and activenodes. However, this could be a time-consuming task and maylead to deviation from the primary research focus while alsoadding overhead to the model construction process.

In contrast to the contribution of Cui et al. [28], ourapproach does not scan the app code. Our EvoCreeper strategyinstead explores the UI of a smart TV app by examiningeach element and observing its reaction. When an element isclickable, the strategy will consider it as a state in the model.Section III presents the details of our strategy.

III. OUR MODEL CONSTRUCTION STRATEGY

In this section, we present our new strategy for automat-ically generating a model of a smart TV app. Our strategyhas been implemented with the Tizen SDK, which includesa smart TV emulator; however, the proposed framework is ageneral one, and it can be applied in combination with otherpossible emerging SDKs in the future. The strategy dependson the black-box approach to model generation and does notrequire knowledge of the internal structure of the app code.

The smart TV app is modeled as a directed graph G =(N,E), where N is a set of nodes, N 6= ∅, and E is aset of edges. E is a subset of N × N possibilities. In themodel, we define one starting node, ns ∈ N . The set Ne ⊆ Ncontains the end nodes of the graph, where Ne 6= ∅. Eachnode corresponds to a UI element (state) of the app. Eachedge corresponds to a possible transition between the statesfocused on by the cursor. These transitions can be triggered

by individual keys on the remote control device. In this study,e ∈ {Dup, Ddown, Dleft, Dright, DOK} for each e ∈ E.

If the model allows parallel edges, formally being a directedmultigraph G′ = (N,E, ns, Ne, s, t), such that N 6= ∅ is afinite set of nodes, E is a set of edges, s : E → N assignseach edge to its source node and t : E → N assigns eachedge to its target node. The node ns ∈ N is the initial/startnode of the graph G and Ne = {ne | ne ∈ N has no outgoingedge } defines nonempty set of end nodes of graph G′. In thisstudy, approach we decided to base the app model on G, whichpractically does not allow parallel edges, to have the proposalin accord with the current path-based testing approaches, forinstance, [29]–[31].

To detect all the necessary states in the app UI for presen-tation in the model, we have developed an algorithm calledEvoCreeper. At present, the concept of state detectors forthe UIs of mobile, desktop, and web apps is rather famil-iar. However, such detectors differ for each app category.As mentioned earlier, algorithms called crawlers have beendeveloped that can crawl a UI and detect states. Because weare following a new approach to state detection, we do notcall our algorithm a crawler. Rather, from a linguistic pointof view, the name ”creeper” is perfectly suited to what thealgorithm does, whereas the word ”crawler” carries a differentmeaning due to its use in web and search engine technologies.Algorithm 1 shows the steps of the EvoCreeper algorithm.

One of the problems that EvoCreeper encounters whenexploring a UI is establishing the position of the naviga-tional cursor. Technically speaking, from the perspective ofa JavaScript developer, this problem arises when a focus pointis not set in the app. For several apps on the store, no focuspoint has been set by the developers. As a result, when suchan app runs on the emulator, there are no preselected statesin its UI. Instead, the user must use the remote control deviceto choose a state. Hence, a starting point for the navigator

5

Input: v1 is the starting or user selected statesOutput: List of states to be modeled Lv

Iteration It←1Maximum Iteration Itmax ← maxwhile ((It < Itmax) ‖ (newV iew 6= null)) do

Use v1 as a start pointFrom v1 generate five possible directions DUp ,DDown , DLeft , DRight, and DOK

foreach direction D doNavigate a stepMonitor emulator log for reactionif newState = Active then

if the state is not duplicate thenAdd newState to Lv

elseBack to the parental state

endRecord the in/out transitions

endendIt++

endend

Algorithm 1: EvoCreeper Steps

is missing. This problem is common with cloud-based smartTV apps because the UI changes in real time with the cloudcontent. Therefore, our strategy starts by checking the initialcursor condition. If a focus point is not set in the app, thestrategy starts by asking the user to choose at least one statein the UI from which to start. From this state, the creeperwill start creeping the UI evolutionarily and incrementally. Ifa focus point is already set, the strategy will proceed withoutasking the user for input.

The algorithm has five directions DUp, DDown, DLeft,DRight, and DOK in which it can move from each state. Whena new state is discovered in each direction (i.e., newState =Active), the algorithm will add it to the list of states to bemodeled, Lv . The algorithm will continue until no new statesare discovered. At this stage, the algorithm will choose tomove back to the parent state. As an alternative stoppingcriterion, the algorithm will perform some preset number ofiterations to avoid the possibility of an endless discovery loopthat is encountered in some special cases of cloud-based apps.

In the post-processing phase, states Lv are converted tonodes of the app model N . In this phase of the strategy, weused the direct mapping between the states and N , practicallyspeaking, for each l ∈ Lv , an original n ∈ N is created.

Figure 3 shows the directed graph model constructed byour strategy for the CineMup smart TV app. To illustrate thedetail of the generated model, Figure 3 also shows a snapshotof three nodes in the graph. Here, each node has a uniqueidentifier number preceded by the actual name of the state onthe smart TV app. The model also records the actual transitionsthat can be performed from a state to another. Each transitionname is preceded by the name of the state that is originatedfrom.

The app model G can be subsequently used for two principal

purposes: (1) model checking, where a potential design sub-optimalities can be detected, and (2) automated path-basedtest case generation, being the major expected use case. Here,the goal is to generate a set of test cases T that satisfy adefined test coverage criteria and counted optimal by a testset optimality criteria.

Respecting the standard approaches used in the field, atest case t ∈ T is a sequence of nodes n1, n2, .., nn, witha sequence of edges e1, e2, .., en−1, where ei = (ni,ni+1),ei ∈ E. The test case t starts with the start node ns (n1 = ns)and ends with a G end node (nn ∈ Ne) [29], [30]. Testcoverage criteria determine the strengths of the T in the senseof number of alternative paths that are toured in the app duringthe tests and vary from low test coverage levels (node coverageor edge coverage) to very intense levels as all paths coverage[29]. For the test set optimality criteria, several options canbe employed, for instance, the number of test cases, |T |,

average length of the test cases, |t| = 1

|T |

|T |∑i=1

|ti|, ti ∈ T ,

or total length of a test set l =

|T |∑i=1

|ti|, ti ∈ T [30], [31].

Such automated generation of test cases, which is subject ofother studies, for instance, [30]–[34] open variety of optionsfor construction of automated test frameworks for Smart TVapps and document the applicability potential of our strategy.

In the following section, we present an example as agraphical proof of concept for Algorithm 1.

IV. PROOF OF CONCEPT

In this section, we present a proof of concept for theEvoCreeper concept introduced in Algorithm 1. Here, weconsider a cloud-based app as a pilot example because this isthe most difficult scenario. As shown in Figure 4, each activewindow has 12 states, and when the user shifts down or tothe right, new states may appear. We consider three iterationsof the algorithm. We assume that the user will choose v1 asthe starting state. In fact, v1 is the worst-case choice amongthe states; we observe that choosing the state in the middleof the window instead may lead to fewer iterations and betterrecognition of the states. From v1, the algorithm will considerfour main directions: DUp, DDown, DLeft, and DRight. Forthis proof of concept, we do not consider DOK because we areinterested in the exploration of the current window, whereasDOK will probably take the algorithm to another window.

In each direction, the creeper algorithm will check fornew states, which are most likely new elements in the UI.Considering the first iteration and starting from v1, the up andleft directions, Du and Dl, do not lead to new states, whilethe right direction, Dr, leads to v2, and the down direction,Dd, leads to v5. In the next iteration, the algorithm will startfrom the newly discovered states (here, v2 and v5). From v2,the new states v3 and v6 are identified by the algorithm. Inaddition, v1 is discovered in the Dl direction; however, thisstate is ignored by the algorithm since it is already included inthe state list. From v5, the states v1, v9, and v6 are discoveredin the three directions Du, Dd, and Dr, respectively; however,only v9 is considered a new state.

6

12/28/2018 WHOLE GRAPH.svg

file:///Users/bestoun/Desktop/WHOLE%20GRAPH.svg 1/1

START

UPCOMING_0

enter

TOPRATED_2

UPCOMING_0_right

NOWPLAYING_1

UPCOMING_0_down

RalphBreakstheInternet_606

UPCOMING_0_enter

POPULARTV_3

NOWPLAYING_1_right

NOWPLAYING_1_up

Venom_486

NOWPLAYING_1_enter

TOPRATED_2_left

POPULAR_5

TOPRATED_2_rightTOPRATED_2_down

DilwaleDulhaniaLeJayenge_246

TOPRATED_2_enter

POPULARTV_3_left

TOPTV_4

POPULARTV_3_right

POPULARTV_3_up

TheFlash_366

POPULARTV_3_enter

TOPTV_4_left

TOPTV_4_up

IAmNotanAnimal_6

TOPTV_4_enter

POPULAR_5_left

POPULAR_5_down

Venom_126

POPULAR_5_enter

BandofBrothers_7

IAmNotanAnimal_6_left

RickandMorty_8

IAmNotanAnimal_6_right

PlayTrailer_121

IAmNotanAnimal_6_enter

IAmNotanAnimal_6_back

DeathNote_9

BandofBrothers_7_left

BandofBrothers_7_right

PlayTrailer_116

BandofBrothers_7_enter

BandofBrothers_7_back

RickandMorty_8_left

PlanetEarth_25

RickandMorty_8_right

PlayTrailer_26

RickandMorty_8_enter

RickandMorty_8_back

TheSopranos_10

DeathNote_9_left

DeathNote_9_right

PlayTrailer_111

DeathNote_9_enter

DeathNote_9_back

PeepShow_11

TheSopranos_10_left

TheSopranos_10_right

PlayTrailer_106

TheSopranos_10_enter

TheSopranos_10_back

StarTrekTheNextGeneration_12

PeepShow_11_left

PeepShow_11_right

PlayTrailer_101

PeepShow_11_enter

PeepShow_11_back

Firefly_13

StarTrekTheNextGeneration_12_left

StarTrekTheNextGeneration_12_right

PlayTrailer_96

StarTrekTheNextGeneration_12_enter

StarTrekTheNextGeneration_12_back

PeakyBlinders_14

Firefly_13_left

Firefly_13_right

PlayTrailer_91

Firefly_13_enter

Firefly_13_back

AvatarTheLastAirbender_15

PeakyBlinders_14_left

PeakyBlinders_14_right

PlayTrailer_86

PeakyBlinders_14_enter

PeakyBlinders_14_back

OvertheGardenWall_16

AvatarTheLastAirbender_15_left

AvatarTheLastAirbender_15_right

PlayTrailer_81

AvatarTheLastAirbender_15_enter

AvatarTheLastAirbender_15_back

Fargo_17

OvertheGardenWall_16_left

OvertheGardenWall_16_right

PlayTrailer_76

OvertheGardenWall_16_enter

OvertheGardenWall_16_back

FullmetalAlchemistBrotherhood_18

Fargo_17_left

Fargo_17_right

PlayTrailer_71

Fargo_17_enter

Fargo_17_back

Sherlock_19

FullmetalAlchemistBrotherhood_18_left

FullmetalAlchemistBrotherhood_18_right

PlayTrailer_66

FullmetalAlchemistBrotherhood_18_enter

FullmetalAlchemistBrotherhood_18_back

TheWire_20

Sherlock_19_left

Sherlock_19_right

PlayTrailer_61

Sherlock_19_enter

Sherlock_19_back

CowboyBebop_21

TheWire_20_left

TheWire_20_right

PlayTrailer_56

TheWire_20_enter

TheWire_20_back

BreakingBad_22

CowboyBebop_21_left

CowboyBebop_21_right

PlayTrailer_51

CowboyBebop_21_enter

CowboyBebop_21_back

PlanetEarthII_23

BreakingBad_22_left

BreakingBad_22_right

PlayTrailer_46

BreakingBad_22_enter

BreakingBad_22_back

StrangerThings_24

PlanetEarthII_23_left

PlanetEarthII_23_right

PlayTrailer_41

PlanetEarthII_23_enter

PlanetEarthII_23_back

StrangerThings_24_left

StrangerThings_24_right

PlayTrailer_36

StrangerThings_24_enter

StrangerThings_24_back

PlanetEarth_25_left

PlanetEarth_25_right

PlayTrailer_31

PlanetEarth_25_enter

PlanetEarth_25_back

ShowImages_27

PlayTrailer_26_right

end_30

PlayTrailer_26_enter

PlayTrailer_26_back

ShowImages_27_left

Close_28

ShowImages_27_right

end_29

ShowImages_27_enter

ShowImages_27_back

Close_28_left

Close_28_back

ShowImages_32


end_35


PlayTrailer_31_back

ShowImages_32_left

Close_33

ShowImages_32_right

end_34

ShowImages_32_enter

ShowImages_32_back

Close_33_left

Close_33_back

ShowImages_37


end_40


PlayTrailer_36_back

ShowImages_37_left

Close_38

ShowImages_37_right

end_39

ShowImages_37_enter

ShowImages_37_back

Close_38_left

Close_38_back

ShowImages_42


end_45


PlayTrailer_41_back

ShowImages_42_left

Close_43

ShowImages_42_right

end_44

ShowImages_42_enter

ShowImages_42_back

Close_43_left

Close_43_back

ShowImages_47


end_50


PlayTrailer_46_back

ShowImages_47_left

Close_48

ShowImages_47_right

end_49

ShowImages_47_enter

ShowImages_47_back

Close_48_left

Close_48_back

ShowImages_52


end_55


PlayTrailer_51_back

ShowImages_52_left

Close_53

ShowImages_52_right

end_54

ShowImages_52_enter

ShowImages_52_back

Close_53_left

Close_53_back

ShowImages_57


end_60


PlayTrailer_56_back

ShowImages_57_left

Close_58

ShowImages_57_right

end_59

ShowImages_57_enter

ShowImages_57_back

Close_58_left

Close_58_back

ShowImages_62


end_65


PlayTrailer_61_back

ShowImages_62_left

Close_63

ShowImages_62_right

end_64

ShowImages_62_enter

ShowImages_62_back

Close_63_left

Close_63_back

ShowImages_67


end_70


PlayTrailer_66_back

ShowImages_67_left

Close_68

ShowImages_67_right

end_69

ShowImages_67_enter

ShowImages_67_back

Close_68_left

Close_68_back

ShowImages_72


end_75


PlayTrailer_71_back

ShowImages_72_left

Close_73

ShowImages_72_right

end_74

ShowImages_72_enter

ShowImages_72_back

Close_73_left

Close_73_back

ShowImages_77


end_80


PlayTrailer_76_back

ShowImages_77_left

Close_78

ShowImages_77_right

end_79

ShowImages_77_enter

ShowImages_77_back

Close_78_left

Close_78_back

ShowImages_82


end_85


PlayTrailer_81_back

ShowImages_82_left

Close_83

ShowImages_82_right

end_84

ShowImages_82_enter

ShowImages_82_back

Close_83_left

Close_83_back

ShowImages_87


end_90


PlayTrailer_86_back

ShowImages_87_left

Close_88

ShowImages_87_right

end_89

ShowImages_87_enter

ShowImages_87_back

Close_88_left

Close_88_back

ShowImages_92


end_95


PlayTrailer_91_back

ShowImages_92_left

Close_93

ShowImages_92_right

end_94

ShowImages_92_enter

ShowImages_92_back

Close_93_left

Close_93_back

ShowImages_97


end_100


PlayTrailer_96_back

ShowImages_97_left

Close_98

ShowImages_97_right

end_99

ShowImages_97_enter

ShowImages_97_back

Close_98_left

Close_98_back

ShowImages_102


end_105


PlayTrailer_101_back

ShowImages_102_left

Close_103

ShowImages_102_right

end_104

ShowImages_102_enter

ShowImages_102_back

Close_103_left

Close_103_back

ShowImages_107


end_110



ShowImages_107_left

Close_108


end_109


ShowImages_107_back

Close_108_left

Close_108_back

ShowImages_112


end_115



ShowImages_112_left

Close_113


end_114


ShowImages_112_back

Close_113_left

Close_113_back

ShowImages_117


end_120



ShowImages_117_left

Close_118


end_119


ShowImages_117_back

Close_118_left

Close_118_back

ShowImages_122


end_125



ShowImages_122_left

Close_123


end_124


ShowImages_122_back

Close_123_left

Close_123_back

TheMeg_127

Venom_126_left

KungFuLeague_128

Venom_126_right

PlayTrailer_241

Venom_126_enter

Venom_126_back

NightSchool_129

TheMeg_127_left

TheMeg_127_right

PlayTrailer_236

TheMeg_127_enter

TheMeg_127_back

KungFuLeague_128_left

RobinHood_145

KungFuLeague_128_right

PlayTrailer_146

KungFuLeague_128_enter

KungFuLeague_128_back

TheHousewithaClockinItsWalls_130

NightSchool_129_left

NightSchool_129_right

PlayTrailer_231

NightSchool_129_enter

NightSchool_129_back

AntManandtheWasp_131

TheHousewithaClockinItsWalls_130_left

TheHousewithaClockinItsWalls_130_right

PlayTrailer_226

TheHousewithaClockinItsWalls_130_enter

TheHousewithaClockinItsWalls_130_back

TheEqualizer2_132

AntManandtheWasp_131_left

AntManandtheWasp_131_right

PlayTrailer_221

AntManandtheWasp_131_enter

AntManandtheWasp_131_back

Peppermint_133

TheEqualizer2_132_left

TheEqualizer2_132_right

PlayTrailer_216

TheEqualizer2_132_enter

TheEqualizer2_132_back

TheGrinch_134

Peppermint_133_left

Peppermint_133_right

PlayTrailer_211

Peppermint_133_enter

Peppermint_133_back

TheNun_135

TheGrinch_134_left

TheGrinch_134_right

PlayTrailer_206

TheGrinch_134_enter

TheGrinch_134_back

Halloween_136

TheNun_135_left

TheNun_135_right

PlayTrailer_201

TheNun_135_enter

TheNun_135_back


Halloween_136_left

Halloween_136_right

PlayTrailer_196

Halloween_136_enter

Halloween_136_back

AvengersInfinityWar_138

RalphBreakstheInternet_137_left

RalphBreakstheInternet_137_right

PlayTrailer_191

RalphBreakstheInternet_137_enter

RalphBreakstheInternet_137_back

MissionImpossibleFallout_139

AvengersInfinityWar_138_left

AvengersInfinityWar_138_right

PlayTrailer_186

AvengersInfinityWar_138_enter

AvengersInfinityWar_138_back

ThePredator_140

MissionImpossibleFallout_139_left

MissionImpossibleFallout_139_right

PlayTrailer_181

MissionImpossibleFallout_139_enter

MissionImpossibleFallout_139_back

BohemianRhapsody_141

ThePredator_140_left

ThePredator_140_right

PlayTrailer_176

ThePredator_140_enter

ThePredator_140_back

Smallfoot_142

BohemianRhapsody_141_left

BohemianRhapsody_141_right

PlayTrailer_171

BohemianRhapsody_141_enter

BohemianRhapsody_141_back

TheSevenDeadlySinsPrisonersoftheSky_143

Smallfoot_142_left

Smallfoot_142_right

PlayTrailer_166

Smallfoot_142_enter

Smallfoot_142_back

FantasticBeastsTheCrimesofGrindelwald_144

TheSevenDeadlySinsPrisonersoftheSky_143_left

TheSevenDeadlySinsPrisonersoftheSky_143_right

PlayTrailer_161

TheSevenDeadlySinsPrisonersoftheSky_143_enter

TheSevenDeadlySinsPrisonersoftheSky_143_back

FantasticBeastsTheCrimesofGrindelwald_144_left

FantasticBeastsTheCrimesofGrindelwald_144_right

PlayTrailer_156

FantasticBeastsTheCrimesofGrindelwald_144_enter

FantasticBeastsTheCrimesofGrindelwald_144_back

RobinHood_145_left

RobinHood_145_right

PlayTrailer_151

RobinHood_145_enter

RobinHood_145_back

ShowImages_147


end_150



ShowImages_147_left

Close_148


end_149


ShowImages_147_back

Close_148_left

Close_148_backShowImages_152


end_155



ShowImages_152_left

Close_153


end_154


ShowImages_152_back

Close_153_left

Close_153_back

ShowImages_157


end_160



ShowImages_157_left

Close_158


end_159


ShowImages_157_back

Close_158_left

Close_158_back

ShowImages_162


end_165



ShowImages_162_left

Close_163


end_164


ShowImages_162_back

Close_163_left

Close_163_back

ShowImages_167


end_170



ShowImages_167_left

Close_168


end_169


ShowImages_167_back

Close_168_left

Close_168_back

ShowImages_172


end_175



ShowImages_172_left

Close_173


end_174


ShowImages_172_back

Close_173_left

Close_173_back

ShowImages_177


end_180



ShowImages_177_left

Close_178


end_179


ShowImages_177_back

Close_178_left

Close_178_back

ShowImages_182


end_185



ShowImages_182_left

Close_183


end_184


ShowImages_182_back

Close_183_left

Close_183_back

ShowImages_187


end_190



ShowImages_187_left

Close_188


end_189


ShowImages_187_back

Close_188_left

Close_188_back

ShowImages_192


end_195



ShowImages_192_left

Close_193


end_194


ShowImages_192_back

Close_193_left

Close_193_back

ShowImages_197


end_200



ShowImages_197_left

Close_198


end_199


ShowImages_197_back

Close_198_left

Close_198_back

ShowImages_202


end_205



ShowImages_202_left

Close_203


end_204


ShowImages_202_back

Close_203_left

Close_203_back

ShowImages_207


end_210



ShowImages_207_left

Close_208


end_209


ShowImages_207_back

Close_208_left

Close_208_back

ShowImages_212


end_215



ShowImages_212_left

Close_213


end_214


ShowImages_212_back

Close_213_left

Close_213_back

ShowImages_217


end_220



ShowImages_217_left

Close_218


end_219


ShowImages_217_back

Close_218_left

Close_218_back

ShowImages_222


end_225



ShowImages_222_left

Close_223


end_224


ShowImages_222_back

Close_223_left

Close_223_back

ShowImages_227


end_230



ShowImages_227_left

Close_228


end_229


ShowImages_227_back

Close_228_left

Close_228_back

ShowImages_232


end_235



ShowImages_232_left

Close_233


end_234


ShowImages_232_back

Close_233_left

Close_233_back

ShowImages_237


end_240



ShowImages_237_left

Close_238


end_239


ShowImages_237_back

Close_238_left

Close_238_back

ShowImages_242


end_245



ShowImages_242_left

Close_243


end_244


ShowImages_242_back

Close_243_left

Close_243_back

GraveoftheFireflies_247

DilwaleDulhaniaLeJayenge_246_left

TheShawshankRedemption_248

DilwaleDulhaniaLeJayenge_246_right

PlayTrailer_361

DilwaleDulhaniaLeJayenge_246_enter

DilwaleDulhaniaLeJayenge_246_back

OneFlewOvertheCuckoosNest_249

GraveoftheFireflies_247_left

GraveoftheFireflies_247_right

PlayTrailer_356

GraveoftheFireflies_247_enter

GraveoftheFireflies_247_back

TheShawshankRedemption_248_left

TheGodfather_265

TheShawshankRedemption_248_right

PlayTrailer_266

TheShawshankRedemption_248_enter

TheShawshankRedemption_248_back

GoodFellas_250

OneFlewOvertheCuckoosNest_249_left

OneFlewOvertheCuckoosNest_249_right

PlayTrailer_351

OneFlewOvertheCuckoosNest_249_enter

OneFlewOvertheCuckoosNest_249_back

SevenSamurai_251

GoodFellas_250_left

GoodFellas_250_right

PlayTrailer_346

GoodFellas_250_enter

GoodFellas_250_back

CinemaParadiso_252

SevenSamurai_251_left

SevenSamurai_251_right

PlayTrailer_341

SevenSamurai_251_enter

SevenSamurai_251_back

ForrestGump_253

CinemaParadiso_252_left

CinemaParadiso_252_right

PlayTrailer_336

CinemaParadiso_252_enter

CinemaParadiso_252_back

OnceUponaTimeinAmerica_254

ForrestGump_253_left

ForrestGump_253_right

PlayTrailer_331

ForrestGump_253_enter

ForrestGump_253_back

FightClub_255

OnceUponaTimeinAmerica_254_left

OnceUponaTimeinAmerica_254_right

PlayTrailer_326

OnceUponaTimeinAmerica_254_enter

OnceUponaTimeinAmerica_254_back

TheDarkKnight_256

FightClub_255_left

FightClub_255_right

PlayTrailer_321

FightClub_255_enter

FightClub_255_back

Psycho_257

TheDarkKnight_256_left

TheDarkKnight_256_right

PlayTrailer_316

TheDarkKnight_256_enter

TheDarkKnight_256_back

PulpFiction_258

Psycho_257_left

Psycho_257_right

PlayTrailer_311

Psycho_257_enter

Psycho_257_back

TheGreenMile_259

PulpFiction_258_left

PulpFiction_258_right

PlayTrailer_306

PulpFiction_258_enter

PulpFiction_258_back

LifeIsBeautiful_260

TheGreenMile_259_left

TheGreenMile_259_right

PlayTrailer_301

TheGreenMile_259_enter

TheGreenMile_259_back

SpiritedAway_261

LifeIsBeautiful_260_left

LifeIsBeautiful_260_right

PlayTrailer_296

LifeIsBeautiful_260_enter

LifeIsBeautiful_260_back

TheGodfatherPartII_262

SpiritedAway_261_left

SpiritedAway_261_right

PlayTrailer_291

SpiritedAway_261_enter

SpiritedAway_261_back

SchindlersList_263

TheGodfatherPartII_262_left

TheGodfatherPartII_262_right

PlayTrailer_286

TheGodfatherPartII_262_enter

TheGodfatherPartII_262_back

YourName_264

SchindlersList_263_left

SchindlersList_263_right

PlayTrailer_281

SchindlersList_263_enter

SchindlersList_263_back

YourName_264_left

YourName_264_right

PlayTrailer_276

YourName_264_enter

YourName_264_back

TheGodfather_265_left

TheGodfather_265_right

PlayTrailer_271

TheGodfather_265_enter

TheGodfather_265_back

ShowImages_267


end_270



ShowImages_267_left

Close_268


end_269


ShowImages_267_back

Close_268_left

Close_268_back

ShowImages_272


end_275



ShowImages_272_left

Close_273


end_274


ShowImages_272_back

Close_273_left

Close_273_back

ShowImages_277


end_280



ShowImages_277_left

Close_278


end_279


ShowImages_277_back

Close_278_left

Close_278_back

ShowImages_282


end_285



ShowImages_282_left

Close_283


end_284


ShowImages_282_back

Close_283_left

Close_283_back

ShowImages_287


end_290



ShowImages_287_left

Close_288


end_289


ShowImages_287_back

Close_288_left

Close_288_back

ShowImages_292


end_295



ShowImages_292_left

Close_293


end_294


ShowImages_292_back

Close_293_left

Close_293_back

ShowImages_297


end_300



ShowImages_297_left

Close_298


end_299


ShowImages_297_back

Close_298_left

Close_298_back

ShowImages_302


end_305



ShowImages_302_left

Close_303


end_304


ShowImages_302_back

Close_303_left

Close_303_back

ShowImages_307


end_310



ShowImages_307_left

Close_308


end_309


ShowImages_307_back

Close_308_left

Close_308_back

ShowImages_312


end_315



ShowImages_312_left

Close_313


end_314


ShowImages_312_back

Close_313_left

Close_313_back

ShowImages_317


end_320



ShowImages_317_left

Close_318


end_319


ShowImages_317_back

Close_318_left

Close_318_back

ShowImages_322


end_325



ShowImages_322_left

Close_323


end_324


ShowImages_322_back

Close_323_left

Close_323_back

ShowImages_327


end_330



ShowImages_327_left

Close_328


end_329


ShowImages_327_back

Close_328_left

Close_328_back

ShowImages_332


end_335



ShowImages_332_left

Close_333


end_334


ShowImages_332_back

Close_333_left

Close_333_back

ShowImages_337


end_340



ShowImages_337_left

Close_338


end_339


ShowImages_337_back

Close_338_left

Close_338_back

ShowImages_342


end_345



ShowImages_342_left

Close_343


end_344


ShowImages_342_back

Close_343_left

Close_343_back

ShowImages_347


end_350



ShowImages_347_left

Close_348


end_349


ShowImages_347_back

Close_348_left

Close_348_back

ShowImages_352


end_355



ShowImages_352_left

Close_353


end_354


ShowImages_352_back

Close_353_left

Close_353_back

ShowImages_357


end_360



ShowImages_357_left

Close_358


end_359


ShowImages_357_back

Close_358_left

Close_358_back

ShowImages_362


end_365



ShowImages_362_left

Close_363


end_364


ShowImages_362_back

Close_363_left

Close_363_back

BlackLightning_367

TheFlash_366_left

TheRealBrosofSimiValley_368

TheFlash_366_right

PlayTrailer_481

TheFlash_366_enter

TheFlash_366_back

Riverdale_369

BlackLightning_367_left

BlackLightning_367_right

PlayTrailer_476

BlackLightning_367_enter

BlackLightning_367_back

TheRealBrosofSimiValley_368_left

TheBigBangTheory_385

TheRealBrosofSimiValley_368_right

PlayTrailer_386

TheRealBrosofSimiValley_368_enter

TheRealBrosofSimiValley_368_back

DCsLegendsofTomorrow_370

Riverdale_369_left

Riverdale_369_right

PlayTrailer_471

Riverdale_369_enter

Riverdale_369_back

GreysAnatomy_371

DCsLegendsofTomorrow_370_left

DCsLegendsofTomorrow_370_right

PlayTrailer_466

DCsLegendsofTomorrow_370_enter

DCsLegendsofTomorrow_370_back

Supergirl_372

GreysAnatomy_371_left

GreysAnatomy_371_right

PlayTrailer_461

GreysAnatomy_371_enter

GreysAnatomy_371_back

TheWalkingDead_373

Supergirl_372_left

Supergirl_372_right

PlayTrailer_456

Supergirl_372_enter

Supergirl_372_back

MarvelsIronFist_374

TheWalkingDead_373_left

TheWalkingDead_373_right

PlayTrailer_451

TheWalkingDead_373_enter

TheWalkingDead_373_back

TheSimpsons_375

MarvelsIronFist_374_left

MarvelsIronFist_374_right

PlayTrailer_446

MarvelsIronFist_374_enter

MarvelsIronFist_374_back

DoctorWho_376

TheSimpsons_375_left

TheSimpsons_375_right

PlayTrailer_441

TheSimpsons_375_enter

TheSimpsons_375_back

GameofThrones_377

DoctorWho_376_left

DoctorWho_376_right

PlayTrailer_436

DoctorWho_376_enter

DoctorWho_376_back

NCIS_378

GameofThrones_377_left

GameofThrones_377_right

PlayTrailer_431

GameofThrones_377_enter

GameofThrones_377_back

Supernatural_379

NCIS_378_left

NCIS_378_right

PlayTrailer_426

NCIS_378_enter

NCIS_378_back

LawOrderSpecialVictimsUnit_380

Supernatural_379_left

Supernatural_379_right

PlayTrailer_421

Supernatural_379_enter

Supernatural_379_back

DupurThakurpo_381

LawOrderSpecialVictimsUnit_380_left

LawOrderSpecialVictimsUnit_380_right

PlayTrailer_416

LawOrderSpecialVictimsUnit_380_enter

LawOrderSpecialVictimsUnit_380_back

Arrow_382

DupurThakurpo_381_left

DupurThakurpo_381_right

PlayTrailer_411

DupurThakurpo_381_enter

DupurThakurpo_381_back

Vikings_383

Arrow_382_left

Arrow_382_right

PlayTrailer_406

Arrow_382_enter

Arrow_382_back

DragonBall_384

Vikings_383_left

Vikings_383_right

PlayTrailer_401

Vikings_383_enter

Vikings_383_back

DragonBall_384_left

DragonBall_384_right

PlayTrailer_396

DragonBall_384_enter

DragonBall_384_back

TheBigBangTheory_385_left

TheBigBangTheory_385_right

PlayTrailer_391

TheBigBangTheory_385_enter

TheBigBangTheory_385_back

ShowImages_387


end_390



ShowImages_387_left

Close_388


end_389


ShowImages_387_back

Close_388_left

Close_388_back

ShowImages_392


end_395



ShowImages_392_left

Close_393


end_394


ShowImages_392_back

Close_393_left

Close_393_back

ShowImages_397


end_400



ShowImages_397_left

Close_398


end_399


ShowImages_397_back

Close_398_left

Close_398_back

ShowImages_402


end_405



ShowImages_402_left

Close_403


end_404


ShowImages_402_back

Close_403_left

Close_403_back

ShowImages_407


end_410



ShowImages_407_left

Close_408


end_409


ShowImages_407_back

Close_408_left

Close_408_back

ShowImages_412


end_415



ShowImages_412_left

Close_413


end_414


ShowImages_412_back

Close_413_left

Close_413_back

ShowImages_417


end_420



ShowImages_417_left

Close_418


end_419


ShowImages_417_back

Close_418_left

Close_418_back

ShowImages_422


end_425



ShowImages_422_left

Close_423


end_424


ShowImages_422_back

Close_423_left

Close_423_back

ShowImages_427


end_430



ShowImages_427_left

Close_428


end_429


ShowImages_427_back

Close_428_left

Close_428_back

ShowImages_432


end_435



ShowImages_432_left

Close_433


end_434


ShowImages_432_back

Close_433_left

Close_433_back

ShowImages_437


end_440



ShowImages_437_left

Close_438


end_439


ShowImages_437_back

Close_438_left

Close_438_back

ShowImages_442


end_445



ShowImages_442_left

Close_443


end_444


ShowImages_442_back

Close_443_left

Close_443_back

ShowImages_447


end_450



ShowImages_447_left

Close_448


end_449


ShowImages_447_back

Close_448_left

Close_448_back

ShowImages_452


end_455



ShowImages_452_left

Close_453


end_454


ShowImages_452_back

Close_453_leftClose_453_back

ShowImages_457


end_460



ShowImages_457_left

Close_458


end_459


ShowImages_457_back

Close_458_left

Close_458_back

ShowImages_462


end_465



ShowImages_462_left

Close_463


end_464


ShowImages_462_back

Close_463_left

Close_463_back

ShowImages_467


end_470



ShowImages_467_left

Close_468


end_469


ShowImages_467_back

Close_468_left Close_468_back

ShowImages_472


end_475



ShowImages_472_left

Close_473


end_474


ShowImages_472_back

Close_473_left

Close_473_back

ShowImages_477


end_480



ShowImages_477_left

Close_478


end_479


ShowImages_477_back

Close_478_left

Close_478_back

ShowImages_482


end_485



ShowImages_482_left

Close_483


end_484


ShowImages_482_back

Close_483_left

Close_483_back

Alpha_487

Venom_486_left

RobinHood_488

Venom_486_right

PlayTrailer_601

Venom_486_enter

Venom_486_back

HunterKiller_489

Alpha_487_left

Alpha_487_right

PlayTrailer_596

Alpha_487_enter

Alpha_487_back

RobinHood_488_left

FantasticBeastsTheCrimesofGrindelwald_505

RobinHood_488_right

PlayTrailer_506

RobinHood_488_enter

RobinHood_488_back

SpeedKills_490

HunterKiller_489_left

HunterKiller_489_right

PlayTrailer_591

HunterKiller_489_enter

HunterKiller_489_back

MortalEngines_491

SpeedKills_490_left

SpeedKills_490_right

PlayTrailer_586

SpeedKills_490_enter

SpeedKills_490_back

AStarIsBorn_492

MortalEngines_491_left

MortalEngines_491_right

PlayTrailer_581

MortalEngines_491_enter

MortalEngines_491_back

Mile22_493

AStarIsBorn_492_left

AStarIsBorn_492_right

PlayTrailer_576

AStarIsBorn_492_enter

AStarIsBorn_492_back

FirstMan_494

Mile22_493_left

Mile22_493_right

PlayTrailer_571

Mile22_493_enter

Mile22_493_back

TheChristmasChronicles_495

FirstMan_494_left

FirstMan_494_right

PlayTrailer_566

FirstMan_494_enter

FirstMan_494_back

NightSchool_496

TheChristmasChronicles_495_left

TheChristmasChronicles_495_right

PlayTrailer_561

TheChristmasChronicles_495_enter

TheChristmasChronicles_495_back

TheHousewithaClockinItsWalls_497

NightSchool_496_left

NightSchool_496_right

PlayTrailer_556

NightSchool_496_enter

NightSchool_496_back

Peppermint_498

TheHousewithaClockinItsWalls_497_left

TheHousewithaClockinItsWalls_497_right

PlayTrailer_551

TheHousewithaClockinItsWalls_497_enter

TheHousewithaClockinItsWalls_497_back

TheGrinch_499

Peppermint_498_left

Peppermint_498_right

PlayTrailer_546

Peppermint_498_enter

Peppermint_498_back

Halloween_500

TheGrinch_499_left

TheGrinch_499_right

PlayTrailer_541

TheGrinch_499_enter

TheGrinch_499_back


Halloween_500_left

Halloween_500_right

PlayTrailer_536

Halloween_500_enter

Halloween_500_back

BohemianRhapsody_502



PlayTrailer_531



Smallfoot_503

BohemianRhapsody_502_left

BohemianRhapsody_502_right

PlayTrailer_526

BohemianRhapsody_502_enter

BohemianRhapsody_502_back

TheSevenDeadlySinsPrisonersoftheSky_504

Smallfoot_503_left

Smallfoot_503_right

PlayTrailer_521

Smallfoot_503_enter

Smallfoot_503_back

TheSevenDeadlySinsPrisonersoftheSky_504_left

TheSevenDeadlySinsPrisonersoftheSky_504_right

PlayTrailer_516

TheSevenDeadlySinsPrisonersoftheSky_504_enter

TheSevenDeadlySinsPrisonersoftheSky_504_back

FantasticBeastsTheCrimesofGrindelwald_505_left

FantasticBeastsTheCrimesofGrindelwald_505_right

PlayTrailer_511

FantasticBeastsTheCrimesofGrindelwald_505_enter

FantasticBeastsTheCrimesofGrindelwald_505_back

ShowImages_507


end_510



ShowImages_507_left

Close_508


end_509


ShowImages_507_back

Close_508_left

Close_508_back

ShowImages_512


end_515



ShowImages_512_left

Close_513


end_514


ShowImages_512_back

Close_513_left

Close_513_back

ShowImages_517


end_520



ShowImages_517_left

Close_518


end_519


ShowImages_517_back

Close_518_left

Close_518_back

ShowImages_522


end_525



ShowImages_522_left

Close_523


end_524


ShowImages_522_back

Close_523_left

Close_523_back

ShowImages_527


end_530



ShowImages_527_left

Close_528


end_529


ShowImages_527_back

Close_528_left

Close_528_back

ShowImages_532


end_535



ShowImages_532_left

Close_533


end_534


ShowImages_532_back

Close_533_left

Close_533_back

ShowImages_537


end_540



ShowImages_537_left

Close_538


end_539


ShowImages_537_back

Close_538_left

Close_538_back

ShowImages_542


end_545



ShowImages_542_left

Close_543


end_544


ShowImages_542_back

Close_543_left

Close_543_back

ShowImages_547


end_550



ShowImages_547_left

Close_548


end_549


ShowImages_547_back

Close_548_left

Close_548_back

ShowImages_552


end_555



ShowImages_552_left

Close_553


end_554


ShowImages_552_back

Close_553_left

Close_553_back

ShowImages_557


end_560



ShowImages_557_left

Close_558


end_559


ShowImages_557_back

Close_558_left

Close_558_back ShowImages_562


end_565



ShowImages_562_left

Close_563


end_564


ShowImages_562_back

Close_563_left

Close_563_back

ShowImages_567


end_570



ShowImages_567_left

Close_568


end_569


ShowImages_567_back

Close_568_left

Close_568_back

ShowImages_572


end_575



ShowImages_572_left

Close_573


end_574


ShowImages_572_back

Close_573_left

Close_573_back

ShowImages_577


end_580



ShowImages_577_left

Close_578


end_579


ShowImages_577_back

Close_578_left

Close_578_back

ShowImages_582


end_585



ShowImages_582_left

Close_583


end_584


ShowImages_582_back

Close_583_left Close_583_back

ShowImages_587


end_590



ShowImages_587_left

Close_588


end_589


ShowImages_587_back

Close_588_left

Close_588_back

ShowImages_592


end_595



ShowImages_592_left

Close_593


end_594


ShowImages_592_back

Close_593_left

Close_593_back

ShowImages_597


end_600



ShowImages_597_left

Close_598


end_599


ShowImages_597_back

Close_598_left

Close_598_back

ShowImages_602


end_605



ShowImages_602_left

Close_603


end_604


ShowImages_602_back

Close_603_left

Close_603_back

WhiteBoyRick_607


TheGrinch_608


PlayTrailer_721



SpiritedAway_609

WhiteBoyRick_607_left

WhiteBoyRick_607_right

PlayTrailer_716

WhiteBoyRick_607_enter

WhiteBoyRick_607_back

TheGrinch_608_left

AStarIsBorn_625

TheGrinch_608_right

PlayTrailer_626

TheGrinch_608_enter

TheGrinch_608_back

MaquiaWhenthePromisedFlowerBlooms_610

SpiritedAway_609_left

SpiritedAway_609_right

PlayTrailer_711

SpiritedAway_609_enter

SpiritedAway_609_back

EverybodyKnows_611

MaquiaWhenthePromisedFlowerBlooms_610_left

MaquiaWhenthePromisedFlowerBlooms_610_right

PlayTrailer_706

MaquiaWhenthePromisedFlowerBlooms_610_enter

MaquiaWhenthePromisedFlowerBlooms_610_back

Mandy_612

EverybodyKnows_611_left

EverybodyKnows_611_right

PlayTrailer_701

EverybodyKnows_611_enter

EverybodyKnows_611_back

Reprisal_613

Mandy_612_left

Mandy_612_right

PlayTrailer_696

Mandy_612_enter

Mandy_612_back

ASimpleFavor_614

Reprisal_613_left

Reprisal_613_right

PlayTrailer_691

Reprisal_613_enter

Reprisal_613_back

Papillon_615

ASimpleFavor_614_left

ASimpleFavor_614_right

PlayTrailer_686

ASimpleFavor_614_enter

ASimpleFavor_614_back

DragonBallSuperBroly_616

Papillon_615_left

Papillon_615_right

PlayTrailer_681

Papillon_615_enter

Papillon_615_back

Bumblebee_617

DragonBallSuperBroly_616_left

DragonBallSuperBroly_616_right

PlayTrailer_676

DragonBallSuperBroly_616_enter

DragonBallSuperBroly_616_back

AXL_618

Bumblebee_617_left

Bumblebee_617_right

PlayTrailer_671

Bumblebee_617_enter

Bumblebee_617_back

AsterixTheMagicPotionsSecret_619

AXL_618_left

AXL_618_right

PlayTrailer_666

AXL_618_enter

AXL_618_back

CreedII_620

AsterixTheMagicPotionsSecret_619_left

AsterixTheMagicPotionsSecret_619_right

PlayTrailer_661

AsterixTheMagicPotionsSecret_619_enter

AsterixTheMagicPotionsSecret_619_back

SpiderManIntotheSpiderVerse_621

CreedII_620_left

CreedII_620_right

PlayTrailer_656

CreedII_620_enter

CreedII_620_back

Aquaman_622

SpiderManIntotheSpiderVerse_621_left

SpiderManIntotheSpiderVerse_621_right

PlayTrailer_651

SpiderManIntotheSpiderVerse_621_enter

SpiderManIntotheSpiderVerse_621_back

HunterKiller_623

Aquaman_622_left

Aquaman_622_right

PlayTrailer_646

Aquaman_622_enter

Aquaman_622_back

MortalEngines_624

HunterKiller_623_left

HunterKiller_623_right

PlayTrailer_641

HunterKiller_623_enter

HunterKiller_623_back

MortalEngines_624_left

MortalEngines_624_right

PlayTrailer_636

MortalEngines_624_enter

MortalEngines_624_back

AStarIsBorn_625_left

AStarIsBorn_625_right

PlayTrailer_631

AStarIsBorn_625_enter

AStarIsBorn_625_back

ShowImages_627


end_630



ShowImages_627_left

Close_628


end_629


ShowImages_627_back

Close_628_left

Close_628_back

ShowImages_632


end_635



ShowImages_632_left

Close_633


end_634


ShowImages_632_back

Close_633_left

Close_633_back

ShowImages_637


end_640



ShowImages_637_left

Close_638


end_639


ShowImages_637_back

Close_638_left

Close_638_back

ShowImages_642


end_645



ShowImages_642_left

Close_643


end_644


ShowImages_642_back

Close_643_left

Close_643_back

ShowImages_647


end_650



ShowImages_647_left

Close_648


end_649


ShowImages_647_back

Close_648_left

Close_648_back

ShowImages_652


end_655



ShowImages_652_left

Close_653


end_654


ShowImages_652_back

Close_653_left

Close_653_back

ShowImages_657


end_660



ShowImages_657_left

Close_658


end_659


ShowImages_657_back

Close_658_left

Close_658_back ShowImages_662


end_665



ShowImages_662_left

Close_663


end_664


ShowImages_662_back

Close_663_left

Close_663_back

ShowImages_667


end_670



ShowImages_667_left

Close_668


end_669


ShowImages_667_back

Close_668_left

Close_668_back

ShowImages_672


end_675



ShowImages_672_left

Close_673


end_674


ShowImages_672_back

Close_673_left

Close_673_back

ShowImages_677


end_680



ShowImages_677_left

Close_678


end_679


ShowImages_677_back

Close_678_left

Close_678_back

ShowImages_682


end_685



ShowImages_682_left

Close_683


end_684


ShowImages_682_back

Close_683_left

Close_683_back

ShowImages_687


end_690



ShowImages_687_left

Close_688


end_689


ShowImages_687_back

Close_688_left

Close_688_back

ShowImages_692


end_695



ShowImages_692_left

Close_693


end_694


ShowImages_692_back

Close_693_left

Close_693_back

ShowImages_697


end_700



ShowImages_697_left

Close_698


end_699


ShowImages_697_back

Close_698_left

Close_698_back

ShowImages_702


end_705



ShowImages_702_left

Close_703


end_704


ShowImages_702_back

Close_703_left

Close_703_back

ShowImages_707


end_710



ShowImages_707_left

Close_708


end_709


ShowImages_707_back

Close_708_left

Close_708_back

ShowImages_712


end_715



ShowImages_712_left

Close_713


end_714


ShowImages_712_back

Close_713_left

Close_713_back

ShowImages_717


end_720



ShowImages_717_left

Close_718


end_719


ShowImages_717_back

Close_718_left

Close_718_back

ShowImages_722


end_725



ShowImages_722_left

Close_723


end_724


ShowImages_722_back

Close_723_left

Close_723_back

Fig. 3: The constructed directed graph model by EvoCreeper for the CineMup

v1 Du

v1 Dd v5

v1 Dl

v1 Dr v2

v1 Du

v1 Dd v5

v1 Dl

v1 Dr v2

v2 Du

v2 Dd v6

v2 Dl v1

v2 Dr v3

v5 Du v1

v5 Dd v9

v5 Dl

v5 Dr v6

v2 Du

v2 Dd v6

v2 Dl v1

v2 Dr v3

v5 Du v1

v5 Dd v9

v5 Dl

v5 Dr v6

v3 Du

v3 Dd v7

v3 Dl v2

v3 Dr v4

v6 Du v2

v6 Dd v10

v6 Dl v5

v6 Dr v7

v9 Du v5

v9 Dd v13

v9 Dl

v9 Dr v10

v3 Du

v3 Dd v7

v3 Dl v2

v3 Dr v4

v6 Du v2

v6 Dd v10

v6 Dl v5

v6 Dr v7

v9 Du v5

v9 Dd v13

v9 Dl

v9 Dr v10

v1 v2 v3 v4

v5 v6 v7 v8

v9 v10 v11 v12

v1 v3 v4

v6 v7 v8

v9 v10 v11 v12

v1 v2 v4

v5 v7 v8

v10 v11 v12

Du

Dd

Dl Dr v2

Du

Dd

Dl Dr v3

Du

Dd

Dl Dr

v6

Du

Dd

Dl Dr

v9

Du

Dd

Dl Dr

v5

Du

Dd

Dl Dr

Fig. 4: Proof of concepts of the EvoCreeper

The third iteration similarly starts from the newly discoveredstates, v3, v6, and v9. In the same way, considering all fourdirections from each state and filtering out repeated states, fournew states are identified: v4, v7, v10, and v13.

EvoCreeper works in an iterative evolutionary manner todiscover new states and events in the app. As mentioned, acloud-based app is considered in this pilot example. Hence,there is no expectation that the app will have a finite numberof states. Consequently, our proposed alternative stoppingcriterion could be useful here. The creeper algorithm will stopafter a certain number of iterations or when no new states arediscovered.

V. EMPIRICAL EVALUATION

To assess the effectiveness of our model generation strategy,we conducted a case study on four smart TV apps. Duringthis evaluation, we attempted to address the following researchquestions (RQs):

• RQ1. Is EvoCreeper capable of exploring and identifyingthe states and transitions of a given smart TV appaccurately relative to manual exploration?

• RQ2. To what extent is the generated model complete interms of the numbers of states and edges? Is the createdgraph valid?

• RQ3. What is the performance of EvoCreeper comparedto that of manual exploration for a given smart TV app?

A. Experimental Objects

Research in the area of smart TV apps is in an early stage.More time may be needed for developers to create and publishsmart TV apps, as it is a new development environment.As a result, not many apps and repositories are availablefor benchmarking. Tizen does maintain a page with severalsimple apps and examples2. However, most of the providedsamples are simple apps with few states. To demonstrate theeffectiveness of our strategy, we chose four different Tizen

2https://bit.ly/2qC5ncS

7

smart TV apps of different sizes from GitHub. These apps arefrom different domains and have varying numbers of states.Table I shows the name and source of each app.

The chosen apps are CineMup, ChessLab TV, Monitor deLoterias, and the game Memory. CineMup is an app forsearching for various types of movies and TV shows. The appcategorizes movies and shows and presents relevant data foreach along with a trailer. ChessLab TV is an app for improvingones personal skills in chess by presenting tactics and solutionsto puzzles. Monitor de Loterias is a Spanish lottery gamingapp. Memory is a gaming app for memory-based puzzles.

B. Experimental Procedure

To address RQ1 and RQ3, we conducted a set of experi-ments. The goal was to compare the models of the benchmarkapps created by EvoCreeper with models created throughmanual exploration. For the experiments, we adopted thefollowing set-up.

We instructed four groups of 60 students from the advancedsoftware engineering study program to download a softwarepackage consisting of a portion of Tizen Studio that allows itsemulator to be run on a desktop or notebook computer. Weextended the remote control module in the emulator to recordthe exploration history in each smart TV app, consisting of thesource UI elements, the particular remote control keys pressed,the target elements and the corresponding timestamps. Eachmember of the group was assigned one app to explore and wasinstructed to export the exploration history log when finished.The stopping criterion was ”the participant considers all partsof the app to have been explored”.

We then processed the recorded logs to construct the appmodels. For each model, we analyzed several properties: thetime needed to create the model (to explore the smart TV appto the extent captured by the model), the number of nodes inthe model, the number of unique nodes in the model (thusenabling the determination of the level of duplicity of thenodes in the model) and the number of edges.

C. Model-Checking Algorithm

To address RQ2, we developed an algorithm to check theconsistency of the graphs generated by EvoCreeper. Eachgenerated model graph was validated with Algorithm 2.

As can be seen from Algorithm 2, during this validation,several steps are performed. The model must not be empty,its starting node must be defined, and all nodes of the modelexcept the starting node must have at least one incoming edge.Furthermore, at least one end node must be present in themodel. From each node, it must be possible to reach an endnode. Finally, each of the nodes must be reachable from thestarting node.

D. Results

As previously mentioned, we created a set of experimentsusing the Tizen SDK environment to assess the effectivenessof our EvoCreeper strategy. Additionally, we developed analgorithm for automatically verifying the correctness of the

Input: GOutput: list of issue tokens Iif G = ∅ then

I ← I ∪ empty modelendif ns not exist then

I ← I ∪ no start node definedendforeach n ∈ N \ {ns} do

if deg-(n) = 0 thenI ← I ∪ node n has not incomming edge

endendif there is not at least one n ∈ N \ {ns} thatdeg+(n) = 0 then

I ← I ∪model has not end nodeendforeach n1 ∈ N \Ne do

if there does not exist a path from n1 to any nodefrom Ne then

I ← I ∪ end node for n1 not reachableend

endforeach n ∈ N \ {ns} do

if there does not exist a path from ns to n thenI ← I ∪ node n not reachable from start

endend

Algorithm 2: The app model validation algorithm

models thus generated. The goal of these experiments was toaddress three main RQs. The following subsections answerthese RQs.

1) RQ1. State detection comparison of EvoCreeper andmanual exploration: To answer this RQ, we compared thenumbers of unique states and edges detected by our strategywith those identified through the manual exploration of theapps. The experimental procedure is described in SectionV-B. Figures 5 and 6 show the box plots of the results forcomparison.

Due to the subjective and nondeterministic nature of theresults obtained by each participant, we compare the resultsusing box plots to ensure fair comparisons and provide moredetails. It is clear from the figures that in contrast to manualexploration, our strategy produces deterministic results. Wecan observe from the results that for large apps, our strategycan produce more accurate results than manual explorationcan. Regarding the numbers of unique states in Figure 5, it isclear that our strategy can generate better results than can beobtained through manual exploration in most cases. In fact, wenotice that manual exploration can also be effective for somemembers of the exploration group, but only for small appswith few states. For example, the best result achieved amongthe participants for the number of nodes is much lower thanthat achieved by our strategy for the CineMup app, as seenin Figure 5b. A similar situation is also clearly observed inFigure 6b for the number of edges discovered. For a smallapplication such as ChessLab TV, as seen in Figures 5a and

8

TABLE I: APPS USED IN THE CASE STUDY

ID App Source

1 CineMup https://github.com/daliife/Cinemup2 ChessLab TV https://github.com/PabloEzequiel/Tizen/tree/master/ChessLabTV3 MonitorDeLoterias https://github.com/brunohpmarques/monitordeloterias-tizentv4 Memory game https://github.com/wissalKhalfi/Brain-Up---Tizen-smart-TV/tree/master/Memoryyyyyy

Manual EvoCreeper9.5

10.0

10.5

11.0

11.5

12.0

No#

Uni

que

Nod

es

(a) ChessLab

Manual EvoCreeper0

200

400

600

800

No#

Uni

que

Nod

es

(b) CineMup

Manual EvoCreeper10

11

12

13

14

No#

Uni

que

Nod

es

(c) Memory game

Manual EvoCreeper0

20

40

60

80

No#

Uni

que

Nod

es

(d) MonitorDeLoterias

Fig. 5: Comparing the number of unique nodes detected by EvoCreeper with the manual exploration

Manual EvoCreeper15

16

17

18

19

No#

Edg

es

(a) ChessLab

Manual EvoCreeper0

500

1,000

1,500

2,000

No#

Edg

es

(b) CineMup

Manual EvoCreeper30

40

50

60

70

80

No#

Edg

es

(c) Memory game

Manual EvoCreeper0

50

100

150

200

No#

Edg

es


Fig. 6: Comparing the number of Edges detected by EvoCreeper with the manual exploration

6a, manual exploration can reveal as many states and edges asautomated exploration can in some cases. In fact, the successof manual exploration depends on the participants knowledgeand experience with the app. While our strategy exploredeach app without any prior knowledge about its operation,the participants were able to explore the apps several timesbefore the recording the exploration logs.

We note that for some apps, the results of our strategyare missing a few states relative to the results of manualexploration. For instance, the output of our strategy is missingtwo states and their related edges for the Memory game app,as seen in Figures 5c and 6c. A few more nodes and edgesare missing relative to those found through manual explorationfor the Monitor de Loterias app, as seen in Figures 5d and 6d.This situation arises due to the nature of these two apps. Bothapps require some manual text entry at some stage duringoperation. This explains the deficiency of our strategy in thesecases. In fact, even in the reverse engineering of desktop andmobile apps, such situations are a critical concern because therequired text is specific to the particular app and may changedepending on the app itself. This difficulty can be addressedby defining the text to be entered into each specific app.

Overall, manual exploration is feasible for model generationfor small apps; however, our strategy is the best choice forsmall, medium and large apps when there are many statesto be explored. Additionally, manual exploration is neither arealistic nor a practical method of model creation.

2) RQ2. Graph verification and correctness: To verifythe correctness of the graphs generated by our strategy, wedeveloped our automated verification method presented in

TABLE II: GRAPH VERIFICATION RESULTS BY ALGO-RITHM 2

ID App No. of not reachable path Result1 CineMup 0 Pass2 ChessLab TV 0 Pass3 MonitorDeLoterias 0 Pass4 Memory game 0 Pass

Algorithm 2. This algorithm checks a graph for incompleteand invalid paths and nodes. In fact, it simply checks thevalidity of a directed graph generated by our strategy. TableII shows the results of this verification for each app used inour experiments.

We can clearly see that for all apps, the graphs weregenerated correctly, and the algorithm could not find anyinvalid paths with a missing state or edge.

3) RQ3. Performance comparison of EvoCreeper with man-ual: Another critical objective of this study was to assessthe performance of our strategy compared to that of manualexploration. As mentioned previously, we measured the timerequired for manual exploration for each participant indi-vidually by calculating the difference between the start andend times of exploration from the log. Figure 7 shows thecorresponding results for each app.

It is clear from Figure 7 that the exploration times variedamong the participants. As mentioned previously, the ex-ploration time may change from one participant to anotherdepending on the usability of the app and that participantsknowledge of the app. To ensure fair comparisons and obtain

9

Manual EvoCreeper0

5

10

15

20

25Ti

me

(Min

s)

(a) ChessLab

Manual EvoCreeper0

20

40

60

80

Tim

e (M

ins)

(b) CineMup

Manual EvoCreeper0

5

10

15

20

25

Tim

e (M

ins)

(c) Memory game

Manual EvoCreeper0

20

40

60

80

Tim

e (M

ins)


Fig. 7: Comparing the exploration time by EvoCreeper with the manual exploration

more details about the results, we used box plots to present thefindings. While the participants times vary, the correspondingrange of variation is much lower for our strategy, as can beclearly seen from the low interquartile ranges in Figure 7.

It is clear from Figure 7d that for a medium-size app, thetime required by our strategy is less than the median timerequired by the participants. For some of the other apps, asseen in Figures 7a and 7c, our strategy even takes less timethan the best time recorded by any participant.

For large apps such as CineMup, for which the results areshown in Figure 7b, our strategy takes more time due to thelarge number of states to explore. In fact, this time is muchhigher than the worst time taken for manual exploration amongthe participants. However, this time is reasonable with respectto the accuracy and completeness of the generated graph. Wenote from Figure 6b that the number of unique nodes found byour EvoCreeper strategy is much higher than the best resultachieved by any participant. Specifically, our strategy found727 unique states in approximately 64 minutes of explorationtime, whereas the best group member found 348 unique statesin approximately 50 minutes of exploration time.

VI. THREATS TO VALIDITY

Like any empirical and experimental study, the validity ofthe evaluation and case studies presented in this paper is notwithout any threats. In fact, there are a few factors that mayaffect the validity of the presented results. We have attemptedto eliminate each of these factors, as described below.

The comparison with the results of manual explorationof the apps could pose an internal threat to validity. Asmentioned earlier, research in the area of smart TV apps isin an early stage. Hence, we could not find another reverse

engineering tool against which to compare our algorithm.Here, the manual exploration is a well-known approach thatis normally used for different testing purposes (e.g., [35]–[37]). However, manual exploration is subjective and couldbe affected by participants biases. We mitigated this threatby considering many participants in our experiment. We alsopresented the results as box plots to visualize the best, worst,and median results.

A possible threat can be identified regarding the simulationof the smart TV apps on an emulator instead of executingthem on a real device. However, for app exploration, thereis no significant difference between the actual device and theemulator. Here, we were simply creating a model, and we donot expect any differences in the model between the two casesbecause it is a platform-independent model.

Another threat could be posed with regard to the recruitmentof students for the experiment. When an app is given toan experienced professional software engineer or smart TVapp developer, for instance, the generated model could becompleted in a slightly shorter time. As mentioned previouslyin the discussion of the results, the usability of an app and priorknowledge about it could affect the exploration time and themodel completeness. We attempted to mitigate these factors byillustrating the app workflow to each student. Additionally, weprovided sufficient time for the students to learn and explorethe apps before the experiment. Each student was also ableto run the experiment several times and report the best timeachieved. However, many other human factors may affect theresults, and indeed, eliminating these factors is an essential aimof our strategy. Automating the model generation process willeliminate the influence of human factors and, consequently, thesubjectivity of the results. Even with our mitigation strategies,the results show that our automated black-box strategy is

10

superior. For a user who does not have any information aboutthe app being explored, we would expect worse results thanthose presented in this paper.

Another threat is related to the fact that some smart TVapps use different remote control keys other than Left, Right,Up, Down, OK and Back. In some cases, this could influencethe completeness of the model. However, this fact does notaffect the reliability of the present experiment because theparticipants were restricted by the logging mechanism in theemulator to creating the manual models of the apps by usingthe Left, Right, Up, Down, OK and Back keys only. The extrakeys on a remote control device are typically app-specific keysand may not strongly affect the general model. Instead, theyare likely to affect the model completeness because some statesand edges could be missing.

Finally, for apps that depend on cloud content, the modelmay change over time. This could lead to differences in themodel between one time and another. Hence, two differentparticipants may generate two different but valid models forthe same app on two different days. We mitigated this threatby ensuring that the participants created their models at thesame time. In fact, this threat reflects another benefit of ourstrategy. Using EvoCreeper will enable the user to create up-to-date models for apps at different points in time withoutrequiring manual effort.

VII. CONCLUSION AND FUTURE WORK

In this paper, we have presented our strategy for au-tomatically reverse engineering smart TV apps consideringinteraction via a remote control device. This strategy involvesnavigating a given smart TV app to explore its UI stateswithout knowing the internal structure of the app code (i.e., itis a black-box strategy). The strategy can extensively explorethe states and transitions in a given app and then generatea directed graph model. We have implemented our strategyin a tool called EvoCreeper that works with the emulator inthe Tizen SDK framework. We have evaluated our strategyusing four medium-size and large apps. The evaluation resultsdemonstrate the effectiveness and good performance of ourstrategy. The strategy can be used to detect states that wouldgo undetected through manual efforts. The generated modelscould be used in various stages of app development for qualityimprovement, testing, identifying missing states, assessing theuser experience and understanding of a given app by providinga visualization of the states and transitions, and even more.

There are many opportunities for future research. An im-mediate step forward is to use the models generated via thepresented strategy to test various apps and identify new faults.Additionally, we are planning to use this strategy for the smoketesting of smart TV applications. Part of this process will beto design an algorithm for generating path-based test casesfrom such a generated model. Another important direction ofresearch that we are planning for the future is to investigatemodel generation for gaze-based interaction with smart TVapps.

REFERENCES

[1] K. Jung, “The prospect of smart tv service,” Information and Commu-nications Magazine, vol. 28, no. 3, pp. 3–7, 2011.

[2] A. Villas-Boas, “Business Insider: Samsung’s latest TVs willcontrol your smart home,” 2016, [Online; accessed 27-October-2017]. [Online]. Available: http://www.businessinsider.com/samsungs-new-smart-tvs-will-control-your-smart-home-2016-1

[3] Global smart tv market. [Online]. Available: https://bit.ly/2HxnMkL[4] C. Chen and Y. Lin, “Study on the interactive interface design of gaze

input smart tv,” in IEEE Int. Conf. Applied System Invention (ICASI),April 2018, pp. 196–199.

[5] Mautilus smart tv sdk. [Online]. Available: https://www.mautilus.com[6] Smart tv alliance. [Online]. Available: http://www.smarttv-alliance.org[7] M. Murgrabia. (2017) Design considerations for vewd app store

applications. [Online; accessed 5-December-2017]. [Online]. Available:https://bit.ly/2IiNb30

[8] B. S. Ahmed and M. Bures, “Testing of smart tv applications: Keyingredients, challenges and proposed solutions,” in Future TechnologiesConf. (FTC) 2018, K. Arai, R. Bhatia, and S. Kapoor, Eds. Cham:Springer International Publishing, 2019, pp. 241–256.

[9] K. Sabina, “Defining a testing platform for smart tv applications,”Bachelor Thesis, Helsinki Metropolia University of Applied Sciences,January 2016.

[10] X. Yuan, M. B. Cohen, and A. M. Memon, “Gui interaction testing:Incorporating event context,” IEEE Trans. Softw. Eng., vol. 37, no. 4,pp. 559–574, Jul. 2011.

[11] J. A. Saddler and M. B. Cohen, “Eventflowslicer: A tool for generatingrealistic goal-driven gui tests,” in Proc. 32nd IEEE/ACM Int. Conf.Automated Software Engineering, ser. ASE 2017. Piscataway, NJ, USA:IEEE Press, 2017, pp. 955–960.

[12] L. T. Long, N. T. Binh, and I. Parissis, “A new test modeling languagefor interactive applications based on task trees,” in Proc. 4th Symp.Information Communication Technology, ser. SoICT ’13. New York,NY, USA: ACM, 2013, pp. 285–293.

[13] P. Mehlitz, O. Tkachuk, and M. Ujma, “Jpf-awt: Model checking guiapplications,” in Proc. 26th IEEE/ACM Int. Conf. Automated SoftwareEngineering, ser. ASE ’11. Washington, DC, USA: IEEE ComputerSociety, 2011, pp. 584–587.

[14] B. S. Ahmed, M. A. Sahib, and M. Y. Potrus, “Generating combinatorialtest cases using simplified swarm optimization (sso) algorithm forautomated gui functional testing,” Eng. Sci. Technol., vol. 17, no. 4,pp. 218 – 226, 2014.

[15] A. Memon, I. Banerjee, and A. Nagarajan, “Gui ripping: Reverseengineering of graphical user interfaces for testing,” in Proc. 10thWorking Conf. Reverse Engineering, ser. WCRE ’03. Washington, DC,USA: IEEE Computer Society, 2003, pp. 260–.

[16] B. N. Nguyen, B. Robbins, I. Banerjee, and A. Memon, “Guitar: aninnovative tool for automated testing of gui-driven software,” Automat.Softw. Eng., vol. 21, no. 1, pp. 65–105, 2014.

[17] P. Aho, T. Kanstrn, T. Rty, and J. Rning, “Chapter two - automatedextraction of gui models for testing,” ser. Advances in Computers,A. Memon, Ed. Elsevier, 2014, vol. 95, pp. 49 – 112.

[18] D. Amalfitano, A. R. Fasolino, P. Tramontana, B. D. Ta, and A. M.Memon, “Mobiguitar: Automated model-based testing of mobile apps,”IEEE Software, vol. 32, no. 5, pp. 53–59, 2015.

[19] A. Mesbah, A. van Deursen, and S. Lenselink, “Crawling ajax-based webapplications through dynamic analysis of user interface state changes,”ACM Trans. Web, vol. 6, no. 1, pp. 3:1–3:30, Mar. 2012.

[20] A. Gimblett and H. Thimbleby, “User interface model discovery: To-wards a generic approach,” in Proc. 2nd ACM SIGCHI Symp. Engineer-ing Interactive Computing Systems, ser. EICS ’10. New York, NY,USA: ACM, 2010, pp. 145–154.

[21] J. Kim, S. Kim, S. Park, and J. Hong, “Home appliances controllingthrough smart tv set-top box with screen-mirroring remote controller,”in Int. Conf. ICT Convergence (ICTC), Oct 2013, pp. 1009–1012.

[22] J. Kim, E. Jung, Y. Lee, and W. Ryu, “Home appliance control frame-work based on smart tv set-top box,” IEEE Trans. Consum. Electron.,vol. 61, no. 3, pp. 279–285, Aug 2015.

[23] C. Fan, Y.-K. Wang, and J.-R. Chen, “Home sleep care with videoanalysis and its application in smart tv,” in IEEE 3rd Global Conf.Consumer Electronics (GCCE), Oct 2014, pp. 42–43.

[24] M. R. Cabrer, R. P. D. Redondo, A. F. Vilas, J. J. P. Arias, and J. G.Duque, “Controlling the smart home from tv,” IEEE Trans. Consum.Electron., vol. 52, no. 2, pp. 421–429, May 2006.

http://www.businessinsider.com/samsungs-new-smart-tvs-will-control-your-smart-home-2016-1

http://www.businessinsider.com/samsungs-new-smart-tvs-will-control-your-smart-home-2016-1

https://bit.ly/2HxnMkL

https://www.mautilus.com

http://www.smarttv-alliance.org

https://bit.ly/2IiNb30

11

[25] D. Vavilov, A. Melezhik, and I. Platonov, “Healthcare application ofsmart home user’s behavior prediction,” in IEEE Int. Conf. ConsumerElectronics (ICCE), Jan 2014, pp. 323–326.

[26] S. Y. Chun and C. Lee, “Applications of human motion tracking: Smartlighting control,” in IEEE Conf. Computer Vision Pattern RecognitionWorkshops, June 2013, pp. 387–392.

[27] E. Erkan, H. R. Ozcalk, and S. Ylmaz, “Designing a smart securitycamera system,” in Proc. 23rd Signal Processing CommunicationsApplications Conf. (SIU), May 2015, pp. 1705–1708.

[28] K. Cui, K. Zhou, H. Song, and M. Li, “Automated software testing basedon hierarchical state transition matrix for smart tv,” IEEE Access, vol. 5,pp. 6492–6501, 2017.

[29] P. Ammann and J. Offutt, Introduction to software testing. CambridgeUniversity Press, 2016.

[30] N. Li, F. Li, and J. Offutt, “Better algorithms to minimize the cost oftest paths,” in Software Testing, Verification and Validation (ICST), 2012IEEE Fifth International Conference on. IEEE, 2012, pp. 280–289.

[31] A. Dwarakanath and A. Jankiti, “Minimum number of test paths forprime path and other structural coverage criteria,” in IFIP InternationalConference on Testing Software and Systems. Springer, 2014, pp. 63–79.

[32] F. Sayyari and S. Emadi, “Automated generation of software testing pathbased on ant colony,” in Technology, Communication and Knowledge(ICTCK), 2015 International Congress on. IEEE, 2015, pp. 435–440.

[33] M. Bures, T. Cerny, and M. Klima, “Prioritized process test: More effi-ciency in testing of business processes and workflows,” in InternationalConference on Information Science and Applications. Springer, 2017,pp. 585–593.

[34] V. Arora, R. Bhatia, and M. Singh, “Synthesizing test scenarios in umlactivity diagram using a bio-inspired approach,” Computer Languages,Systems & Structures, 2017.

[35] P. Raappana, S. Saukkoriipi, I. Tervonen, and M. V. Mntyl, “The effectof team exploratory testing – experience report from f-secure,” in IEEEInt. Conf. on Software Testing, Verification and Validation Workshops(ICSTW), April 2016, pp. 295–304.

[36] T. D. Hellmann and F. Maurer, “Rule-based exploratory testing ofgraphical user interfaces,” in Agile Conf., Aug 2011, pp. 107–116.

[37] M. Bures, K. Frajtak, and B. S. Ahmed, “Tapir: Automation support ofexploratory testing using model reconstruction of the system under test,”IEEE Trans. on Reliability, vol. 67, no. 2, pp. 557–580, June 2018.

Bestoun S. Ahmed obtained his B.Sc. degree inElectrical and Electronic Engineering from the Sala-haddin University-Erbil in 2004, his M.Sc. degreefrom University Putra Malaysia (UPM) in 2009, andhis Ph.D. degree from University Sains Malaysia(USM), Software Engineering, in 2012. He spentone year doing his post doctoral research in theSwiss AI Lab IDSIA, Switzerland. Currently, he is asenior lecturer at the department of mathematics andcomputer science, Karlstad University, Sweden andalso a part time assistant professor at the department

of computer science, Czech Technical University in Prague. His main re-search interest include Combinatorial Testing, Search Based Software Testing(SBST), Applied Soft Computing, and quality assurance of smart devices andIoT systems.

Miroslav Bures received his Ph.D. at Czech Tech-nical University in Prague, Faculty of ElectricalEngineering, where he currently works as a re-searcher and a senior lecturer in software testing andquality assurance. His research interests are model-based testing (process and work flow testing, dataconsistency testing) efficiency of test automation,and quality assurance methods for Internet of Thingssolutions, reflecting specifics of this technology. Heis a member of Czech chapter of the ACM, CaSTB,and ISTQB Academia work group.

Automated Black-Box Model Generation for Smart TV ... - arXiv

Documents

Transcript of Automated Black-Box Model Generation for Smart TV ... - arXiv