Ambiguities in sketch-based interfaces

10
Ambiguities in Sketch-Based Interfaces Danilo Avola, Istituto di Ricerche sulla Popolazione e Politiche Sociali, CNR, Via Nizza 128, 00198 Roma danilo.avola@irp ps.cnr.it Maria Chiara Caschera, Istituto di Ricerche sulla Popolazione e Politiche Sociali, CNR, Via Nizza 128, 00198 Roma mc.caschera@irp ps.cnr.it Fernando Ferri, Istituto di Ricerche sulla Popolazione e Politiche Sociali, CNR, Via Nizza 128, 00198 Roma fernando.ferri@ir pps.cnr.it Patrizia Grifoni, Istituto di Ricerche sulla Popolazione e Politiche Sociali, CNR, Via Nizza 128, 00198 Roma, patrizia.grifoni@i rpps.cnr.it Abstract Sketch-based interaction is an intuitive, simple communication method. However, it has several critical aspects, due to difficulties during the interpretation step by the computer side. These mainly derive from the semantic gap between the user’s communicative intention and how he/she is able to convey it. The interpretation must also consider the deletion and over-tracing actions. This introduces further ambiguities that make it difficult to interpret the user’s communicative intention. To deal with these problems, this paper provides a classification of ambiguities for sketch-based interfaces, based on experimental observations from a group of users. Ambiguities are caused by inaccuracy, approximation of the represented reality, user’s indecision and deletion or retracing of parts of the sketch. This paper presents some useful situations to give an overview on ambiguities by considering information strictly related to the user’s behaviour, considering spatial and temporal information in the drawing process and the user’s intention. 1. Introduction People use sketches to represent concepts and communicate in different contexts. Sketches materialise abstract ideas using symbolic graphical elements and the spatial constraints among them. Sketch-based software applications have undergone strong development in recent years. For example in [1] a complex sketch-based environment has been developed to help users query for geographic databases, while in [2] the sketch interaction is the core of the environment enabling the creation of web-based interactive design. Symbolic graphical elements do not usually represent the complexity of “real life” objects, but represent a simplified idea or related concepts. Sometimes the use of sketch can be effective to represent both the real and a more abstract world. In fact, a user commonly draws only the relevant aspects of the real world and/or ideas that he/she is explaining during sketching process. In spite of a growing interest in human-computer interfaces and tools for freehand sketching and writing, the cognitive complexity of the sketching process is still not fully understood. In other words, the information content of a generic sketch is highly complex. The user expresses information content by including spatial and temporal information within the sketch and its drawing process. Its complexity is closely related to the sketch’s typical use to support creative activities. This paper provides a classification of ambiguities for sketch-based. It is based on experimental observations from a group of users. Several studies have examined sketch ambiguities in a given graphical context [3][4][5]. Their common consensus is that context is a fundamental element to resolve ambiguity. For example [6] discusses ambiguities in sketches drawn in complex technical environments. But the simple intuition of using the context to resolve the different ambiguities is a simple artifice used in various contexts, such as electronic design, mechanical engineer design, technical domains such as ERD, DFD, and so on. Furthermore, creativity and the proposition of new ideas conveyed by the sketch are intrinsically ambiguous, although they express complex information contents. In [7] for example the ambiguities arising during the Proceedings of the 40th Hawaii International Conference on System Sciences - 2007 1 © 1530-1605/07 $20.00 2007 IEEE Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07) 0-7695-2755-8/07 $20.00 © 2007

Transcript of Ambiguities in sketch-based interfaces

Ambiguities in Sketch-Based Interfaces

Danilo Avola, Istituto di

Ricerche sulla Popolazione e

Politiche Sociali, CNR, Via Nizza

128, 00198 Romadanilo.avola@irp

ps.cnr.it

Maria Chiara Caschera, Istituto di Ricerche sulla

Popolazione e Politiche Sociali, CNR, Via Nizza

128, 00198 Romamc.caschera@irp

ps.cnr.it

Fernando Ferri, Istituto di

Ricerche sulla Popolazione e

Politiche Sociali, CNR, Via Nizza

128, 00198 Roma fernando.ferri@ir

pps.cnr.it

Patrizia Grifoni, Istituto di

Ricerche sulla Popolazione e

Politiche Sociali, CNR, Via Nizza

128, 00198 Roma, patrizia.grifoni@i

rpps.cnr.it

Abstract

Sketch-based interaction is an intuitive, simple communication method. However, it has several critical aspects, due to difficulties during the interpretation step by the computer side.

These mainly derive from the semantic gap between the user’s communicative intention and how he/she is able to convey it.

The interpretation must also consider the deletion and over-tracing actions. This introduces further ambiguities that make it difficult to interpret the user’s communicative intention.

To deal with these problems, this paper provides a classification of ambiguities for sketch-based interfaces, based on experimental observations from a group of users.

Ambiguities are caused by inaccuracy, approximation of the represented reality, user’s indecision and deletion or retracing of parts of the sketch. This paper presents some useful situations to give an overview on ambiguities by considering information strictly related to the user’s behaviour, considering spatial and temporal information in the drawing process and the user’s intention.

1. Introduction

People use sketches to represent concepts and communicate in different contexts. Sketches materialise abstract ideas using symbolic graphical elements and the spatial constraints among them. Sketch-based software applications have undergone strong development in recent years.

For example in [1] a complex sketch-based environment has been developed to help users query for geographic databases, while in [2] the sketch

interaction is the core of the environment enabling the creation of web-based interactive design.

Symbolic graphical elements do not usually represent the complexity of “real life” objects, but represent a simplified idea or related concepts. Sometimes the use of sketch can be effective to represent both the real and a more abstract world. In fact, a user commonly draws only the relevant aspects of the real world and/or ideas that he/she is explaining during sketching process.

In spite of a growing interest in human-computer interfaces and tools for freehand sketching and writing, the cognitive complexity of the sketching process is still not fully understood. In other words, the information content of a generic sketch is highly complex. The user expresses information content by including spatial and temporal information within the sketch and its drawing process. Its complexity is closely related to the sketch’s typical use to support creative activities.

This paper provides a classification of ambiguities for sketch-based. It is based on experimental observations from a group of users.

Several studies have examined sketch ambiguities in a given graphical context [3][4][5]. Their common consensus is that context is a fundamental element to resolve ambiguity. For example [6] discusses ambiguities in sketches drawn in complex technical environments. But the simple intuition of using the context to resolve the different ambiguities is a simple artifice used in various contexts, such as electronic design, mechanical engineer design, technical domains such as ERD, DFD, and so on. Furthermore, creativity and the proposition of new ideas conveyed by the sketch are intrinsically ambiguous, although they express complex information contents. In [7] for example the ambiguities arising during the

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

1©1530-1605/07 $20.00 2007 IEEEProceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007

interpretation of a simple sketch based on a stick figure are highlighted.

Ambiguities are usually determined by different representations of the same concept.

Figure 1. An example of sketch

For example, the sketch of Figure 1 can be interpreted in several ways. It could represent two overlapped regions, or three closed regions, or only one region (the region identified by boundary), and so on.

In this paper we consider the user’s drawing behaviour to support the sketch interpretation. The user chooses how to create each object (e.g. which parts to draw first), the relationships among them (or parts of them) and their attributes (such as: size, position, direction).

When a user draws a sketch to explain a concept he/she uses elements belonging to a precise domain. Simple and complex concepts can be brought back to the elements of a specific application domain and their inter-relationships. For example if a user describes an ER (Entity-Relationship) diagram representing an employee database he/she will use the basic elements of the related domain to express relationships such as “employees (entity) have (relationship) an income (entity)”.

The user’s intention must be correctly interpreted to avoid ambiguities and “mistakes” in interpreting the sketch, considering the complexity of the sketched reality and simplifications introduced during sketching. The interpretation must take into account the semantic gap between the communicative user’s intention and how he/she is able to convey it. It must also consider deleting and over-tracing actions, which introduce ambiguities that make it difficult to interpret the user’s intention. The paper is organised as follows. Section 2 describes some experimental observations, section 3 considers the various kinds of ambiguities in the sketch, and section 4 concludes the paper.

2. Experimental observations

This section describes some experimental observations of behaviour during the freehand drawing process. The experiments involved thirty people aged eighteen to sixty-five, with different knowledge of the application domain. They were all from cultures, which write sentences from left to right.

Each person took the test separately, in order to minimise reciprocal influences. Experimental observations involved three specific drawing objectives:

representing an object, representing an abstract concept, representing a scheme in specific contexts.

They were classified according to the experimental subjects’ knowledge of the particular context.

One group of users drew different kinds of sketch and another group evaluated the level of comprehension. The experiment involved the following types of information:

Objects of daily reality concerning a specific domain. Metaphors involved in a concept that cannot be ascribed to a particular domain. Geometrical elements concerning several figures (2D and/or 3D) and the relationships among them. Circuit symbols that are a specific context. The test subjects were aware of circuit symbols and had to copy several circuits. Data Flow Diagram (DFD) symbols that are another specific context. The test subjects knew how to use DFD and had to build several diagrams, starting from a verbal description of the executive streaming of the processes.

The experiment showed that user behaviour depends on:

knowledge of context (application domain), kind of experiment:

o if a drawing process follows a verbal description,

o if the user copies the sketch.

Behaviour was also assessed by: orientation of each polyline, polylines and areas drawing sequence , object drawing sequence,

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

2Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007

the observed orientation in each drawing sequence of objects.

Orientation provides information about freehand sketch drawing, as a user may draw different objects: 1) from left to right; 2) from top to bottom; 3) from inside to outside and 4) vice versa.

User behaviour essentially follows two fundamental principles:

the part-by-part hand drawing principle, the structural components priority principle.

The part-by-part hand drawing principle states that:

During the hand-drawing process of one or more than one objects whose parts are clearly identifiable, the user generally ends one part or object before drawing another.

Observation of behaviour during the sketch process revealed that 81% of subjects with knowledge of the application domain and 62 % of those with no knowledge of the application domain adopt the part-by-part principle.

The structural components priority principle distinguishes components representing the structure from object details and states that:

During the hand-drawing process of one object (simple or complex) associated with a concept and its presentation in its context, the user generally adopts a drawing strategy, which assigns priority to the structural component of the object.

Observation of behaviour during the sketch process revealed that 75% of subjects with knowledge of the application domain adopt the structural components priority principle.

The following section analyses user behaviour, focusing on and classifying the ambiguities observed during the sketching process.

3. Sketch ambiguities

The main aim of this paper is to identify classes of ambiguities arising during sketch interpretation.

We now introduce some preliminary concepts that will be used in the following.

In this paper a sketch results from a set of drawing actions (strokes). This concept is expanded by introducing the concept of generalised stroke and by

considering deleting actions and spatial and temporal discontinuities, as explained below.

Another concept is the interpretation of elementary components. Drawing and deleting actions are associated with elementary components - the basic elements of the sketch - and are formed by a subset of generalised strokes. Therefore the interpretation of graphical objects relates elementary components to graphical objects.

Finally, the sketch is a pair of graphical objects and a set of admissible relations among them.

Many ambiguities must be solved in acquiring and interpreting the sketch. They arise from the lack of a bi-univocal relationship between the sketch and symbols during interpretation of the sketch.

In fact the sketching process is generally full of imprecision, which can lead to indecision in interpretation due to inaccuracy, approximation of the represented reality, user’s indecision and deletion or retracing of parts of the sketch. Some useful situations are discussed below to give an overview.

In this section two elementary actions are considered: drawing a stroke and deleting an area. A drawing action (stroke) is spatially characterised by the trajectory, starting from when the pen tip begins to touch the tablet/paper and ending at the time it leaves the tablet/paper (the stroke). A deleting action is spatially characterised by an area. In addition to their spatial characterization, the two elementary actions can be characterised temporally; in fact, time is very important in interpreting the sketch correctly. Actions are temporally characterised by the temporal interval in which they are performed and the velocity of the action for each point x of the trajectory. For this reason the drawing of a stroke is described by a trajectory p (approximated by an oriented polyline), a temporal interval t and the velocity v(x,t). The deletion of an area is described by an area a and a temporal interval

t.

= (p, t,v(x,t))

= (a, t)

A sketch is then described by the set of drawn strokes and the set X of deleted areas. ’ indicates the set of strokes considering deleting actions. Each deleting action can completely delete or partially modify strokes in the sketch. As a user may introduce discontinuities in correspondence to the angles during the drawing of polygons or polylines, it is important to consider any spatial and/or temporal discontinuity that appears during the drawing actions. For these reasons

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

3Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007

the generalised stroke concept is introduced to consider strokes modified by deleting actions and any spatial and/or temporal discontinuity while drawing. A formal definition of generalised stroke can thus be introduced.

Def. 1 A generalised stroke ' of ' isdescribed by a trajectory p (approximated by an oriented polyline), a temporal interval t, the velocity v(x,t), and the set X of deleted areas.

'= '(p, t,v(x,t),X)

In this approach a generalised stroke ends at any spatial and/or temporal discontinuity, including deleting actions.

Figure 2 gives an example of generalised strokes. In the stroke (t) a temporal discontinuity appears in correspondence to the angle (a), thus it is possible to identify a pair of generalised strokes (b) and (c).

Figure 2. Generalised strokes resulting from temporal ambiguities

Similarly, given the stroke (s), the deleting action (e) introduces a spatial discontinuity in the stroke (s), producing a pair of generalised strokes (d) and (f).

Figure 3. Generalised strokes resulting from spatial ambiguities

The trajectory p of a generalised stroke is described by a set of points representing straight lines or curves. A curve is an infinite set of points having an angle and a radius of curvature. It can be approximated by a sequence of straight lines, each with an infinite radius of curvature and a constant angle. Generalised strokes have the aim to represent objects of the interest domain and each object of the interest domain is represented by a set of strokes.

From a static point of view (without considering drawing actions), the elementary sketch components are: points, straight lines, open curves, closed curves (rings), polygons and closed regions. These are not necessarily objects in the interest domain but can represent a part of them.

Introducing a simplification due to the finite dimensions of the sketching area, a line is formed by a finite set of points and a polygon is formed by a finite set of lines. A closed region is intercepted by a polygon, and each open or closed line is intercepted by a polyline.

The elementary components are not necessarily drawn by an elementary action of the user (a single stroke). They may result from more than one stroke and deletion.

Consequently, the interpretation I is a function of the set of drawn strokes and the set of deleted areas, and relates strokes to the elementary components.

Def. 2 An interpretation I of elementary components, which relates strokes to the elementary components, is a function of and , where each elementary component c of C is formed by a subset of generalised strokes ’ belonging to ’.

I: ( , ) Cc= ’,I>

Obviously, a deletion is not associated with an elementary component but with strokes only.

These elementary components are not part of the alphabet; the latter is in fact formed by objects, which in their turn are formed by elementary components. A set of composition rules to define the alphabet elements is defined.

A sketched object is described by its interpretation, which relates elementary components to objects. Each graphical object is formed by a subset of elementary components. Below a formal definition is introduced.

Def. 3 An interpretation J of graphical objects relates elementary components of C to graphical objects og of OG defining them by a

b

c

a

t

d

ef

s

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

4Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007

subset c of elementary components of C. The interpretation J is therefore a function of the elementary components C.

J: (C) OG og=< c,J>

At the end of the interpretation process in its specific domain a sketch corresponds to a set of graphical objects, as the following definition shows.

Def. 4 A sketch is a pair defined by the set of graphical objects OG and the set R of admissible relations between those objects.

=<OG,R>

Some problems in correctly assigning drawing actions to the elementary components of the sketch may be due to the fact that a user can draw a component as one stroke or a sequence of strokes, and he/she can erase and introduce discontinuities that are difficult to interpret.

The user draws strokes and may also delete some parts of strokes; these tracing or deleting actions enable identification of generalised strokes. These elements can be associated with elementary components. If we consider a specific domain, it is possible to associate elementary components with elements of objects in the application domain.

We can now deal with the main focus of this paper, that is ambiguities’ analysis in sketch recognition. In the following section we examine the classification of ambiguities in the temporal step.

3.1. Ambiguities due to crosses in a stroke

The first class of ambiguities arises when users draw more then one symbol by only one stroke (pen down, pen movement, pen up sequence). In fact, some configurations are susceptible to more than one interpretation.

For example, the sketch in Figure 4a is composed by only one stroke a.

a= a(pa, t a,va(x,t)) Where :

pa is the trajectory. t a is the temporal interval in which the

drawing action is performed. va(x,t) is the velocity of the gesture for each point x of the trajectory during time t.

Neither has the user made any deletions:

X=

Considering spatial and temporal discontinuity, the three generalised strokes '1, '2 and '3 can be considered as represented in Figure 4b.

'i = 'i(pi, i,vi(x,t),X) i=1, 2, 3

Considering the Figure 4a and associated generalized strokes of Figure 4b, several different interpretations are possible. Among them the three interpretations I1, I2, I3 are described below.

The user may intend to identify: 1) one polygon and two polylines (Figure 4c), 2) one polygon and one polyline (Figure 4d) or 3) one polyline only (Figure 4e).

Figure 4. Interpretations of configurations produced when more than one symbol is drawn by only one stroke

Three possible interpretations are expressed as follows:

I1: '1, '2, '3) (c1,c2,c3)where c1= '1, c2= '2, c3= '3 (c)

I2: '1, '2, '3) (c1,c2) where c1= '1 '3, c2= '2 (d)

I3: 3 '1, '2, '3) (c1)where c1= '1 '2 '3 (e)

Figures 4c, 4d, and 4e show the three interpretations I1, I2 and I3, giving the elementary components.

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

5Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007

Generally, a stroke can compose several polygons and polylines in a very complex way, as in Figure 5. Different interpretations can be provided for these configurations.

Figure 5. Polygons and polylines drawn by one stroke only

3.2. Ambiguities due to the intersection of two polygons

Another class of ambiguities can be produced by the intersection of two polygons.

Let us draw two strokes b and c in sequence as in Figure 6a, defined as below:

b= b(pb, tb,vb(x,t))

c= c(pc, tc,vc(x,t))

In this case too, the user has not made any deletions:

X=

Therefore considering spatial and temporal discontinuity, four generalised strokes '1, '2, '3 and

'4 can be considered, as represented in Figure 6b.

'i= 'i(pi, i,vi(x,t),X) i=1, 2, 3, 4

Possible interpretations of the sketch in Figure 6a are i) three different polygons A, B, and C (Figure 6c) ii) two overlapped polygons (Figure 6d) iii) two polygons and a polyline (Figure 6e) and iv) a second configuration of two polygons and a polyline (Figure 6f). These interpretations are formally expressed as follows:

I1: 1 '1, '2, '3, '4) (c1,c2,c3

where c1= '1 '2, c2= '2 '3, c3= '3 '4 (c)

I2: 2 '1, '2, '3, '4) (c1,c2)where c1= '1 '3, c2= '2 '4 (d)

I3: 3 '1, '2, '3, '4) (c1,c2,c3)where c1= '1 '3, c2= '3 '4, c3= '2 (e)

I4: 4 '1, '2, '3, '4) (c1,c2,c3)where c1= '1 '2, c2= '2 '4, c3= '3 (f)

Figure 6. An area C resulting from A and B areas intersection

3.3. Ambiguities due to the generation of undesired polygons and polylines

A different class of ambiguities is observed when the inaccuracy produces undesired polygons, as in Figure 7a, in which two strokes d and e represent the sketch and the second stroke produces a small third polygon due to inaccuracy in the user’s drawing.

d= d(pd, td,vd(x,t))

e= e(pe, te,ve(x,t))

Again, the user has not made any deletions:

X=

Considering the spatial and temporal discontinuity, five generalised strokes '1, '2, '3, '4 and '5 are considered in Figure 7b:

'i= 'i(pi, ti,vi(x,t),X ) i=1, 2, 3, 4, 5

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

6Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007

Possible interpretations of the sketch in Figure 7a are i) three different polygons A, B, and C (Figure 7c) ii) two polygons (Figure 7d) iii) one polygon and a polyline (Figure 7e).

Figure 7. Ambiguity due to the generation of an undesired polygon

The main three different interpretations are formally expressed as follows:

I1: 1 '1, '2, '3, '4, '5) (c1,c2,c3)where c1= '1 '4 '5, c2= '5 '2, c3= '3 '4 (c)

I2: 2 '1, '2, '3, '4, '5) (c1,c2)where c1= '1 '4 '5, c2= '2 '5 (d)

I3: 3 '1, '2, '3, '4, '5) (c1,c2)where c1= '1 '4 '5, c2= '2 '3 (e)

In this case, the most likely interpretation is I2,

consisting of two different polygons (A and B). However the user may actually wish to draw a small polygon near the boundary between A and B.

Such inaccuracies may also produce undesired small polylines, which are indistinguishable from intentional small polygons.

3.4 Ambiguities due to the over-tracing of different strokes

If a user over-traces two or more different strokes, a further class of ambiguities arises.

These occur when users over-trace one set of pixels by a different sequence of pixels. In this case over-traced pixels are incoherent with contiguous pixels as the drawing times are not consecutive.

Figure 8 gives an example to explain this type of ambiguity.

The stroke f (Figure 8a) is characterised by the set of time values 1-8:

f = f (pf, tf,vf (x,t))

The second stroke g (Figure 8a) over-traces the stroke f and is characterised by the set of time values 11-16:

g= g(pg, tg,vg(x,t))

The user has not made any deletions:

X=

Considering spatial and temporal discontinuity, four generalised strokes '1, '2, '3 and '4 are shown in Figure 8b:

'i= 'i(pi, ti,vi(x,t),X) i=1, 2, 3, 4

X=

In this case it is necessary to introduce the temporal layer concept and in particular a temporal layer for the stroke f and a second layer for g. Each temporal layer stores different time values for each pixel.

Considering the temporal layers and temporal discontinuity, the interpretation associates generalised strokes with two elementary components.

I: '1, '2, '3, '4) (c1,c2)where c1= '1, c2= '2 '3 '4

Figure 8. Over-tracing of pixel sequences

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

7Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007

This kind of ambiguity can also appear in some pictorial configurations involving polygons and polylines, as in Figure 9.

Figure 9. Examples of critical configurations

3.5. Ambiguities due to the deleting and retracing actions

Given two different polygons obtained by drawing two different strokes h and k, a different class of ambiguities arises, such as when the common boundary between them is deleted and one or more strokes are then drawn as a new boundary between the two polygons.

Figure 10a shows these two strokes:

h= h(ph, th,vh(x,t))

k= k(pk, tk,vk(x,t))

In this case the user has performed a deleting action :

X=

Figure 10b shows five generalised strokes '1, '2,'3, '4 and '5 considering the spatial and temporal

discontinuity introduced by the deleting action :

'i= 'i(pi, ti,vi(x,t),X )

X= i=1, 2, 3, 4, 5

Figure 10a shows two polygons with boundary pixels in common, while in Figure 10b these common pixels are deleted and new ones are redrawn. The boundary of A’ is intercepted by '1 and '2. The boundary of B’ is composed of '2, '3, '4 and '5.

The most common interpretation (I1) considers the two different areas A’ and B’ (Figure 10c). However,

other interpretations are possible, such as I2 and I3

which interpret the sketch as: 1) a polygon and a polyline (Figure 10d), or 2) three polylines (Figure 10e).

The interpretation of Figure 10b is formally expressed as follows:

I1: 1 '1, '2, '3, '4, ) (c1,c2)where c1= '1 '2, c2= '2 '3 '4 '5 (c)

I2: 2 '1, '2, '3, '4, ) (c1,c2)where c1= '1 '3 '4 '5, c2= '2 (d)

I3: 3 '1, '2, '3, '4, ) (c1,c2,c3)where c1= '1,c2= '2, c3= '3 '4 '5 (e)

Figure 10. Deleting and retracing of the boundary polyline between two polygons

These ambiguities can be generalised considering one or more polylines intercepting a polygon. They have at least one intersection point and one set of points that are neither internal nor boundary points of the polygon.

3.6. Strategies to solve ambiguities

The ambiguities classified above cause obvious difficulties in correct sketch interpretation. Users apply different criteria to express their intentions, and sketches are in any case only approximate and imprecise representations of the reality. Finally, deleting and re-drawing actions exacerbate these difficulties. The interpretation of the sketch should consider three types of information, closely related to the user’s intention: spatial, temporal and domain (syntactic and semantic). These can be considered separately by dividing the interpretation into three steps: [8] [9] [10] [11]:

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

8Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007

spatial step: it intercepts simple geometrical elements such as closed regions and polylines;temporal step: it composes geometrical elements of the spatial step according to temporal information; domain step: it composes objects according to knowledge of the application domain.

As they have spatial, temporal, syntactic/semantic continuity, generalised strokes can be base elements for the three previous steps. A generalised stroke is described by one oriented polyline and one temporal interval and is part of only one object (or two objects if forming part of their boundary).

The spatial step involves static information of the freehand sketch, which is observed as an image. This step identifies geometrical elements that are either objects of a specific context or parts of such objects. Temporal information, in the temporal step, enables simple geometrical objects to be composed as parts of a more complex one. Knowledge of the context allows these objects to be analysed while taking the overall perception of the sketch into account.

Objects can either match the set of symbols making up the domain’s alphabet (context), or can be combined to intercept these symbols according to their spatial adjacency.

The assumption that users preferably tend to draw each elementary component without discontinuities, or minimise their number, allows it to be affirmed that elementary components usually consist of temporally contiguous generalised strokes.

Starting from the definition of interpretation I, a measure of discontinuity in elementary components can be associated with this interpretation, considering the number of discontinuities and the waiting time between the generalised strokes constituting the elementary components.

The assumption that users tend to complete one object before beginning another, allows the affirmation that objects are usually made up of temporally contiguous elementary components.

Starting from the definition of interpretation J, a measure of discontinuity in objects can be associated with this interpretation, considering the waiting time between the elementary components constituting objects.

Obviously, the most correct interpretation should minimise these temporal measures and maximise spatial correspondence, so that each stroke has to be part of an interpreted object and each object has to be as close as possible to the spatial model of reference.

4. Conclusions This paper proposed some strategies for the

interpretation of ambiguities in human-computer interaction using a freehand drawing approach. The sketch recognition process starts by observing the user’s behaviour during this process and analysing its general characteristics.

In this way, we classified the possible ambiguities introduced by the user while sketching. Behaviour, producing an ambiguity in the sketch is thus attributed to a given ambiguity class.

Five classes are considered: 1) ambiguities due to crosses in a stroke; 2) ambiguities due to the intersection of two polygons; 3) ambiguities due to the generation of undesired polygons and polylines; 4) ambiguities due to the over-tracing of different strokes; 5) ambiguities due to the deleting and retracing actions.

The difficulties that these cause in interpreting sketches can be overcome by taking into account the intentions expressed by the user in the sketch. The user informative content is expressed by spatial and temporal information that the user transposed inside the sketch while he/she was drawing. Strategies adopted to solve the different ambiguity classes are defined and two assumptions are made which enable the most probably interpretation to be obtained.

These steps in the interpretation process enable ambiguities to be classified and resolved in order to match the sketch produced with the user’s real intention.

5. References

[1] A.D. Blaser, M. J. Egenhofer, “A Visual Tool for Querying Geographic Databases.” Proc. ACM AVI 2000, Palermo, pp. 211-216, 2000.

[2] Lin, M. Thomsen, J.A. Landay, “A Visual Language for Sketching Large and Complex Interactive Designs” Proc. ACM CHI 2002, Minneapolis, pp. 307-314, 2002.

[3] Ellen Yi-Luen Do, Mark D. Gross, “Drawing as a means to design reasoning”. AI and Design, 1996.

[4] Ellen Yi-Luen Do, Mark D. Gross, “Ambiguous intentions: a paper-like interface for creative design”. In Proceedings of UIST 96, pages 183–192, 1996.

[5] Jennifer Mankoff, Scott E Hudson, Grefory D. Abowd, “Providing intergrated toolkit-level support for ambiguity in recogntiion-based interfaces”. In Proceedings of the CHI 2000 conference on Human factors in computing systems, pages 368–375, 2000.

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

9Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007

[6] Christine Alvarado, Randall Davis, “Resolving Ambiguities to create Natural Computer-Based sketching Environment”, MIT Artificial Intelligence Laborator, In Proceedings of the IJCAI, 2001.

[7] J.V. Mahoney and M. P. J. Fromherz, “Three main concerns in sketch recognition and an approach to addressing them”, AAAI Spring Symposium on Sketch Understanding, Stanford, CA, March 2002.

[8] A. Rosenfeld, “Algorithms for Image/Vector Conversion”, Computer Graphics, n. 12, pp.135-139, 1978.

[9] A. Rosenfeld and A.C. Kak, “Digital Image Processing”, New York; Academic Press, Cap 11, 1987.

[10] Ferri F., Grifoni P.: “Interpretation and ambiguity resolution in Sketch-Based Interfaces”, Proceedings of the Ninth International Conference on Distributed Multimedia Systems, DMS 2003, KSI 2003, Miami, USA, 2003.

[11] Ferri F., Grifoni P., “Vectorization of Graphical Components in Sketch-Based Interfaces”, Databases in Networked Information Systems, DNIS 2003, Lecture Notes in Computer Science 2822 Springer 2003.

Proceedings of the 40th Hawaii International Conference on System Sciences - 2007

10Proceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS'07)0-7695-2755-8/07 $20.00 © 2007