Designing and Evaluating Embodied Learning Experiences for children with Autism and Intellectual

Disabilities in Developing Countries1st Author

1st author's affiliation 1st line of address 2nd line of address

Telephone number, incl. country code 1st author's E-mail address

2nd Author 2nd author's affiliation

1st line of address 2nd line of address

Telephone number, incl. country code 2nd E-mail

3rd Author 3rd author's affiliation

1st line of address 2nd line of address

Telephone number, incl. country code 3rd E-mail

ABSTRACT Autistic children and challenged learners in developing countries are often marginalized, but in larger cities several non-profit special schools are opening up to technological interventions. Working with two schools with such initiatives in New Delhi, India, we explored embodied interaction and learning experiences within the classroom environment, using computer supported cooperative gameplay. We designed, deployed and evaluated two gesture based games –a collaborative balloon game for children in the ASD spectrum and a virtual shopping game for challenged learners. The development of these games was informed by a field study, to identify problems, methods and guidelines for designing them for this audience. Our research goals were to investigate the general perception of embodied learning, and if these system actually affects a child’s learning, reactions and behavior. Our results indicate a strong preference for embodied interactions for learning. Furthermore, these games illustrate the potential of motivating social interaction by virtue of joint attention, and collaborative learning in terms of their mathematical ability.

Categories and Subject Descriptors H.1.2 [User/Machine systems]: Human factors and Human information processing.

General Terms Design and Human Factors

Keywords Embodied learning experiences, embodied interaction, Autism Spectrum Disorder and joint attention, special needs children and educational technology

1. INTRODUCTION The earliest Autism intervention in the classroom environment focused on visual cards representing tasks and schedules, which moved to tablet based visualization of tasks and schedules. The

TOBY Playpad [16] takes the technology paradigm even further by providing interactive game-based educational interventions, using the iPAd, focused on improving cognitive abilities of children with Autism; such as attention, memory, object recognition and learning of complex activities by imitating videos of people.

With emerging embodied technology solutions, such as the Microsoft Kinect, gaining momentum in the educational space, current Autism research is also moving towards these immersive, engaging and collaborative interaction methodologies [1, 2, 5, 7 and [9]. Research by the Lakeside Center for Autism [11], MEDIATE [14], and Kinems.com [1], taps into the potential of embodied interaction and its inherent affordances for embodied learning experiences within the classroom environment. The focus is towards encouraging embodied interaction and kinesthetic learning mechanisms for autistic children rather than providing visual flash cards for memory recall and recognition.

Unfortunately, most of the emerging research in Autism is based on the developed world, leaving behind its counterparts in the developing world. Autism in developing countries, like India, is still under diagnosed, and treated much like a social taboo [6]. Even though, India’s urban cities such as Delhi and Mumbai have several special schools dedicated to differently abled children, to the best of our knowledge none of those schools use embodied interaction technologies.

In hopes of bridging this technological divide for educational interventions for Autism using embodied technologies in developing countries, we collaborated with two special schools in New Delhi to design, develop and deploy an interactive educational system using gestures as the primary mode of interaction. Our research goal was to design and develop applications that support embodied learning for cognitively challenged individuals, including Autistic children, and evaluate whether the learning also translate to real world scenarios. Our applications encourage social interaction, peer collaboration and teach educational content though immersive and engaging games.

The first special school was for challenged learners; including high functioning Autistic children, children with Down’s syndrome, Asperger's syndrome and intellectual disabilities (ID). These children attend classroom teaching based on India’s largest open school curriculum, the National Institute of Open Schooling curriculum (NIOS) [13] and participate in nationwide NIOS exams for grades 8 and 10. These children learn educational content similar to their peers in public schools as set by the Indian

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CSCW’15, March 14-18, 2015, Vancouver, Canada. Copyright 2010 ACM 1-58113-000-0/00/0010 …$15.00.

Education Ministry. Subjects range from Hindi, English, homes Science to with vocational training in bakery, file making or Microsoft office among others. In the second school low functioning autistic children are taught basic life skills and are socially integrated with typically developed children, from the nearby slums, in a classroom style setting. The school also has an Autistic Center for research where researchers mediate educational interventions. The project started with a preliminary user study following participatory design approach where discussions with experts from the school, including teachers, researchers, occupational therapists and psychologist, were held to chalk out the possible application content. A researcher also spent time with the teachers, educationists, various therapist and the children from both schools in classroom settings and outdoor extracurricular activities. Based on these discussions, two applications, that used a Microsoft Kinect for motion tracking, were developed.

This paper first presents the related work in this domain, followed by a detailed description of our preliminary user study and its results. Next, we explain our system description and games that were developed: Money Game and balloon Game, and their evaluation methodology and results, followed by observations and discussion of the user study.

2. RELATED WORKS There is a huge body of research focused on educational interventions for children on the Autism Spectrum., most of them claim joint attention pays a pivotal role in early Autism detection and has been linked to language acquisition and social interaction in the later stages of neurological development of a child [4]. (Google Scholar shows about 48,700 research results for autism + joint attention as of 4th of June 2014). Befittingly, early Autism interventions focus on improving joint attention and shared experiences. From the early 90s, these interventions have increasing become technology driven employing computer mediated interactions [9]. Previous research strongly suggests that technology, including tablets, mobile phones, TV and video games, is appealing to autistic children [3 and 12]. With current emerging interactive technologies, such a touch-screens and gesture-recognition using the Microsoft Kinect, these educational interventions have become more pervasive in western special school and their learning environments.

The TOBY (Therapy Outcomes By You) Playpad is an iPad based educational intervention for Autistic children that “aims to empower parents to commence early intervention to maximize their child’s development” [16]. TOBY offers over 300 flash card inspired games covering over 50 cognitive skills for children with Autism. In 2013, the TODY initiate was awarded an Excellence in International Education award for its work in India. Although hugely successful, using a touch-based interaction method tends to limit the collaborative and cooperation nature of social interactions and confine its users to static positions reducing kinesthetic learnings opportunities. Gesture–based interaction allows for free form full body interaction and introduces game-like elements into learning, merging the lines between playing and studying. There are several advantage of using body gestures over touch screens tablets; the learning from gesture-based interaction is applicable to real world

scenarios where selection via pointing and moving of physical objects can be a part of everyday life.

Employing gesture-based interaction for embodied learning in Autism has gained favor in the recent years with many researchers working towards developing new and innovative embodied interactions to motivate and encourage Autistic children to learn and interact socially. The Lakeside Center for Autism in Washing state, US, started a Kinetix Academy in 2012 to “create a comprehensive curriculum of educational and therapeutic games specifically designed for children with autism to support their development of motor, speech and language, cognitive, academic, and social skills” [11].

MEDIATE [14, 15] is a Multisensory Environment Design for an Interface between Autistic and Typical Expressiveness. Setup in the United Kingdom, it consists of several multisensory applications present in one large space. The Mediate team included high functioning Autistic children in their participatory design process. The interactive environment “generates real time visual, aural and vibrotactile stimuli” for low functioning Autistic children. The children are allowed to explore the spaces at their own pace and there is no time limit to the interaction. Results from user studies, conducted in London, Hilversum (Netherlands), Barcelona and Portsmouth, have shown a high acceptance of its interactive space by Autistic children, who are able to express themselves creatively using the several multimodal applications presented in the space.

A more recent online community kinems.com [1], spread across the US, Greece and Netherlands, develops Kinect based games to improve cognitive abilities of children with learning disabilities, including hand-eye coordination, memory and attention and basic problem solving. Their finding suggest that even though suck kinesthetic interaction induce fatigue, it provides an immersive and engaging learning environment that can help children concentrate, thus outweighing the benefits.

Other such individual efforts include, but are not limited to, a gesture based game to match visual facial expressions [5], using existing Kinect sports games to improve hand-eye coordination, attention and focus [2]. Another innovative use of the Kinect camera and gesture recognition was used to detect repetitive behavior or tantrums [7].

What must be noted here is that institutions such as the Lakeside center, MEDIATE and Kinems.com, and the individual efforts mentioned above, are spread across the developed world. Most of the developing world is yet to include such embodied technological interventions for its special need children user group. With this emotion in mind, we worked with a school in New Delhi to understand the implication and potential of using these technological interventions in the developing classroom environments. In our knowledge, this is the first of its kind of research in India.

3. PRELIMENARY USER STUDY This section reports the setup and finding from the preliminary user study, which provided the basis of designing two gesture-based games discussed further in the paper. The preliminary user study was carried out in two schools for special children in New Delhi, India for a period of 2 months. While one school was dedicated to impart education to children with intellectual disabilities (ID) students, the other school mostly

had children with low functioning Autistic Spectrum Disorder (ASD). Following the participatory design process, initial sessions were held with the 12 schools educators, 2 occupational therapist, 2 psychologist, 2 speech therapist, 3 researchers and a few parents, who attend the schools with their children. A variety of methods were employed in the user study: group interviews, in-depth Interviews, participant observations, school transect walks and study notes. The group interviews, in depth interviews and informal discussion varied between 30 minutes to 3 hours of time duration with different stakeholders. This 2-week user study aimed to understand every stakeholder’s pain-points in general classroom pedagogy. Following the initial interview sessions, a researcher spent another two weeks in the school, attending daily classroom lectures and educational interventions. The goal of these sessions was to find the course content for the students, intervention techniques and methods of teaching in class. Researcher also observed general behavior of students in the class and how school educator use to communicate and handle the children. From the sessions, it was also noted that children of the schools were not familiar with gesture-based interaction (Xbox or Wii games) and none of the children from school had played any such game before. Although previous research suggests technology is appealing to autistic children in general [12], Autism by its nature is not welcoming to changes or new experiences. Thus, it became imperative to understand how the children would respond to embodied interaction, whether they would shun away from it or embrace it and whether the abstract nature of free form body gestures to trigger system response did not hinder interaction. To help understand the acceptance of gesture-based interaction, we conducted a field trail of simple gesture based games. The goal of these sessions was to understand how children would respond to gesture based interaction, to define a gesture interaction vocabulary comfortable for this user group and identify problem areas or topics for further investigation using embodied interaction mechanisms for technology supported cooperative work within the classroom environment. An annual school event “Diwali Mela” (school fest celebrating the festival of Lights) was chosen for the above purpose. The event was open to children and parents from all schools. The gestural gaming system was setup in a designated area of the school, to introduce children and parents to embodied interaction. The purpose of the installation was many folds; first to capture the initial reaction to such a system and then define a gesture vocabulary for three main interaction goals, namely navigation, selection and object manipulation. This provided an opportunity to understand gestures, which are fun, enjoyable and can be performed by a large range of children. The gestures that were most natural and intuitive for them while interacting with the system through gameplay mechanisms were also identified. The trials were recorded in the form of videos, photos and field observations and were later analyzed by the researchers. We included gestures relating to these three main goals of interaction in our applications to derive our gesture vocabulary, for our future user study. The first application was a free form painting interaction to understand how the children navigated using free form gestures, and the second was an embodied version of flash cards matching game, which the children are familiar with already, to understand object selection and manipulation

interaction goals and their accompanying gestures. The details of the applications and our results from the preliminary user study are described in the subsequent sections.

3.1 Field Trials The gestural gaming system was setup in a designated area of the school. Using a Kinect, a projector and a laptop running our games, two gaming applications were showcased to the Diwali Mela visitors. These applications were free-form painting (Figure 1), and an animal matching game (Figure 2). These games were developed using our in-house technological framework mentioned in section 4.1.

Figure 1 Diwali Mela Setup showing a researcher interacting

with the free form painting Free-form Painting was used as an introduction to a simple gesture-based interaction and there were no tasks in this game. The right hand had red color while the left hand had blue. The width of the hand-brush was controlled by moving the hand away (thinner) or towards (thicker) the onscreen canvas. Children could draw any shape or image they liked. The objective was to observe natural, comfortable and intuitive free form hand gestures, which is a relatively new form of interaction and rarely used in educational settings in India, and then derive the gestures that support navigation. Animal matching game was based on flash cards; something used extensively in Autism interventions, for example in the TOBY Playpad [16]. The game used pointing, with a dwell time of one second for selection, and drag-n-drop for object manipulation as mid-air gestures. This aim of this game was to introduce pointing as a selection mechanism and the concept of object manipulation. There were three cards on the top row, as shown in Figure 2, and three placeholders on the bottom row. The top row of cards were initially placed one of top of each other, as a deck of cards.

Figure 2: Flash card based Animal matcing game

The top row cards opened up (Figure 2), after a swipe gesture, as one would expand a deck of cards. Children point to and select a card which gets attached to their right hand. The attached card needs to be put to the right placeholder in the row below. When an animal picture was near the correct placeholder, it got magnetically attracted and flied on top of the placeholder, leaving

the participant’s hand. It was not possible to attach the animal picture to an incorrect placeholder. To make the game simpler for students who didn’t know spellings, the flash cards could also be matched by visual cue provided by the background color. Parents who visited the installation were encouraged to interact with it, after watching a short demo from the researcher or by watching another interacting child. Children interacted in the presence of their parents and the audience was instructed to clap for the child, after every game. This was derived from one of the insights from the field study; applauds motivated students to try out non-familiar tasks. The order of the tasks in the games was randomized for each child. Parents were asked to fill in a feedback questionnaire based on their experience with their child and their initial reaction to the games. They were also requested to provide comments and suggestions. Assistance was provided to parents in filling in the questionnaire, if required. Table 1 lists the questions and response options of the questionnaire.

Table 1 : Parent Questionnaire

Question Response Options Did you/the child enjoy using the system?

Yes or No

Was it tiring for you/her/him to use the system?

Yes or No

Have you/she/he used gestures based interaction before?

Yes or No

What are your initial reactions to the system?

Confused, Excited, Impressed, Skeptical or Other

How do you feel about using gestures for educational interventions?

Enthusiastic, Skeptical, Can’t Say or Other

What do you think will be the benefit of using gestures for educational interventions?

1. encourages social interaction by gestures and/or 2. encourages collaboration by gameplay and/or 3. makes learning more enjoyable and fun and/or 4. combines physical agility with learning and/or 5. doesn't seem to offer any benefit and/or 6. hinders learning and/or other

What is your overall impression of the system?

Positive, Not positive or negative, Negative

Would you consider using gestures for educational interventions in the future?

Yes, No or only if it worked better

3.2 Results 3.2.1 Findings from Participatory design As mentioned earlier, the children in the first school included high functioning ASD children, children with Down’s syndrome, Asperger's syndrome and challenged learners. These children were taught NIOS [13] open school curriculum and take nationwide exams for grades 8 and 10. These children learn content similar to

their peers in public schools as set by the Indian Education Ministry. From the initial user study, we gathered a broad range of areas in which the students faced a difficulty in learning. These included basic concepts involving in monetary transactions, time management, planning or scheduling of events, working with constraints such as time or a budget, concept of currency (mainly because the size/shape/weight of coins and notes does not relate to its value). While most of them did appear in the nationwide exams, not many were able to clear it, demotivating them to learn in school.

While the second school, catered to low functioning ASD children and several of whom were not verbal communicators and were accompanied by their parents or nannies to school. The broad areas in which students faced problems included personal hygiene such as washing hands or taking a shower, concepts of collaborative gameplay and team work, lack of motivation for social interaction. These children occasionally displayed repetitive behavior, when stressed or uncomfortable and even when enjoying a certain activity. This behavior can be isolating for the child and requires special attention by educators. After the analysis of interviews from different stakeholders in participatory design discussions, we consolidated a set of broad guidelines to follow while designing our embodied educational application. These guidelines were:

1. Introduce the game or application. There should be a clear start and end of each interactive session.

2. The interaction should be consistent yet fun for children.

3. Combine the subject matter with real world scenarios and real life social interactions and cultural norms to provide an easier transition from virtual to real world.

4. There should be enough repetition to allow slow learners to remember the concepts easily.

5. The interaction should empower and involve the child in decision making.

6. Include colorful graphics with smooth visual movements and illustrations to captivate attention for longer time spans.

7. Provide adequate narration of core concepts and to visual images to help users with impaired visual modality.

8. Games should focus on general life skills help make the child independent and confident.

For both the Schools, onscreen cursor based navigation gestures and selection using a longer dwell time was used.

The previously mentioned findings guided our design process explained in section 4.

3.2.2 Findings from field trials In total, 18 students participated in the field trials with the system. Out of 18, 14 responses were for the children who interacted with both the games. Rest 4 either briefly interacted with either one of the games or the parent was in a hurry and so did not fill up the questionnaire.

From the questionnaire data, all of the parents agreed that their child enjoyed using the system and only 21% said it was tiring to use. Three out of the 14 children had used some form of gesture interact before. Initial reaction of parents included excitement (50%) or being impressed (50%) by the games. 78% of the parents were enthusiastic about using gestures for educational interventions, while only one parent was skeptical and another didn’t comment. All (92%) but one parent had a positive overall impression of the system and that one parent had neither negative nor positive impression. 12 out of the 14 parents (85%) said they would consider using gestures for educational interventions in the future while one parent wished the system worked better and another one did not comment. In tune with the overall positive acceptance of our system and gesture based interaction for education interventions, the parents selected a number of benefits of using gestures for educational interventions, as shown in Figure 3. Parents also appreciated the inherent physical nature of the interaction, and commented that such games are good for body movement and encouraging physical activity. They hoped to see more rewards and motivation in the gameplay to help their child perform better and suggested focusing on real world scenarios such basic arithmetic. As one parent exclaimed, such systems ‘will leave an impact for a longer time and improve understanding’ of more abstract concepts.

Figure 3: benefits of using gestures for educational

interventions

3.2.3 Identification of Gestures As mentioned in previous sections, we presented a system with two games during the school’s Diwali Mela: free-form painting and animal matching. These two games enabled us to identify gestures required for selection, navigation and object manipulation, key inputs for our interactive embodied learning educational system, described in section 4.

Navigation: for both the games, a cursor was present on the screen to help navigate the screen space. This was more evident in the free–form painting game where hand motion left a red or blue imprint on the plain white canvas. From our observations, the children found it easiest to form circles using both the hands, moving their hand anti-clockwise and left hand clockwise. Several children, on the insistence of the parents, tried to write their name on the white canvas and were able to write several alphabets with ease.

Selection: the animal matching game used a one second dwell time for object selection, which was found to be too quick for the children. Once an object was attached to the onscreen cursor, the children were able understand the concept of dragging. Surprisingly, the system could recognize a child on a wheelchair, without using the Kinect’s seated mode options, and he could play both the games without any difficult.

Figure 4: Participant creating midair circles

Figure 5: Selection gesture with a one second dwell time

Object Manipulation: the animal matching game required moving a selected animal picture to a placeholder on the bottom row using the desktop drag and drop metaphor. Although all children matched the three animal pictures to their placeholders, it was observed to be rather by chance than by choice, since placeholders were magnetic. Thus object manipulation, drag and then drop, was the hardest gesture for the children and there was no consensus on which would be the most suitable gesture for the same. Further research is required depending on the game tasks to identify this group of gestures.

Figure 6: Parent helping a child with drag and drop gesture

Given the wide spread of the Autism Spectrum and therefore children’s abilities, our discussions focused on developing two distinct and individual applications, one for each school. Based on our findings from our participatory design discussions and the

“Diwali Mela” user study, we identified focus areas for each school and designed two gesture based games, which we will discuss in the following section.

4. SYTEM DESCRIPTION Informed by our preliminary user study, the participatory design discussions and Diwali Mela installation, we designed and developed two games, cooperative money game for children of the first school and collaborative balloon game for children of second school. The technology used and the actual games are described in subsequent sections.

4.1 System Implementation The games were based on an in-house application framework consisting of three main processes; a Kinect service, graphics engine and application core logic. The Kinect service is a thin client over the Microsoft Kinect SDK that connects to the core logic over sockets. Microsoft Kinect is used to track the user’s body movements. Kinect SDK tracks user joints in 3D coordinate space and we use upper body joints to identify touching and pointing gestures. All audio and graphical content is rendered using the Panda 3D graphics engine. The core logic is a Python based application that consists of an input/output management module, which handles all the inter-process communication and acts as an interface between the core logic, the Kinect and the graphics engine. Thus, the Kinect data is taken as the input, based on which the relevant content is displayed via the graphics engine.

4.2 Cooperative Money Game The cooperative money game focuses on introducing the concept of money and purchasing, and building confidence to visit a grocery store among high functioning autistic and developmentally challenged children studying in the first school. A participant starts the game by standing in front of the Kinect and is greeted by a female voice welcoming the child to the virtual shop. Inspired by Delhi’s Kirana shops (small local grocery shop), the virtual shop has two shelves behind a shop table, as shown in Figure 4. These shelves contain food items that can be bought by the participant. The items are randomly arranged for each session, removing learnability of item placements.

Figure 7: Cooperative Money Game Screen

The right side of the screen has the participants wallet or available money with total available displayed at the bottom. The left side of the application shows the real time billing of items as they are bought and paid for by the participant. To select and item, the participant hovers over the item using her left hand with a dwell time of 1.5 seconds. A selection is made when the item flies to the table and its price is added to the bill on the left.

Once there is at least one item to be bought, that is, placed on the shop table, the participant could select any denomination of money to pay for the item(s) using her right hand. This two-handed selection mechanism was used to encourage more bodily movement and isolate the item selection concept from the money selection concept as advised by the school’s researchers. To support multiple modalities, each item and money is accompanied by its audio name and each transaction process also has audio feedback. For example, when balance is returned, the female voice says here is your balance and if the participant runs out of money the female voice says you have no money left.

Figure 8: Buying items from the table and paying using 10

rupees note The application aims to clarify and simplify the process of paying and taking the balance, if any, to instill confidence about money transactions. Based on the Diwali Mela findings in section 3.2, drag and drop gestures were not included as part of the interaction. Instead items and money flew to pre-designated spaces on the table or bag. To provide control flexibility for customization, the type of item, its price and total available money in the wallet on the right hand side could be changes easily between sessions.

4.3 Collaborative Balloon Game The collaborative balloon game focuses on encouraging low functioning autistic children to explore proto-declarative pointing [10] leading to shared experiences by virtue of joint attention, a crucial skill for Autistic children [4]. The application requires a two member team that works together to select the same virtual balloon simultaneously out of three possible options, further encouraging the need for social interaction, verbal or non-verbal, between the team members. A snapshot of the game with one player’s right hand is shown in Figure 9.

Figure 9: Collaborative balloon game screen with one user

If one team member selects a balloon, a star is shown on the screen as a reward. If both team members select the same balloon,

a rainbow growing in size is shown on the screen with background music. The rainbow and star is shown in Figure 10. Since the balloon game requires two players to simultaneously select one balloon, the dwell time for selection was increased to 3 seconds.

Figure 10: Rainbow and star as rewards

The teams consist of one autistic child and one typically developed child studying in the second school, thus also emphasizing on socially integrating autistic children with typically developed children by team building gameplay.

5. EVALUATION We performed studies aimed at exploring the extent to which the embodied interactions and our system will be useful to special children in enhancing the learning. We had three research questions:

1. How do a child and an educator perceive the concept of Embodied Learning?

2. Does the gestural system actually affect Child’s Reaction and Behavior?

3. Does Embodied Interaction derive Embodied Learning?

To help understand these questions we carried out a two fold exploratory field trial over a period of a month. The first phase of the experiment involved recruiting 18 differently abled students having different categories of Intellectual Disability. The participants were given practice sessions with the cooperative Money Game installed in the school to make them learn the monetary concept. The researchers listed the quantitative and qualitative data through system logs, stopwatch, qualitative questionnaire and observations.

In the second phase of the study, we investigated the Collaborative balloon game with 10 differently abled students suffering from severe autism. The aim of the application was to make them practice collaboration with Joint attention, joint pointing, turn taking and oral communication.

The following section describes the field trial carried out; detailing our methods, findings and followed by discussions of the results.

5.1 Experimental Methodology Both the evaluation experiment was conducted in three phases over a period of five weeks. We will now present the methodology and demographics for both the experiments.

5.1.1 Cooperative Money Game The experiment involved recruiting 18 differently abled students (14 Males; 4 Females) having different categories of ID or Autism (Down syndrome, learning disabilities, Epilepsy, Asperger etc.) from a school for special children in New Delhi, India. Students

were aged between 16 to 35 years and their mental age ranged between 6 years to 19 years. The participants were students of Class X to Class Y. We recruited students from this background because most of students in our user group were at least, physically not dependable and yet had sufficient scope for learning. Also a variance in term of their Intellectual disability helped us to randomize the sample and analyze the results in term of diversified range of cases.

The participant is given a grocery list by the teacher and a fixed amount of virtual money which he/she must use to purchase the items from the virtual shop. To evaluate whether the learning is translated to real world scenarios, at the end of the training sessions with the virtual application, the teams playing on the virtual balloon application will be encouraged to select real balloons placed around the school while the children playing the virtual shopping application will be encouraged to purchase groceries from an actual store near the school.

Methodology

For the evaluation, we conducted manual tests twice, both pre and post evaluation, as a part of class exercises. These tests aimed to check the mathematical ability and performance on monetary transaction of the participant. These were in form of different Mathematical exercises, which they had to solve in their notebooks. Later the researchers evaluated these exercises manually. While these scores do not map exactly to the scores provided by the game, they relate well to how the teachers measure these concepts at school. These scores formed the baseline for evaluation. The second phase was experimental trial of the Kinect Game. The kinect based game setup was installed in the play area of the school as shown in Figure 11.

Figure 11: Researcher helping a child with the Money game

The money game trials were conducted with students for period of three weeks. We gave following task to students: (i) You need to buy a breakfast from the Items available in

the shop. Select your items and put them on the table. (ii) You have 100 Rupees and need to pay a bill for the items

you buy. Also check the change returned by shopkeeper. We maintained system logs of the usage of the game by students during this period. At the end of every week, a field study was done to collect task completion time; success rate, behavior analysis and other observations. This was done to determine the longitudinal learning offered by the system. To evaluate learning offered by the games, no classes related to any of the above-mentioned concept were offered by the school during that period. In the third phase, we re-evaluated the participants with mathematical ability and monetary transaction performance tests with the notebook exercises.

5.1.2 Collaborative Balloon Game The Second study was done with 10 differently abled children (6 Males; 4 Females) having Autism Spectrum Disorders (ASD). All of these participants had problem in social interaction, communication (both verbal and non verbal), behaviors and interests. All participants were students from the second school and had Autistic Spectrum Disorder.

Methodology

For the evaluation of the balloon application, we conducted a balloon test both pre and post evaluation to check participant’s ability for Joint Attention and Pointing. Since, most of these children have a different capacity to do physical task and an aptitude to learn, the result may have had an influence of individual performance. To make analyze the results more generically, the first test was done with 18 participants. This allowed the researchers to collect an average normalized data for further analysis.

Physical balloons of different colors were arranged on the School’s wall in a group of three different colors. Participants were asked to form a team of two. Depending on their behavior and comfort level, they were given freedom to team up either with a fellow student, class teachers or even with moderators. The participants were given the following task:

(1) Member-I from team had to select three different colored balloons, in the pattern instructed by moderators and points his finger towards it. The Moderators made sure, that there was enough randomization in the positions of the selected balloons in the wall.

(2) Member-II had to point to the same balloon, and then go and touch that balloon.

The task was repeated atleast three times with different colored balloon. The task completion time, success rate and behavior analysis were collected by the researchers and later analyzed. User trails are shown in the figures below:

Figure 12: Team of students playing the physical balloon game The second phase was experimental trial of the Kinect Balloon Game with 10 participants. These participants had an emerging Joint attention skill and were able to tolerate presence of the external person, to a certain extent and perform a task. The kinect based game setup was installed in the play area of this school. The balloon game trials were conducted with the teams everyday for a period of two weeks. We maintained a system logs of the usage of the game by students during this period as well as usage was observed and noted down by the moderators. We collected task completion time; success rate, behavior analysis and other observations during the trails to determine the longitudinal

improvement offered by the system. No Exercise related to concept of joint attention, joint pointing and turn taking was offered to them during this period.

Figure 13: Team of students playing the Kinect balloon game

In the third phase, we re-evaluated the 10 participants with a similar physical balloon test as we did in phase-I. The task completion time, success rate and behavior analysis were collected by the researchers and later analyzed. In the section below we will describe the results from field trials carried out; followed by discussions and implication of these results.

6. RESULTS 6.1.1 Cooperative Money Game As shown in Figure 14, we observed a difference in the mathematical ability and performance on monetary transaction of the participant through the notebook test. In the phase–I (pre-trial) of the test, the highest score was 80 and lowest was 30 (maximum 100 marks). The average score for 18 students was 56.94 with a high standard deviation of 16.25. In the phase–III (post-trial), the highest score was 95 and lowest was 50 (maximum 100 marks). The average score for 18 students was 71.38 with a standard deviation of 12.45.

Figure 14: Comparative analysis of Phase-I and Phase-III

absolute scores Comparing Phase I and Phase III of the experiment, a relative improvement of 25.36% was seen. We plot relative percentage of improvement for each of these 18 students in relation to number of items they bought on the Kinect system (Figure 15). The Pearson correlation of the improvement in mathematical ability with number of items bought on the system was 0.74. All 18 participants played the game at least 4 times, with an average of 2.5 hours (S.D. = 1.2 hours) total session time per participant. All the participants successfully bought breakfast for

themselves, spending an average of 65 Rupees, and buying an average of 4 items.

Figure 15: Percentage relative improvement and number of

items bought Figure 16 shows the comparative analysis of time spent to buy one item at the end of the week evaluation. The figure establishes that most of the bars (i.e. students) finish with their lowest time in the third week, thus establishing the fact that their ability to use the gestural system as well as mathematical ability did improved through the process of practicing. The most significant evaluation of the game was to compare the relative percentage improvements of the mathematical ability tests with the amount of money spent on the student on the application. We observed a strong positive correlation between the two-R=0.80 (r square=0.64), which means that a higher the amount of money spent on the system, higher the relative improvement scores in mathematical ability tests (and vice versa).

Figure 16: Comparative analysis of time spent to buy one item

at the end of the week

6.1.2 Collaborative Balloon Game As described earlier, Phase–I of the experiment had diversified participant with a lot of variance in intellectual capacities (including some low functional ASDs). In the phase–I (pre-trial) of the test with, 18 participants, the average time to complete a task was 66.16 sec with a very high standard deviation of 65.23 sec. For Phase II and III we selected 10 participants who had an emerging Joint attention skill and were able to tolerate presence of the external person, to a certain extent and perform a task. In phase–III (post-trial), average time to complete a task was 7.27 sec with standard deviation of 5.42 sec. Comparing Phase I and Phase III of the experiment, a relative improvement of 88.98% was seen. Although there was a higher difference in intellectual capabilities among the participants in Phase–I, we believe all 18

students would have improved, if motivated to practice the balloon game. Some of them may have required more time to show a significant improvement, to the extent it happened in the case of 10 participants in Phase-III. All 10 participants tried the game atleast 12 times. System log reported an overall more than 36 hours of testing the system. On an average each student tried 3.6 hours of the system in total. All the participants successfully completed the task, at times with the intervention of the moderators, jointly pointing to an average of 2 balloons. We plot relative percentage of improvement for each of these 10 students in relation to number of balloons they pointed together on the Kinect system (Figure 17). The Pearson correlation of the improvement in mathematical ability with number of balloons pointed on the system was 0.8336 (r square = 0.6949). Out of total 280 successful attempts for joint pointing, 146 (52.12%) required help and intervention by the moderators. The rest 134 (47.89%) attempts were successfully done alone by the participants.

Figure 17: Number of balloons jointly pointed and percentage

improvement in the time of physical task (Phase-III) The analysis of the experiment results revealed that (a) the students were able to play the game (b) the student’s having ASD, improved in their ability for joint attention and pointing as a result of playing Collaborative Balloon game with help and intervention from the Moderators. (c) Student’s having ID, improved in their ability of managing Monetary Concepts as result of practicing the cooperative Money Game. The improvement was reflected in both the manual evaluation as well as in the task completion time they achieved in the games. Enthusiasm level to these games was high and these games did improve the collaborative interactions.

7. OBSERVATIONS and DISCUSSION

7.1 Reaction, Emotions and Behavioral changes Positive emotions All the participants were excited after playing the game. Most of them were seen making happy comments all throughout the trials. At some instance, it was observed that they wanted to play the game again, and the moderators had to pursue many of them to leave the installation for the next class. The system had an audio feedback in form of applaud. Once the task was successful, many of the participants were seen clapping for themselves with the system audio. At another instance a participant having Autism

Spectrum Disorder asked the moderators – “Don’t you have more colors in balloons?” Negative Emotions While almost all the participants in both the game were excited and happy playing, two of them reported a slight pains in their hand. We observed them swinging their hands. During the Money app trial, at some instances, the moderator had asked students to use their left hand to pick the items they want to buy and their right hand to pay the bill. This was done to reduce the student’s confusion. We observed, some participants faced difficulty, using left hand to select the objects. One of the participants was upset, as she was not called often for the trails and wanted trials to be placed in her timetable. Time taken for selecting items that were places in the middle of application UI was more. We observed that most of the participants felt bad when they were not able to do the tasks properly or they felt they are doing it slowly. At one instance, we noticed a participant slaps her left hand to express the frustration. Other Observations and communication with the mediators In the first few days of the trials for balloon game, the participants did not show active interest. Once they observed their peer performing the trials, they came forward to participate. At another instance, we saw students touching the entire sequence for the balloon, twice, once they had completed the round one. In the money app, we observed the performance getting drastically better with more trials. Participants navigated more carefully to objects in the later trials. We observed, air conditioned environment ideally suited the trials. Surprisingly, as the temperature in the room increased, few students preferred selecting liquid objects than solid. Few even requested for cold drinks as against solid items described in the system. During the trial, a grid was drawn on the floor to enable participants to know the exact location from where to operate. In the later trials, we observed an excellent improvement in terms of understanding where to stand and reposition themselves.

7.2 Collaboration and Team Work In both the trials, the games gave rise to collaborative interactions. In the balloon game, participants were observed expressing both their happiness and frustration with moderators & other team members, in the context of high levels of excitement generated by the trials. Most of them patiently worked with researcher, hugged the moderators on success. There were exchange of both physical prompts and verbal prompts from moderators to participants during the trials. Owing to the familiarity with the touch screens, some were observed to touch the computer monitor to answer. At one instance, we noticed a participant got hyper after he could not complete the task and started beating peers. He was finally sent back to the class. Similarly, in the money game, many a times, the participants exclaimed and shared their success with the moderators. With progress in trials, participants were observed to be much more expressive and verbal than usual. Their speech too became slightly better than usual. At times, the other students who were just observing the trials intervened between the task, making the attempts more collective.

7.3 Embodied Interaction Design Guidelines Based on our participatory design discussion for designing and developing the Money and Balloon games, we consolidated several guidelines for embodied interaction when designing for children with special needs.

Provide serial and structured animated or visual media content Children on the Autism spectrum and with ID find it difficult to notice simultaneous changes. If the animation and other visual media content follow a serial step by step change, it becomes easier for the children to notice all the changes and follow the progress of the interaction. In the Money Game, the purchasing transition animation was slowed down to follow one movement after another, that is, at any given time; there was only one item or money movement, from the shelves or the wallet on the right hand side, to the table. Once that was the animation finished, the balance was returned back from the table to the wallet on the right hand side. Updating the bill and total amount was also in sync with these serial animations. In the balloon game, there was a pause between two consecutive selections, individual selections and joint selections, so that while the rainbow and star animations are played, there can be no other changes in the system. Follow real world scenarios For the money game, the bill on the left hand side was so that children can see what they have bought and their total bill. The bill format was in line with their tuck shop activity where kirana shop keepers give a written bill back for all items purchased. Provide multimodal feedback Children find it easier to select items that have both visual (change in size) and auditory (name of the item is said aloud) feedback. This reinforced multimodal feedback helps overcome visual or auditory impairments, if any, and provided multiple stimuli for attention. Both the money and balloon games provided multimodal feedback for items and rewards. Provide clear start and end of gameplay To avoid ambiguity and confusion during interaction, it is extremely important to provide clear start and end of gameplay sessions. For instance, in both the games, when the user moves out of the Kinect’s interaction zone, the game screen turns black to indicate an end of interaction. The money game also catered to various end of game scenarios such as not enough money left to buy an items, or running out of money or items to buy. Each of these scenarios stated the reason for the game end, followed by a well done visual and audio feedback to conclude the gameplay. Both games also had started with a welcome audio message followed by music and a change of the background screen color from black to white. Provide control to the teacher or moderator Due to wide range of abilities of the children, the teacher or moderator present should be able to customize the game based on individual capabilities and interest. In the money games, the number, type and price of items, and the total amount of money in the wallet could be changed easily between sessions. For the balloon game, the balloons could be customized to float towards the top of the screen, making selection a little more difficult for children who found the static arrangement easy or boring. Potential of embodied technology to support collaborative and cooperative work Using free form body gestures provides a mechanism for more inclusive social interaction and team work when compared with touch screens that have limited form factor and surface area to support multiple users. The inherent nature of such embodied interaction makes for an immersive and engaging shared user

experience, as designed in the balloon game. For the money game, even though it was a single player game, the interaction was visible to the teacher and moderators who could then prompt and encourage, indirectly involving them in the gameplay through cooperation.

8. CONCLUSION The project extended our understanding of embodied learning experiences for special needs children and provided a mechanism to design and develop future embodied learning applications. These embodied learning methodologies enable teachers and educationists to focus on applications that impart basic life skills to cognitively challenged individuals bridging the gap between classroom learning and real world scenarios. By supporting collaborative and cooperation interaction paradigms, children with special needs can be introduced to subtle social interactions, building confidence and independence, and providing a mechanism to be integrated with typically developed individuals.

9. ACKNOWLEDGMENTS Our thanks to the schools for their valuable time and guidance, and a special thanks to all the children involved.

10. REFERENCES [1] Altanis, G., Boloudakis, M., Retalis, S., & Nikou, N.

Children with Motor Impairments Play a Kinect Learning Game: First Findings from a Pilot Case in an Authentic Classroom Environment. www.kinems.com

[2] Bartoli, L., Corradi, C., Garzotto, F., & Valoriani, M. (2013, June). Exploring motion-based touchless games for autistic children's learning. In Proceedings of the 12th International Conference on Interaction Design and Children (pp. 102-111). ACM.

[3] Bauminger-Zviely, N., Eden, S., Zancanaro, M., Weiss, P. L., & Gal, E. (2013). Increasing social engagement in children with high-functioning autism spectrum disorder using collaborative technologies in the school environment. Autism, 17(3), 317-339.

[4] Charman, T. (2003). Why is joint attention a pivotal skill in autism?. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 358(1430), 315-324.

[5] Christinaki, E., Triantafyllidis, G., & Vidakis, N. (2010). A gesture-controlled Serious Game for teaching emotion recognition skills to preschoolers with autism.

[6] Daley, T. C. (2004). From symptom recognition to diagnosis: Children with autism in urban India. Social Science & Medicine, 58(7), 1323-1335.

[7] Firth, N. I. A. L. L. (2012). Kinect cameras watch for autism. New scientist, 214(2863), 17.

[8] Frauenberger, C., Good, J., Alcorn, A., & Pain, H. (2013). Conversing through and about technologies: Design critique as an opportunity to engage children with autism and broaden research (er) perspectives. International Journal of Child-Computer Interaction, 1(2), 38-49.

[9] Goldsmith, T. R., & LeBlanc, L. A. (2004). Use of Technology in Interventions for Children with Autism. Journal of Early and Intensive Behavior Intervention, 1(2), 166-178.

[10] Goodhart, F., & Baron-Cohen, S. (1993). How many ways can the point be made? Evidence from children with and without autism. First Language, 13(38), 225-233.

[11] LAKESIDE CENTER FOR AUTISM, http://lakesideautism.com/tag/kinect/

[12] Mineo, B. A., Ziegler, W., Gill, S., & Salkin, D. (2009). Engagement with electronic screen media among students with autism spectrum disorders. Journal of autism and developmental disorders, 39(1), 172-187.

[13] NIOS http://www.nios.ac.in/ [14] Parés, N., Carreras, A., Durany, J., Ferrer, J., Freixa, P.,

Gómez, D., ... & Sanjurjo, À. (2005, June). Promotion of creative activity in children with severe autism through visuals in an interactive multisensory environment. In Proceedings of the 2005 conference on Interaction design and children (pp. 110-116). ACM.

[15] Parés, N., Carreras, A., Durany, J., Ferrer, J., Freixa, P., Gómez, D., ... & Sanjurjo, A. (2004). MEDIATE: An interactive multisensory environment for children with severe autism and no verbal communication. In Proc. of the Third International Workshop on Virtual Rehabilitation.

[16] Porayska-Pomsta, K., Frauenberger, C., Pain, H., Rajendran, G., Smith, T., Menzies, R., ... & Lemon, O. (2012). Developing technology for autism: an interdisciplinary approach. Personal and Ubiquitous Computing, 16(2), 117-127.

[17] Venkatesh, S., Phung, D., Duong, T., Greenhill, S., & Adams, B. (2013, April). TOBY: early intervention in autism through technology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 3187-3196). ACM.