A cognitive prosthesis and communication support for people with dementia

International Workshop on Cognitive Prostheses and Assisted Communication (CPAC 2006) In conjunction with 2006 International Conference on Intelligent User Interfaces (IUI 2006) Sydney, Australia January 29, 2006 Organizers:

Norman Alm, University of Dundee (United Kingdom) Shinji Abe, ATR IRC Labs. (Japan) Noriaki Kuwahara, ATR IRC Labs. (Japan)

Program Committee:

Sue Balandin, Sydney University (Australia) Joseph F. Coughlin, MIT AgeLab (U.S.A.) Joy Goodman, University of Cambridge (United Kingdom) Motoichiro Kato, Keio University (Japan) Henry Kautz, University of Washington (U.S.A.) Kazuhiro Kuwabara, ATR IRC Labs. (Japan) Michael J. Lyons, ATR IRC Labs. (Japan) Andrew Monk, University of York (United Kingdom) Siriwan Suebnukarn, Asian Institute of Technology (Thailand) Toshiyo Tamura, Chiba University (Japan) Yasuyuki Yanagida, Meijo University (Japan) Kiyoshi Yasuda, Chiba Rosai Hospital (Japan)

Preface Various prostheses have been developed to help people with physical impairments, by replacing lost or underdeveloped abilities, and these have played an important part in facilitating the personʹs social participation. Providing prostheses for lost or underdeveloped cognitive abilities, however, presents a challenge of a different order. Some work on memory aids and daily schedule management systems has shown promise, but progress in this field will depend upon keeping up to date with our advancing knowledge about cognition generally as well as emerging technologies which could be platforms for various kinds of cognitive support. An interdisciplinary approach to this programme of work will be essential in order to produce:

• Intelligent systems for augmenting or replacing impaired or absent cognitive

functions. • User interfaces that make it possible for cognitively impaired people to make

effective use of the support offered. • Context‐awareness that recognizes the situation of the cognitively impaired person

and detects appropriate moments for offering support.

This workshop will offer the opportunity for researchers in the fields of assistive technology, cognitive psychology, user interface design and context‐awareness to present the state of the art in each field and to discuss an approach and a research agenda for realizing effective cognitive prostheses. While the intended audience for this workshop includes those with experience or interest in Assistive Technology especially for dementia people and their care, it is also crucial to have participation by those with expertise in other areas such as Artificial Intelligence, Intelligent User Interface Design, Context‐aware systems, and Cognitive Psychology.

Table of Contents Personalized Assistance (9:00 – 10:30)

Using Semantic Web Technologies for Cognitive Prostheses in Networked Interaction Therapy ...................................................................................................................................... 1

Kazuhiro Kuwabara (ATR), Noriaki Kuwahara (ATR), Shinji Abe (ATR), and Kiyoshi Yasuda (Chiba Rosai Hospital)

Towards a Wearable Cognitive Prosthesis to Support “What” and “Who” Type Memory Activities .................................................................................................................. 6

Tatsuyuki Kawamura, Yasuyuki Kono, and Masatsugu Kidode (NAIST) Vision‐based Behavior Detection for Monitoring and Assisting Memory‐Impaired People ..................................................................................................................................... 10

Akira Utsumi (ATR), Daisuke Kanbara (Ritsumeikan University), Shinjiro Kawato (ATR), Shinji Abe (ATR), and Hironori Yamauchi (Ritsumeikan University)

Interface Design (11:00 – 12:30)

Developing an engaging multimedia activity system for people with dementia ......... 16 Arlene Astell (University of St. Andrews), Norman Alm (University of Dundee), Gary Gowans (Duncan of Jordanstone College of Art), Maggie Ellis (University of St. Andrews), Richard Dye (University of Dundee), and Phil Vaughan (Duncan of Jordanstone College of Art)

Networked Reminiscence Content Authoring and Delivery for Elderly People with Dementia.................................................................................................................................. 20

Noriaki Kuwahara, Shinji Abe, and Kazuhiro Kuwabara (ATR) An easy‐to‐use Interface for Evidential Reasoner.............................................................. 26

QingXiang Wu, David Bell, and Rashid Hafeez Khokhar (Queen’s University Belfast)

Communication and Cognitive Psychology (14:00 – 16:00)

A cognitive prosthesis to support communication by people with dementia ............... 30 Norman Alm (University of Dundee), Richard Dye (University of Dundee), Arlene Astell (University of St. Andrews), Maggie Ellis (University of St. Andrews), Gary Gowans (Duncan of Jordanstone College of Art), and Jim Campbell (University of Dundee)

Promoting Virtual Social Interaction with Older People .................................................. 37

Peter Bagnall, Guy Dewsbury, Victor Onditi, and Ian Sommerville (Lancaster University) Talking with Individuals with Dementia on Video Phone: A Preliminary Study for Networked Interaction Therapy ........................................................................................... 43

Kiyoshi Yasuda (Chiba Rosai Hospital), Kazuhiro Kuwabara (ATR), Noriaki Kuwahara (ATR), Shinji Abe (ATR), and Nobuji Tetsutani (Tokyo Denki University)

The Relationships among Cognition, Parental Speech Patterns and Foreign Language Ability, and Multiple Intelligence Theory based Foreign Language Teaching .............. 47

Keiko Hayashi (Hiroshima Jogakuin University)

Using Semantic Web Technologies for CognitiveProstheses in Networked Interaction Therapy

Kazuhiro KuwabaraATR

2-2-2 Hikaridai, Seika-choSoraku-gun, Kyoto 619-0288 Japan

[email protected]

Noriaki KuwaharaATR


[email protected]

Shinji AbeATR


[email protected]

Kiyoshi YasudaChiba Rosai Hospital

2-16 Tatsumidai-HigashiIchiharashi, Chiba 290-0003 Japan

[email protected]

ABSTRACTThis paper explores the concept of a ‘semantic desktop’ as a plat-form for Networked Interaction Therapy, which is a framework forproviding support for people (clients) with dementia or memory-impairment. Since personalized services are essential in networkedinteraction therapy, there is a need for both clients and caregiversto handle personal data. The semantic desktop can be used as aplatform for handling these personal data. Application scenariosinclude supporting the creation of multimedia content to be usedin therapy, such as reminiscence videos, and storing and analyzingcare sessions, as well as providing a collaboration infrastructure in-tended for remote care sessions. The networked interaction therapysystem is composed of a number of applications, and the meta-data attached to the application data is utilized for sharing the dataamong different applications.

Categories and Subject DescriptorsD.2.2 [Design Tools and Techniques]: User interfaces

General TermsDesign, Human Factors

KeywordsCognitive Prostheses, Assisted Communication, Networked Inter-action Therapy

1. INTRODUCTIONAs the number of elderly people among the population increases,caring for them, especially those who suffer from dementia and/or

memory-impairment is becoming an important problem. In addi-tion, a brain injury or disease sometimes results in cognitive impair-ment. Networked Interaction Therapy is intended to support care-givers and handicapped people (clients) through networked com-munication [5]. The communication, here, includes presenting propercontent to keep the clients’ attention focused. For example, a rem-iniscence video, which is a slide show video made from old photoalbums of the clients, is used to stimulate past memories. Its clin-ical effectiveness has been reported [14]. We created an authoringtool that utilizes a Semantic Web framework to support the creationof reminiscence videos [7], where annotations attached to the pho-tos are used to determine what kind of visual effect is to be addedto make the reminiscence video more attractive.

Since the reminiscence video involves personal content, such asthe client’s own photos from the past, it can be viewed from theaspect of personal information management. Similarly, networkedinteraction therapy involves other kinds of personal data, such asthe client’s schedule (for arranging care sessions), a history of caresessions (to prepare for better care sessions in the future). Thus,a unified approach for personal information management is quiteuseful in networked interaction therapy.

This paper explores the potential of the semantic desktop [11] asa basis for the networked interaction system. The next sectionbriefly introduces the networked interaction therapy framework andthe concept of the semantic desktop. The subsequent section de-scribes application scenarios that are related to the semantic desk-top scheme.

2. NETWORKED INTERACTION THERAPYBasically, a system for networked interaction therapy is intendedto keep the attention of people (clients) with dementia occupied.Figure1 illustrates an overview of networked interaction therapy.It is assumed that a client sits in front of a video monitor, which isused to present interesting content, such as a reminiscence video,as well as a video phone terminal. The client is connected to thenetwork, and companions (often volunteers) can talk with the clientover the network using a video phone so that the care session canbe conducted remotely. When a human companion is not available,a screen character may be used instead. Sensing devices, such as

1

VolunteersScreen

Characters

Friends

Client

Network

Alternative Interaction Companions

Figure 1: Overview of Networked Interaction Therapy

a camera, are also employed to monitor the status of the client. Inaddition to visual and audio stimulus (that is, video content), wemay use the olfactory or haptic sense to provide a more engagingstimulus.

When we consider the use of personal information management,the overall architecture of the networked interaction therapy sys-tem can be as shown in Figure2. The system is designed around apersonal data repository that uses the Resource Description Frame-work (RDF) [13], which is one of the basic semantic web technolo-gies for handling metadata. The personal data repository holds per-sonal data, such as old photos (for reminiscence videos) and a listof acquaintances. It also contains the client’s schedule to arrangefor care sessions. Additionally, the history of care sessions may bestored for the future analysis. These kinds of data are inherentlypersonal, and are used in a variety of applications that comprise thenetworked interaction therapy system.

On top of the personal data repository, various applications suchas an annotation tool, an authoring tool (for reminiscence videos),and a scheduler are employed. The application may include a videophone to communicate with companions (caregivers) over the net-work. The personal data repository allows the data to be sharedwith other personal data repositories. This is useful, for example,for arranging remote care sessions with potential companions.

This data centric view of the network interaction therapy systemrequires personal information management for both the client andthe caregivers. Since the metadata plays an important role, the se-mantic desktop [11] concept can be applied. The core idea of thesemantic desktop is to bring semantic web technologies to desktopcomputers [10]. Various kinds of applications concerning personalinformation management are integrated by making use of metadatadescribed using ontologies in the semantic desktop. Its personal-ized aspect, i.e., the fact that the personal model of informationhandling is respected, and its social aspect, i.e., the fact that infor-mation sharing among people is inherently facilitated though com-mon ontologies, make the semantic desktop well suited as a plat-form for networked interaction therapy. In the next section, severaluse case scenarios of networked interaction therapy are presented.

3. APPLICATION SCENARIOS3.1 Content Creation3.1.1 Reminiscence videos

Video and audio are inherently powerful and effective stimuli fornetworked interaction therapy clients. The reminiscence video,which is mentioned in the introduction section, is a typical example

of this. The reminiscence video is intended to help elderly clients toremember old times and enjoy peaceful moments. It is a slide showvideo made from old photos. We developed an authoring tool for areminiscence video, which utilizes annotations attached to photos(Figure3) [7].

The process of creating a reminiscence video is as follows. First,the photos to be used are annotated with information provided bythe client’s family members. We used the following vocabularyto describe annotations: Dublin Core, FOAF, and Image Regions.Dublin Core is used to describe properties of the photo itself, suchas the title and the date the photo was taken. FOAF is used todescribe people in the photo. Since a reminiscence video often in-volves people whom the client knows in the present and past, thisvocabulary plays a central role in the photo annotations. ImageRegions are used to describe certain regions in the photo. Thisis used to represent the portion of the photo that depicts a certainperson. The depicted person is then described by using FOAF. Inaddition, since the relationship between people in the photo and theclient (viewer) needs to be described in detail, we used the RELA-TIONSHIP vocabulary to describe the kinship relationship, suchas parent, son, and daughter. Figure4 shows an example of photoannotations using this vocabulary.

The video creator then selects the photos to be used in the reminis-cence video from the annotated photo archive. The Jena toolkit isused to store the annotations in RDF format, and RDQL is used toquery the photo archive for suitable photos. The photos are thensorted according to certain criteria, such as the date that the photowas taken.

In order to make a reminiscence video more effective, visual effectslike zooming and panning are added. The visual effects to be ap-plied are automatically selected based on the annotation attachedto the photo. The visual effect templates determine what kinds ofvisual effects should be added.

The reminiscence video is a typical example of tools that can be ofhelp to caregivers and clients. Other examples include the CIRCAproject, which utilizes multimedia content to aid conversations be-tween the caregivers and the clients [3]. In addition, the conceptof a video respite is proposed in dementia care and utilized in thecare setting [8]. The respite video is intended to give rest to thecaregivers (often family members) while the client is watching thevideo.

As these examples indicate, multimedia content can play an impor-tant role in dementia care. Since effective content is different from

2

authoring tool

annotation tool scheduler

videophone…

desktop user interface

client (handicapped) caregiver

communication

data sharing

RDF-based personal data repository

reminiscence videos

schedule data

care session records

photos acquaintances

…

Figure 2: Personal Data Repository for Networked Interaction Therapy

Dublin CoreFOAF

Image Regions

Figure 3: Annotation tool for creating reminiscence videos

person to person, personalized content should be employed. If per-sonal content could be compiled semi-automatically from a set ofdata in the personal data repository, it would be beneficial both tothe client and the caregivers. In the case of the reminiscence video,different videos can be created from the same set of photos takinginto consideration the viewer’s preferences, which may be recordedin his/her personal data repository.

3.1.2 Instructional contentFor patients with memory-impairment, it is often difficult to plantheir everyday activities. In order to cope with this problem, it isimportant to provide a cue to the client in a timely fashion. Ex-amples include navigational content that guides the client througha certain task, such as cooking. Another example is a navigationsystem for taking public transportation [4]. Since everyday lifeactivities are different from person to person and the content andtiming of the cues need to be provided to satisfy the client’s per-sonal needs, they are often better created with the help of familymembers who live with the client.

One of the possible ways to create such ‘content’ is to use a slideshow in a similar way to the reminiscence video described above.In this case, a photo will be taken at an important step of the tar-get task by a caregiver or volunteer, and stored in the personal datarepository. Annotations that describe the step that the photo depictsare then added. The navigational slide show video can be createdfrom these photos using an authoring tool similar to the reminis-cence video authoring tool by making use of the annotations at-tached to the photos.

3.2 Collaboration infrastructure3.2.1 Scheduling remote care sessions

In networked interaction therapy, a care session is conducted overthe network using a network communication tool, such as a videophone. In this case, scheduling a remote session is one of the essen-tial functions of the networked interaction therapy system. In orderto facilitate scheduling of the remote care session, the personal datarepository of the client can exchange the schedule information withthat of potential companions. The iCalendar format [2] can be used

3

female Haruka

(x1,y1)

(x2,y2)

h1

w1They are very

happy.

at Disney

x1,y1,w1,h1

x1,y1,x2,y2

20040716

sample.jpg

rel:grandchildOf

dc:datedc:title

foaf:Image

rdf:type

dc:description

imgReg :hasRegion

imgReg:hasRegion

imgReg:Rectangle

imgReg :regionDepict

imgReg:boundingBox

imgReg:coords

foaf:namefoaf:gender

foaf:Person

rdf:type

rdf:type

rdf:type rdf:type

Viewer

at Disney Animal Kingdom

20040716

::

:

::

Literal

Resource

Figure 4: Example of photo annotation

to represent such kind of data, and its RDF based representationhas also been proposed [1]. The RDF representation is suitable forthe personal data repository.

In addition, when the companion talks with the client over the net-work, the background information on the client, such as his/herinterests and hometown, might be helpful. If this kind of private in-formation could be stored in the personal data repository and sharedwith the companion, it would help the companion to conduct a caresession more smoothly.

3.2.2 Remote collaborative annotationThe remote care session described above can also be used for addingannotations to the photos. That is, the companion and the clientcan talk over the network using a video phone. While the com-panion asks questions about a photo to the client, the companioninputs photo annotations to the system from the answers given bythe client. This process assumes the functionality that the samephotos can be displayed in different locations in a synchronizedway. Overhead involved in the annotation process can be lessened,since the process of adding annotations is incorporated in the caresessions [6].

Presenting the content in a synchronized way remotely would beuseful not only for the photo annotation process, but for other typesof remote collaboration. This functionality would be essential whenthe semantic desktop is applied as an infrastructure for remote col-laboration.

3.3 Content presentation and analysis of thecare sessions

3.3.1 Playback environmentThe semantic desktop can possibly be used as an interface to pro-vide a slide show video to the client. Of course, the GUI needs to

be modified for the client, for example, by incorporating a touch-panel [3]. At the same time, since the degree of impairment isdifferent from client to client, how the GUI should be customizedis better described in a declarative way, and can be stored in thepersonal data repository.

3.3.2 Capturing the interaction and its analysisSince the system can potentially include a video camera to take avideo of the client, it is relatively easy to obtain the responses ofthe client to the content presented. When the response is recorded,what kind of content is being presented can also be recorded atthe same time. This information is added as an annotation to therecorded video and can be used later to analyze the response of theclient to the content. The resultant video is useful for evaluatingthe effectiveness of the content presented. For example, a caregivercan analyze whether the client is interested in the video [12]. Byincorporating an image understanding technique, the system mayautomatically mark the point at which the response of the clientchanges. These marks will help a human user (in this case, thecaregiver or medical staff) to look back at the client’s responsesand interpret the consequences [9]. The results of this analysis canalso be added to the video as an annotation.

Moreover, by combining the results of the viewer with the personalschedule of the client, it makes it easier to further analyze the ef-fects of the content, which may be dependent on the condition ofthe client. By gathering much evidence, it might be possible to ex-tract knowledge or know-how on the content to be presented to theclient using a data-mining technique.

4. CONCLUSION AND FUTURE WORKThis paper discussed the potential use of a semantic desktop con-cept in networked interaction therapy. Since personal data withmetadata plays a crucial role, the semantic desktop can be a basisfor the networked interaction therapy system.

4

There are a number of issues to be resolved in order to develop thesystem according to the application scenarios described here. Forexample, an access control mechanism needs to be implementedsince very personal data is handled in networked interaction ther-apy. In addition, a vocabulary (or ontology) needs to be developedto describe data, such as the responses of the client to the videocontent. The use of metadata is a key for connecting the variousapplications that comprise networked interaction therapy. Further-more, we need to provide two different kinds of user interface: onefor the client (the handicapped person) and the other for the care-giver. The same data might be represented differently accordingto the type of user, and different modes of interaction need to bedesigned.

Currently, the system presented here is being developed. Variouskinds of applications are possible in addition to those describedhere, which make use of the personal data repository. Through theapplication to the networked interaction therapy system, we plan tofurther explore the potential of the semantic desktop.

5. ACKNOWLEDGMENTSThe authoring tool of the reminiscence video was designed and im-plemented by Koji Saito. This research was supported by the Na-tional Institute of Information and Communications Technology.

6. REFERENCES[1] Rdf calendar workspace.http://www.w3.org/2002/12/cal/.

[2] Internet calendaring and scheduling core object specification(iCalendar).http://www.ietf.org/rfc/rfc2445.txt,1998.

[3] G. Gowans, J. Campbell, N. Alm, R. Dye, A. Astell, andM. Ellis. Designing a multimedia conversation aid forreminiscence therapy in dementia care environments. InCHI’04: CHI ’04 extended abstracts on human factors incomputing systems, pages 825–836, New York, NY, USA,2004. ACM Press.

[4] J. James F. Sullivan and G. Fischer. Mobile architectures andprototypes to assist persons with cognitive disabilities usingpublic transportation. In26th International Conference onTechnology and Rehabilitation, 2003.

[5] K. Kuwabara, N. Tetsutani, N. Kuwahara, S. Kawato, M. J.Lyons, A. Utsumi, Y. Yanagida, and K. Yasuda. Networkedinteraction therapy: Supporting people with mild dementiaand their family members with internet. In20th InternationalConference of Alzheimer’s Disease International Kyoto2004, Abstracts, page 377, 2004.

[6] N. Kuwahara, S. Abe, and K. Kuwabara. Networkedreminiscence content authoring and delivery for elderlypeople with dementia. InWorkshop on Cognitive Prosthesesand Assisted Communication (this proceedings). 2006.

[7] N. Kuwahara, K. Kuwabara, N. Tetsutani, and K. Yasuda.Using photo annotations to produce a reminiscence video fordementia patients. In3rd International Semantic WebConference (ISWC2004) Demo Papers, 2004.

[8] D. Lund, R. Hill, M. Caserta, and S. Wright. Video respite:an innovative resource for family, professional caregivers,and persons with dementia.The Gerontologist,35(5):683–687, 1995.

[9] M. J. Lyons, M. Funk, and K. Kuwabara. Segment andbrowse: A strategy for supporting human monitoring offacial expression behaviour. In M. F. Costabile andF. Paterno, editors,Interact 2005: Tenth IFIP IC13International Conference on Human-Computer Interaction(LNCS 3585), pages 1120–1123, 2005.

[10] L. Sauermann, A. Bernardi, and A. Dengel. Overview andoutlook on the semantic desktop. In S. Decker, J. Park,D. Quan, and L. Sauermann, editors,Proceedings of the 1stWorkshop on The Semantic Desktop. 4th InternationalSemantic Web Conference (Galway, Ireland), 2005.

[11] Semantic desktop.http://www.semanticdesktop.org/.

[12] A. Utsumi, S. Kawato, K. Susami, N. Kuwahara, andK. Kuwabara. Face-orientation detection and monitoring fornetworked interaction therapy. InProceedings of Joint 2ndInternational Conference on Soft Computing and IntelligentSystems and 5th International Symposium on AdvancedIntelligent Systems, 2004.

[13] W3C. Resource description framework.http://www.w3.org/RDF/.

[14] K. Yasuda, A. Iwamoto, T. Nakamura, and K. Kuwabara.Reminiscence video for higher brain dysfunctions. InProceedings of General Conference of Japan Society forHigher Brain Dysfunctions, page 109, 2004.

5

http://www.w3.org/2002/12/cal/

http://www.ietf.org/rfc/rfc2445.txt

http://www.semanticdesktop.org/

http://www.w3.org/RDF/

Towards a Wearable Cognitive Prosthesis toSupport “What” and “Who” Type Memory Activities

Tatsuyuki KawamuraNara Institute of Science and

Technology8916-5 Takayama, Ikoma,

Nara 630-0192, JAPAN

[email protected]

Yasuyuki KonoNara Institute of Science and



[email protected]

Masatsugu KidodeNara Institute of Science and



[email protected]

ABSTRACTIn this paper, we introduce a wearable cognitive prosthesis1

we have implemented previously [6, 5, 9, 12, 7, 8]. The tech-nology for recording and sharing experiences such as thoseused in augmented memory has been extensively studied inrecent years [1]. The aim of these studies is to enable usersto receive the following benefits; 1) less time wasted in ev-eryday life, 2) a recycling of experiences, and 3) a facilitationof human-to-human interactions. Previously designed sys-tems, including ours were designed for people with ordinarycognitive processing abilities. These systems are not suit-able for people with dementia because the memory processof people with dementia is different from the memory pro-cess of a person with ordinary cognitive processing. In thispaper, we discuss the design of a system for people with de-mentia. Our existing systems must be partially or totallyredesigned in order to realize a cognitive prosthesis that canperform for the sufferers of dementia.

1. INTRODUCTIONWearable computing technology has been investigated ex-tensively in recent years resulting in smaller and smallercomputers with larger and larger storage capacities. An ad-vantage of a wearable system is that a user can get variousservices “anytime” and “anywhere”. This advantage is goodfor memory aid because human memory activities take placeanytime and anywhere. Interesting research in recording auser’s experiences has been discussed in such examples aslife-log [3]. J. Gemmell et al. have attempted trying todigitize their lives as much as possible in the MyLifeBitsproject [4]. The main point of view of the life-log project isthe method of recording and analyzing experiences.

In addition to people with ordinary cognitive processing,

1We usually call our memory aid system “augmented mem-ory”. In this study, we term this kind of system “cognitiveprostheses”.

people with dementia and people with amnesia are also stud-ied as target users in memory aid research [2]. Unfortu-nately, however, sufferers of dementia or amnesia have notbeen studied in term of computer-aided memory prosthe-ses as much as ordinary, unremarkable users. At least tworeasons exist for this: One is the difficulty of design and eval-uation. Generally, almost all system developers, researchers,and their colleagues are cognitively ordinary people. There-fore, designing a system useful for the sufferer2 is difficult.Moreover an evaluation of the designed system is even moredifficult than designing the system itself because few oppor-tunities for experiments using test subjects with dementiaor amnesia under the controlled conditions. Another rea-son for less design and evaluation for sufferers of dementiais the gap in market size. The population of people withordinary cognitive processing is larger than the populationof people with dementia or amnesia. Market size obviouslydetermines what research will be done and what will not.

Facilitating “ordinary people” with “people with demen-tia/amnesia” interaction is necessary to both group livingtogether because both must interact in many situations to-gether daily. One of possibility of making this interactionsmoother and more comfortable is a universal design of acognitive prosthesis from ordinary people through peoplewith dementia or amnesia. The universal design would ex-pand market size. In order to achieve universal cognitiveprostheses, researchers and developers must employ an or-dinary person and a person with dementia or amnesia asusers to design services and interfaces.

In this paper, we first introduce designs of wearable cogni-tive prostheses for people with ordinary cognitive processingand then discuss these designs and how they may be usedfor sufferers of dementia or amnesia. We have designed andimplemented four types of systems. The systems are respec-tively called Residual Memory [6], Ubiquitous Memories [5],I’m Here! [9, 12], and Nice2CU [7, 8]. We discuss especiallythe I’m Here! system and the Nice2CU system. The I’mHere! system supports an object finding task (a ‘what-type’memory activity). The Nice2CU system enables a user toremember a person’s name, birthday, hobby, etc. The sys-tem supports a recollection of an event that is triggered bya person who is in front of the user (a ‘who-type’ mem-ory activity). Both memory activities are important for not

2Note that we term simply a person with dementia or am-nesia “a sufferer” in this paper.

6

(i) (ii)

(iii) (iv)

Figure 1: Registration mode of the I’m Here!.

only ordinary people but also for people with dementia oramnesia.

2. OVERVIEW OF OUR SYSTEMS2.1 I’m Here!This system retrieves the last recorded video of a user’s view-point including a handled object from a video database. TheI’m Here! system is composed of a wearable camera [11],which is specially provided for detecting an object handledby the user, a sleeve-on display, and a sleeve-on switch. TheI’m Here! system includes three operation modes as follows:

Registration: A user first selects this mode (Figure 1 (i)and (ii)) (STEP-I). The user then picks up an objectthat the user wants to register (STEP-II). The usermoves and rotates the object in front of the wearablecamera (Figure 1 (iii)) (STEP-III). The system startscapturing appearance data of the object as images(Figure 1 (iv)) (STEP-IV). The system then extractsfeatures for recognizing the registered object (STEP-V). Finally, the system records both the feature dataand a thumbnail including the object (STEP-VI).

Observation: The system has been recording a video whilea registered object has been in the view of the wear-able camera (STEP-I). The system then updates thedatabase just after the object disappears from the cap-tured image (STEP-II).

Retrieval: The user selects this mode (STEP-I). The userchecks thumbnails (Figure 2 (i) and (ii)) (STEP-II).The user selects one thumbnail including the objectthat the user wants to find (STEP-III). The systemreplays the last recorded video including the handledobject (STEP-IV). Finally, the user finds the object(Figure 2 (iii)) (STEP-V).

2.2 Nice2CUThis system retrieves the last recorded video of a user’s view-point including the person who is in front of the user. TheNice2CU system is composed of a wearable camera for de-tecting the person in front of the user, and a wrist-on RFID

(i) (ii)

(iii)

Figure 2: Retrieval mode of the I’m Here!.

Figure 3: Observation and retrieval mode of theNice2CU

(Radio Frequency ID) tag reader. The Nice2CU system in-cludes the following two operation modes:

Registration: A person comes to meet a user. The persongives the user a business card attached to an RFID tag(STEP-I). The user reads the information recorded inthe RFID tag by using the wrist-on RFID tag reader(STEP-II). The system registers the person’s informa-tion data (name, birthday, hobby, and etc.) and fea-ture data that are for recognizing the person using thewearable camera (STEP-III).

Observation and Retrieval: The user meets the personagain. The system recognizes the person (STEP-I).The system has been recording a video while the regis-tered person has been in the view of the wearable cam-era (STEP-II). The system then updates the databasejust after the person disappears from the captured im-age (STEP-III). The system displays the person’s in-formation data, and replays the video recorded when

7

the user last met the person (Figure 3) (STEP-IV).

3. DISCUSSION OF SYSTEM DESIGNS3.1 Distinguishing an Operator from a UserA system design model distinguishing an operator from auser is important in achieving a wearable cognitive pros-thesis for sufferers. Both the I’m Here! system and theNice2CU system have registration, observation, and retrievalmode. It is especially hard for the sufferer to understandhow he or she can operate the system in the registrationmode and the retrieval mode. A caregiver including thefamily must control the system as an “operator” in thesetwo modes. The subject person in this case is the suffererwho records his or her own experiences. The sufferer thenimplicitly requires service as a user.

Figure 4: A system design model distinguishing anoperator from a user

Figure 4 illustrates the system design model which distin-guishes an operator from a user. An person with ordinarycognitive processing behaves as a user in conjunction withan operator. The system supports the user directly. On theother hand, a sufferer is only a user. The sufferer’s systemestablishes communication with a caregiver’s system. Thecaregiver’s system supports the caregiver to be able to carefor the sufferer by using the information from the sufferer’ssystem. A system developer can design both systems witha system configuration. Performing rules for input and out-put depending on the user is important if the core design isuniversalized.

3.2 Arranging Articles for Everyday UseIn terms of a wearable system, methods exist for recogni-tion of objects: 1) an appearance-based object recognitionmethod and 2) a tag-based object recognition method. Inthe case of appearance-based object recognition, a wearablesystem extracts an object image from an image captured byusing a wearable camera. On the other hand, a wearablesystem can employ a visual tag or an RFID tag to detectan object. In this paper, we select an RFID tag system forrecognizing an object.

A person with ordinary cognitive processing arranges phys-ical objects by him or her self. Objects can be classified bythe frequency type of their use. High-use objects are articlesfor everyday use. Low-use objects represent objects storedin a warehouse, cupboard, etc. for a long time. The totalnumber of registered objects should be over 100 even thougha user limits objects to those he or she wants to find quickly.In the case of a sufferer, the sufferer would find it difficult

to complete an object finding task. A person doing the ob-ject finding task instead of the sufferer would be a caregiverincluding the family. The number and variation of objectsthat must be managed are fewer than in the case of peoplewith ordinary cognitive processing. Objects, which shouldmanage carefully, are valuables for the sufferer. These ob-jects are sometimes “troubled objects” between the suffererand the caregiver. The priority registration of troubled ob-jects would lift the burden of finding objects that the suffererwants from the caregiver. We assume that the number oftroubled objects for everyday life is 10 at most.

In the case of the appearance-based object recognition method,a user uses the system to capture many appearances of anobject so that there are many appearances when the objectposture is changed. Also, the user must associate the nameand property of the object with the features calculated fromthe set of the appearances of the object. In comparison withthe appearance-based method, the tag-based object recog-nition method allows the user to register only the nameand property of the object with an RFID tag. Most ob-jects would be represented by attached or implanted RFIDtags, respectively only if a tag-based ubiquitous society waswidely extended from what it is today. Also, the tag wouldinclude the name and property of the object with the tag inadvance. However, a user with the appearance-based objectrecognition method never omits an operation for an associa-tion of the name and property of the object and the featurecomputed from the set of the appearances of the object.

The tag-based object recognition method is more suited toperforming as a cognitive prosthesis for sufferers than theappearance-based object recognition method. In the caseof the tag-based method, there are some objects to whichRFID tags cannot be attached. Nonetheless, RFID tagscan be attached to all the sufferer’s objects if a caregiverattentively selects objects to buy for the sufferer.

3.3 Controlling Inter-Personal CommunicationA sufferer might feel a higher calmness of mind by interact-ing with ordinary people. The person with ordinary pro-cessing emphasizes exchanging correct information to keepcommunication channels open with other people. The suf-ferer, in contrast, finds it difficult to establish communica-tion with an another person, such as an acquaintance. Inorder to establish a connection, both the sufferer and theacquaintance must be aggressive. Facilitation of a commu-nication channel and keeping it open is an important issuein allowing the person with ordinary processing to performaction for the connection.

The Nice2CU system we have designed for ordinary peopleactivates thier communication channels by allowing the ex-change of information between communicators. Four typesof information about a person are essential to this system:

A PROFILE includes current information about the per-son (e.g. person’s face, name, sex, birthday, bloodtype, birthplace, current address and affiliation.)

An EXPERIENCE contains an event memorized by acertain person, and includes the context of what hap-

8

pens in an event that was experienced together by arecipient and a target person.

A MESSAGE includes a temporary memorandum similarto a “post-it” for delivering a message to the targetperson. For instance, the recipient suddenly remem-bers that he or she wants to repay money to the targetperson when the recipient accidentally runs into thistarget person.

HUMAN RELATIONS consists of the above elements:PROFILES, EXPERIENCES, and MESSAGE. For ex-ample, the target person was the recipient’s formerboss when he was in college.

The system selects and displays appropriate information forfacilitating communication by calculating the latest relation-ship among the user, the target person, and other people.The ordinary person can interact with others by consider-ing displayed information correctly. On the other hand, thesufferer finds it difficult to establish communication withothers by using only the displayed information. Knowinghow to get information from a sufferer is necessary for keep-ing communication channels open with the sufferer. To getsuch information, the system must give appropriate infor-mation to activate communication to “a sufferer”, “an ac-quaintance”, and “a caregiver” respectively. If a sufferer asa user meets a person he or she has met before, the sufferercannot establish communication with the person because thesufferer will find it difficult to recognize who the person is.The system must give the latest past-information in all in-formation that the sufferer can remember, e.g. events inthe childhood. For instance, a system, which can recog-nize when they have met, similar to the Nice2CU system,displays photos or videos recorded in the childhood. Also,the system must let the acquaintance understand that thepast-information is useful in making conversation with thesufferer. The recorded video of when the sufferer met theacquaintance is also available for the caregiver. The care-giver can talk about the event with the acquaintance of thesufferer. Additionally, recorded video would be more usefulonce the caregiver edits the video and annotate semanticdata on the video [10].

4. CONCLUSIONSIn this paper, we have introduced wearable cognitive pros-theses that we have implemented in the past. Especially,we explain the I’m Here! system for supporting an object-finding task and the Nice2CU system for remembering anacquaintance’s information. We also discussed the designsof these systems in regard to people with disease associ-ated with dementia or amnesia so as to improve or redesignthese systems for people with dementia or amnesia. We haveproposed a system design model distinguishing an operatorfrom a user. It is important for developers to achieve acognitive prostheses for both people with ordinary cognitivefaculties and people with dementia or amnesia.

5. ACKNOWLEDGMENTSThis research is supported by Core Research for EvolutionalScience and Technology (CREST) Program “Advanced Me-dia Technology for Everyday Living” of Japan Science andTechnology Agency (JST).

6. REFERENCES[1] http://www.ii.ist.i.kyoto-u.ac.jp/ sumi/pervasive04/.

[2]http://www.media.mit.edu/wearables/mithril/memory-glasses.html.

[3] K. Aizawa, T. Hori, S. Kawasaki, and T. Ishikawa.Capture and efficient retrieval of life log. In Pervasive2004 Workshop on Memory and Sharing Experiences,pages 15–20, 2004.

[4] J. Gemmell, R. Lueder, and G. Bell. Living with alifetime store. In ATR Workshop on UbiquitousExperience Media, pages 69–76, 2002.

[5] T. Kawamura, T. Fukuhara, H. Takeda, Y. Kono, andM. Kidode. Ubiquitous memories: Wearable interfacefor computational augmentation of human memorybased on real world object. In 4th InternationalConference on Cognitive Science (ICCS2003), pages273–278, 2003.

[6] T. Kawamura, Y. Kono, and M. Kidode. A novelvideo retrieval method to support a user’s recollectionof past events for wearable information playing. In 2ndIEEE Pacific-Rim Conference on Multimedia(PCM2001), pages 24–31, 2001.

[7] T. Kawamura, Y. Kono, and M. Kidode. Nice2cu:Managing a person’s augmented memory. In 7th IEEEInternational Symposium on Wearable Computers(ISWC2003), pages 242–243, 2003.

[8] T. Kawamura, C. C. Lin, Y. Kono, and M. Kidode.Rating human relations for recommendation of anaugmented video memory using human profile dataand meeting logs. Technical ReportNAIST-IS-TR2003017, 2003.

[9] T. Kawamura, Y. Miyake, T. Shibata, K. Suzuki,K. Nishino, N. Saiwaki, Y. Kono, and M. Kidode. Acasual wear with an object-finding support system. InExtended Abstract of the 9th IEEE InternationalSymposium on Wearable Computers (ISWC2005),2005.

[10] N. Kuwahara, K. Kuwabara, N. Tetsutani, andK. Yasuda. Using photo annotations to produce areminiscence video for dementia patients. In 3rdInternational Semantic Web Conference (ISWC2004),2004.

[11] T. Ueoka, T. Kawamura, S. Baba, S. Yoshimura,Y. Kono, and M. Kidode. Wearable camera device forsupporting object-triggered memory augmentation. In3rd CREST/ISWC Workshop on Advanced Computingand Communicating Techniques for WearableInformation Playing, pages 46–53, 2004.

[12] T. Ueoka, T. Kawamura, Y. Kono, and M. Kidode.I’m here!: a wearable object remembrance supportsystem. In 5th International Symposium on HumanComputer Interaction with Mobile Devices andServices (MobileHCI2003), pages 422–427, 2003.

9

Vision-based Behavior Detection for Monitoring andAssisting Memory-Impaired People

Akira UtsumiATR Intelligent Robotics andCommunication Laboratories

2-2-2 Hikaridai SeikachoSorakugun

Kyoto 619-0288, Japan

[email protected]

Daisuke KanbaraRitsumeikan University

1-1-1 Nojihigashi, Kusatsu CityShiga 525-8577, Japan

[email protected]

Shinjiro KawatoATR Intelligent Robotics andCommunication Laboratories



[email protected]

Shinji AbeATR Intelligent Robotics andCommunication Laboratories



[email protected]

Hironori YamauchiRitsumeikan University

1-1-1 Nojihigashi, Kusatsu CityShiga 525-8577, Japan

[email protected]

ABSTRACTIn this paper, we describe our methods to detect human be-havior for monitoring and assisting memory-impaired peo-ple. To assist human activities in daily life, we must be ableto understand a user’s situation/state in the current task:what information is useful for the user now. Our posture-detection system using IR cameras and IR pattern projec-tors detects the user’s state as the 3-D appearance (posture)of him or her. In the system, human behavior is modeledas a distribution of 3-D appearances, and the results of thisbehavior detection are used to determine instructions to begiven to the user. We show an sample implementation ofthe system for a toilet task. In addition, we also address amethod to display video content to people with dementia. Inthe method, user behavior, including facial movements andbody motions, is detected with vision-based methods. Userattention to the displayed content is then estimated based onthe facial orientation and body motions. The estimated at-tention level is used for content control to attract and main-tain the attention of the viewer to the display system. Ex-perimental results suggest that the content-switching mech-anism does indeed effectively attract user interest.

Categories and Subject DescriptorsH.5.2 [INFORMATION INTERFACES AND PRE-

SENTATION (e.g., HCI)]: User InterfacesInteraction styles

General TermsVision-based HCI

KeywordsBehavior Detection, Action Assistance, Content Switching,Computer Vision

1. INTRODUCTIONFor a computer system to efficiently support human ac-

tivities, it has to recognize users’ behaviors and understandtheir intentions. Therefore, the ability to recognize humanbehavior by using sensors embedded in living environmentsis becoming an increasingly important research endeavor.

The same situation prevails in our ”networked interactiontherapy system,” which entertains and effectively attractsthe attention of memory-impaired people and lightens theburden of helpers or family members [1]. The main goal ofnetworked interaction therapy is to support the daily activ-ities of memory-impaired people by enhancing their qualityof life and reducing the burden on their family. For this pur-pose, we give networked interaction therapy the capabilityto automatically detect the intentions of a memory-impairedperson. This provides the necessary information/entertainmentfor guiding the individual to a more comfortable conditionon behalf of his or her family before the occurrence of suchbehavioral problems as wandering at night, incontinence, fitsof temper, and so on.

Various sensory devices can be used to detect user be-haviors. Vision-based detection of human behavior fits ourneeds since it does not require any special attachments [2,3, 4, 5]. The system can remotely detect human motions,and users do not need to be aware of the sensory systems.

10

In this paper, we describe a method to detect human be-havior for monitoring and assisting memory-impaired peo-ple. To assist human activities, we have to accurately knowthe user’s situation/state, and to do this we use an IR-basedrange finder for estimation. In the system, human behavioris modeled as a distribution of 3-D appearances, and theresults of this behavior detection are used to determine in-structions to be given to the user. In addition, we estimateuser attention based on facial orientation and body motionsrelated to visual/aural stimuli in displayed video/audio con-tents detected with a vision-based system. In the proposedmethod, if user attention strays from the target medium, thesystem changes the contents of the displayed information orprompts the user to lure him or her back.

In the next section, we describe our action assistancemethod using 3-D posture observation. Section 3 outlinesthe framework of our attention estimation, and in Section4, we give our conclusions.

2. ACTION ASSISTANCE USING HUMANPOSTURE OBSERVATIONS

In the daily activities of memory-impaired people, it be-comes a serious problem that caregivers must always watchand assist each action made such as using the toilet, shower,etc. Some of these actions are routine tasks that are some-times difficult and often boring for caregivers due to theirrepetitive nature.

Figure 1: IR range finder

To achieve action assistance, we have to first confirm thestate of the user (i.e., a memory-impaired person). Here, weconsider a scenario where the system gives voice instructionsto a user for him/her to achieve a typical daily task (usingthe toilet, changing clothes, etc. (Figure 2)). In these situa-tions, normal vision systems are not suitable due to privacyconsiderations. In addition, a method that is robust againstchanges in clothing color/illumination conditions is required.Therefore, we have employed an IR-pattern-based systemto detect the 3-D geometries of multiple sample points. Byusing the system, we can robustly detect 3-D positions ofmultiple points on the user’s body without detecting tex-ture (normal camera image) information. This capability issuitable for our purpose.

Figure 1 shows an overview of the implemented system.Here, we used DLP projectors (resolution 800 × 600 pixels)to project IR patterns. CCD cameras are used for imageobservations. Both projectors and cameras have IR (infra-red) filters on their lens units.

The 3-D measurement method using projected patterns ispopularly used for detecting 3-D object shapes in computervision [6, 7]. Here, we use multiple IR spot patterns to de-

Figure 2: Behavior Detection by IR Pattern Projec-tion

tect 3-D positions on the target object. To avoid interferenceamong projected spot patterns in the camera observations,we calculate the geometrical relations between every twospot patterns and then form groups of spot patterns so thata separate spot observation is guaranteed within each group.By projecting the group of spot patterns, we can avoid con-fusion among multiple spot patterns and thus simplify thespot detection process.

2.1 Behavior Detection using IR Detection sys-tem

In our method, we can model the typical behavior of usersand detect the current state by using matching based on the3-D model. Regarding behavior detection, our system de-tects 3-D shape patterns of the scene around the detectedhuman location, and compares them with the pre-storeddata (Fig. 3).

Camera

Projector

IR pattern

Observe

BehaviorDatabase

Recognition

Camera

Projector

IR pattern

Observe

IR pattern

Observe

BehaviorDatabaseBehaviorDatabase

RecognitionRecognition

Figure 3: Behavior Detection

Here, let Pi be the i-th 3-D point in the model P and Qj

be the j-th point of the current 3-D patterns.

Pi =ˆxpi ypi zpi

˜′, (1)

Qj =ˆxqj yqj zqj

˜′. (2)

Accordingly, the distance between the two sets of 3-Dpoints P and Q can be defined as follows.

Distance(P, Q) =NX

i=1

minj∈{1,...,M}

(Pi −Qj)′(Pi −Qj)/N. (3)

After comparison of the input 3-D shape with all storedmodels, the model that best fits the current input is selectedas the current user’s state.

2.2 Example: Toilet Task Instruction usingBehavior Detection

We applied the above system to a real scene: a toilet.As we mentioned, there is difficulty with installing image

11

Table 1: Measurement Errors (cm)X Y Z

Ave. 6.37 -1.42 3.89S.D. 3.56 0.99 2.42

sensors to private spaces like toilets due to invasions of pri-vacy. Our method, however, only observes 3-D positions ofdiscrete spot patterns and fits them to the situation. Fig-ure 4 shows the configuration of the implemented system.Here, an IR range finder is placed above the toilet and asmall LCD display is placed on the side wall to display aCG character that gives voice instructions to users (Figure5).

We designed the system to instruct on toilet usage to usersin a stepwise manner. We divided the toilet task into sixsmall subtasks (i.e., “open the top cover,” “turn around,”“take pants down and sit down,” “flush the toilet,” “standup and pull pants up,” and “close the top cover”). When auser has completed one subtask, the system then providesinstructions for the next subtask. If the user does not un-dertake a subtask within a certain time, system providesthe same instruction again. If a user takes a different actionand deviates from the task scenario, the system will alert thecaregiver(s). The behavior in each subtask is modeled as aseries of 3-D posture transitions (the number of 3-D posturesfor one subtask is one to three here). In the next section,we confirm the system’s feasibility with a larger number ofbehavior sets.

2.3 ExperimentsTo confirm the effectiveness of the proposed system, we

conducted the following experiments.First, we measured an arbitrary 3-D plane with our sys-

tem and evaluated the system’s accuracy. Table 1 shows theresults of the evaluation. It contains the average and stan-dard deviations of errors for each axis, revealing that oursystem can measure 3-D locations with errors of less than10 cm.

Next, we reconstructed the 3-D shapes of real objects,with Fig. 6 showing an example of 3-D reconstruction. Here,one absence scene and four typical human postures in thetoilet task are measured by our system. Fig. 6 bottom showsthe result of 3-D reconstruction. As can be seen, the 3-Dshape of a person is accurately reconstructed.

We also conducted experiments on behavior recognition

IR Range Finder

CG Character Display

IR Range Finder

CG Character Display

Figure 4: System Configuration (Toilet)

Figure 5: CG Character Display

Absence Stand Open Cover Rotate SitAbsenceAbsence StandStand Open CoverOpen Cover RotateRotate SitSit

Figure 6: 3-D shape reconstruction (top: targetscene, bottom: reconstruction results)

with a larger number of categories. Figure 7 shows represen-tative appearances for eight different pose categories: ”standfrontal (STF),” ”stand leftward (STL),” ”stand rightward(STR),” ‘sit frontal (SIF),” ”sit leftward (SIL),” ”sit right-ward (SIR),” ”dressing (DR), and ”reaching action (RE).”

STF STL STR SIF

SIL SIR DR RE

STF STL STR SIF

SIL SIR DR RE

Figure 7: Eight Categories

Table 2 shows the recognition results for 48 samples.Finally, used our system to continuously observe a per-

son’s behavior. Figure 8 shows the results, where the hor-izontal axis denotes the time and the vertical axis corre-sponds to the category selected for each frame. As can beseen, our system properly recognizes the user’s behaviors.

3. CONTENT SWITCHING FOR ATTRACT-ING AND MAINTAINING THE ATTEN-TION OF VIDEO WATCHERS

In this section, we address our content-switching methodfor attracting and maintaining the attention of video-watchingpeople. Further details, including a pilot observation on ac-tual behavior of memory-impaired patients watching videos,is described in [8]. Since the human vision system plays animportant perceptual and communicative role in our dailylife, gaze can be a strong cue for estimating user attention.In our attention monitoring, we first focus on the orientationof the face. Strictly speaking, facial orientation is differentfrom human gaze due to the lack of eye information. How-

12

Table 2: Recognition Rate (%)STF STL STR SIF SIL SIR DR RE

STF 50.0 0.0 0.0 0.0 0.0 0.0 50.0 0.0STL 0.0 83.4 16.6 0.0 0.0 0.0 0.0 0.0STR 0.0 16.6 83.4 0.0 0.0 0.0 0.0 0.0SIF 0.0 0.0 0.0 100.0 0.0 0.0 0.0 0.0SIL 0.0 0.0 0.0 0.0 100.0 0.0 0.0 0.0SIR 0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0DR 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0RE 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0

Absence

Open Cover

Stand

Rotate

Sit

TIME

STATE

Actual Motion Recognition Result

Absence

Open Cover

Stand

Rotate

Sit

TIME

STATE

Actual Motion Recognition Result

Figure 8: Recognized Sequential Behaviors

ever, in most cases a loss of visual attention is accompaniedby head movement. Therefore, we still consider facial orien-tation information useful for attention estimation.

Another means of estimating human attention is body mo-tions, especially synchronous motions in visual/aural signalsthat appear in displayed content, which should represent theuser’s level of absorption in the content. In this paper, wedeal with the extraction of synchronous beat actions frommusic programs.

Detected attention can be used for controlling audio/videocontents displayed to users. For example, such positive syn-chronous reactions as looking at the TV and/or keeping timewith one’s body can be used as the basis to make a systemthat shows the contents for longer. In contrast, such neg-ative behavior as a user turning his/her head away fromthe TV can be used as a trigger for the system to switchcontents.

3.1 Vision-Based Behavior Extraction

Face Orientation TrackingVision-based face tracking is a popular research field thatmany researchers have investigated [9, 10, 11]. However, few

-1

-0.5

0

0.5

1

0 50 100 150 200[frame]

-1

-0.5

0

0.5

1

0 50 100 150 200[frame]

-1

-0.5

0

0.5

1

0 50 100 150 200[frame]

Figure 9: Example of facial orientation estimation

systems have achieved both robustness and high-speed pro-cessing. In an effort to solve this problem, we have developeda real-time and stable face tracking system via SVM-basedface detection coupled with prefiltering with SSR filters.

Figure 9 (left) shows an example of face tracking. Graph-ics are overlayed on the facial image, indicating detectedeyes and the tip of the nose. The detected facial orientationis indicated by the bold line in the narrow region along thetop of the image. Figure 9 (right) shows the results of faceorientation estimation (horizontal). Here ±1 in the verticalaxis denotes the limit of detection, and for this subject itcorresponds to about ±45 degrees.

Detecting Body MotionTo detect body motion, we currently employ an inter-framesubtraction scheme. Here, the magnitude of motion is mea-sured as the number of moving pixels N .

Nt =X

i

si,t, (4)

si,t =

j0 ((ci,t − ci,t−1)

′(ci,t − ci,t−1) < threshold)1 (otherwise)

(5)Here, ci,t denotes a color vector of the i-th pixel in an ob-served image at time t.

This method is simple but still valid for extracting tempo-ral patterns of motion magnitude. For instance, a ”keepingthe beat” action can be detected as ”frozen” points wherethe motion suddenly stops in a motion-magnitude sequence.

beat =

j1 (

dNt−1

dt< 0 and dNt

dt� 0)

0 (otherwise).(6)

Figure 10 shows an example of an extracted beat action.Here, the person was expressing the beat by hand.

3.2 Contents ControlOur system displays video contents to users whose be-

havior (facial orientation and body motions) is observedby cameras and other sensory systems, with displayed con-tent characteristics simultaneously extracted by the system.Currently, only the beat structures of audio signals are ex-tracted in this process. After beat structure extraction, thetemporal correlation between the beat structure and usermotion is evaluated. User attention to the displayed con-tents is then estimated from the correlation and facial ori-entations.

13

0 100 200 300 400 500[frame]

Figure 10: Example: Extracted Beat Actions

If the system recognizes a loss of attention to the presentedmedium, it switches contents to regain the user’s attention.

3.3 ExperimentsWe conducted the following experiments to confirm the

feasibility of our proposed method. All experiments de-scribed in this section were performed using healthy adults.

First, we applied our facial orientation detection to asequence of head rotation images. Figure 11 denotes theresults of attention estimation. Here, the subject mainlywatched our TV system and sometimes looked down at amagazine on his lap. Figure 12 shows typical sample imagesof the two states: ”attention on (looking at the TV set)” and”attention off (looking in other directions).” The results in-dicate that in this case, our system accurately detected theloss of attention away from the TV system.

Next we performed detection of synchronous beat behav-ior related to displayed musical signals. Beat structures inmusical signals are extracted based on frequency analysis[12, 13]. In this work, we also formalized beat-structure de-tection based on a set of signal power banks consisting ofseveral different frequencies. Input signals are first fed intoa frequency analyzer (FFT) and separated into discrete filterbanks. In our current implementation, guard frequencies are0-250 Hz, 250-500 Hz, 500-1 kHz, 1-2 kHz, and 2-4kHz. Foreach bank, the power transition is examined, an envelopesignal is produced, and the beat structures are extractedas a rising point using differential operations. Figure 13shows both the detected beat from the displayed musicalsignal and the human beat motions. Here, several popularsongs were used for the experiment. At the bottom of Fig.13, we find the extracted synchronous motion (beat action).Furthermore, Fig. 14 shows the result of synchronous/non-synchronous behavior detection. Here on the left side, a

OFF

ON

30 35 40 45 50 55 60 65 70 75

Est

imat

ed A

ttent

ion

time [sec]

Figure 11: Results of Attention Estimation

Figure 12: Two states: ”attention on (looking at TVset)” and ”attention off (looking at other objects)”

1280 1300 1320 1340 1360 1380[frame]

0-250Hz

250-500Hz

500Hz-1KHz

1-2KHz

2-4KHz

Body Motion

Sound Signal

ExtractedSynchronous Beat

1280 1300 1320 1340 1360 1380[frame]

1280 1300 1320 1340 1360 1380[frame]

0-250Hz

250-500Hz

500Hz-1KHz

1-2KHz

2-4KHz

Body Motion

Sound Signal

ExtractedSynchronous Beat

Figure 13: Synchronous ”beat” behavior extraction

subject is listening to music and is instructed to keep thebeat. On the right side, he moves his hand randomly, indi-cating that our system can clearly distinguish between thetwo states.

Finally, we examined the effect of content control basedon attention monitoring (here, only face orientation is used.)We placed two TV sets in front of a subject. One employedthe content-switching mechanism based on facial orienta-tion, whereas the other had no attention monitoring; videocontents were just switched at regular intervals in a prede-fined order. Table 3 and Fig. 15 compare the total viewingtime of the two TV sets for six subjects. As we can see,subjects watched the controlled TV significantly longer, sug-gesting that content switching can effectively attract userinterest.

4. CONCLUSIONSWe have described methods to track human behavior us-

30

35

40

45

50

55

60

65

70

75

randombeat action

Num

ber o

f Syn

c. B

eats

Per

Min

ute

Figure 14: Distinguishing Beat Action and Non-Beat Action

14

Table 3: Watching time and number of missed detectionsSubject a b c d e f Avg.

Watching Time (sec) with control 528 526 439 422 444 294 442*without control 289 314 391 288 336 426 340*

detection error false-negative 0 3 6 0 3 1 2.2false-positive 2 0 0 4 0 0 1

*p=0.06

250

300

350

400

450

500

550

W/O ControlWith Control

Tota

l Vie

win

g Ti

me

[sec

]

Figure 15: TV Watching Time with/without Con-tent Control

ing vision-based system to monitor and assist human ac-tivities in daily life. In the action assistance, human be-havior is modeled as the change of 3-D appearances. wedetect the 3-D appearance by using IR pattern projections,it achieves widespread use and impose low visual impact onusers, taking into account the human need for privacy. Wealso described a method to estimate human attention basedon face orientation and body actions, employing an SVMto model human faces, and determined facial orientation byusing the geometrical relation between the eyes and the tipof the nose. Body motions (”beat actions”) were also ex-tracted and examined in terms of synchronization with thebeat elements of displayed musical signals. In experiments,we controlled the timing of content switching to attract userattention to the presented video contents. Experimental re-sults demonstrated the effectiveness of human tracking andbehavior monitoring by the proposed methods.

5. ACKNOWLEDGMENTSThis research was supported in part by the National In-

stitute of Information and Communications Technology.

6. REFERENCES[1] N. Kuwahara, K. Kuwabara, A. Utsumi, K. Yasuda,

and N. Tetsutani. Networked interaction therapy:Relieving stress in memory-impaired people and theirfamily members. In Proc. of IEEE Engineering in

Medicine and Biology Society, 2004. to be appeared.

[2] Akira Utsumi and Jun Ohya. Multiple-camera-basedhuman tracking using non-synchronous observations.In Proceedings of Fourth Asian Conference on

Computer Vision, pages 1034–1039, 2000.

[3] D. M. Gavrila and L. S. Davis. 3-d model-basedtracking of humans in action: a multi-view approach.In Proc. of Computer Vision and Pattern Recognition,pages 73–80, 1996.

[4] Q. Cai and J. K. Aggarwal. Tracking human motionusing multiple cameras. In Proceedings of 13th

International Conference on Pattern Recognition,pages 68–72, 1996.

[5] Jakub Segen and Sarma Pingali. A camera-basedsystem for tracking people in real time. In Proceedings

of 13th International Conference on Pattern

Recognition, pages 63–67, 1996.

[6] Kosuke Sato and Seiji Inokuchi. Range-imagingsystem utilizing nematic liquid crystal mask. In Proc.

of 1st International Conference on Computer Vision,pages 657–661, 1987.

[7] K. Hattori and Y. Sato. Handy rangefinder for activerobot vision. In Proc. of Int. Conf. Robotics and

Automation, pages 1423–1428, 1995.

[8] A. Utsumi, S. Kawato, and S. Abe. Attentionmonitoring based on temporal signal-behaviorstuructures. In LNCS3766, Proc. of ICCV2005

Workshop on HCI, pages 100–109, 2005.

[9] J. Yang, R. Stiefelhagen, U. Meier, and A. Waibel. Areal-time face tracker. In Proc. 3rd IEEE Workshop on

Application of Computer Vision, pages 142–147, 1996.

[10] J. C. Terrillon, M. David, and S. Akamatsu.Automatic detection of human faces in natural sceneimage by use of skin color model and invariantmoment. In Proc. of IEEE Int. Conf. on Automatic

Face and Gesture Recognition, pages 112–117, 1998.

[11] J. Heinzmann and A. Zelinsky. 3-d facial pose andgaze point estimation using a robust real time trackingparadigm. In Proc. of IEEE Int. Conf. on Automatic

Face and Gesture Recognition, pages 142–147, 1998.

[12] E. D. Scheirer. Tempo and beat analysis of acousticmusical signals. Journal of Accoustical Society of

America, 103(1):588–601, 1998.

[13] M. Goto. An audio-based real-time beat trackingsystem for music with or without drum-sounds.Journal of New Music Research, 30(2):159–171, 2001.

15

Developing an engaging multimedia activity system for people with dementia

Arlene J. Astell1 Norman Alm2 Gary Gowans3 Maggie Ellis1 Richard Dye2 Phil Vaughan3 1University of St. Andrews, 2University of Dundee, 3Duncan of Jordanstone College of Art School of Psychology Division of Applied Computing Division of Graphic Design, South Street, St. Andrews Faculty of Engineering and Design School, Dundee Fife, Scotland, KY16 9JU Physical Sciences Dundee, Scotland, DD1 4HN Scotland, DD 4HN, +44 1334 462056 +44 1382 345596 +44 1382 348273 [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] Abstract Finding ways to engage people with dementia in stimulating but safe activities would be beneficial both to them and to their caregivers. We have previously demonstrated that a simple but carefully designed computer-based touch screen interface can capture and hold the attention of people with dementia who are normally difficult to engage [2]. In the previous project, people with dementia used the system together with a caregiver to provide a shared activity. We are now developing a similar hypermedia system to be used alone by a person with dementia without caregiver assistance. The system is intended to capitalise on the ‘living in the moment’ state of people with dementia and make the most of their preserved skills. At the same time, the system will compensate for diminished abilities, such as short-term memory. We have piloted three navigable virtual environments – a garden, a museum and a pub – with both people with dementia and professionals in the field of dementia care. We report the results of this pilot work and consider the further questions to be addressed in developing an engaging, multimedia activity for people with dementia to use independently. Categories and Subject Descriptors J.3 [Computer Applications] : Life and medical sciences - Health ; Social and Behavioural Sciences - Psychology General terms : Design, Experimentation, Human Factors

General Terms Design; Experimentation Keywords Dementia; activity; engagement. 1. Introduction Dementia is a progressive, irreversible disorder and age is the greatest risk factor for developing it. The most common cause of dementia is Alzheimer’s disease, which is

characterised in the early stages by memory problems and failure to learn new information. Over time, all aspects of cognitive function are affected and daily living skills erode. People with dementia gradually lose the ability to keep safe, keep busy and care for themselves. Left to their own devices, people with mild to moderate dementia may become lost and confused if they go out of the house, and at home may start activities, such as cooking or running a bath, which they fail to complete. These situations present considerable risks and reducing these places significant demands on caregivers, who are typically family members. As dementia progresses, people are commonly confused and disorientated. Many attempt to leave the unfamiliar nursing or residential settings full of apparently unknown people. Their agitation and distress present a challenge to caregivers and highlight the difficulties of keeping people occupied. Finding safe, stimulating activities that engage people at all stages of dementia would therefore be beneficial not only to them but also to their caregivers [3]. Participation in activities is important for stimulating and maintaining cognitive and social skills [8]. In addition, previous studies have shown positive effects of activity on reducing agitation and engendering a sense of well-being in people with dementia [4]. Engaging activity for people with dementia also provides caregivers with well-earned breaks from constant caring. In our previous work (The CIRCA Project; 1,2). We demonstrated that people with dementia are comfortable in front of a touch screen computer and will use the system when prompted. We have shown that a carefully designed system can tap into well-rehearsed cognitive and social scripts and use them to engage people with dementia [1,2]. In CIRCA we developed such a system to prompt and support social interactions between people with dementia and caregivers using a hypermedia structure. Hypermedia is ideal for tackling the short-term memory problem of people with dementia, as there is no penalty for ‘losing the place’ [5,6]. Users are encouraged to explore and get lost in the system to make the most of the material. Thus wherever the user is within the interface is the right

16

place to be. This is excellent for people with dementia whose short-term memory is extremely impaired, and who are unable to remember where they have just been. Using multimedia technology it is possible to include text, photographs, graphics, sound recordings and film recordings. These contents can be linked in various dynamic and flexible ways to promote the user’s engagement. The user is invited to interact with the material presented in a more lively way than by just looking at text and pictures on a page. In addition hypermedia provides abundant opportunities for users to make decisions as they explore the system. The way that hypermedia operates fits well with the Multiple Trace Theory of how memories are stored and accessed [7]. In this theory older memories are accompanied by multiple traces within the brain, which facilitate retrieval by providing more ways to access the memories. However, people with dementia have extreme difficulty accessing memories at will. Hypermedia provides

multiple prompts from across a range of stimuli to tap into these multiple access routes and facilitate retrieval of stored memories. Hypermedia offers further possibilities for creative interventions for people with dementia to access self-generated activities, which are safe and enjoyable. The main challenges in developing such a system are impaired short-term memory and problems learning new skills and information. Our solution with CIRCA was to take the here-and-now state of people with dementia and design the system around this. We are repeating this approach in the new project, ‘living in the moment’. We have thus developed three virtual environments for users to navigate. We wanted to assess the usability of these for people with dementia. We also wanted to investigate ways to prompt people with dementia to touch the screen. Finally, we wanted feedback from professionals in the field of dementia care on the suitability of the contents.

Figure 1: Virtual botanic garden

17

2. Pilot study (i) - Dementia professionals Method 2.1 Participants Respondents were recruited at the UK Alzheimer’s Society Annual Conference. Thirteen people completed a questionnaire after trying out the virtual environments. 2.2 Materials Three navigable virtual environments were created using Macromedia Director 8.5 as the authoring software. The following additional software were used to provide the content: Adobe Photoshop 6.0; Adobe Illustrator 9.0.1; Adobe Premier 6.0; QTVR Authoring Studio 1.0; SoundEdit 16 version 2; Infini-D 4.5. The environments were run on an Apple G4 laptop and were presented through a 20-inch touch-screen monitor. SONY SRS-T77 speakers were used to output stored music. The VR environments were viewed at a resolution of 1280 x 1024 pixels. Written prompts were included to indicate places to touch the screen (see Figure 1). A brief questionnaire about the virtual environments was compiled comprising the following questions: Did you try out the virtual reality environments?; Which, if any, did you prefer and why? Do you think they add to the experience?; Why?; Any additional comments; Suggestions for other VR environments. 2.3 Procedure Participants were invited to use the computer by themselves for as long as they liked. When they had finished each user was asked to complete the brief questionnaire about the virtual environments. 3. Results Most respondents gave general comments about the VR environments. One respondent commented, ”I like to see things that are related to the person with dementia’s life”. Another commented on the nature of the VR contents, stating that it was “Good to see moving pictures”. Of the thirteen respondents, eight expressed a preference for one of the three VR environments. Of these eight, five preferred the botanical gardens. The reasons for this included reference to the contents such as “beautiful to look at”; “excellent music”; and “lovely scenery”. Finally, one respondent commented, “the range of possibilities is greater than the other environments”. Eleven respondents commented on the additional value of having VR environments. Responses were addressed to three areas. First was the experience itself where respondents commented that it was ‘very realistic’ and “the garden – more real looking and multidimensional. Provides a choice of where to go and what to look at more closely, etc.”. A second group of comments referred to the engaging properties of the VR environments, such as “an escapism into a relaxing, enjoyable and engaging environment”. Further comments in this vein were “holds the attention’ and “stimulates memories, gives enjoyment through several senses – colour, movement and sound”. The third group of comments referred to the appeal of the VR

experience and the possibility for giving back some control to people with dementia. These included “greater variety and therefore more possibilities for everyone”; “it brings fun and a sense of achievement - you can make things happen; and “you can control what you are looking at”. One respondent additionally comment that the activity provided “Confidence building opportunities”. Other additional comments referred to the ease of navigation, the size of the control buttons and the ion screen instruction. Some respondents made suggestions for additional VR environments, which included: shopping; dancing; roller skating; fairground rides; animals, i.e. a zoo; and seasonal scenes. 4. Pilot study (ii) People with dementia 4.1. Participants Five people with dementia (2 men) were invited to look at the virtual environments in a day care setting. 4.2 Materials The laptop and touch screen as described above (section 2.2) were taken to a Day Centre for people with a diagnosis of dementia. 4.3 Procedure Each participant was invited to look at the VR environments with a member of the research team. In addition, each participant was encouraged to use the touch screen individually and explore the environments. When they had finished each participant was interviewed. 5. Results All of the people with a diagnosis of dementia appeared to enjoy looking at the VR environments and 3 out of the 5 tried using it independently. All participants spontaneously commented about the experience both when using it and afterwards. It was apparent that the botanic garden was the most popular environment with this group, just as it had been with the dementia care professionals. The participants with dementia commented on the contents, for example “lovely garden”, “beautiful flowers”, and “looks like a good pint “ (in the pub VR environment). One person went further in relating to the experience with the comment “I’ll sit here (bench in garden) for a while”. Another person commented on the totality of the experience, saying “the music is in time with it all” (in the VR botanic garden). A final group of comments reflected the user’s reaction to the experience including “amazing”, “wonderful”, “clever”, and “interesting”. 6. Discussion Evaluation of our three initial VR environments by a group of dementia care professionals and a group of people with a diagnosis of dementia was positive and provided useful feedback for the development of the Living in the Moment project. Of particular interest were the comments relating to the engaging properties of the stimuli. The dementia care professionals directly commented on the way the system

18

captured attention and provided an engaging experience. These observations were further supported by the comments of the users with a diagnosis of dementia, who were able to imagine themselves in the environments, for example sitting on a bench in the botanic garden or having a pint of beer in the pub environment. A number of useful comments were also made about the ease with which one could navigate around the environments. These included comments about the size of the arrows and on screen prompts for where to touch. We are currently carrying out detailed investigation of how best to prompt people with dementia to initiate and maintain interaction with the touch screen when they are using it alone. It is intended that the finished Living in the Moment system will provide a way for people with dementia to independently engage in activities that they can no longer participate in outside of the care environment, for example shopping and cooking. VR is one way to provide people with a diagnosis of dementia with opportunities to re-experience activities they previously enjoyed in a safe and risk-free situation. The evidence from this pilot work suggests that people with dementia respond positively to the environments and with adequate prompting can explore them independently. As such we plan to develop an extensive network of interconnected VR environments each containing activities to occupy and stimulate people with dementia. It is further hoped that Living in the Moment will be an enjoyable experience, which provides opportunities for people with dementia to exercise autonomy over their activities. This would assist families and others who care for people with mild to moderate dementia by providing safe and positive ways to occupy their relatives, creating a well-earned respite from the heavy demands of keeping people with dementia active and stimulated. In addition, it will help to address the problems of safety for people with dementia with as little infringement on their rights and autonomy as possible.

Acknowledgements We are grateful to everyone at the Alzheimer’s Society Annual Conference in York (UK) and at the Alzheimer Scotland Dundee Project who took part in this pilot work. The Living in the Moment project is funded by grant GR/T09279/01 to the first author and GR/T09262/01 to the second (Alm) and third authors (Gowans) from the EPSRC. References [1] Alm, N., Astell, A., Ellis, M., Dye, R., Gowans, G., &

Campbell, J. (2004). A cognitive prosthesis and communication support for people with dementia. Neuropsychological Rehabilitation, 14 (1-2), 117-134.

[2] Astell, A. J., Ellis, M. P., Alm, N., Dye, R., Campbell, J., & Gowans, G. (2004). Facilitating communication in dementia with multimedia technology. Brain and Language. 91(1), 80-81.

[3] Gori, Pientini, & Vespa. (2001). The selection of meaningful activities as a treatment for day-care in dementia. Arch Geron Ger, supp &. 207-212.

[4] Lou, A. (2001). The use of music therapy to decrease agitated behaviour of the demented elderly: the state of the science. Scandinavian Journal of Caring Sciences, 15(2), 165-173.

[5] Maddy, D. Brouwer-Janse (1996) User centred design, the contour of multimedia, theoretical and empirical developments, Academia Research Monograph 19. Ch12. Luton: John Libbey Media.

[6] McKerlie, D and Preece, J. (1992) The hypermedia effect: more than just the sum of its parts, in Proceedings of the St.Petersburg HCI Conference, pp 115-127.

[7] Nadel, L. and Moscovitch, M. Consolidation, retrograde amnesia and the hippocampal formation. Current Opinion in Neurobiology, 7: 217-227, 1997.

[8] Tzrinski & Higgins. (2001). Therapeutic play activities: building cohesion and socialization among nursing home residents. Activities, Adaptation and Ageing, 25(3-4), 121-135.

19

Networked Reminiscence Content Authoring and Delivery for Elderly People with Dementia

Noriaki Kuwahara [email protected]

Shinji Abe [email protected]

Kazuhiro Kuwabara [email protected]

Department of Assisted Communication ATR Intelligent Robotics and Communication Laboratories

Keihanna Science City, Kyoto, +81-0774-95-1401

ABSTRACT Providing good care at home to people with dementia is becoming an important issue as the elderly population increases. One main problem in providing such care is that it must be constantly provided without interruption, which puts a great burden on caregivers who are often family members. Networked Interaction Therapy is what we call our methods designed to relieve the stress of people suffering from dementia as well as their family members. This therapy aims to provide a system that interacts with people with dementia by utilizing various engaging stimuli. One such stimulus is reminiscence videos created from old photo albums, a promising way to capture a person’s attention for a long time. In this paper, we first present reminiscence videos and an authoring tool to assist in their production. Next, we explain the Networked Reminiscence Content Authoring and Delivery service that uses our authoring tool as a key component and show how this system is used by both caregivers and dementia sufferers.

Categories and Subject Descriptors H.4.3 [Communications Applications]: Computer conferencing, teleconferencing, and videoconferencing

General Terms Design

Keywords Interactive Reminiscence Contents, Content Sharing, Photo Annotation, Semantic Web, Dementia

1. INTRODUCTION Providing good care to people with dementia is becoming an important issue, especially as the aging of society inevitably increases their numbers. Care for the elderly must be constantly provided, placing a great burden on caregivers, who are often family members. The survey we conducted with four support groups indicates that caregivers urgently need a system that holds the attention of their loved ones so that they can enjoy short breaks [1]. Networked Interaction Therapy is the name for our proposed methods to relieve the stress of both people with dementia and their family members. These methods use a system that interacts with users to hold their attention. Human caregivers can thus take a break while the user is engaged with the system. A Networked Interaction Therapy system needs to get people with

dementia involved and keep their attention. Figure 1 illustrates an example of the home terminal for Networked Interaction Therapy. Currently, we plan to provide our service by using a large screen TV and a set-top box that controls network communication along with a camera and microphone for monitoring user status. One of our services provides an audio-visual stimulus because it is the most effective way to hold user attention. In addition to audio and visual stimuli, we are also considering an air cannon and a vibrator in the sofa to activate olfactory [2] and haptic senses [3].

Camera and MicrophoneAir Cannon for

Olfactory Sense

TV Set for Displaying Images

and Sounds

Vibrator in the Sofa for Haptic Sense

Figure 1. The illustrative Example of the Terminal Equipment

of Networked Interaction Therapy In this paper, we first present a reminiscence video and an

authoring tool to assist its production. Next, we explain the Networked Reminiscence Content Authoring and Delivery service, whose key component is our authoring tool, and show how this system is used by both caregivers and dementia sufferers.

2. REMINISCENCE VIDEO Much research has presented audio-visual stimuli to people with dementia. For example, the ‘video respite’ concept [4] was created to capture and maintain the attention of people with dementia while giving caregivers time to rest. Respite videos often feature a character who talks to the person with dementia in a way that attracts his or her attention. Furthermore, Project CIRCA utilized multimedia technology to stimulate the long-term memory of people with dementia [5]. People with dementia sometimes have problems with short-term memory even while retaining long-term memory. This project, in a sense, augments reminiscence therapy with multimedia technologies.

From the viewpoint of stimulating the long-term memory of people with dementia, we selected a so-called ‘reminiscence video’ as an audio-visual stimulus. This is a slide show video

20

produced from the user’s old photo albums. Its effectiveness was experimentally shown [6]. In addition, once created, it is relatively easy for caregivers to use the video.

2.1 Requirements for Reminiscence Videos First, we have already conducted several experiments [7] that clarified the following as the most important requirements for reminiscence video to satisfy caregiver needs:

1. Adding narration to produce an attractive reminiscence video

2. Zooming in on familiar faces in a photo

3. Narration and zooming synchronization. When the narration is about a specific person in the photo, it should be dictated while that region is being zoomed in on and that corresponds to this person.

Creating such a video, however, is not easy because it involves many steps: first, photo albums are collected from the user’s family. Next, photos are selected from the photo albums that should ideally invoke distant memories of the user. Then the photos are shot with a video camera, often using panning and zooming effects (the so-called ‘Ken Burns effect’ [8]). Finally, narration is added to increase user involvement in the video. These processes are usually performed by skillful volunteers, since they are not the kinds of tasks family caregivers can do while looking after people with dementia. Moreover, for some users, showing the same reminiscence video again and again may erode the video’s original effects. Thus, a simple method for making different reminiscence videos from the same set of photos is needed. To solve this problem, we developed an authoring tool that uses photo annotations [9, 10]. Annotation attached to a photo includes a theme, such as ‘travel’ or ‘school,’ and the time it was taken. In addition, ‘region’ in the photo can be specified by annotation. Each region will represent, for example, an area of a person’s face or some interesting objects in the photo. These regions will be used when panning and zooming effects are added. These are popular when making an attractive slide show video from a set of still photos. Figure 2 shows the overall reminiscence video production flow with consideration given to the above-mentioned features of our system.

START

Annotating Photos

Querying Photos for the Reminiscence Video and Choosing proper set

of the photos

Selecting BGM and Narrations for the

Reminiscence Video

Preview Video?

Rendering Video by addingKen Burns Effect, BGM and Narrations with Semantic

Synchronization

FINISHYES

NO

Finish editing?

YES

NO

Generating Movie File of the Reminiscence Video

Con

tinue

to e

dit R

emin

isce

nce

Vid

eo

Figure 2. Reminiscence Video Production Flow

Our tool achieves these features that make it possible to automatically transform one type of content (photo) to another kind of content (slide show video – more specifically, reminiscence video) by annotation data. Furthermore, as mentioned above, content distribution and sharing on the Web will become much more common and help make reminiscence videos more interesting for elderly people.

2.2 Overview of Authoring Tool Figure 3 depicts the overall process of slide show generation.

The architecture is based on the RDF database. Our photo annotation tool can annotate photos and store the annotations in an RDF format.

RDF Database (Jena2.1 +Postgresql 7.4)

PhotosPhotosDublin Core`

Image Regions

NarrationsNarrations

Annotations

Annotating Photos

Selecting BGM and Narrations

Rendering Video

Use Cases

BGMPlaylist

Photo Annotation Tool (J2SE 1.4.2_04)

RenderingTemplatesRenderingTemplates

ReminiscenceVideo

RELATIONSHIPFOAF

RELATIONSHIPFOAF

Querying Photos

Refer

Slide-Show Generating Tool (J2SE 1.4.2_04)

Generating Movie File

Figure 3. Overview of System Architecture

female Haruka

(x1,y1)

(x2,y2)

h1

w1They are very happy.

at Disney Animal Kingdom

x1,y1,w1,h1

x1,y1,x2,y2

20040716

sample.jpg

rel:grandchildOf

dc:datedc:title foaf:Image

rdf:type

dc:description

imgReg:hasRegion

imgReg:hasRegion

imgReg:Rectangle

imgReg:regionDepict

imgReg:boundingBox

imgReg:coords

foaf:namefoaf:gender

foaf:Person

rdf:type

rdf:type

rdf:type rdf:type

Viewer

LiteralClass ResourceInstance Resource

Figure 4. Annotation of Photo Data

For photo annotation, annotations basically focus on the people in the photo because photos containing relatives and/or friends are often included in creating a reminiscence video. We employed de facto standard vocabularies as much as possible to describe annotations such as Dublin Core [11], FOAF [12], and Image Regions [13] because using these standards is a great advantage when the creator wants to use materials from the

21

Internet or to publish their own materials. We show an illustrative example of photo annotation in Figure 4, where Dublin Core is utilized to describe various properties of the photo itself. The date, a title, and a description of the photo are represented with this vocabulary. FOAF is mainly used to describe a person included in the photo. The person’s name, gender, age, and contact information are represented by FOAF. Image Regions are used for storing each photo’s region data that correspond to the person in the photo. We also employ a vocabulary called RELATIONSHIP [14], which is a refined vocabulary of the “knows” attribute of FOAF. It represents the relationship between the persons in the regions of the photo specified by Image Regions.

In addition to photo annotation, we annotated the narration stored in the database. Figure 5 shows an illustrative example of the annotation of narration data. First, we defined our original vocabulary for annotating narration. In Figure 5, “na” represents the name space of this vocabulary. The narration text of “Narration1.wav” is specified by “na:text,” and “Narration1.wav” includes several instances that represent the keyword specified by “na:keyword.” In the figure, the left instance represents the person described by the narration text and indicates an instance of “foaf:Person” by using “na:referTo.” On the other hand, the left instance indicates the person’s appearance, while “dc:description” points to the Literal that represents the word used for generating the narration text.

female

Anonymous

Smile

Narration1.wav

foaf:name

foaf:gender

na:keyword

LiteralClass ResourceInstance Resource

urn:atr:person:daughter

URI

Viewer

rdf:type

urn:atr:appearance:smile

rel:childOf

dc:description

Your daughter is smiling.

na:text

na:keyword

foaf:Personrdf:type

na:referTo daughterdc:description

Figure 5. Annotation of Narration Data

3. NETWORKED REMINISCENCE CONTENT DELIVERY We are now developing the Networked Reminiscence Content Authoring and Delivery service by using our authoring tool as the key component. In this section, we will present this service’s design goals and an overview of our prototype system.

3.1 Design Goals of the Platform When designing our service, we defined the following goals: 1. Dementia sufferers do not need to operate terminals to start

communication or to share reminiscence contents with caregivers.

2. Caregivers can get feedback from dementia sufferers on the content, i.e., the date, the place, and names of each person in the photo, and so on.

3. The terminals of dementia sufferers require no installation of special hardware or software.

4. Remote caregivers and dementia sufferers should feel as if they are enjoying reminiscence contents and talking about them while sitting beside each other.

One motivation for defining these goals acknowledges the results of interviews with family caregivers who participated in our experiments. They recognized the efficiency of the reminiscence videos used in experiments to hold their dementia family members’ attention, and they also found that reminiscence videos greatly increased dementia sufferers’ willingness to talk. This encourages family caregivers to talk more with their dementia family members about memories by using their old photos. However, the number of elderly people living alone is rapidly increasing in developed countries, reducing opportunities for elderly people to communicate with others and increasing the development of various physical and mental disorders. Finally, these disorders are more likely to cause dementia. If technologies can offer elderly people living alone more chances for communication with others about their memories by using old photos, it might be possible to prevent such disorders. These technologies also provide opportunities to communicate with remote relatives and friends of dementia sufferers, greatly improving their quality of life.

Another motivation was cost reduction of photo annotation for producing reminiscence videos because collecting information about every photo requires a lot of deliberate interviews with dementia suffers and their relatives. Otherwise, inappropriate photo narration destroys the warm atmosphere of a reminiscence video. If a remote caregiver can talk with the dementia sufferer and family members by sharing old photos on the Internet during such a session, the caregiver will be able to gather information on such photos to annotate them.

3.2 Overview of Prototype System To achieve our goals, first we made the photo database function of our reminiscence video authoring tool accessible from HTTP clients so that both the caregiver’s and the dementia sufferer’s terminals can retrieve photos without special software.

Secondly, we developed a mechanism that makes it possible to publish and subscribe to messages between web browsers by using HTTP (Pub/Sub over HTTP) that works as a wrapper of simple web contents, such as showing photos in the photo database. The web browser on the dementia sufferer’s terminal is fully controlled by the web browser on the caregiver’s terminal to load contents for showing photos and to specify the URLs of photos to be displayed. Furthermore, we plan to use a touch panel screen for the terminal of the dementia sufferer, and the caregiver encourages the dementia sufferer to point to persons or objects of interest in the photo. Then, using the above Pub/Sub over HTTP, the dementia sufferer’s web browser sends a message that includes the data of the pointing position on the photo, which the caregiver’s terminal can display by an icon that shows this position on the photo.

22

Thirdly, we use an IP TV phone that supports H.323 protocol for communication between the dementia sufferers and caregivers. Combined with this communication device and photo sharing functions, we create an atmosphere where they can enjoy reminiscence contents and talk about them under the illusion of sitting beside each other. Figure 6 shows an illustrative example of our prototype system. The dementia sufferer and caregiver are remotely sharing and talking about the same photo.

Dementia Sufferer Remote CaregiverInternet

Pointing the interest Person by Finger

Reminiscence Photo Database Feedback from Dementia Sufferer

Communication by using IP TV Phone

Figure 6. Overview of Prototype System

3.3 Use Cases We show GUI examples of our implementation along with use cases where the caregiver opens a communication channel with the dementia sufferer, shares the photo, gets feedback from the dementia sufferer, and annotates the photo.

3.3.1 Opening Communication Channel between the Dementia Sufferer’s and the Caregiver’s Terminals

Caregiver’s Terminal

Dementia Sufferer’s Terminal

Caregiver’s Image

Dementia Sufferer’sImage

Photo Database

Figure 7. Opening Communication Channels

The dementia sufferer’s terminal is set up to start the web

browser and access the URL for subscribing to the message from the caregiver’s terminal.

The caregiver starts the reminiscence video authoring tool and connects to the photo database to retrieve photos related to the dementia sufferer with whom the caregiver is now opening communication channels.

The caregiver calls the dementia sufferer’s terminal to start communication by using an IP TV phone, and the dementia sufferer’s terminal automatically responds.

3.3.2 Sharing Photos between the Dementia Sufferer’s and the Caregiver’s Terminals

The caregiver selects a photo from the photo database and drops and drags a thumbnail of it into the main panel

The reminiscence video authoring tool sends a command message to the dementia sufferer’s web browser so that it loads the web page for photo sharing and displays the corresponding photo on the browser by using this page.

The dementia sufferer’s web browser automatically executes the command from the caregiver’s terminal and displays the photo on the web browser. Figure 8 shows an example of the terminal GUI just after sharing the photo between the two terminals.



Drag & Drop

Thumbnail Images of Photos in the Photo Database

The Main Panel of the Authoring Tool

Caregiver’s Image

Figure 8. Sharing a Photo

3.3.3 Getting Feedback from the Dementia Sufferer’s Terminal

The caregiver talks with the dementia sufferer through the verbal and visual communication provided by the TV phone and encourages the dementia sufferer to point to a person of interest in the photo.

The dementia sufferer’s terminal has an eight-inch touch panel screen, and the position data on the screen where the dementia sufferer points are sent to the caregiver’s terminal.

Then an icon showing the position on the photo where the dementia sufferer has pointed is displayed on the photo shown on the caregiver’s terminal. Figure 9 shows examples of receiving feedback from the dementia sufferer.

23

Dementia sufferer points to a person of interest.



Caregiver gets feedback on a person of interest by an icon on the photo.

Figure 9. Getting Feedback from Dementia Sufferer

3.3.4 Photo Annotation

After getting feedback from the dementia sufferer, the caregiver annotates the photo. To collect information on the person in whom the dementia sufferer’s shows interest, the area corresponding to that person in the photo is zoomed up on to promote conversation, as shown in Figure 10.



Caregiver selects this area to be annotated

Annotated area is zoomed up automatically to promote conversation.

Annotation Form

Figure 10. Photo Annotation

3.4 Implementation To implement our prototype system, we chose NTT Flet’s Phone [15], a commercial NTT product that works as an IP TV phone by H.323 protocol, as the dementia sufferer’s terminal because its eight-inch touch panel screen design is as easy to use as home electric appliances. Also, it supports Internet Explore 6.0, and our

Pub/Sub over HTTP for exchanging command messages between the dementia sufferer’s and the caregiver’s terminal is available. Figure 11 shows the system architecture of Pub/Sub over HTTP.

XML RPC

Asynchronous HTTP Request

Asynchronous HTTP Request

XML RPC

XML RPC

Web Content Wrapper

HTTP Protocol

Java Application

Web Content

Pub / Sub Manager

Web BrowserReminiscence Video

Authoring Tool

Web Server

NotifyCommand Subscribe

Command

PublishCommand

Figure 11. Pub/Sub over HTTP

The dementia sufferer’s terminal has been implemented as Web Content that displays the photo specified by URL and can pan and zoom up in the rectangular region on the photo given as X, Y coordinate.

A Web Content Wrapper can invoke these Web Content functions according to the command notified from the Reminiscence Video Authoring Tool. But first, the Web Content Wrapper needs to request the Pub/Sub Manager to subscribe to commands from the authoring tool. The Pub/Sub Manager, implemented as a servlet in the Web Server, stops the request process until the authoring tool publishes the command.

The authoring tool publishes the command to the Pub/Sub Manager, which restarts the subscribe request process from the Web Content Wrapper; consequently, the command is notified. Every command is represented by an XML format, and we use XML RPC for encoding and decoding commands.

After receiving the command, the Web Content Wrapper immediately sends a subscribe request and simultaneously invokes a Web Content function corresponding to the command. Web Content Wrapper uses an Asynchronous HTTP Request to send the subscribe request, and then Web Content can continue to perform such functions as photo display, panning to a specific area of the photo, and zooming without waiting for responses from the Web Server.

This Pub/Sub mechanism is symmetric between the Web Browser and the Reminiscence Video Authoring Tool. To obtain the point of interest position of the dementia sufferer in the photo from the Web Browser, the authoring tool sends the request for subscribing commands from the Web Browser, which publishes the command that includes an X, Y coordinate of the pointing position of the dementia sufferer in the photo when the dementia sufferer pushes the terminal’s touch panel screen.

By using this Pub/Sub mechanism, we achieved our three design goals in section 4.1. The dementia sufferer’s terminal does not need special software to be remotely operated from the caregiver’s terminal. There is no need to set up a broadband router for IP port forwarding. The dementia sufferer and his/her family members simply prepare an IP TV phone and access our web contents for reminiscence photo sharing from the web

24

browser. Then remote caregivers can fully control his/her terminal and talk about the reminiscence photos.

4. CONCLUSION AND FUTURE WORK In this paper, we presented our Reminiscence Video Authoring Tool and Networked Reminiscence Content Authoring and Delivery service that uses an authoring tool as a key component. We defined our design goals for developing the Networked Reminiscence Content Authoring and Delivery service and showed that we achieved our three design goals from section 4.1 by using Pub/Sub over an HTTP mechanism.

Currently, our prototype system can only handle reminiscence photos. However, our content sharing mechanism is quite versatile, and we are enhancing it to handle reminiscence videos, music, and other types of interactive reminiscence content, such as CIRCA [5].

Now we are conducting preliminary experiments on the possibility of communication between caregivers and dementia sufferers by TV phones and the effectiveness of reminiscence contents during such communication. From these experiments, we will evaluate whether our system satisfies our design goals. Also, we are planning large scale experiments with the cooperation of a senior citizens care facility where terminals for dementia sufferers will be set up in each private room and remote caregivers will take care of dementia sufferers by TV phone. Through this experiment, we will examine whether this type of service satisfies all participants, including the dementia sufferers, their family members, caregivers in the facility, and remote caregivers.

5. ACKNOWLEDGMENTS We would like to thank the participants of the reminiscence video experiments. We also would like to thank Koji Saito for the design and implementation of the authoring tool. This research was supported by the National Institute of Information and Communications Technology.

6. REFERENCES [1] Kuwahara, N., et al.: Networked Interaction Therapy:

Relieving Stress in Memory-Impaired People and Their Family Members, Proceedings of the 26th Annual International Conference IEEE EMBS (2004).

[2] Yanagida, Y., Kawato, S., Noma, H., Tetsutani, N., and Tomono, A.: A Nose-Tracked Personal Olfactory Display, SIGGRAPH 2003 Sketches & Applications (2003)

[3] Kakita, M., Yanagida, Y., and Tetsutani, N.: Presentation of A Character Using Distributed Vibro-Tactile Display, IEICE Society Conference A-16-23 (2003) 318 (in Japanese).

[4] Lund, D.A., et al.: Video Respite: an innovative resource for families, professional caregivers, and persons with dementia, The Gerontologist, Vol. 35, Issue 5 (1995) 683-687.

[5] Gowans, G., Campbell, J., Alm, N., Dye, R., Astell, A., and Ellis, M.: Designing a multimedia conversation aid for reminiscence therapy in dementia care environments, Extended abstracts of the 2004 conference on Human factors and computing systems (2004) 825 - 836.

[6] Yasuda, K., et al.: Reminiscence Video for Higher Brain Dysfunctions, Proceedings of General Conference of Japan Society for Higher Brain Dysfunctions, (2004) (in Japanese).

[7] Kuwahara, N., Kuwabara, K., Tetsutani, N., and Yasuda, K.: Reminiscence Video - Helping At-Home Caregivers of People with Dementia, Home-Oriented Informatics and Telematics, Proceedings of the IFIP WG 9.3 HOIT 2005 Conference (2005) 145-154.

[8] http://en.wikipedia.org/wiki/Ken_Burns [9] Kuwahara, N., Kuwabara, K., Tetsutani, N., and Yasuda, K.:

Using Photo Annotations to Produce a Reminiscence Video for Dementia Patients, 3rd International Semantic Web Conference (ISWC2004) Demo Papers (2004).

[10] Kuwahara, N., Kuwabara, K., Shinji Abe, Yasuda, K., and Tetsutani, N.: Semantic Synchronization: Producing Effective Reminiscence Videos, 4th International Semantic Web Conference (ISWC2005) Demo Papers (2005).

[11] http://dublincore.org/ [12] http://www.foaf-project.org/ [13] http://www.bnowack.de/w3photo/pages/image_vocabs [14] http://vocab.org/relationship/ [15] http://www.fletsphone.com/index_f.html

25

An easy-to-use Interface for Evidential Reasoner QingXiang Wu

School of Computer Science Queen’s University Belfast Belfast, BT7 1NN, N.I. UK

Tel:(44)2890974783

[email protected]

David Bell School of Computer Science Queen’s University Belfast Belfast, BT7 1NN, N.I. UK

Tel:(44)2890974165

[email protected]

Rashid Hafeez Khokhar School of Computer Science Queen’s University Belfast Belfast, BT7 1NN, N.I. UK

Tel:(44)2890974783

[email protected]

ABSTRACT In this paper, we propose the system for cognitive prostheses by using AI technology through a simplified interface. Evidence theory is often used as an uncertainty handling mechanism when applied to complex reasoning systems. On the other hand, this theory can be applied to compensate human brain’s limitations when making decisions in daily life, business, or political arguments. Here the very simple interface is designed for the reasoner based on evidence theory so that it can be simply used to help human to make decisions in different situations with only a few keyboard clicks. A couple of examples for the applications are demonstrated in this paper.

Keywords Cognitive prostheses, evidential reseaner, decision making, uncertainty reasoning.

1. INTRODUCTION

There are limitations (Wang, 2004) of Bayesianism and shortcomings of other techniques for handling vagueness (Shi and Bell, 1996). An evidence-based system is proposed for carry out uncertain reasoning. The evidence theory can be embedded in complex systems, and it can be applied to daily life, business, or political arguments. In other words, evidential reasoning can be used to make decisions for cognitive prostheses [3] where AI systems are regarded as a system [4,5] that interacts with humans and extends human’s capability, rather than as an isolated problem solving system. For example, a businessman has to make dozens of important decisions every day, politician need to make decisions to approval for huge projects according to expert’s opinions. Evidence theory is a good way to support such decision-making. The research results show that the theory has very high potential applications in widely varying decision-making domains (Bell, 1993; Fixsen et al, 1997). However, there are few simple

software systems for such applications. The purpose of this paper is to study the simplified interface for applications of evidence theory and provide a simple reasoner for such applications.

In Section 2, Demspter’s evidence combination rule is reviewed. In Section 3, a basic reasoner is designed with simple interfaces. A user guide of the reasoner is provided to demonstrate the operations in Section 4. In Section 5 some examples are presented, and finally conclusions are given in Section 6.

2. THE DEMSPTER’S RULES In order to handle uncertain and imprecise information, Bayesian belief networks, Rough set theory, evidence theory and fuzzy logic are among the techniques that have been proposed in the artificial intelligence domain. Evidence theory which can be regarded as uncertain reasoning using beliefs. Based on a mass function and uncertain rules, evidence theory (Dempster-Shafer, 1976; Barnett, 1991; Bell, 1993; Fixsen et al, 1997) has applied to solve a variety of problems in uncertain reasoning.

Let Θ represent a set of hypotheses in a hypothesis space. A function m: 2Θ→ [0,1] is defined as a mass function if it satisfies

M1. ( ) 0m ∅ =

M2. ( ) 1X

m X⊆Θ

=∑

Here ∅ represents an empty set. A function bel: 2Θ→ [0,1] is defined as a belief function if it satisfies

B1. ( ) 0bel ∅ = .

B2. ( ) 1bel Θ = .

B3. For any collection A1, A2, …, An(n ≥ 1) of subsets of Θ, | | 1

1 1{1,..., }

( ... ) ( 1) (In i

I nbel A A A bel A+

∈⊆

∪ ∪ ∪ ≥ − ∩∑ ).I i

Based on these functions, a set of evidential reasoning operations is defined. The fundamental operation of evidential reasoning is the orthogonal sum of evidential function, which is known as Dempster’s rule for combining evidence. Bayesian method take account of an event in two situations, true or false. Evidence theory takes account of a piece of evidence in three situations, true, false and unknown. Therefore evidence theory is more powerful.

26

For example, a shop manager has done a series of survey for product A to see if the product satisfied customers. The answer from one area C1 is 50% satisfied, 30% unsatisfied, 20% not sure. Another answer from area C2 is 40% satisfied, 50% unsatisfied, 10% not sure. In this case, hypothesis space can be regarded as

, S for satisfied and U for unsatisfied. Two sets of opinions are regarded as two pieces of evidence C1 and C2 respectively. It is very often required to combine the two sets of opinions. According to evidence theory, the combination of two pieces of evidence can be represented in Table 1.

{ , }S UΘ =

Table 1. Combination of two customer surveys

C1⊕C2 C1{S} 0.5 C1{U}0.3 C1{S,U}0.2

C2{S}0.4 0.4x0.5=0.2 ∅ 0.4x0.2=0.08

C2{U}0.5 ∅ 0.5x0.3 =0.15 0.5x0.2=0.10

C2{S,U}0.1 0.1x0.5=0.05 0.1x0.3=0.03 0.1x0.2=0.02

In the combination, we have applied the Dempster rule as follows.

1 2 1 20( ) ( ) 1 ( ) ( )

X Y X YN m X m Y m X m Y

∩ ≠∅ ∩ == = −∑ ∑

=0.2+0.08+0.15+0.10+0.05+0.03+0.02 =0.63

1 2 1 2{ }( )({ }) (1/ ) ( ) ( )

X Y hC C h N m X m Y

∩ =⊕ = ∑

1 2( )({ }) (1/ 0.53)(0.2 0.05 0.08)0.524

C C S⊕ = + +=

1 2( )({ }) (1/ 0.53)(0.15 0.10 0.03)0.444

C C U⊕ = + +=

1 2( )({ , }) (1/ 0.53)(0.2)0.032

C C S U⊕ ==

The combination result is 52.4% satisfied, 44.4% unsatisfied, and 3.2% unknown. For a few of such surveys, the calculation can be done manually. For a large number of customers, a large matrix needs to be processed. Therefore, a light reasoner with simple interfaces is designed for different spot applications in daily life, business, and political arguments. In order to avoid processing a dynamic matrix, we have proposed an incremental combination algorithm that is implemented in the reasoner.

3. EVIDENTIAL REASONER The architecture of the reasoner is shown in Figure 1. In order to use the reasoner easily, it is constructed with three modules: knowledge acquisition module, reasoning module and the core of reasoner. Two simple interfaces are designed to interact with the users. We can input arguments or expert’s opinions through the knowledge acquisition interface. After the reasoner has acquired the arguments or expert’s opinions, a user can go to the reasoning

interface. The user can collect evidence, conduct reasoning and attain the results.

Knowledge acquisition interface

Core of the reasoner

Reasoning operation interface

Figure 1. Architecture of evidence reasoner

Two stages are indicated in use of the reasoner. At first stage, knowledge or expert’s opinions must be entered into the reasoner. Then it is said that the reasoner has knowledge. At second stage, the reasoner can be used to make decisions.

3.1 Knowledge acquisition interface

The reasoner is designed as one executable file in Microsoft windows system. The reasoner can be started with a double click the file name. The ‘Evidence’ tab can be seen on the menu bar. Click ‘Evidence’ and ‘Knowledge’ items. A knowledge input window is opened. The interface is shown in figure 2

Figure 2. Knowledge acquisition interface Expert’s opinions can be easily inputted to the reasoner through the interface in Figure 1. Each piece of evidence can support multiple hypotheses, and each hypothesis can be supported by multiple pieces of evidence. Each pair of evidence and hypothesis forms an uncertain argument. The argument can be added to the reasoner and display in the argument list box. A list of buttons beneath the argument list box can be used to edit the arguments. For example, a user can select an argument in the list box and then click the ‘EditSlect’ button. The argument goes to argument

27

input box. The user can edit the argument and click ‘EndEdit’ button to confirm the edited argument. If the user clicks the button ‘AddasNew’, the edited argument will be regarded as a new argument and add in the argument list. The user can delete an argument or all the arguments in the argument list.

3.2 Reasoning Interface Reasoning interface is designed to make decisions based on the knowledge input through the knowledge acquisition interfaces. Two list boxes at top show the evidence names and hypothesis names that refer to the input knowledge. Clicking the display button, all the names can be listed. Double clicking on the evidence or hypothesis, the evidence or hypothesis will be selected to the list box at middle row. Click the reasoning button; the final results from the selected evidence to the selected hypothesis can be obtained. See the example in Figure 3.

Figure 3. Reasoning interface Based on this interface, a user can calculate different evidence combinations or different hypothesis. If the reasoner has knowledge, reasoner always can give a result despite how many evidences are obtained.

4. APPLICATIONS OF THE REASONER

4.1 Decision in a directorate committee Traditionally, the number of directorates is an odd number so that the rule “the minority obeys the majority” can be applied to some argument issues. If each directorate can represent his/her opinion with numbers, their opinions can be easily combined with the reasoner and the result will show supports or against despite there are odd number of directorates or not. For example, there are 10 directors in a committee. They have very different opinions for an investment Project B. Their opinions are represented by number shown in Table 2.

Table 2. Opinion represented by number

Earning Losing Unknown

Director 1 0.9 0 0.1

Director 2 0.6 0.2 0.2

Director 3 0.2 0.7 0.1

Director 4 0.5 0.4 0.1

Director 5 0.2 0.5 0.3

Director 6 0.7 0.1 0.2

Director 7 0 0.8 0.2

Director 8 0 0.7 0.3

Director 9 0.4 0.6 0.1

Director 10 0.4 0.3 0.3

From Table 2, we can see that the opinions are very different. Directorate 1 said 90% earning, no losing, only 10% not sure. However, Director 7 said that no earning money, 80% losing, and 20% not sure. Using human’s brain, it is not easy to judge whether the project will earn money or lose money. If these data are entered into the reasoner, the answer is obtained by several clicks. The results are shown in Figure 4. The result is 46% for earning money, and 54% for losing.

Figure 4. Result obtained from the reasoner There is a chance that someone suggestion may not be significant, and his/her opinion can be taken away and get new results cpompanies. For example, click Directorate 2 in the left list box in the middle row; then click remove button. Directorate 2’s opinion is taken away. Click the reasoning button. The new result are shown in Figure 5. The result is 30% earning, and 70% losing.

28

Figure 6. Results for house evaluation

5. CONCLUSION

In this paper, we proposed to design simple human-machine interface to apply an AI technology for cognitive prostheses. A simplified, easy to use, evidential reasoner has been designed in a straightforward way and can be applied to make decisions for daily life and business. In fact, the reasoner can be applied to different situations to help human’s decision making. A very important issue is how to make the meaningful input data. Then result can be explained using the same set of inteface. The examples in Section 5 show how to assign mean up to the percentage numbers and how to get the percentage numbers so that the results have very clear mean. This paper only focuses on evidence combination. The other operations from evidence theory are easy to integrate into the reasoner.

Figure 5. New result after taking away Directorate 2’s opinion

4.2 Decision in daily life Suppose that you are going to buy a house at location C. You can ask your friends to evaluate whether your decision is good or not by using evidential reasoner. You can say to your friend, “Please evaluate the house at location C with percentages: which percentage for advantages, which percentage for disadvantages, and which percentage not sure.” Then you will get a list of numbers from your friends. Suppose the numbers are shown in Table 3. Inputting the data into our reasoner, the combination results can again be obtained by several clicks. The final results are shown in Figure 6. The results are 90% for advantages and 10% for disadvantages.

6. ACKNOWLEDGMENTS

The authors would like to thank Northern Ireland Investment Ltd. to fund this project.

7. REFERENCES [1] Pei Wang, The Limitation of Bayesianism, Artificial

Intelligence, 158,1( 2004), 97-106. Table 3. Evaluation of a house [2] Shengli Shi and D.A.Bell, An approach to system control

based on combination-update of belief, Journal of Control and Cybernetics, 25, 2(1996).

Advantages Disadvantages

Unknown

Friend 1 0.8 0.1 0.1

Friend 2 0.7 0.2 0.1

Friend 3 0.3 0.5 0.2

Friend 4 0.4 0.4 0.2

[3] Ford, Kenneth M. and Patrick J. Hayes. Cognitive Prostheses. Invited talk FLAIRS-99, Orlando,1999.

[4] P. Gregor and A. F. Newell, "Information technology for rehabilitation and as cognitive prostheses", Journal of the International Neuropsychological Society, 6, 4 (2000)419-420.

[5] Eric Neufeld, David Callele, David Mould, Sonje Kristtorn,

Raymond Rabu, A Contribution to the Theory and Practice of Cognitive Prostheses, LNCS, Springer-Verlag GmbH, Volume 2733 (2003) 241 – 250.

[6] Bell, D.A., From Data Properties to Evidence, IEEE Transactions on Knowledge and Data Engineering, 5, 6 (December 1993) 965-969.

[7] Fixsen, D., Mahler, R.P.S., The Modified Dempster-Shafer Approach to Classification, IEEE SMC-A, 27, 1( January 1997) 96-104.

[8] Shafer, Glenn, A Mathematical Theory of Evidence. Princeton University Press, 1976.

29

http://iris.usc.edu/Vision-Notes/bibliography/author/mah.html#Mahler, R.P.S.

A cognitive prosthesis to support communication by people with dementia

Norman Alm

University of Dundee Division of Applied Computing

University of Dundee, Dundee, UK Tel: +44 (0) 1382 385596

[email protected]

Maggie Ellis University of St Andrews

School of Psychology University of St Andrews, St Andrews,

UK Tel: +44 (0) 1382 385119

[email protected]

Richard Dye University of Dundee

Division of Applied Computing University of Dundee, Dundee, UK

Tell: +44 (0) 1382 384150 [email protected]

Gary Gowans

University of Dundee Department of Computer-Aided Design University of Dundee, Dundee, UK

Tel: +44 (0) 1382 345571 [email protected]

Arlene Astell University of St Andrews

School of Psychology University of St Andrews, St Andrews,

UK Tel: +44 (0) 1334 462056 [email protected]

Jim Campbell

University of Dundee Department of Computer-Aided Design University of Dundee, Dundee, UK

Tel: +44 (0) 1382 385471

ABSTRACT Given the devastating impact of dementia, developing cognitive prostheses for elderly people with this condition offers the possibility of making significant improvements in their lives. We report on a project that has developed a hypermedia based communication support system for older people with dementia, based on reminiscence material. The system has been designed to be usable by people with short-term memory deficits. The system has proved remarkably effective in supporting the communication of people with dementia.

Categories and Subject Descriptors J.3 [Computer Applications] : Life and medical sciences - Health ; Social and Behavioural Sciences - Psychology General terms : Design, Experimentation, Human Factors

General Terms

Design, Experimentation, Human Factors

Keywords Dementia, Alzheimer's disease, working memory, short-term memory, multimedia, hypermedia

1. DEMENTIA AND WORKING (SHORT-TERM) MEMORY Dementia consists of a group of symptoms involving progressive impairment of cognitive abilities. The two major causes of dementia are Alzheimer's disease, which is a progressive loss of nerve cells, and vascular dementia, which is loss of brain function due to a series of small strokes. The gradual loss of working (short term) memory is often the first symptom of the disease to be noticed. Caring for people with dementia will be a major challenge in all countries of the world in the 21st century. The number of people with dementia is projected to increase due to demographic changes which are taking place worldwide, with the number of older people proportional to the younger population growing steeply. The percentage of the population with dementia increases from the age of 65 to reach 25% of people over 85 [1]. The loss of working memory in elderly people as a result of dementia is a very serious problem for the person and for their family and carers. Many social activities and interactions become increasingly difficult as the condition progresses, since they depend on short term memory for effective participation. As a result people with dementia can become socially isolated and deprived of the range and variety of social interactions that characterise everyday life for unimpaired people. This can have a profound effect on their mental wellbeing, and put severe strains on family and carers.

30

The potential of using computers as cognitive prostheses would seem to be well worth exploring. If a way could be found for a person with dementia to be able to interact with a helpful computer system, this may have a significant impact on the quality of their lives, and that of their carers. Systems might then be developed to assist the person with dementia in carrying out conversations with others, and to provide them with stimulation, entertainment, and other types of computer-based support. None of these interventions would be expected to offer an improvement in the dementia itself. But they would enable the person to maximise their positive skills, empower them, and restore to a degree their participation in social life. As a starting point in creating a range of systems which are usable by people with short-term memory deficits, we have developed a hypermedia conversation support system based on reminiscence that can be used by a person with dementia along with a carer. Our plans are to follow this work by devising ways in which the person with dementia could themselves take direct control of navigating through such a system.

2. THE DIFFICULTY OF COMMUNICATION WITH DEMENTIA Everyday conversation is fundamental to human social interaction. Through conversation we learn to talk, we are socialised, we make friends and form lifelong partnerships, and we establish ourselves in the social hierarchy. We in fact create our sense of identity and sense of self worth primarily through our conversational interactions with others [2].

Holding a conversation is very difficult without a correctly functioning working memory. People with dementia will often repeat what they have only just recently said, and can give the appearance of not listening to what others are saying. This can be disempowering for the person and distressing for relatives and friends. Ways of structuring conversations have been investigated, for instance with the use of specially prepared memory books that contain autobiographical material, daily schedule information, and problem resolution information. When such material is individually prepared, and when staff are trained in using it, communication improves between the carers and people with dementia [3]. Reminiscence sessions are also a useful way to structure interaction so as to maximise the positive contribution that can be made by a person with dementia. Although their working memory is impaired, the long term memory of a person with dementia is often still functioning even at the latter stages of the disease. By guiding and supporting the person to take advantage of long term memory they can be helped to take a more active part in conversations [4,5,6,7,8]. While very valuable, reminiscence activities involve prior planning and gathering of material, which is time consuming for busy

relatives and carers. Also the process of conducting these sessions entails a great deal of effort. Reminiscence sessions can be pleasurable and empowering for the person with dementia, but the experience for the carer is far from a relaxed natural interaction. One hypothesis of this research is that suitably presented hypermedia can be helpful in supporting people with dementia in taking part in conversations. An obvious advantage of a computer based system over using traditional materials is the bringing together of the various media into one easily accessible system. But we also wanted to explore ways in which the hypermedia could in some way mimic the way memory is used in conversations. It is inherently plausible that the associative links in memory [9,10] and in free-flowing conversation [11] could be roughly modelled with hypermedia links. We have developed a hypermedia-based system that takes this approach.

3. A SUPPORT FOR COMMUNICATION BASED ON HYPERMEDIA AND REMINISCENCE MATERIAL It is unrealistic to expect people with dementia to acquire new skills, as their working memory problems normally preclude this. For the current generation of older people, therefore, using a mouse to interact with a system ( or a keyboard, if the person has not been a typist) would not be suitable. A touchscreen, however, has proved to be a suitable input device for an application such as this. It is intuitive to use and does not require fine muscle control to operate. Previous work, by ourselves and others, has established that people with dementia can usually manage to make use of touchscreens. The direct sense of manipulation they provide seems to offer enough affordance that, with encouragement, they work very well even with people who have fairly severe dementia [12,13].

Because of potential vision problems with elderly people, and since the display was to be shared during a reminiscence session it was decided to use the largest available (21”) flat panel touchscreen. This size of screen was also chosen to create a more engaging experience than that offered by standard TV screens, but still staying on the table top, with a screen that suggested active use, and not just passive viewing of a large public screen on the wall. Sound output was provided using a standard computer sound card together with speakers. An iterative design approach was used to develop the first prototype. Forty people with dementia were identified who expressed an interest in helping us to design and evaluate the system. Thirty relatives and carers also agreed to take part as advisors and evaluators for the project. As a first step, we asked the group to comment on suitable content for the system. Ideas for themes the system might include (such as national events, local industries, street life, celebrations) were suggested by means of high quality photographs. Using the photographs as prompts, discussions were held with people with dementia and

31

their carers to help decide on the best choices for content for the system. The system was developed using Macromedia Director. Director was chosen because it is a cross-platform development package that allows rapid development of complex multimedia systems. It also has its own programming language which allows it to be interactive and to be connected to a database. The completed system was given the name CIRCA, standing for Computer Interactive Reminiscence and Conversation Aid. 3.1 Description of the interface Consultation with experts on dementia had indicated that the interface would need to be as simple as possible while still appearing attractive and encouraging interaction. One problem that people with dementia have is an inability to cope with too

many items that compel attention. In this state the user often will focus on one item and stay with it, not being able to scan easily over the other possibilities. To cope with this, we designed the interface so that the background and navigation features are in muted colours. In most cases only one item of interest is shown at a time. In this way, even a black and white photograph stands out clearly as the point at which the user's attention should be directed. In order to keep the navigation process as simple as possible It was decided to have just three themes to choose from, and for each theme to have there media types that could be chosen. Each theme is associated with a different colour scheme. When a theme is selected the background colour and colour of all the buttons are changed to reflect the selected theme. When the carer or the person with dementia selects one of the three themes, they are then offered a choice of photographs, video or music.

Figure 1. Screen shot of the Photographs section in the Dundee Life theme. A title and brief description are given as a help to the carer, who might not recognise what is shown. The menu bars of the touchscreen are at the bottom for easier reaching for older users. Visible are the three themes, the three media choices, and simple navigation buttons.

32

3.2 Media content of the system Based on feedback from our user population, the three themes chosen for the prototype were; Recreation, Entertainment and Dundee Life. The Recreation theme focused on UK material with an emphasis on Dundee and Scotland. The Entertainment theme contained both US (movies and music) and UK material (TV and music). The Dundee Life theme focused on local material. In total the system contained 80 photographs, 10 videos, and 23 music items. Photographs are often used in reminiscence sessions as they are relatively easy to source. There are many photograph archives and library available, covering many themes and time periods. It was therefore relatively easy to add a large number of photographs to the prototype. As it is usual to view photographs in short sequences, the photographs within

each theme are grouped into categories. Two navigating buttons, “Next picture” and “Previous picture” are provided for stepping through the photograph sequences (see Figure 1). The system allows the users to select and play video clips related to the theme selected. People with dementia may not be able to follow a long video clip because of their working memory problems. The video clips have a short duration for this reason and also because they are intended to act as conversation prompts and not be too immersive. The system allows the users to select and play music related to the theme selected. Rather than simply having a static screen while the music plays, a display is presented of the type of device that particular music would normally be coming from. A record player, radio or tape recorder is displayed depending on the theme selected, as shown in Figure 2. This representation of the music producing device can also act as a conversation prompt in itself.

Figure 2. Music is accessed through representations of the type of player that might have produced it in times past. The record player and the reel-to-reel tape recorder are animated to give the impression of working while the song is played.

33

4. EVALUATING THE PROTOTYPE An evaluation was performed to determine if people with dementia were able to make sense of the system, and understand its use well enough to direct a carer in operating it, or to operate it themselves. In the evaluation sessions, the system was compared with traditional reminiscence sessions. In a traditional session, the carer takes the responsibility for guiding the session, and at all points compensating for the short-term memory problems of the person with dementia. We designed the CIRCA system so that it hopefully would be able to take over this role of supporting the cognitive abilities of the person with dementia, freeing the carer to take part in the session more naturally. Eighteen people with dementia and their carers took part. Each person with dementia and their carer undertook a 20 minute reminiscence session: nine using the prototype, and nine using traditional reminiscence methods. As expected, the carers did most of the direct operation of the touch screen, but they were often prompted and directed by the person with dementia. With encouragement a number of people with dementia also made direct use of the touch screen to make selections. The carers were asked to complete a questionnaire at the end of each session. All the sessions were video recorded, and all the items accessed using the prototype were logged. A protocol was developed for observing and recording behaviour during the sessions, to identify the degree of participation, engagement and enjoyment of the experience. Eighteen people with dementia took part, 13 women and 5 men. Nine participated in CIRCA sessions and nine participated in traditional reminiscence sessions). The age range was 65-95 yrs. The Mini Mental State Examination (MMSE) is a rough measure of degree of severity of dementia ranging down from 30 (no measured cognitive impairment) to 0. The MMSE range was from 2 to 23. Profile of participants : Carers (N = 12) CIRCA sessions Age Mean 81.88 Standard Deviation 7.93 Range 65-95 MMSE Mean 14.88 Standard Deviation 6.14 Range 2-22 Traditional sessions Age Mean 82.00 Standard Deviation 6.17

Range 71-89 MMSE Mean 16.00 Standard Deviation 6.71 Range 5-23 A between subjects protocol was used (Mann Whitney test). All the people with dementia were able to make sense of the material the system was presenting, and showed an understanding of how it worked. Some spontaneously commented several times that they were enjoying using CIRCA. One person said, "It takes you back and refreshes your memory". Other spontaneous comments were : "This covers everything", "Good thing, this", "It's good to remember things", and "That's entertainment!'" One person said she certainly would like to use CIRCA again. She said she thoroughly enjoyed it and found it very interesting and "something new". Another person commented that she enjoyed using the system herself.

In comparing CIRCA with using traditional reminiscence aids, we found that the person with dementia was offered a choice of reminiscence subject/materials more often when using CIRCA (U = 1.50, p<0.001). We also found that the person with dementia chose reminiscence subjects/materials more often when prompted when using CIRCA (U = 3.60, p<0.001). The traditional sessions were characteristically a series of one question from the carer followed by one response from the person with dementia. The CIRCA sessions were more of a conversation, with each person contributing an equal amount, and control of the direction of the conversation being shared. The carers asked more direct questions when using traditional reminiscence methods (U = 5.00, p=0.01).

5.XXCONCLUSIONS CIRCA proved to be usable by people with dementia. Their problems with short-term memory did not stop them from understanding the material presented to them, and working out how to navigate to other material, either by directing the carer or using the touchscreen themselves. CIRCA facilitated a more equal participation in communication for the person with dementia, as compared with traditional reminiscence. In offering a richness of choice, the carer could hand over more of the control of the session to the person with dementia. By choosing reminiscence items, the person with dementia guided the session. In this way, people with dementia could set the agenda and pace of the session. This was enhanced by the fact that people at earlier stages of dementia could use the touchscreen themselves.

By providing an engaging focus of joint attention, CIRCA allowed the carer and the person with dementia to explore a wide range of topics provided in various media. This created subjects of conversation for both parties which could be changed easily and quickly. Moreover, periods of silence which

34

might create awkwardness in traditional reminiscence sessions were filled continuously by CIRCA. What was supplied was a third element in the interaction that at the same time relieved the carer of the sole responsibility for directing the session and provided stimulating possibilities for conversation. The person with dementia could use CIRCA in different ways. They could tell a story or simply enjoy the experience of exploring the presentations with another person. By ‘filling in the gaps’ in a typical conversation, CIRCA encouraged a failure-free social interaction. CIRCA encouraged the person with dementia to take a more active part in creating the reminiscence session, and carers as a result asked fewer direct questions in CIRCA sessions, compared with traditional reminiscence work. Several carers mentioned that they felt less pressure to maintain the conversation when using CIRCA. Moreover, in contrast with traditional reminiscence sessions, time spent using CIRCA did not need to be planned by the carer. Thus, the experience of reminiscence could be more relaxed for both parties. This project has been a multidisciplinary effort, with participants from software engineering, psychology and design. We feel this is an essential aspect of this work. One way this is evidenced is in the attractive and interesting look that the system has, which has been produced by an experienced team of designers. The look and feel of the system is polished, professional and inviting. We are convinced that the look of the system invites engagement in a way that a worthy but less aesthetically pleasing interface might not. This relates to what some designers have called the 'aesthetic-usability effect'. Aesthetic designs are perceived as easier to use than less-aesthetic designs [14,15,16]. And since the aim of this system is to engaged the continued attention and involvement of people for whom this has proved difficult, this is not just a matter of a desirable extra feature.

6. FURTHER WORK Following the lessons learned from the evaluation of the system, improvements and additions are being made to the system as follows : (1) The system will be made modular with a separation of form and content, to allow the creation of new material easily. This will allow for localisation of the material, and also for the potential to include personal material, where this is suitable. (2) The system will be extended to allow for a larger number of themes and individual items to be available, with a random choice of three of these being made by the system for each session, in order to keep the continued interest of relatives and carers. Similarly, the number of potential media items will be increased, with a random selection being made among these

each time the system is used. This was felt to be particularly important given the fact that the equal control over the interaction was dependent on both conversation partners sharing a unique experience, so that the carer did not need to feel ‘in charge’ of the conversation. Plans are also underway to extend what we have learned about interfaces that are usable by people with dementia to develop an interactive entertainment system which they can use on their own, without a carer to assist them. Whereas the CIRCA system was developed as a conversation support, and not as a provider of entertainment, we would like to develop a range of systems that can engage and hold the involvement of people with short-term memory deficits in a way which is stimulating and entertaining. CIRCA has shown that technology can help restore a degree of lost social functioning. What we would now like to do is to develop ways of helping people with dementia to enjoy themselves without the need for the constant presence of a carer or relative. This could be significant help, since one of the principle pressures on families and care faculties is the inability of people with dementia to occupy themselves in satisfying activities without the continual involvement and guidance of another person.

7.XXACKNOWLEDGEMENTS We would like to acknowledge the essential participation in this work of the people with dementia, their relatives, and carers. Alzheimer Scotland and Dundee City Council Social Work Department are active and supportive partners in the project. REFERENCES [1] Jorm, A.F., Korten, A.E., Henderson, A.S. (1987) The prevalence of dementia : a quantitative integration of the literature. Acta Psychiatrica Scandinavica. 76, pp 465-479.

[2] Beattie, G. (1983) Talk: An Analysis of Speech and Non-verbal Communication. Milton Keynes : Open University Press.

[3] Bourgeois, M., Dijkstra, K., Burgio, L., Allen-Burge, R. (2001) Memory aids as an augmentative and alternative communication strategy for nursing home residents with dementia. Augmentative and Alternative Communication. Vol 17, pp 196-210.

[4] Baines, S., Saxby, P., Ehlert, K. (1987) Reality orientation and reminiscence therapy: a controlled cross over study of elderly confused people. British Journal of Psychiatry, 151, pp 222-231.

[5] Feil, Naomi (1993) The Validation Breakthrough. Maryland: Health Professions Press.

[6] Finnema, E., Dröes, R-M, Ribbe, M., Van Tilburg, W. (1999) The effects of emotion-oriented approaches in the care for persons suffering from dementia: a review of the literature. International Journal of Geriatric Psychiatry, Vol 15, pp 141-161.

35

[7] Woods, R. T. (1994) Management of memory impairment in older people with dementia. International Review of Psychiatry, Vol 6, pp 153-161.

[8] Woods, R. T (1999) Psychological therapies in dementia. In R. T. Woods (Ed.) Psychological Problems of Ageing. Chichester: Wiley.

[9] Conway, M. A., & Pleydell-Pearce, C. W. (2000). The

construction of autobiographical memories in the self-memory system. Psychological Review, 107 (2), pp 261-288.

[10] Brown, N. R., & Schopflocher, D. (1998). Cueing, event

clusters and the temporal distribution of autobiographical memories. App.Cog. Psych., 12 (4), pp 305-319.

[11] Jefferson, G. (1984) On stepwise transition from talk about

trouble to inappropriately next-positioned matters, in Atkins, J. and J. Heritage, (Eds.) Structures of Social Action -- Studies In Conversation Analysis. London: Cambridge University Press. pp 191-222.

[12] Alm, N., Dobinson, L., Massie, P., Hewines, I. (2001)

Computers as cognitive assistants for elderly people. In Proceedings of Human-Computer Interaction - INTERACT’01. (Tokyo, July 2001). Michitaka Hirose, Ed. Amsterdam: IOS Press. pp 692-693

[13] Alm, N., Ellis, M., Astell, A., Dye, R., Gowans, G., Campbell, J. (2004) A cognitive prosthesis and communication support for people with dementia. Neuropsychological Rehabilitation. Vol 14 (1/2), pp 117-134.

[14] Lidwell, W., Holden, K., Butler, J. (2003) Universal Principles of Design. Gloucester, Massachusetts : Rockport Publishers.

[15] Kurosu, M., Kashimura, K. (1995) Apparent usability vs. inherent usability : Experimental analysis on the determinants of the apparent usability. Proceedings of CHI95. pp 292-293.

[16] Norman, D. A. (2002). Emotion and design: Attractive things work better. Interactions Magazine, ix (4), pp 36-42.

36

Promoting Virtual Social Interaction with Older People

Peter Bagnall, Guy Dewsbury, Victor Onditi, Ian Sommerville,

Computing Department, InfoLab21 South Drive, Lancaster University

Lancaster, LA1 4WA +44 1524 510352

bagnall | dewsbury | onditi | is @comp.lancs.ac.uk

ABSTRACT This paper charts the development of a software platform and applications designed specifically for older people to promote communication and sociability. Through this we attempt to help alleviate isolation or loneliness by providing appropriate informal communication and virtual interaction. The software has been developed using a person-centred design approach that specifically favours simplicity and flexibility of use over functionality and power. Early participation from the older people was in the form of adapted cultural probes [8]. As the design progressed participation took the form of iterative criticism of the design, and informal user tests with older people. The system allows older people to easily discover who else is currently using the system, and therefore potentially available to chat with, interact with or play games with. The games also allow voice chat so players can socialise while playing. The application of this software has the possible use of encouraging intergenerational communication.

General Terms Person-Centred design, Experimentation, Human Factors.

Keywords Person-Centred Design, Older People, Cultural Probes

1. INTRODUCTION An increasing concern with modern society is that the rise in technology has empowered and supported those people who have been able to take advantage of the available technologies, whilst leaving the remainder technologically alienated and isolated [1]. One group for whom this is certainly a possibility is older adults who are increasing in proportion as well as becoming more isolated through family breakdowns [4] [10]. This paper reports on the work being undertaken at Computing Department at Lancaster University in the UK where some software is being developed to enable seniors to communicate informally with each other for example their friends and families through a computer without the complexity associated with traditional computing platforms. Our research originated as part of the DIRC (www.dirc.org.uk) initiative, a multi-institutional investigation funded by the EPSRC on dependability in computing systems and continues through Microsoft Research funding. On this project, we worked with a number of older people in the community to design a technology system that would be suitable for them. In the process of this

investigation, our original conceptions of designing some form of smart or ubiquitous home was found to be misguided by the participants whose concerns were mundane, relating to undertaking everyday tasks and avoiding isolation and loneliness. Our investigations found that, even in housing specifically designed for elders with available social facilities, older people felt lonely and isolated. They used the artefacts such as the television as a means of avoiding the consequence of their isolated experience, the television taking the place of the lost partner, in a silent static form. As a response to the isolation that many of the participants were experiencing, we started to investigate how we could make use of computer-mediated communications as a way to facilitate non-intrusive, informal communications between older people. Critical to this was the development of a system that was both usable by people suffering from many of the normal infirmities of old age and which did not look like a conventional computer with all of the negative preconceptions that this entailed.

2. PROBLEM IDENTIFICATION “Unfortunately, our common metaphors for computer interaction lead us away from the invisible tool, and towards making the tool the center of attention.” [12]

Lancaster University’s Computing Department is renowned for having a multidisciplinary staff membership consisting of computer experts and social scientists. This multidisciplinary mix has proven extremely successful in a number of software developments and has recently been focused on developing assistive technology and ‘smart homes’ for senior and disabled people as well as ‘CATCH - a Compendium of Assistive Technology Checklists for the Home’, for professionals to determine technology requirements for supporting independence in the home (www.smartthinking.ukideas.com/CATCH.html). The research team undertook ethnographic field studies with seniors in their own homes over a period of two years. A number of new methods were piloted and adapted to achieve this task including adapting cultural probes [10] [5], technology tours [2] [11], observations, qualitative interviews and open questionnaires [7]. The researchers have worked with a many seniors throughout the UK and allowed them to determine their own technology requirements. As a result of this fieldwork, it was revealed that the seniors’ were less concerned with the introduction and use of high-end technology, such as smart homes, and more concerned with everyday mundane routines and activities such as cleaning their homes, meeting other people and being isolated for much of the time. Participants explained this isolation was due to family

37

and friends not visiting or having passed away. Many of the older adults found that at weekends they could be alone all weekend. On further investigation this time period expanded considerably such that some older people could spend up to a week without seeing another person socially. Many participants had circumnavigated this by employing people to come in to their homes and clean or do other jobs, which provided some level of company at a high price, but many participants were still, at times, chronically lonely.

3. DERIVING OUR SOLUTION There was no obvious technology solution to preventing or alleviating isolation without bringing more people into the equation. Loneliness and isolation cannot be solved without contact with other people. Yet for many older people in our study, the contact with other people was a source of concern and worry and hence the cycle was extended. Over many months of work with the older participants, we determined that an informal communication system was what the older people wanted. A system that would indicate when other friends or family were available to be contacted was what the older people required. The system needed to show who was on the network but this should not place any obligation on the user to make contact if they did not want to. Contact with others was therefore voluntary. Our participants informed us that often knowing that other people are there and available is enough to console the isolated person and make them feel less alone. Similarly we found in the fieldwork that many seniors were reluctant to make overt contact even with neighbours as they did not wish to cause ‘bother’ to them and did not wish for visitors themselves as this meant a lot of preparation beforehand. Social communications, therefore, needed to be informal, and not require the users to enter into more disclosure than they were happy to. For this reason cameras were ruled out, as the seniors did not want to be seen by others although they would like to see the other people themselves. Avatars were considered as a possibility to replace real pictures but this becomes over complex and possibly of little use to the older people themselves, where as a still picture of the person they are communicating with does help them feel they are talking to a real person. Our initial intention was to build software package that could be used on a standard PC but we soon discovered that there are numerous problems with traditional desktop or laptop computers for seniors. Firstly and most important, the older people in our study were by and large computer illiterate and feared anything that resembled a computer. In practical terms of hardware, desktop computers require significant amounts of space, often at a premium. Input via a keyboard assumes either some keyboard skills or great patience, and mouse input requires learning a coordination skill, and is awkward for many older people even once they have learnt. Interaction idioms such as double clicking are mechanically difficult for many older people. Laptops suffer from most of these problems, but exacerbate them by having a smaller keyboard, and even more difficult pointing devices. Modern operating systems are highly complex, and require substantial training or experience to use effectively. For seniors who have limited energy to devote to learning new skills this is

daunting and inappropriate. What’s more, most of the functionality offered by such systems is not relevant to seniors.

4. FROM REQUIREMENTS TO DESIGN The hardware and software needed to be simplified. For the hardware a Tablet PC form factor was chosen, specifically an NEC T400, a slate style tablet, which has no keyboard (although a keyboard can be connected via USB). This particular machine is relatively light, at 1kg (2.2lb) and has a reasonable sized screen, which is important to be able to present information at a useful size. As with all Tablets, it offers stylus-based input. The styluses (figure 1) that are provided with the tablet come in two sizes. One is about the size of a normal biro, the other much smaller. Most older people were much more comfortable using the larger stylus, but so far the selection of available styluses is less than ideal for older people.

Figure 1

The actual design was informed in large part by observations of older people using computers and prototypes at various stages. From this a number of simple design guidelines were laid down. Having seen that they were disinclined to explore an interface in the way experienced computer users do, one principle was that functionality should be clearly visible, not hidden inside GUI widgets such as dropdowns or in dialog boxes. To keep the complexity to a minimum only a small set of widgets are being used. Buttons, which can be tapped with the stylus, have been tested with older people, and cause few problems. Initially many older people tried, very logically, to tap the button with their finger. Having been told this didn’t work they quickly became used to using the stylus instead. Ideally the hardware would support using a finger for tapping buttons, but pragmatically the problem appears to be solvable by documentation and training since it was very readily learnt. Writing with a stylus is also relatively familiar, although making it obvious where they on the screen they are ‘allowed’ to write is an important design consideration. Using a consistent visual style for regions that can be written on is important, but it appears that older people are less adept at noticing these relatively subtle cues in the interface. Again, training and documentation may help, but the design needs to assume that these subtle visual cues are not understood, so redundant mechanisms may be required. The last interaction style that is allowed is dragging. Mechanically this is identical to drawing a line. While we don’t have as much experience with seniors using a drag interaction there have, as yet, been no cases of seniors having trouble with this idiom. However, the objects to be dragged must be very clearly indicated. Another guideline related to the labelling of user interface elements. Fiske et al suggest [8] that since icons are rarely well understood, textual labels are better. Observations of older users confirm this finding. This lead to a design guideline that all buttons must have a textual label. Iconic labels are optional.

38

This simple set of guidelines is quite restrictive, but this has proved to be an asset rather than a hindrance, since it serves to concentrate the design on those functions that are of greatest importance.

4.1 Platform Design As one of the concerns being addressed by the designs was the reluctance to intrude, a presence awareness interface, similar in principle to a Microsoft Messenger Buddy List was developed for the system. The “Chooser” (figure 2), by which we refer to the main screen, displays other users, indicating whether they are online and allows each user of the system to write a short phrase to convey their emotional state or whatever information they wish other users to know. This means users can see at a glance who is available to connect with. Alongside the “buddy list” is a list of potential activities. The exact set of activities depends on which programs have been installed on the device. In accordance with the established guidelines, the buttons for the activities are large, with large text labels, to make them easy to read and easy to tap with the stylus. This means that there is a limit to the number of activities that can be accommodated on the interface, but again this in fact is a useful limitation, since it helps provide an incentive to keep the number of activities (and hence the level of complexity) down. It was also determined by the seniors that if

they found that they had reached a point where they required more functionality than the platform could provide then it would be time to start using the full functionality that is provided by Microsoft Windows. When seniors use the chooser to select an activity, in most cases they will be required to select another user to participate in the activity. To begin an activity the user is required to make a minimum of taps on the screen. Tapping an activity selects it, tapping another user selects them. Once selected tapping ‘start’ activates and initiates the activity. Tapping a selected user or activity deselects it, which allows for a straightforward way to correct errors.

4.2 Chat Design The design for Chat (Figure 3), although inspired to some extent by instant messenger (IM) applications, deviates from this is a few important ways. Firstly there is an “invitation” step, on entering the chat activity the user writes a short invitation to the person they with to chat to or determine if the user is ready for communication. This extra step is to allow a way for older users to politely decline a conversation through not accepting the invitation. This step is essential as a major barrier to communication is a sense of not wanting to intrude on another person. If the chat activity simply launched directly into chat,

Figure 2 – The Chooser User Interface

39

without giving the other person a way of declining it is more likely that this would be considered intrusive, and therefore the same barriers might reappear, in a similar way that a telephone call can be annoying when it is from someone you do not wish to speak with. Once an invitation has been written and sent, the receiver sees the invitation on screen and can either accept it, in which case the chat begins, or decline, in which case they can if they wish write a brief message saying why, again to allow polite behaviour. Invitations deliberately do not time-out. It was considered better to let the user decide when to stop waiting rather than impose some arbitrary time limit. This is especially true for older people where it may take a few minutes for them to respond if they are away from the tablet when an invitation arrives. The actual chat is essentially a shared white-boarding system. This has some advantages over conventional IM. It avoids the threading problem within a conversation, of utterances getting out of order, because the position of writing indicates clearly what relates to what. It is also more powerful in that users can draw, scribble, and even play simple games. Of course screen space is limited, and this solved by using a paging system. One user can suggest that a new page is created. When the other user agrees both move to the new page together. This rather involved interaction ensures that both parties are aware of new pages being

created, so it avoids one person writing on one page, while the other writes on a different page. The users can look back over old pages at any time by using paging buttons to move back and forth. Scrolling was avoided because it is mechanically awkward, conceptually somewhat confusing (novices often find it upside-down), and it is open to the failure of having the two participants looking at entirely different parts of the system, and therefore not communicating. It would also introduce another interaction style, whereas the paging model can be achieved with button, a widget that is already in use in many other parts of the system.

5. GAMES AS A COMMUNICATIONS FACILITATOR There are many other applications that we are developing for the platform including email / Internet access and photograph sharing options, but we turn to consider the development of a card game application - FreeCard. As an icebreaker and a common shared experience games can support a means of obviating isolation. It is not required that one needs to know or like a person to play a virtual game with them and get enormous pleasure from doing so. Chris Crawford, remarked in his book "The Art of Computer Game Design" that:

Figure 3 – The Chat User Interface

40

"Games are frequently used (especially by adults) as social lubricants. The game itself is of minor importance to the players; its real significance is its function as a focus around which an evening of socializing will be built. Card games and some light board games serve this function.” [6]

Traditional games provide a social environment where conversations can arise. The game provides a reason for meeting, and an excuse for conversation. Between friends, conversation around a game table will generally not relate to the game itself, although from time to time undoubtedly the game will be referred to. That social experience of gathering with friends is what the game environment aims to replicate. When strangers meet to play a game, conversation may initially revolve more around the game, but it is less certain that this design is so well suited towards strangers playing meeting and playing games since. The game environment has been designed, for now, to support card games, as these provide a wide range of games with a wide audience. However, when exploring which games older people played it was found that, amongst the participants, there was a great variation in the games played, and for those games that were known to many, there were significant variations in the rules. The variation of rules and the general desire to keep the design simple lead to the concept of a games environment that is agnostic towards rules. In effect the game environment, ideally, should behave as little more than a virtual table. As with real cards, the rules are agreed between the players, and enforced by the players. This is a departure from typical computer-mediated games, where the computer often acts a referee. The advantages of this are considerable. It allows players to play a wider range of games, since they decide on the rules. It simplifies the game interface, since the system doesn’t have to enforce rules and it means the selection of games is simpler, since there is no need to select which rules are being used. It also makes it possible for players to cheat, which in the context of a grandparent playing a game with

a grandchild may actually be an advantage too! Since the system will not referee the game, the need for players to communicate increases. They need to prompt people when it’s their turn, to enforce rules, and to decide which game will be

played. But since the purpose of the system is to increase sociability, increasing the need for conversation is actually a benefit, not an inconvenience. The ideal medium for this communications channel is clearly voice. It is rapid, simple, and doesn’t clutter the virtual game-table interface as a visual communication medium (such as the chat activity) would. The tablet hardware has both speakers and a microphone, so it is possible to use voice communication without additional hardware. It is possible that older people will find this somewhat unnatural, since they are not generally familiar with hands-free telephony style systems. For this reason, and to gain improved audio quality it may be better to provide a headset, although this then becomes more cumbersome. There is also the issue of compatibility with hearing aids, so it is very likely that a range of solutions are required to cater for a range of needs and preferences. Currently we are exploring the use of Skype (www.skype.com) for supporting a multiple-player audio conference. Skype’s audio conferencing facility is currently limited to a maximum of five participants. As a consequence, FreeCard can be played by a maximum of five players even though theoretically, more players could be supported. Our practical experience using the facility is that the sound quality degenerates dramatically with an increase in participants. The quality is very impressive for two players.

The game platform automatically instigates the creation of a Skype session without any additional work on the part of the users. From the users perspective audio conferencing is simply part of the games activity.

The user interface design of FreeCard (figure 4) aims to be reminiscent of a game table. One of the aspects of this is the relative positions of the various players. With a real card game the players hand is held between the player and the table, and so with the interface the natural place for the players hand is at the bottom of the screen, closest to the player when the tablet is resting in the lap. Since in many games where players sit relative to one another

is significant it is important that the interface makes it possible for people to ‘sit’ where they choose. For example, it often defines the order of play. To keep the interface as simple as possible however, it was decided to not allow players to move ‘seats’ once a session had started. The problem is that if players are allowed to

Figure 4 - FreeCard Game Prototype

41

move ‘seat’ midway through a session, and a player always appears at the bottom of the screen, moving from one seat to another would mean the table would need to be rotated. It is likely that this would be somewhat confusing since the player would appear to return to where they were originally. Not allowing this is the simplest way to avoid the issue, although other solutions may be tried in the future.

The participants also expressed views on how their hand should be laid out. They preferred that cards should be laid out with their full face showing, rather than allowing a card to partly obscure another. It is possible that this reflects a difficulty with reading the smaller numbers in the corner of the cards. Future card designs will aim to address this issue with alternative card designs, as well as layouts.

6. CONCLUSION It is difficult to draw firm conclusions at this time, although we know the system works and that older people in the UK are enjoying using it. We are currently putting Broadband connections in to participant’s homes in order to undertake longer and more rigorous studies of its efficacy and usage. A clear indication at this stage is that the systems will have much wider applications that its initial design specification. For this reason we will be developing an installation package CD ROM to be given to friends and relatives of the users, so that they can use the system. This will take considerable work as this will require allowing keyboard and mouse support and usage on a standard PC. Moreover, the implementation of the platform and the activities that it supports separate the user interface from most of the underlying engineering [3]. The user interface that is being deployed includes some configurability for colour schemes but future plans include allowing UIs that are more suited to desktop or laptop computers, without stylus capability. At the moment the software can be used with Wacom style tablets without modification. We also consider that this disconnection of the UI is advantageous as we are able to make special UIs for people with specific eye conditions, or people who speak other languages. It is also envisaged that part of the system could be implemented on mobile devices, such as mobile phones or PDAs, which would allow much greater contact between young and old generations. One possibility here includes allowing younger users to take photographs with a mobile phone and upload it directly to a senior relative. At this stage we can see many ways in which this devise can be further developed to support communication between older people, but we also consider the software has considerable applicability for use with children, people with learning difficulties and disabled people who have limited functionality.

7. ACKNOWLEDGMENTS The authors would like to thank Age Concern Carnforth and MHA Care Group for their assistance and support in this project. This work was partially funded in the UK by the Engineering and Physical Sciences Research Council as part of the DIRC Interdisciplinary Research Collaboration and partially funded by

Microsoft Research as part of their ‘Create, Play and Learn’ collaborative research programme.

8. REFERENCES [1] Richard Adler -

http://www.seniornet.org/php/default.php?PageID=6694 [2] Baillie, L. and Petersen, M. G. (2001) Emerging Themes in

Designing Household Environments. In Proceedings of the OIKOS2001 Workshop, Aarhus University Press, pp. 44-46.

[3] Bagnall P, Dewsbury G and Sommerville I, (2004) ‘Easy For Everyone: Using components to offer specialised interfaces for software’ in Baxter G and Dewsbury G (Eds), (2004) HEAT 2004: Proceedings of the First “HEAT: The Home and Electronic Assistive Technology Workshop”, Computing Department, Lancaster University.

[4] Batty D (2004) Older people face a lonely future, thinktank warns, Guardian Politics, 30June, 2004, http://politics.guardian.co.uk/thinktanks/story/0,10538,1250281,00.html

[5] Cheverst K, Clarke K, Dewsbury G, Hemmings T, Hughes J and Rouncefield M, (2003) 'Design With Care: Technology, Disability and the Home', in Harper R (Ed) Inside The Smart Home, Chapter 9, pp163-180, Springer-Verlag, London Ltd.

[6] Crawford C, (1982) The Art of Computer Game Design, http://www.vancouver.wsu.edu/fac/peabody/game-book/Coverpage.html

[7] Dewsbury G, Clarke K, Rouncefield M, and Sommerville I (2004) Depending on Digital Design: Extending Inclusivity, Housing Studies Vol 19, No 5, September 2004, ISSN 0267-3037

[8] Fiske, A., Rogers, A., Charness, N., Czaja, S., & Sharit, J. (2004). Designing for older adults: Principles and creative human factors approaches. Boca Raton, Florida, CRC Press.

[9] Gaver, W.H., Dunne, A. and Pacenti, E. Design: cultural probes, Interactions, vol. 6 (1), 1999, 21-29.

[10] Help The Aged, (2005) Social Isolation and Loneliness, Help the Aged Policy Statement 2005, available from http://policy.helptheaged.org.uk/NR/rdonlyres/e7nauzeva2yba5cjgr7nerhygd64mfr3yjf2gokljerpxglx4t2zixflnmlqbdhx2jkxrtux6g3oo3pqg2mzqkllnkb/index_social_isolation_loneliness_01_05.pdf

[11] Petersen, M. G. and Baillie, L. (2001) Methodologies for Designing Future Household Technologies. In Proceedings of the OIKOS2001 Workshop, Aarhus University Press, pp. 47-49.Petersen, M. G., Madsen, K. H. and Kjær, A. (2002) Usability of Everyday Technology – Emerging and Fading Opportunities. In ACM Transactions. on Computer-Human Interaction, Vol. 9, No. 2, June 2002, pp. 74-105

[12] Weiser M (1993) ‘The world is not a desktop computer’, http://www.ubiq.com/hypertext/weiser/ACMInteractions2.html

42

Talking with Individuals with Dementia on a Video Phone: A Preliminary Study for Networked Interaction Therapy

Kiyoshi Yasuda Chiba Rosai Hospital

Chiba, Japan

[email protected]

Kazuhiro Kuwabara ATR

Kyoto, Japan

[email protected]

Noriaki Kuwahara ATR

Kyoto, Japan

[email protected]

Shinji Abe ATR

Kyoto, Japan

[email protected]

Nobuji Tetsutani Tokyo Denki University

Saitama, Japan

[email protected]

ABSTRACT The purpose of networked interaction therapy is to provide individuals with dementia various types of intervention using a video phone, appropriate video contents, and scheduling prompts. As part of the preliminary study to investigate the effectiveness of a video phone, we have conducted an experiment to compare how long individuals with dementia can talk with a partner on a video phone and in a face-to-face session. In the experiment, nine individuals are requested to reply to 20 questions asked by a partner. The results reveal that there is little difference in the total response time to these questions between the two sessions. This suggests that the video phone conversation can potentially have the same effects as face-to-face conversion from the viewpoint of dementia care. In addition, we discuss the use of several kinds of contents that can augment a video phone session.

Categories and Subject Descriptors J.3 [LIFE AND MEDICAL SCIENCES]: Health

General Terms Human Factors

Keywords Dementia, intervention, video phone, talking, conversation

1. INTRODUCTION The onset of dementia usually begins with mild forgetfulness and often develops into a variety of behavioral disturbances such as wandering, agitation, illusion, and incontinence. Because of these behavioral disturbances, caring for dementia sufferers is a difficult and frustrating task, especially for home caregivers [2]. Since caregivers bear a heavy burden, they are in need of some regular respite time [7]. In addition to pharmacotherapy, several

social services are available to relieve the stress of home caregivers. Various therapeutic approaches have also been undertaken such as memory training, music therapy, behavioral therapy, and reminiscence therapy [3].

Nowadays the use of computer-based intervention is also being investigated [1, 5]. We are currently proposing a networked interaction therapy intended to provide various types of communicative intervention such as talking on video phones, applying appropriate contents, and conducting daily-life management through a TV monitor [5]. This system uses a set-top box with a camera and microphone for monitoring the user’s status and controls various types of intervention. Additionally, an air cannon is also considered for stimulating the olfactory sense. If these forms of intervention can increase individuals’ quality of life (QOL) and reduce behavioral disturbances, caregivers will be able to gains some respite while individuals with dementia are engaged in networked interaction therapy.

Communication, especially verbal conversation, is an easy, common, and enjoyable activity for most of us. Dementia sufferers, however, tend to be alone and poorly informed, with few chances to converse. One of the most important forms of intervention in networked interaction therapy is to provide them with talking partners on the Internet, such as friends, volunteers, and family members living in remote locations. A simple interface would thus enable individuals with dementia to talk with partners on the video phone whenever they want.

However, there are no studies on how long individuals with dementia can talk with a partner on the video phone, in comparison with the face to face settings. Since the concept of networked interaction therapy is based on the use of video phone-based intervention, it is important to verify that the dementia sufferers can converse sufficiently for a certain duration on the video phone.

In this study, individuals with dementia were requested to reply to 20 prepared questions from a partner in the video phone session and in a face-to-face session. Total response times to these questions are compared between the two sessions. Furthermore, we plan to deliver several contents during the video phone session in order to augment talking on the video phone. In the latter part of this paper, we introduce several such contents that we consider promising for video phone sessions.

43

2. VIDEO PHONE EXPERIMENT 2.1 Subjects Nine individuals with mild and moderate dementia, whose ability to converse still remains were selected as subjects. They were diagnosed to be suffering dementia by their physician (neurologist) in charge.

Table 1. Profiles of five subjects

Subjects Gender Age MMSE Etiology

OI M 81 10 Alz, Infar

NK F 71 23 Levy, Park

YM M 76 23 Alz, Infar

KM M 87 15 Alz, Infar

SS F 83 17 Alz

IS F 76 25 Alz

SN F 65 23 Alz

FS F 73 20 Alz

KM F 79 17 Alz, Infar

Note: MMSE: score of the Mini Mental State Examination, Alz: Alzheimer's disease, Infar: infarction, Levy: Levy-type dementia, Park: Parkinson’s disease.

2.2 Experimental Setup We used commercially available TV conferencing software that runs on a personal computer (PC)1. We prepared PCs, 19-inch monitor, USB camera, microphones and speakers for the subjects’ use and for their partner. The face image approximately the same size as the real face of each person was displayed on the monitor. PCs were connected by a LAN at a speed of 100 Mbps.

2.3 Questions Twenty questions were prepared for this experiment. All questions were easy, common, and concerned with each subject’s younger days, such as their hometown, family members, favorite foods, and primary school activities. The talking partner in this experiment was a speech therapist who was in charge of the subjects’ evaluation and training. He asked them the questions in a slow, relaxed, and gentle tone. One subject (OI) with hearing difficulty wore earphones.

2.4 Procedures The experiment was carried out in speech therapy rooms of a hospital that the nine subjects visited. The AB or BA design was applied, where A was the video phone session, and B was the face-to-face session. The AB design was applied to five subjects, while the BA design was applied to the other four. The intervals between A and B, or B and A, were varied from a week to two months. The partner asked each subject the 20 prepared questions. After asking each question, he only listened to the subject’s reply passively, asking no additional questions, just responding with some subtle, customized gesture (e.g. nodding, smiling) or vague

1 http://www.polycom.com/common/pw_cmp_updateDocKeywor

ds/0,1687,4550,00.pdf

utterances such as “Oh, I see”. If the subject uttered nothing for five seconds and no additional comments were forthcoming, the partner proceeded to the next question. When the subject tried to utter further comments, the partner waited for a maximum of ten more seconds. Two minutes were allocated to each question as the maximum responding time; accordingly, the total responding time was 40 minutes at maximum longest. When a subject took more than two minutes on a question, the partner interrupted the subject from talking and proceeded to the next question. Their faces and gestures as well as utterances were recorded by a digital video camera. Before the experiment, the partner explained its aim and procedure to each subject and their caregiver, and obtained their consent. After the partner chatted with the subjects on common opening topics such as the weather and their health condition about for three minutes, he started to ask the prepared questions, beginning with “Now, please let me hear about your past.” We compared the total response time between the video phone session and the face-to-face session.

2.5 Results The experimental hypothesis was that there is no difference in the total response time between the video phone session and the face-to-face session. The total response time of each subject for the two sessions was calculated, with the results presented graphically in Fig. 1.

The average response time was 16.5 minutes in the video phone sessions but 14.1 minutes in the face-to-face sessions. These results demonstrate there was no significant difference in the total response time between the two sessions.

3. AUGMENTING WITH VIDEO CONTENTS Some individuals with dementia cannot comprehend words and sentences, nor can they recall old memories. These impairments often disturb continuing conversations. To cope with these impairments, then, presentation of various visual, auditory, verbal supportive contents are a great help. The following contents can be delivered as augmentative contents for video phone conversations. We plan to evaluate their effectiveness in the near future.

3.1 Personalized Reminiscence Photo Video Reminiscence intervention is one type of intervention suggested as being effective for decreasing behavioral disturbances [4]. In

0 5

10 15 20 25 30 35 40

OI NK YM KM SS IS SN KM FS

Video Phone Session Face to Face Session

Figure 1. Total response time (minutes) of each subject in the videophone and face-to-face sessions

44

spite of its effectiveness, regular reminiscence intervention is difficult to perform at home. Therefore, we have created a personalized reminiscence photo video that consists of an individual’s old photographs with background music (BGM), narration and pan/zoom visual effects as a convenient reminiscence intervention [6]. The photo video helped home caregivers to perform the reminiscence intervention in the home. It could also be performed repeatedly even without the caregivers’ attendance [9].

3.2 “Talk Video” Several videotape series have been on the market for the generic use of individuals with dementia. In a series of videos titled “Video Respite,” an actor or actress talks to the viewer about favorite activities such as gardening, Christmas, and family gatherings, in a calm and friendly tone [7]. They also sing old songs in the video. We created two kinds of a similar generic video named “Talk Video,” in which a woman talks on various topics about her hometown and the changing of the seasons, and sings old songs. This video can be delivered when the partner on the network is too busy to participate in a video phone session.

3.3 Vocabulary Data File To facilitate conversation for people with severe or moderately severe communication deficits, a vocabulary data file called “Rakuraku Jiyu Kaiwa (Easy, Natural Conversation)” has been compiled [8]. This file contains more than 50,000 words, mainly proper names (for example, all the titles of the Beatles’ songs, titles of old movies etc.). These words function as key words in conversations (for example, “What is your favorite Beatles song?”). The words are classified into ten categories, which have a four-level tree structure, and by accessing a website (http://rousai.tripod.co.jp/), users can easily search for words related to current topics. Furthermore, by using these words, related pictures or photographs can also be collected easily from the Internet.

3.4 Reminiscent Short Movies The DVD movie series “Omoide no Tamatebako (Reminiscent Treasure Box),” published in Japan, contains about 20 short movies depicting things such as old foods, customs, children’s play, and traditional events in Japan. Each movie is about three minutes long and features soothing music.

3.5 Historical Social News Movies A collection movies featuring Japanese social news originally made after World War II were edited and published in a DVD series named “Syowa Nippon (Japan 1925-1988).” The series consists of about 20 DVDs that are arranged chronologically. Each DVD is about 60 minutes long and contains about 50 short movie clips.

3.6 Categorized Photograph Files Many generic photographs are sold on the market as royalty-free photographs. These photographs are often categorized according to themes such as Japanese people, landscapes, and animals. Such photographs may be used to elicit memories from subjects when the partner asks about their hobbies.

4. DISCUSSION AND CONCLUSION If we can provide some types of intervention that can attract and hold the attention of individuals with dementia, it can potentially reduce the behavioral disturbances and increase caregivers’ free time [7]. In this study, we investigated how long individuals with dementia can talk with a partner on a video phone. The results revealed that individuals with dementia can talk on the video phone as long as they engage in face-to-face conversation. Therefore, video phone conversation can be used as an effective form of intervention to support them. Furthermore, the experiment result also indicates that the concept of networked intervention is a promising one. To increase the opportunities for individuals with dementia to participate in a conversation, we consider the video phone to be a useful tool.

As an individual’s dementia worsens, normal conversation becomes difficult to perform. At this stage, it would be better to provide a variety of contents as augmentative contents to promote conversation. Presentation of the contents that we introduced in the previous chapter will enable them to more easily comprehend verbal questions, stimulate old memories, and activate conversation. Although several commercially available on-line services are used in practice for health care, these services are limited to basic physical/behavioral checks or to conducting short conversations. Networked interaction therapy, on the other hand, will extend networked supportive intervention for individuals with dementia more widely and systematically by providing an interface that is simple for them to use.

5. ACKNOWLEDGMENTS This study was supported in part by the National Institute of Information and Communications Technology (NICT).

6. REFERENCES [1] Alm, N., Astell, A., Ellis, M., Dye, R., Gowans, G., and

Campbell, J. A. Cognitive prosthesis and communication support for people with dementia. Neuropsychological Rehabilitation, 14, 117-134. 2004.

[2] Davis, R. N., Massman, P. J., and Doody, R. S. Cognitive intervention in Alzheimer disease: A randomized placebo-controlled study. Alzheimer Disease and Associated Disorders, 15, 1-9. 2001.

[3] De Vreese, L. P., Neri, M., Fioravanti, M., Belloi, L., and Zanetti, O. Memory rehabilitation in Alzheimer's disease; A review of progress. International Journal of Geriatric Psychiatry, 16, 794-809. 2001.

[4] Finnema, E., Droes, RM., Ribbe, M., and Van Tilburg, W. The effects of emotion-oriented approaches in the care for persons suffering from dementia: A review of the literature. International Journal of Geriatric Psychiatry, 15, 141-161. 2000.

[5] Kuwabara, K., Tetsutani, N., Kuwahara, N., Kawato, S., Lyons, M. J., Utsumi, A., Yanagida, Y., and Yasuda, K. Networked Interaction Therapy: Supporting People with Mild Dementia and their Family Members with Internet. 20th International Conference of Alzheimer’s Disease International Kyoto 2004, Abstracts, 377. 2004.

45

[6] Kuwahara, N., Kuwabara, K., Utsumi, A., Yasuda, K., and Tetsutani, N. Networked Interaction Therapy: Relieving Stress in Memory-Impaired People and Their Family Members. Proc. 26th Annual International Conference of the Engineering in Medicine and Biology Society (EMBS 2004), 3140-3143. 2004.

[7] Lund, D. A., Hill, R. D., Caserta, M. S., and Wright, S. D. Video RespiteTM; An innovative resource for family, professional caregivers, and persons with dementia. The Gerontologist, 35, 683-687. 1995.

[8] Yasuda, K., Nemoto, T., Takenaka, K., Mitati, M., and Kuwabara, K. The effectiveness of vocabulary data file, electronic, encyclopedia, and Internet homepages as a conversation support system for people with moderate-severe aphasia. Aphasiology, in submission.

[9] Yasuda, K., Kuwabara, K., Kuwahara, N., Abe, S., and Tetutani, N. Personalized Reminiscence photo video for individuals with dementia. Neuropsychological Rehabilitation, in submission.

46

The Relationships among Cognition, Parental Speech Patterns and Foreign Language Ability, and Multiple Intelligence Theory based Foreign Language Teaching

Keiko Hayashi Hiroshima Jogakuin University

(home telephone: 81-072-833-6828 e-mail:[email protected])

ABSTRACT This paper reports (1) the study on the relationships among intelligence, cognition, and foreign language ability, and discusses (2) the Multiple Intelligence (MI) Theory based teaching method for developing communicative ability.

Categories and Subject Descriptors A.1 [Introductory and Survey]: The influence of the parental speech on cognition and foreign language (FL) learning, and the introduction of the Multiple Intelligence (MI) into FL teaching.

General Terms Languages, Theory, Performance, and Experimentation

Keywords Cognition, Parental Speech, Foreign language learning, the Multiple Intelligence (MI), Cooperative Learning

I. INTRODUCTION The study of parental speech patterns indicates that the linguistic input is an essential factor for developing FL ability, and children’s success or failure in FL learning depends on the way of parental input. Interpersonal and intrapersonal intelligences developed in childhood affect EF learning. Using the MI theory developed by H. Gardner, this workshop shows the effect of the debate task through cooperative learning in FL classrooms.

II. THE RELATIONSHIP BETWEEN INTELLIGENCE AND COGNTION It is often said that intelligence affects language skill and aptitude, but it is not clear if intelligence is related to language or cognition. For example, the TOEFL score in the Netherlands is the highest in the world. According to some researchers in the Netherlands, this is because Dutch people are more intelligent than other people (cf. Kee de Bot, 1999). However, the result of PISA2003 indicates that there is little relationship between intelligence and language ability. Although Finland shows that all three subjects are the highest in the world, the Netherlands and Japan show that the mathematics is higher than that of other countries, but literacy is lower. The correlation between IQ and language aptitude is 0.43 (Gardner and Lambert, 1972) and 0.44 (Skehan,1998).The relationship between general cognitive abilities and the second language ability is 0.65（Skehan,1998）.1 It is difficult to define precisely what “intelligence” means. IQ score is often referred to intelligence, but it seems that cognitive abilities influence on foreign language (FL) learning due to the necessity of inference of rules and meaning from context. According to H. Gardner, each person possesses all eight intelligences. He proposes a pluralistic view of the mind, recognizing many different and discrete facets of cognition and acknowledging that people have different cognitive strengths (Gardner 1993). It is necessary for us to study on how intelligences and cognitive abilities develop and how MI theory integrates into FL learning. 1 See Hayashi, K. (2004:22)

47

III. THE INFLUENCE OF ENVIRONMENT ON COGITIVE DEVELOPMENT Every parent in the world hopes that their own children have success in academic pursuit, and want to know how to bring up their children to be successful. There may be genesis for developing child’s intelligence, but there is also a large possibility that a child is influenced by the environment. Children are influenced by the parental attitude. From the psychological point of view, it is said that every person does not like to listen their own reproachful words and criticism, and likes to listen praise words, applaud their achievements, victories, etc. Parental positive attitude towards their children’s curiosity and ideas about making something must be most important for child development. As for general cognitive development, Piaget classified cognitive development into six stages from sensory-motor intelligence level to formal operations. The formal operations begin towards eleven to twelve years. Eleven to twelve years old children have the possibility of reasoning by means of a hypothesis, and they will develop the deductive operation until about the age of fifteen. However, individual differences in cognitive abilities are large. Children possess the mechanisms of cognition as well as language, but whether the mechanism functions to bring about a rapid-learner or a slow learner depends on the environments. The environments involve the speech patterns and attitude of parents, teachers, and peers. Their speech patterns have positive effects and negative effects on children's cognitive development.

IV. PARENTAL SPEECH AND FOREIGN LANGUAGE (ENGLISH) ABILITY

Referring to the Bernstein’s theories (elaborated cord and restricted cord developed in 1967), I have examined the influence of parental speech on cognitive development and FL proficiency using 153 Japanese university students. Table 1 Correlations between Parental Speech Patterns and FL Proficiecy

I. Father's Speech II. Mother's Speech III. L2 Proficiency

1. 2. 3. 4. 5. 6. 7. 8. 9.

I. Father's Speech

1. Dem 1 -0.18* -0.41** -0.58** 0.47** -0.25** -0.20** -0.29** 0.46**

2. Imp. -0.18* 1 0.03 -0.55** -0.09 0.31** 0.02 -0.13 -0.06

3. Neg -0.41** 0.03 1 -0.15 -0.27** 0.16* 0.42** -0.06 -0.42**

4. Nof -0.59** -0.54**-0.15 1 -0.23** -0.06 -0.03 0.39** -0.18*

II. Mother's Speech

5. Dem 0.47** -0.09 -0.27** -0.24** 1 -0.60** -0.43** -0.38** 0.31**

6. Imp -0.25** 0.31** 0.16* -0.05 -0.60** 1 0.12 -0.26** -0.13

7. Neg. -0.20** 0.02 0.42** -0.03 -0.44** 0.12 1 -0.15 -0.22**

8. Nof -0.29** -0.13 0.06 0.39** -0.38** -0.26** -0.15 1 -0.17*

III. L2 Proficiency

9. skills 0.46** -0.06 -0.42** 0.18* 0.31** -0.13 -0.22** -0.17* 1

Dem.: Democratic interaction; Imp.: Imperative/Authoritarian;

Neg.: Negative/Reproachful; Nof.: Non-feedback

Skills: including reading,listening,vocabulary, and grammar.

*p<0.05 **p<0.01 (Hayashi 1997).

Table 1 shows that there are significant correlation coefficients between FL proficiency and parental speech patterns. The democratic interaction speech pattern has a highly significant correlation with FL proficiency. The imperative/authoritarian, negative/reproachful, and non-feedback speech patterns have negative correlations with FL proficiency.

Advanced level learners have a high rate of democratic interaction with both their fathers and mothers and a low rate of negative/reproachful interaction with both their fathers and mothers. Intermediate level learners have the same frequency of the mother's democratic pattern as that of advanced level learners but the frequency of the father's democratic pattern is lower than that of the advanced level. The next significant point of parental speech patterns is the negative/reproachful pattern. Beginning level learners reported the highest frequency of both father and mother's negative patterns. It is also noteworthy that the proportion of the non-feedback pattern for intermediate and beginning level learners is higher than that of advanced level learners. The speech pattern indicates that children need input data, and at the same time, the way of parental input is crucial to facilitate their children’s cognitive development such as guessing ability, high mentality, concentration, and reasoning. The students with democratic interaction use contextualization strategies guessing the meaning of words and sentences from context in FL learning. They use skimming and scanning strategies for understanding the passage. Students with the imperative/authoritarian interaction learn by rote memory/ imitation. They are unwilling to understand abstract meaning. The students with negative/reproachful interaction have high affective filter and empathy, and confuse and frustrate about what to do in FL learning. Finally, they fear or blame others for an incomprehensible explanation. The non-feedback/non-interference speech pattern is important in consideration of the children’s independent study but children cannot develop cognitive abilities with a let-alone policy. The children who are left alone do not know how to speak due to lack of a phonetic coding ability and are unwilling to understand meanings and grammatical rules. The learner’s success or failure in FL learning is related to their parental speech. Some children are interactive and cooperative with others, while others do not. Children do not develop only with parents’ interaction, but with social interaction. Hence, MI theories suggest collaboration with others and synthesize mind to get some ideas about how to learn and teach foreign language in the classroom.

V. INTEGRATIONG MI THEORIES INTO FL TEACHING It is often said that many Japanese university students keep their accustomed silence in the classroom. They never ask any questions or give any comments on what teachers say in the lesson. Thus, no communicative ability develops. This is because teacher-centered instruction is more popular than learner-centered instruction from elementary school. In the case of teacher-centered instruction students get knowledge about the rules and words, but they have little opportunity to interact with others in the classroom. Learner-centered instruction develops students’ creativity and understanding through interaction in the classroom, but some students do not like to do group work. MI theories offer

48

EFL teachers a way to examine their best teaching techniques and strategies in light of human differences (ref. Christison,1998; Kagan & Kagan, 1998).

According to H. Gardner, humans possess at least eight intelligences: linguistic, logical-mathematical, musical, bodily/kinesthetic, spatial, naturalist, interpersonal, and intrapersonal intelligences. Each intelligence is useful for FL learning and instructional practice in the classroom, but everyone has weakness and strength of the intelligence. In what ways should teachers apply MI Theory in FL classrooms? There are many ways to build each intelligence strength in the tasks. The purpose of using of MI theory in FL teaching is to enhance FL and communicative abilities. We can use all intelligences in the classroom. Table 2 shows 1) what is each intelligence, 2) how can each intelligence nurture, and how teachers can use the intelligence in the activities. Table 2 What is each intelligence? How can you nurture? The activities to enhance FL and communicative abilities?

(1) Linguistic - the ability to use language to express meanings, understand others, tell a simple story, and learn new vocabulary or a FL that is used naturally. Nurture: make sure your program is rich with language opportunities and that children’s linguistic expression is listened to and appreciated. Activity: tell jokes, riddles, or puns. Read, write, or tell stories. Play word games, crossword puzzles, and word searches. Create poems and stories. Keep diary. Have students interest in debates.

(2) Logical-mathematical - the ability to understand the basic properties of numbers, appreciate principles of cause and effect, one-to-one correspondence, and recognizes and solves problems using reasoning skills; the ability to predict, as in which objects will float, sink, etc. Nurture: work with numbers, figure things out, and analyze situations; use simple machines to help children think about cause and effect. Activity: work with numbers, figure things out, and analyze situations. Work in situations in which there are clear black-and-while solutions. Construct Venn diagrams. Use games of strategy. Have students demonstrate understanding using concrete objects. Establish time lines and draw maps. Logic puzzles and games.

(3) Musical - the ability to produce and recognize simple songs; sensitive to nonverbal sounds in the environment, and aware of patterns in rhythm, and pitch. Nurture: incorporate music daily, use a tape recorder for listening, singing, and recording songs. Activity: play with melodies, varying speed and rhythm. Sing, hum, and move to music. Rewrite song lyrics to teach a concept. Teach history through music of the period.

(4) Bodily/kinesthetic - the ability to use the body or parts of the body (hands, feet, etc.) to communicate and solve problems, as in playing a ballgame, dancing, or making things with the hands. Nurture: provide opportunities for physical challenges throughout the day, not just outdoors. Activity: Play sports and be physically active. Use body language. Provide tactile and movement activities. Offer role playing and acting opportunities. Allow the students to move.

(5) Visual-Spatial - the ability to be able to form a mental image of large and local spatial layouts; think three-dimensionally; find one’s way around a new building. Nurture: provide many opportunities for mapping of the classroom and beyond; explore new spaces and encourage children to vary the arrangements of

materials in the space. Activity: doodle, paint, or create three-dimensional representations. Draw charts, maps and mazes. Take things apart and put them back together. Photographs, videos, slides, movies.

(6) Naturalist - the ability to recognize species of plants or animals in one’s environment. Nurture: play games in which children recognize fine distinctions among members of a plant or animal group; explore the outdoors regularly and bring the outdoors in; provide ample books, visuals, and props related to the natural world. Activity: use the outdoors as a classroom. Create a nature area on the playground.

(7) Interpersonal - the ability to understand other people and work effectively with them and to notice who plays with whom at school, and why. Nurture: give children lots of opportunities to talk about one another and their social interactions, and to problem-solve conflicts together; play games in which one has to figure out the knowledge or intentions of other players. Activity: Use cooperative learning. Assign group projects. Give students opportunities for peer teaching. Use debate task in group work, and brainstorm solutions to problems. Create situations in which students are given feedback from others.

(8) Intrapersonal - the ability to understand things about oneself, how one is similar to, different from others; know own strengths and weaknesses and how to soothe oneself when sad. Nurture: let children express their own unique emotions, preferences, and strategies; help them understand their own wishes and fears and how best to deal with them. Activity: Assign individual, self-directed projects. Provide opportunities for students to get feedback from one another. Involve the students in journal writing and other forms of reflection (ref. Christison,1998) Table 2 shows that each intelligence develops through interaction, and plays the important role for developing FL ability. However, individual difference in each intelligence, particularly the interpersonal and intrapersonal intelligences, is large. Some students like to do group work, while others do not. The activities in cooperative learning attempt to better understand concepts and strategies, and enhance interpersonal and intrapersonal intelligences needed for developing FL ability, rather than simply reading and writing sitting at tables. For cooperative learning, elementary and junior high school children should be given a lot of opportunities to express their own ideas in ways that respect the individual interest and strengths while talking among group members. For senior high school and university students, debate task seems to be most appropriate for better understanding the concepts of the topics and strategies for communication.

VI. COOPERATING LEARNING 1. INVESTIGATION OF DEBATE TASK For developing students’ communicative abilities debate tasks have been administered to my university students majoring in linguistics, education, and sociology. In 2001, I have examined the effect of the collaborative learning involved in the debate task using 119 Japanese sophomore students majoring in linguistics. The result shows that for some students the debate activities are

49

effective to improve FL proficiency, while for others there was little improvement. In order to develop reading and grammatical abilities extensive reading task is more effective than debate task. However, the survey on the students’ reactions to the debate task shows that only 2 students out of 119 had a negative impression. That is, 98% of the students showed a feeling of satisfaction with debate task. They wrote as follows:

I learned how to find materials, and could reflect on other points of view.

I learned how to make others understand my opinion. I learned to think about the topic deeply, to write my

opinions in English, and to cooperate with others. I learned logical thinking, how to discuss and write, and the

importance of group work. I took a lot of time to prepare, but I found it very useful for

learning English. I heard others’ views too. I came to understand my topic in depth.

Even though there were many things to be prepared, I really enjoyed it. We got together and worked hard.

Debating in English is very difficult, but it’s a good way to study language. We have to tell our opinions to others.

It’s one of the most interesting classes and tasks that I have ever done with group work.

Also, 98% of the students majoring in education showed the same positive and satisfactory reaction to the debate task. On the other hand, nearly 50% of the freshmen majoring in sociology showed the negative reaction to the debate task. They are not autonomous learners. They are usually expected to receive teacher-instruction, neither tell any opinions against others in the debate activity. In order to become independent learners, the students should be accustomed to interact and communicate with people from elementary school days.

2. THE EFFECTS OF COOPERATIVE LEARNING ON FL TEACHING

Thinking about how to communicate effectively in written and spoken form for understanding goals, the debate task was given in the FL classroom. It is a partial activity of the cooperative learning for “Teaching for Understanding” concept orientation of MI theory. From unit title the students engage with the current events topics. The debate task builds a supportive classroom environment in the following. As listeners and readers, students analyze experiences, ideas, information, and issues presented by others. As speakers and writers, they use oral and written language from a variety of perspectives to help them understand their opinions and judgments on these performances, ideas, etc. While speaking and writing, the students interpret and apply the concepts of words or phrases to transmit information. To develop students’ understanding of the various aspects of listening and reading, students will collect data, facts, and ideas, and discover relationships, concepts, and generalization. To become an independent learner, students learn reading strategies. They understand how discussion with group members helps to improve their own skills by asking questions and answers to the questions. They learn how to speak in public and how to argue against specific points of view. The debate task develops knowledge

building, problem solving and understanding alternative point of view. In traditional FL classes, teacher talk dominates, but cooperative learning is learner-centered. In group work, the students take responsibilities to collect reading materials suitable for the topic by themselves, talking with group members, thinking through ideas, and listening to others’ opinion. The students learn a lot of vocabulary and ideas through interaction with group members. They often misunderstand ideas when reading by themselves, but they become aware of the appropriate interpretation through interaction. Cooperative learning increases the quantity of speech. When teachers select the group members carefully, reticent students also have the chance to speak out. Cooperative learning from childhood helps students to be more positive in their learning attitudes and interaction.

VII. CONCLUSION Intelligences related with cognitive abilities develop from childhood through interaction. Parental speech patterns indicate that the roles of parent and teacher feedback are crucial to developing cognitive abilities and intelligences. As Gardner (1993) suggests, everyone has the capacity to develop all eight intelligences to a reasonably high level of performance with appropriate encouragement, enrichment, and instruction. Each intelligence has its own developmental time. Although it is said that linguistic intelligence can peak very late, the interpersonal and intrapersonal intelligences will develop through interaction from childhood. The two intelligences are important for developing communicative and linguistic abilities.

Giving children opportunities for peer teaching to talk about one another and to get feedback from one another in cooperative learning help children express their own ideas, emotions, preferences, and strategies. For senior high school and university students cooperative learning such as debate task is effective to enhance understanding ability of concepts, logical thinking, strategies, and communication ability in FL learning.

ACKNOWLEDGEMENTS This study was funded by a Kaken-Houga Research Grant No. 17652063 from the Japan Society for the Promotion of Science (jsps). I would also like to thank Professor Kuwahara Noriaki and other anonymous reviewers of this International Workshop on Cognitive Prostheses and Assisted Communication 2006 for their helpful comments on this paper.

REFERENCES [1] Budden, J. Multiple Intelligences. British Council, Barcelona.

Teaching English. 2005. [2] Christison, M.A. Multiple Intelligences and Learning Styles

in Second Language Teaching and Learning. TESOL Academy. Seattle, Washington, August 14-16, 1998.

[3] Gardner, H. Multiple Intelligences: The Theory and Practice. New York: Basic Books. 1993.

[4] Haley, M.H. Learner-Centered Instruction and the Theory of Multiple Intelligences With Second Language Learners.

50

Teachers College Record Volume 106, No.1, Jan.2004. pp.163-180.

[5] Harvard Graduate School of Education. The Project Zero Classroom 2005: Views on Understanding. July 2005. Hayashi, K., The Influence of Cognitive Development in EFL/ESL(L2) Learning and Environmental Variables. Communication, Cognition, and Second Language Acquisition Journal., Belgium. 1997.

[6] Hayashi, K., The Effectiveness of Collaborative Learning. The Japanese Learner. England: University of Oxford. 2001.

[7] Hayashi, K. Gaikokugogakusyu ni Eikyou o oyobosu Oya to Ko no Komyunikeisyon. Tokyo: Kazamasyobo.2004.

[8] Hayashi, K. Oranda no Gaikokugokyouiku no Doukou. Paper presented at Symposium, Nihon Gengo Seisaku Gakkai. 2005.

[9] Kagan, S.& Miguel K. Multiple Intelligences. 1998. [10] Kee de Bot. Foreign Language Teaching and Learning in

the Netherlands. http://www.nlconference. org/docs.2004.

51

A cognitive prosthesis and communication support for people with dementia

Documents

Transcript of A cognitive prosthesis and communication support for people with dementia