Comparison of Displaying with Vocalizing on Preliminary-Announcement of

Mobile Robot Upcoming Operation

Takafumi Matsumaru Graduate School of Information, Production and Systems

Waseda University, Japan

1 Introduction  

This research aims at examining the method and equipment of preliminary-announcement of the upcom-ing operation of a mobile robot moving on a two-dimensional plane to those who are in the surroundings. The speed and direction of movement are the contents to inform. This paper presents the result of the experimental examination in an interactive situation between a user and a robot comparing displaying with vocalizing as the announcement means.

2 Related Researches and Previous Works

2.1 Background

Robotics and mechatronics are expected to spread through society not only to help and assist the elderly and the physically challenged but also to support and entertain ordinary people in their daily life. There-fore the number of the human synergetic robot operating in the same living or working space as users is increasing (Milgram et al., 1993; Potkonjak, 2002; Lee & Makatchev, 2009; Kim & Kwon, 2010). The human synergetic robot should avoid contact and collision with users. Consequently there have been many researches on the user safety and reassurances against robots (Burri & Helander, 1986; Schweitzer, 1996; Zaatri et al., 2000; Tzafestas & Tzafestas, 2010; Cernetic, 2003). However it is difficult for a user to estimate the upcoming operation of a robot only from its outward appearance without special treatment because it is a technical artifact. Each robot has various features and performances so that people cannot keep in mind its performances as a sort of common sense. One of the solutions might be the function to preliminarily announce the upcoming operation of a robot while operating in a human-coexisting envi-ronment.

2.2 Previous and Relating Works

Most research on nonverbal interfacing between a user and a machine has been focused on the infor-mation transmitted from a user to a machine to operate it. There have been researches involved in, for example, imaging of the body language or gestures (Ueda et al., 2001; Bretzner et al., 2002) and the faci-al expressions or glances (Kuno et al., 2003; Ohno & Mukawa 2004) focusing on the human motion and his intention. Some researches have emphasized on analyzing human emotion based on the psychological findings about facial expressions (Liu et al., 2002; Tian et al., 2005).

Comparatively a few researches have concerned the information transmission from a machine to a user especially on robotic systems. As an indication of robot's internal state, a mobile robot displaying remaining battery charge, internal temperature, etc. in a screen is reported (Ogata & Sugano, 2000). For a communication of a robot upcoming movement, the projection to common space between a manipulator and an user is proposed (Wakita et al., 2001). Some experiments on the industrial robot with LEDs sup-porting eating by the physically challenged is also presented (Ikeura et al., 2000). Recently there are sev-eral researches focusing on robot’s attention and intention (Yamazaki et al., 2007; Mutlu et al., 2009; Kim et al., 2009; Jee et al., 2010; Park et al., 2010; Shindev et al., 2012; Hirota et al., 2012), but those mainly focus on informational aspect of human-robot interaction.

2.3 Approaches

We proposed four approaches categorized into two types to preliminarily announce the speed and direc-tion of a mobile robot upcoming movement (Matsumaru, 2004). The first type announces the state of operation at a time just after the present, (a) lamp method and (b) blowout method. The second type indi-cates operations from the present to some future time continuously, (c) light-ray method and (d) projec-tion method. The validity and timing to preliminarily announce are evaluated using a computer simula-tion (Matsumaru et al., 2003), and four prototype robots are developed to make the proposed approaches embodiment (Table 1 and 2).

Method (a) Lamp (b) Blowout

Prototype robot PMR-2R, eyeball robot

PMR-6R, arrow robot

Announce. device Omni-directional display Flat-panel display

Mobile mech. (max. trans./rot.)

Two-wheel drive 36 cm/s, 41.4 rad/s

Two-wheel drive 36 cm/s, 41.4 rad/s

Size D47 × W48 × H94 cm D47 × W48 × H44 cm

Weight 24.5 kg 22.0 kg

Table 1: Proposed method and prototype robot: type (1) announcing state of operation just after the present.

Flat- panel display

Omni- directional display

Table 2: Proposed method and prototype robot: type (2) indicating operations from present to some future time.

The eyeball robot PMR-2 using an omni-directional display (eyeballs are shown on the display)

(Matsumaru et al., 2005) and the arrow robot PMR-6 using a flat-panel display (an arrow is presented on the display) (Matsumaru, 2007a) are developed for the former type after the lamp method and the blow-out method respectively. The light-ray robot PMR-1 using laser pointers (the schedule route is drawn on running surface as movement afterimage of radiant from the laser pointers) (Matsumaru et al., 2006) and the projection robot PMR-5 using a projector (various information can be informed on the two-dimensional frame which is projected on running surface from the projector) (Matsumaru, 2006) are cre-ated for the latter type after the light-ray method and the projection method respectively. These four pro-totype robots were exhibited at the 2005 International Robot Exhibition (iREX05) held in Tokyo. We had visitors answer the questionnaire in a five-stage evaluation (Matsumaru, 2007b). The result shows that the understandability is different among four prototype robots. The projection robot PMR-5 received the highest evaluation score among the four robots on each gender and age groups. The examination of dif-ferences by gender and age suggests that some people prefer the friendly expressions by the eyeball robot PMR-2 and the simple and minimum information presented at one time by the arrow robot PMR-6.

Method (c) Light-ray (d) Projection

Prototype robot PMR-1R, light ray robot

PMR-5R, projection robot

Announce. device Laser pointer Projector

Mobile mech. (max. trans./rot.)

Two-wheel drive 36 cm/s, 41.4 rad/s

Two-wheel drive 36 cm/s, 41.4 rad/s

Size D46 × W48 × H91 cm D50 × W44 × H100 cm

Weight 30.0 kg 25.0 kg

Laser pointer

Mirror

Pan motor Tilt motor

Scheduled route

Occupied width

Mirror

Projector

Scheduled route

Mirror

Laser pointer

Pan motor

Tilt motor

3 Mobile Robot with Preliminary-Announcement Function

Displaying and vocalizing are considered as the preliminary-announcement means of the operational state after a certain period of time from the present.

3.1 PMR-6R

Here we focus on the arrow robot PMR-6R, because this concept is the most applicable to, for example, the autonomous robot cleaner existing with more modest adjustments. PMR-6R indicates its upcoming operational state by showing the sign (arrow) and characters on a liquid crystal display.

The mobile mechanism of PMR-6R is two-wheeled drive driven by a DC servomotor with reduc-tion gears and encoders respectively. A ball-caster with suspension is used as trailing wheel back and forth so that the robot can rotate around the robot center. In order to restrict the kind of operation to an-nounce, the speed is made into two steps, high speed and low speed, and the direction is set at four stag-es, a going straight, a loose turn, a tight turn and a pivot turn (Figure 1). The on-board computer with expansion boards is carried to control the operation but AC 100V is supplied through the cable as an ex-ternal power source. The liquid crystal display is installed on the upper surface to display the direction and speed of movement after a certain period of time by an arrow. A commercial product of 17 inches with thin bezel (GH-PLG170SV, Green House Co., Ltd.) is selected. The robot height is designed as low as possible so that the displaying can be seen also by the person separated a few from the robot.

The instruction acquired from a joystick or a game pad every 50 ms is displayed or vocalized to preliminarily announce the operational state in future. Thirty instructions are stored and the operation by an instruction inputted 1.5 s before is performed (Figure 2). That is, the operation is preliminarily an-nounced 1.5 s before it is performed actually. The timing to preliminarily announce, 1.5 s before the ac-tual operation, is determined based on the result of the simulation experiment so as to be comprehensible for users (Matsumaru et al., 2003). In order to arrange the condition on displaying and vocalizing, the robot movement in the experiment is limited to the combination of three kinds of speed (high speed: translational velocity V = 36 cm/s, low speed: V= 18 cm/s, and stop: V = 0 cm/s) and four kinds of direc-tion (going straight: turning radius (R) = infinity, loose turn: R = 100 cm, tight turn: R = 50 cm, and pivot turn: R = 0 cm).

3.2 Preliminary-Announcement of Operation by Displaying

The sign (arrow) and characters are shown on the display using OpenGL (OpenGL.org organization) as the operational state at 1.5 s before the actual operation. The speed of movement is expressed as the size (length and width) and color (base on traffic signals) of the arrow: big blue at high speed, small yellow at low speed, and red characters when stopped (see Figure 1). The direction of movement is indicated with the curved condition of the arrow: straight when going straight, curved when making loose turn, swerved when making tight turn, and rounded when making pivot turn (see Figure 1). The arrow expression is direct and intuitive and it is intelligible for everyone because there is no ground for translation and inter-pretation between the sign and the movement even at first sight.

Forward left,

tight turn

Forward left, loose

turn

Forward

Forward right,

loose turn

Forward right,

tight turn

Slow, forward left, tight turn

Slow, forward left, loose turn

Slow, forward

Slow, forward

right, loose turn

Slow, forward right, tight turn

Left, turn

Slow, left, turn

Stop

Slow, right, turn

right, turn

Slow, forward left, tight turn

Slow, backward left, loose turn

Slow, backward

Slow, backward right, loose turn

Slow, backward right, tight turn

Backward left,

tight turn

Backward left,

loose turn

Backward

Backward right,

loose turn

Backward right,

tight turn

Figure 1: Robot operation with displaying and vocalizing (in Japanese).

3.3 Preliminary-Announcement of Operation by Vocalizing

The operational state at 1.5 s later is vocalized using AquesTalk (Aquest Corp.), a free middleware for the text-to-speech-synthesis in Japanese. The AquesTalk features are as follows:

1) A high-quality sound is generated with a small amount of resource and throughput.

2) It can be used free of charge regardless it is commercial and nonprofit.

3) An intonation and an accent can be set up finely according to preferences.

Input section

Operation discrimination

Translation factor Jy

Rotation factor Jx

Command buffer

Latest 1.5 s Delete oldest data

Announcement section

Target speed of both wheels

Motor drive

Target value for motor driver

* Stop * On-the-spot rotation

Encoder

Motion control section

Command input

* Going straight * Loose turn * Tight turn

Displaying Vocalizing

OpenGL AquesTalk

Figure 2: Structure of program.

4) It supports Japanese, ordinary language of Japanese users.

5) It also supports both synchronous and asynchronous processing.

Japanese users are supposed here so that the language is set into Japanese instead of English. The asyn-chronous processing is mounted so that a text might be vocalized even if interrupting a present text spo-ken.

The order of the information vocalized is made into (a) the speed of movement, (b) the back-and-forth (and right-and-left if necessary) direction, and (c) the degree of turn. For example, it is ''forward'', ''forward, right, loose turn'', ''slow, left, pivot turn'', etc. (see Figure 1). Firstly the vocalization time should be shortened as much as possible. The information about the speed of movement may be im-portant on a mobile robot. It doesn't inform anything at high speed as the default setting, whereas it vo-calizes ''slow'' at low speed. The back-and-fore (and right-and-left) direction seems the second important and it should be prior to the degree of turn.

The back-and-fore and right-and-left directions are set from the robot's viewpoint. Although there is an opinion that the direction vocalized should be set from the user's viewpoint to receive the infor-mation, the contents vocalized should not be different according to the situation, such as the user position. Moreover, it is difficult to respond to everyone in an arbitrary direction to the robot and it is impossible to deal with more than a user at the same time.

4 Comparison Experiment of Displaying and Vocalizing

The position prediction experiment by trial subjects is carried out to compare the validity and effect of displaying and vocalizing the upcoming operation of a robot quantitatively.

4.1 Experimental Description: Position Prediction

4.1.1 Experimental Setup

The subject predicts the robot position at 1.5 s later referring to the preliminary-announcement of upcom-ing operation, while the robot runs at random speed and in random direction in a test area (Figure 3). We have the subject point out the robot position at 1.5 s later with the tip of a stick (115 cm length) during the testing time. Both the tip position of the stick and the actual movement of the robot are measured us-ing a motion capture system at 30 Hz. The position pointed by the subject reflects what the subject rec-ognizes referring to the preliminary-announcement. Therefore the effect of the preliminary-announcement and the features both of displaying and vocalizing are estimated with comparing the dis-tance gap between the position pointed by the subject and the actual position of the robot at 1.5 s later.

Figure 3: Position prediction experiment.

Subject Robot

Predicting the position at 1.5 s later

Pointing the predicted position

?

?

4.1.2 Experimental Condition

The robot movement for 30 s at random speed in random direction in the test area of 4.5-meter width by 3.0-meter width bounded by lines is designed using a random function. The subject is told that the robot does not go out from the area before the experiment. Some space outside the test area is prepared as much as possible taking care that the subject motion might not be interfered.

The experiments are carried out on four kinds of announcement setting: displaying (D), vocalizing (V), displaying plus vocalizing (D+V), and announcement-less (NO). Five patterns of robot movement for 30 s are prepared. Aiming to eliminate the influence of practice as much as possible, a subject tries all of the five movement patterns on each of the four announcement settings, i.e. at twenty kinds of settings in total. Moreover, the order of the announcement settings and movement patterns for a subject is made different arbitrarily taking care that the data might not be biased.

4.1.3 Measured Data

The experimental situation is shown in Figure 4. The subjects are six male students in their early twenties who look at the robot almost for the first time. The robot movement by manual operation on four kinds of announcement setting is shown to the subject to understand the function and performance of the robot at first. And then the experiment is carried out at five movement patterns for each announcement setting on each subject.

Figure 4: Experimental situation.

4.2 Experimental Result

The evaluation index is the distance gap between the position pointed by the subject and the actual posi-tion of the robot at 1.5 s later.

4.2.1 Average Value during 30 s

The average and standard deviation of the average value on individual subject are shown in Table 3 and Figure 5. The significant difference is shown on all combinations of the four announcement settings ex-cept the pair of D+V and D and that of D and V when applying a t-test at the 1 % level of confidence. Even though running a t-test at the 5 % level of confidence, it does not show a significant difference on these two pairs.

D+V Disp. Vocal. No Av. cm 25.53 26.19 28.75 31.22 SD cm 4.37 6.08 3.18 1.93

Table 3: Average value during 30 s.

D+V Disp. Vocal. No

Av. cm 25.53 26.19 28.75 31.22 SD cm 12.80 13.18 13.63 16.02

Table 4: All data for 30 s.

Figure 5: All data for 30 s. Figure 6: Average value during 30 s.

4.2.2 All Data for 30 s

The average and standard deviation of all data for 30 s by all subjects are shown in Table 4 and Figure 6. The significant difference is shown on all combinations of the four announcement settings when applying a t-test at the 1 % level of confidence.

4.3 Discussion

4.3.1 Comparison of V with NO

The distance gap is smaller on the preliminary-announcement by vocalizing rather than announcement-less. It shows the effect of vocalizing as the preliminary-announcement of a mobile robot upcoming op-eration.

0

10

20

30

40

50

D+V Disp. Vocal. NoSetting of preliminary-announcement

Ave

rage

gap

cm

n.s. n.s. p<0.01

0

10

20

30

40

50

D+V Disp. Vocal. NoSetting of preliminary-announcement

Gap

cm

p<0.01 p<0.01 p<0.01

4.3.2 Comparison of D with V

The distance gap is smaller on the preliminary-announcement by displaying rather than vocalizing. It indicates that displaying is more effective than vocalizing to understand the mobile robot upcoming oper-ation. It is due to the fact that displaying a sign (arrow) can transmit more than one kind of information, i.e. the speed of movement, the back-and-forth (and right-and-left), and the degree of turn in this case, at once, while vocalizing takes more time to inform the equivalent information according to its serial form. Moreover, the direction of movement is sometimes vocalized in the opposite direction for a user depend-ing on the relative position between the robot and the user so that he may be confused, although a user can understand the correct direction of robot movement by displaying regardless of his position.

4.3.3 Comparison of D+V with D

The distance gap is smaller on the preliminary-announcement by displaying plus vocalizing rather than only displaying. It suggests that the additional vocalizing to displaying makes more impact on the prelim-inary-announcement of a mobile robot upcoming operation. This is attributed to the fact that the vocaliz-ing makes the user unnecessary to concentrate his attention to the robot and the timing to change the ro-bot operation is recognized with the sound.

4.3.4 Findings

The findings are summarized as follows:

• The effect of vocalizing on the preliminary-announcement of a mobile robot upcoming operation is confirmed,

• Displaying is more effective to understand the upcoming operation in comparison with vocaliz-ing, and

• Additional vocalizing to displaying makes more impact on the preliminary-announcement of a mobile robot upcoming operation.

Those are corresponding with the previous studies concerning visual and auditory cues on various human-machine interfaces (Gaver, 1989; Akamatsu et al., 1995; DiFranco et al., 1997; Petzold et al., 2004; Maltz & Shinar, 2004; Ho & Spence, 2005; Ho & Spence, 2006; Ho et al., 2006; ), following the knowledge in psychology (Sumby & Pollack, 1954; Erber, 1969; Brown et al., 1989; Corneil & Munoz, 1996; Mondor & Amirault, 1998; Cohen et al., 2004), physiology and medicine (Stein et al., 1989; Law-rence, 1994; Suteerawattananon et al., 2004; Rochester et al., 2005). Displaying can transmit various kinds of information correctly at once with devising such as using a symbolic sign. On the other hand, vocalizing could transmit information to everyone around, but it takes much time to transmit a lot of in-formation and has lower exactness and reliability. Therefore, sound should be used to call an user's atten-tion when a robot changes its operation rather than to transmit the detailed information about the robot operation. For example, a robot calls attention of those around with a signal sound (beep sound, buzzer sound, chime, etc.) before it changes its operation. The signal sound is more acceptable rather than vocal-ized words even for those not wanting or needing the information. And then the user who wants or needs can acquire the detailed information from the displayed sign by looking at the direction of the sound source.

5 Conclusions and Future Works

This paper presented the result of the experimental examination about the preliminary-announcement of the upcoming operation of a mobile robot moving on a two-dimensional plane in an interactive situation between a user and a robot comparing displaying with vocalizing as the announcement means. The find-ings are summarized as follows: the effect of vocalizing on the preliminary-announcement of a mobile robot upcoming operation is confirmed, displaying is more effective to understand the upcoming opera-tion in comparison with vocalizing, and additional vocalizing to displaying makes more impact on the preliminary-announcement of a mobile robot upcoming operation.

Future plan includes continuing the experiment by various subjects to generalize the results. We should also examine in various conditions, such as the case that traffic lines are obliquely crossed not facing each other but meeting suddenly due to a wall or the case that several obstacles are scratched on the way. We have to examine more about vocalizing, such as the timing to announce, the classification and abbreviation of the contents to inform, the speed to vocalize, the vocal sound including the signal sound, and so on. We would like to install the preliminary-announcement function in actual human-synergetic robots like a remote-operated mobile robot.

Acknowledgement

Gratitude is expressed to laboratory members including Mr. Yuichi Ito (now, at Hitachi Construction Machinery Co., Ltd.) and Mr. Hiroyuki Hayashi (now, at Juroku Bank, Ltd.) who greatly contributed both on mounting the speech synthesizer in the robot and performing the evaluation experiment. This research is supported partly by the Hamamatsu Scientific and Technological Research Promotion Associ-ation (fiscal 2009, Murata foundation) to which our gratitude is expressed here.

References

Akamatsu, M., Mackenzie, I. S., & Hasbroucq, T. (1995). A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type device. Ergonomics, 38(4), 816-827.

Aquest Corp. AquesTalk. available at http://www.a-quest.com/aquestalk/

Bretzner, L., Laptev I., & Lindeberg, T. (2002). Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering. In Proc. 5th IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG'02), 405-410.

Brown, M. L., Newsome, S. L., & Glinert, E. P. (1989). An experiment into the use of auditory cues to reduce visual work-load. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems (CHI '89), 339-346.

Burri, G. & Helander, M. (1986). Implications of product/process design on human and robotic workstations. In Proc. 1986 IEEE Int’l Conf. on Robotics and Automation, 3, 1121-1125.

Cernetic, J. (2003). On cost-effectiveness of human-centered and socially acceptable robot and automation systems, Robot-ica, 21(3), 223-232.

Cohen, Y. E., Cohen, I. S., & Gifford III, G. W. (2004). Modulation of LIP Activity by Predictive Auditory and Visual Cues. Cerebral Cortex, 14(12), 1287-1301.

Corneil, B. D. & Munoz, D. P. (1996). The influence of auditory and visual distractors on human orienting gaze shifts. J. of Neuroscience, 16(24), 8193-8207.

DiFranco, D. E., Beauregard G. L., & Srinivasan, M. A. (1997). The effect of auditory cues on the haptic perception of stiffness in virtual environments. In Proc. the ASME Dynamic Systems and Control Division (DSC-Vol.61), 17-22.

Erber, N. P. (1969). Interaction of audition and vision in the recognition of oral speech stimuli. J. of Speech and Hearing Research, 12(2), 423-425.

Gaver, W. W. (1989). The SonicFinder: An Interface That Uses Auditory Icons. Human-Computer Interaction, 4(1), 67-94.

Hirota, K., Yamazaki, Y., & Dong, F. (2012). Mascot Robot System Based on Fuzzy Control Technology, In Combining Experimentation and Theory. Trillas, E., et al. (eds.), Springer, 137-149.

Ho, C. & Spence, C. (2005). Assessing the Effectiveness of Various Auditory Cues in Capturing a Driver's Visual Atten-tion. J. of Experimental Psychology: Applied, 11(3), 157-174.

Ho, C. & Spence, C. (2006). Verbal interface design: Do verbal directional cues automatically orient visual spatial atten-tion? Computers in Human Behavior, 22(4), 733-748.

Ho, C., Tan, H. Z., & Spence, C. (2006). The differential effect of vibrotactile and auditory cues on visual spatial attention. Ergonomics, 49(7), 724-738.

Ikeura, R., Kawakita, Y., Ito, M. & Mizutani, K. (2000). Previous notice method of robotic arm motion for suppressing threat to human, In Proc. 9th IEEE Int. Workshop on Robot and Human Interactive Communication (RO-MAN 2000), 276-280.

Jee, E.-S., Jeong, Y.-J., Kim, C. H., & Kobayashi, H. (2010). Sound design for emotion and intention expression of social-ly interactive robots, Intelligent Service Robotics, 3(3), 199-206.

Kim, H.-R. & Kwon, D.-S. (2010). Computational Model of Emotion Generation for Human-Robot Interaction Based on the Cognitive Appraisal Theory. J. of Intelligent and Robotic Systems, 60(2), 263-283.

Kim, W. H., Park, J. W., Lee, W. H., Kim, W. H.,& Chung, M. J., (2009). Synchronized multimodal expression generation using editing toolkit for a human-friendly robot, In Proc. of the 2009 IEEE Int’l Conf. on Robotics and Biomimet-ics (ROBIO 2009), 706-710.

Kuno, Y., Shimada, N., & Shirai, Y. (2003). Look where you're going: a robotic wheelchair based on the integration of human and environmental observations. IEEE Robotics and Automation Magazine, 10(1), 26-34.

Lawrence, W. (1994). Supramodal and modality-specific mechanisms for stimulus-driven shifts of auditory and visual attention. Canadian J. of Experimental Psychology, 48(2), 242-259.

Lee, M. K. & Makatchev, M. (2009). How do people talk with a robot?: an analysis of human-robot dialogues in the real world, In Proc. CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09), 3769-3774.

Liu, Y., Schmidt, K., Cohn, J.F., & Weaver, R.L. (2002). Human facial asymmetry for expression-invariant facial identifi-cation, In Proc. 5th IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG'02), 208-214.

Maltz, M. & Shinar, D. (2004). Imperfect in-vehicle collision avoidance warning systems can aid drivers. Human Factors, 46(2), 357-366.

Matsumaru, T., Endo, H. & Ito, T. (2003). Examination by Software Simulation on Preliminary-Announcement and Dis-play of Mobile Robot's Following Action by Lamp or Blowouts. In 2003 IEEE Int. Conf. on Robotics and Automa-tion (2003 IEEE ICRA), 362-367.

Matsumaru, T. (2004). The human-machine-information system and the robotic virtual system. J. SICE, 43(2), 116-121. (in Japanese)

Matsumaru, T., Iwase, K., Akiyama, K., Kusada, T. & Ito, T. (2005). Mobile robot with eyeball expression as the prelimi-nary-announcement and display of the robot's following motion. Autonomous Robots, 18(2), 231-246.

Matsumaru, T., Kusada, T., & Iwase, K. (2006). Mobile Robot with Preliminary-Announcement Function of Following Motion using Light-ray. In 2006 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS 2006), 1516-1523.

Matsumaru, T. (2006). Mobile Robot with Preliminary-announcement and Display Function of Following Motion using Projection Equipment, In 15th IEEE Int. Symp. on Robot and Human Interactive Communication (RO-MAN 06), 443-450.

Matsumaru, T. (2007a). Mobile Robot with Preliminary-announcement and Indication Function of Forthcoming Operation using Flat-panel Display. In 2007 IEEE Int. Conf. on Robotics and Automation (ICRA'07), 1774-1781.

Matsumaru, T. (2007b). Development of Four Kinds of Mobile Robot with Preliminary-announcement and Display Func-tion of its Forthcoming Operation. J. of Robotics and Mechatronics, 19(2), 148-159.

Milgram, P., Zhai, S, Drascic, D., & Grodski, J. (1993). Applications of augmented reality for human-robot communica-tion, In Proc. of the 1993 IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems (IROS '93), 3, 1467-1472.

Mondor, T. A., & Amirault, K. J. (1998). Effect of same- and different-modality spatial cues on auditory and visual target identification. J. of Experimental Psychology: Human Perception and Performance, 24(3), 745-755.

Mutlu, B., Yamaoka, F., Kanda, T., Ishiguro, H., & Hagita, N. (2009). Nonverbal leakage in robots: Communication of intentions through seemingly unintentional behavior, In 4th ACM/IEEE Int’l Conf. on Human-Robot Interaction (HRI 2009), 69-76.

Ogata, T. & Sugano, S. (2000). Emotional communication between humans and the autonomous robot WAMOEBA-2 (Waseda Amoeba) which has the emotion model, JSME Int. J., Series C, 43(3), 586-574.

Ohno, T., & Mukawa, N. (2004). A free-head, simple calibration, gaze tracking system that enables gaze-based interaction, In Proc. of Eye Tracking Research & Application Symposium 2004 (ETRA 2004), 115-122.

OpenGL.org organization. OpenGL - The Industry Standard for High Performance Graphics. available at http://www.opengl.org/

Park, J.-C., Song, H., Koo, S., Kim, Y.-M., & Kwon, D.-S.(2010). Robot's behavior expressions according to the sentence types and emotions with modification by personality, In 2010 IEEE Workshop on Advanced Robotics and its So-cial Impacts (ARSO), 105-110.

Petzold, B., Zaeh, M. F., Faerber, B., Deml, B., Egermeier, H., Schilp, J., & Clarke, S. (2004). A study on visual, auditory, and haptic feedback for assembly tasks. Presence, 13(1), 16-21.

Potkonjak, V., Radojicic, J., Tzafestas, S., & Kostic, D. (2002). Modeling Robot “Psycho-Physical” State and Reactions ? A New Option in Human-Robot Communication Part 2: Modeling and Simulation, J. of Intelligent and Robotic Systems, 35(4), 353-364.

Rochester, L., Hetherington, V., Jones, D., Nieuwboer, A., Willems, A.-M., Kwakkel, G., & Wegen, E. V., (2005). The Effect of External Rhythmic Cues (Auditory and Visual) on Walking during a Functional Task in Homes of People with Parkinson’s Disease. Archives of Physical Medicine and Rehabilitation, 86(5), 999-1006.

Schweitzer, G. (1996). Mechatronics for the design of human-oriented machines, IEEE/ASME Trans. on Mechatronics, 1(2), 120-126.

Shindev, I, Sun, Y., Coovert, M., Pavlova, J., & Lee, T. (2012). Exploration of intention expression for robots, In 7th ACM/IEEE Int’l Conf. on Human-Robot Interaction (HRI 2012), 247-248.

Stein, B. E., Meredith, M. A., Huneycutt, W. S., & McDade L. (1989). Behavioral Indices of Multisensory Integration: Orientation to Visual Cues is Affected by Auditory Stimuli. J. of Cognitive Neuroscience, 1(1), 12-24.

Suteerawattananon, M., Morris, G.S., Etnyre, B.R., Jankovic, J., & Protas, E. J. (2004). Effects of visual and auditory cues on gait in individuals with Parkinson's disease. J. of the Neurological Sciences, 219(1-2), 63-69.

Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. J. of the Acoustical Society of America. 26(2), 212-215.

Tzafestas, S. G. & Tzafestas, E. S. (2001). Human-machine interaction in intelligent robotic systems: A unifying consider-ation with implementation examples, J. of Intelligent and Robotic System, 32(2), 119-141.

Tian, Y.-L., Kanade, T., & Cohn, J. (2005). Facial expression analysis. In Handbook of Face Recognition, Li, S.Z., & Jain, A.K. (ed.), Springer.

Ueda, E., Matsumoto, Y., Imai M., & Ogasawara, T. (2001). Hand pose estimation for vision based human interface. In Proc.10th IEEE Int. Workshop on Robot and Human Communication (RO-MAN 2001), 473-478.

Wakita, Y., Hirai, S., Suehiro, T., Hori, T., & Fujiwara, K. (2001). Information sharing via projection function for coexist-ence of robot and human. Autonomous Robots, 10(3), 267-277.

Yamazaki, Y., Dong, F., Masuda, Y., Uehara, Y., Kormushev, P., Vu, H. A., Le, P. Q., & Hirota, K. (2007). Intent expres-sion using eye robot for mascot robot system, In 8th Int’l Symp. on Advanced Intelligent Systems (ISIS2007), 576-580.

Zaatri, A., Waarsing, B. J. W., Van Brussel, H., Tuytelaars, T., & Van Gool, L. (2000). Design and Implementation of a Supervised Intelligent Function. In Proc. Int’l Conf. on Artificial Intelligence (IC-AI’2000), 1-7.