Heterogeneous Data Fusion for an Adaptive

Training in Informed Virtual Environment

Loıc Fricoteaux, Indira Mouttapa Thouvenin and Jerome Olive

Heudiasyc Laboratory

University of Technology of Compiegne

Compiegne, France

Email: firstname.lastname@utc.fr

Abstract—This paper presents an informed virtual environ-ment (environment including knowledge-based models and pro-viding an action/perception coupling) for fluvial navigation train-ing. We add an automatic guide to a driving ship simulatorby displaying multimodal aids adapted to human perceptionfor trainees. To this end, a decision-making module determinesthe most appropriate aids according to heterogeneous datacoming from observations of the learner (his/her mistakes, therisks taken, his/her state determined by using physiologicalsensors, etc.). The Dempster-Shafer theory is used to merge theseuncertain data. The purpose of the whole system is to managethe training almost autonomously in order to relieve trainersfrom controlling the whole training simulation. We intend todemonstrate the relevance of taking the learner’s state intoaccount and the relevance of the heterogeneous data fusion withthe Dempster-Shafer theory for decision-making about the bestlearner guiding. First results, obtained with a predefined set ofdata, show that our decision-making module is able to propose aguiding well-adapted to the trainees, even in complex situationswith uncertain data.

I. INTRODUCTION

Training for steering, driving or piloting has been made with

simulators for many years. Their advantage is to train learners

without risks with a cost and training time reduction [1]. This

is possible thanks to the different parameters of the simulation

which can be easily modified.

As a first platform, we use a fluvial-navigation training

simulator, called SimNav [2], developed by the CETMEF (In-

stitute for inland and maritime waterways) and the University

of Technology of Compiegne (Fig. 1). It allows learners to

experiment barge steering in a virtual environment by using

real controls. This simulator includes a trajectory computing

module, Navmer, allowing knowing the position, the direction

and the speed of the boat in real time according to the

controls operated and the environmental conditions (currents,

winds, banks). Such a simulator allows learners to experiment

navigation, to familiarize themselves with the controls and to

visualize their effects on the behavior of the ship thanks to the

physical engine. However, during a training session with such

a simulator, trainees do not know if they have just made a

This work has been funded by the European Union and Picardie regionunder the OSE project (fOrmation for fluvial tranSport with an informedvirtual Environment). Europe is committed in Picardie with the EuropeanRegional Development Fund.

wrong maneuver, if they have forgotten to perform an action,

if their steering is risky, etc. A trainer has to be present for

each learner during the use of the simulator to help them and

to control the simulation (to make a break, to redo a difficult

passage, etc.).

Fig. 1. The fluvial-navigation simulator SimNav

Our proposal is an adaptive training system, added to

SimNav, which can automatically display aids according to

the learner’s performance. We do not want to suppress the

trainer presence, but to support him. This is possible by

monitoring the learner to extract heterogeneous data about

his/her performance. These data are then merged to take a

decision about the best guiding to display to the learner. This

system is intended to relieve the trainer from controlling the

whole training simulation.

In this paper, we present in section II an overview of existing

adaptive training systems for learner guiding and an overview

of the feedbacks used. Then, section III describes our adaptive

training system, GULLIVER, with an emphasis on the learner

monitoring and the heterogeneous data fusion for decision-

making. Section IV gives an application of our approach with

the heterogeneous data fusion for learner guiding. Section V

presents first evaluations of this approach. We conclude with

its advantages.

II. RELATED WORKS

A. Adaptive Training Systems for learners’ guiding

Several training systems propose an automatic guiding for

the learners. For example, TRUST [3], a truck-driving training

simulator, allows the trainer to add explanations/indications

which will automatically appear during the simulation to help

the learners. In this case, the guiding is automatic but the

system is non-adaptive. Indeed, the explanations are predefined

and will appear no matter how the learners are driving.

Therefore, they will not be appropriated to each learner. Too

much help will be present for some learners and not enough

for others.

What we use is an adaptive training system, like PE-

GASE [4] which is a generic intelligent system for virtual

reality learning. It can guide learners according to their errors,

their profile and domain knowledge. However, this system

can only be used for procedural tasks, because it needs to

know the actions that learners must perform. In our case,

fluvial navigation, it is not possible to have procedures about

good actions to perform for steering, because, in this case, the

situations are too complex. Hence, we use a non-deterministic

approach, as in TELEOS [5] which is a learning environment

for orthopedic surgery. With this environment, the learners

build their knowledge by interacting with the system. The

environment gives a learning situation to the learner in order

to allow him/her to take decisions to acquire more experience.

Therefore, the system must respond pertinently to the learner’s

actions so as to encourage him/her to think about his/her

actions.

More precisely, TELEOS provides feedbacks to the learners

based on their actions, their knowledge level and a standard

knowledge model. The decision-making module used is based

on an influence diagram which takes the uncertainty on the

learner model into account. In our case, we must also make

a decision based on uncertain data, coming from the learner’s

activity recognition and the learner’s state recognition. How-

ever, we do not use an influence diagram for two main reasons.

First, conditional probabilities necessary to build an influence

diagram are not obvious to draw up for an expert in fluvial

navigation [6]. Secondly, with the use of probabilities, we

cannot model ignorance properly [6] and this is necessary in

our case since we have incomplete data (the situation is not

perfectly known). For these reasons, we use the Dempster-

Schafer theory for decision-making.

In comparison with PEGASE, TELEOS and training sim-

ulators with a decision-making module like [7] and [8], we

propose to take not only learners’ actions and their profiles

but also their state (stress state, cognitive load, attention

level, etc.) into account by using physiological sensors. We

use the Dempster-Shafer theory for decision-making on these

heterogeneous data, in order to choose among the available

feedbacks those which will be triggered.

B. Multimodal Feedback Based on Human Perception

To guide the learners throughout the training session, help-

ing feedback (visual metaphors, audio messages, etc.) must

be triggered by the decision-making module. For example,

the car-driving training simulator VDI (Virtual Driving In-

structor) [7] uses a rule-based system to trigger feedbacks.

However, these feedbacks are only audio messages, thus trig-

gered one at a time. In [8], a truck-driving training simulator,

a decision-making module can decide to trigger multimodal

feedbacks: one feedback per canal (visual or audio). Neverthe-

less, these feedbacks are only warning messages, explanations,

complementary information or motivating messages. The feed-

backs are not linked to the virtual environment, for example

by highlighting a traffic sign. This allows triggering more

feedbacks at a time without disturbing the learners because

they do not have to read an explanation or pay attention to

an audio warning message. In PEGASE, the feedbacks are

integrated in the environment. For example, it is possible to

make some important objects flash. In our work, we have

followed this approach.

As multimodal feedbacks, we use visual and sound

metaphors to guide the learners. Visual metaphors are used

to highlight important objects to take into account in steering

(ex: other boats, beacons, obstacles, etc.) [9], [10], [11], [12],

[13]. We also add virtual objects in the virtual environment.

For example, we display the optimal path to follow on the

river, called the highway metaphor [9], [10], [11], [12], [13]

(Fig. 2). We also enrich the environment by adding virtual

instruments which are unavailable in reality. They bring addi-

tional information useful for steering (ex: information about

the position, the tilt, etc.) [10], [12].

Fig. 2. An example of visual aids: the highway metaphor

However, the multiplication of these aids can bring negative

side effects on the simulation:

• learners’ cognitive overload: too much information simul-

taneously, and learners cannot analyze all of them and

do not know which are the most important for the actual

situation.

• learners’ attention focuses on aids rather than on the

simulation: learners spend too much time in analyzing

aids (which are not available in a real situation), to the

detriment of the analysis of the simulated situation.

That is why in our system we automatically moderate the

amount of aids according to human’s perception. We use the

following solutions to avoid these negative side effects.

To avoid catching user attention on the HUD (Head Up Dis-

play) rather than on the environment, information is collocated

in the environment (displaying in the foreground forces to

change the focus between the HUD and the environment) [11].

Information is also gathered in the user’s field of view to

avoid forcing him/her to turn the head so as to save time for

information reading [10].

The visual canal is mainly used to add aids to a training

simulation. In order to not overload this canal to improve the

visibility of the simulation, several techniques exist like the

dynamic level of details of information. Just as the levels of

details of a geometric mesh, levels of details of information

allow an adjustment of the quantity of information displayed

(semantic zoom-in or zoom-out) [14]. This is done according

to several factors like the spatial distance between the user and

information collocated, the importance of this information, etc.

To reduce the amount of information on the visual canal,

the virtual environment can be represented in a simplified

way. In this case, users learn progressively by starting with

an environment which requires a low cognitive load for its

analysis. For example, this can be done by representing

the environment with a cognitive map, which is helpful for

orientation [9].

To display several different data simultaneously without

overloading the visual channel, other modalities can be used

like sound [15], [16] or haptic [17], [18] to display informa-

tion. In our case, we use visual and audio feedbacks.

In our system, we will use all the previous solutions, which

moderate the amount of aids, in order to avoid a cognitive

overload of the learner and to prevent the learner from focusing

on the aids rather than on the simulation.

III. TOWARD HETEROGENEOUS DATA FUSION FOR AN

ADAPTIVE TRAINING IN INFORMED VIRTUAL

ENVIRONMENT

A. User Monitoring

We monitor the learner by a user’s activity detection mod-

ule, a user’s state recognition module and a personalized

profile continuously updated (Fig. 3). This monitoring gives

heterogeneous data used to make a decision about the best

guiding for the learner.

The user’s activity detection module (not yet implemented)

has to determine learner’s mistakes and the risks taken. As nav-

igation is a difficult process to model, it cannot be expressed

as a set of procedures to follow. Therefore, mistakes cannot be

detected by comparing with a set of good actions to perform,

which is a deterministic approach. Instead, this module will be

realized with a non-deterministic approach: the system has to

answer in a credible manner according to the learner’s actions.

The response of the system is not expected to be realistic but

only credible, that is to say the learner must feel present in the

virtual world. He or she can interact freely with the system,

perceive the response of the environment, interact again, etc.

This allows a training by action/perception [19] thanks to the

strong coupling between the user and the system.

Steering errors and risks taken will be detected according

to the future position of the boat, extrapolated from the actual

position of the boat and the actual state of the ship controls.

For example, it will be detected that the boat will collide with

a bridge. This information is uncertain since it is based on

an extrapolation of the future position of the boat in the case

that the learner does not change the boat trajectory. However,

this uncertainty decreases gradually as the boat approaches the

bridge. A risk indicator will be also computed from the room

of maneuver to avoid the next obstacles.

Errors and risks will be detected with a predictive model,

as explained before, but also with a set of rules. For example,

navigation rules about the traffic signs will be used. There

will be also some rules about the use of ship controls. The

learner’s gestures on the ship controls will be tracked and

wrong maneuvers will be detected. We intend to also detect

errors with an eye tracking system allowing knowing where

the learner is looking at. Some elements have to be seen by the

learner to make good decisions (ex: traffic signs). Thus, this

is an error to not have seen such elements and, for example,

the decision-making module can decide to highlight these

elements so that they will be seen.

The user’s state recognition module is based on physiolog-

ical sensors. Currently, a heart-rate variability sensor is used

and we plan to use an eye tracking system (and possibly other

sensors) to determine the learner’s state. This gives indications

about stress, cognitive load, attention level and situational

awareness. The eye tracking system, which will be used to

know where the learner is looking at, will also be used to

determine the learner’s visual attention and his/her cognitive

load thanks to pupillometric measurement [20]. Heart-rate

variability is measured to detect cognitive overloads. Of course

these data are uncertain and this is taken into account by the

decision-making module.

Throughout the learner’s training, a personalized profile is

recorded. It contains the current level of the learner (novice,

intermediate, experienced) and his/her usage history (previous

errors, efficient aids, inefficient aids, etc.).

All the data coming from the learner’s profile, the user’s

state recognition module and the user’s activity detection

module are sent to the decision-making module (Fig. 3).

B. Heterogeneous Data Fusion

This incoming heterogeneous data (the learner’s mistakes,

the risks taken, his/her state determined by using physiological

sensors) have to be expressed in a common formal framework

so that they can be merged and allow taking a decision.

As a common formal framework, we use the Dempster-

Shafer theory. The belief is distributed on triggerable elements:

aids (ex: Fig. 2) and events (ex: adding some danger, like

dead trunks to avoid). Each piece of information (about the

learner’s state, his/her errors, etc.) assigns belief masses for

each triggerable element on the utility of the triggering, the

inutility or the ignorance of the triggering. Thus the frame

of discernment is Ω = utility, inutility and the power set

is 2Ω = ∅, utility, inutility, ignorance with ignorance =Ω. A basic belief assignment (bba) m is a function that assigns

belief masses to the subsets of 2Ω with∑

A⊆Ω

m (A) = 1.

Each information applies a bba to each triggerable element.

As soon as every belief mass has been assigned, they are

combined for each triggerable element using the Dempster’s

rule of combination (1). Then, the pignistic probability of the

utility (2) is computed for each triggerable element in order

to obtain an indicator about the priority of the triggering of

the aids and the events.

∀A ⊂ Ω, (m1 ∩©m2) (A) =∑

B∩C=A

m1 (B)m2 (C) (1)

BetP (utility) =∑

A⊆Ω,utility∈A

m (A)

(1−m (∅)) |A|(2)

Among the aids and events available, the decision-making

module must choose the best set to trigger, that is to say the

set which satisfies as many constraints as possible:

• Elements to trigger must have a high BetP (utility);• The sets of elements to trigger must not overload the

learner’s sensory canals and his/her cognitive load;

• Elements to trigger must be mutually compatible;

• Elements to trigger must be adapted to the learner’s level

and must respect his/her preferences;

• . . .

The trainer can intervene in the system by adding constraints

about the choice of aids or events to trigger (for example, it is

possible to forbid an aid). The system has to find a combina-

tion of triggerable elements that satisfies as many constraints

as possible among all these ones. To resolve this problem,

constraints are divided into two categories: strong constraints

(which necessarily have to be respected) and weak constraints

(which can be ignored but the solution must respect as many of

them as possible). A score is allocated for each rule respected

and the best solution is the one with the highest score (a similar

method is used in [8]). This is a constraint satisfaction problem

which can have a high complexity depending on the number

of aids/events available. In the case that the complexity is too

high to have an answer in real-time, a metaheuristic is used:

a genetic algorithm. This provides a good solution in a time

limit chosen but nothing ensures that this is the best solution.

Later, we will try to use a constraint programming solver in

order to have the best solution as fast as possible.

The computation of the aids/events to display is contin-

uously made so that the guiding is adapted to the current

situation. For example, a visual aid become useless, since the

associated error is not made anymore, will be removed, which

will let space on the visual canal to possibly add another aid.

In another case, if the learner makes a serious mistake, the

associated aid(s) will be displayed and maybe several less

important aids will be removed so as to not overload the

learner.

C. GULLIVER: a Model for an Adaptive Training in Informed

Virtual Environment

We built a model, GULLIVER (GUiding by visuaLization

metaphors for fluviaL navigation training in an Informed

Virtual EnviRonment), to propose an adaptive training in

informed virtual environment (IVE). An IVE is a ”virtual

environment including a knowledge-based model and in which

it is possible to both interact and allow behaviors by inter-

preting dynamic and static representations” [21]. GULLIVER

provides automatic guiding thanks to a decision-making mod-

ule based on heterogeneous data obtained by monitoring the

learner.

GULLIVER takes as input SimNav output data which are

the position, the direction and the speed of the barge controlled

by the learner thanks to the controls associated (Fig. 3).

From these data, the position of the barge is updated in the

virtual environment. Actions (boat movements) and events

(collisions, etc.) are transmitted by the virtual environment to

the module of the user’s activity detection. Information about

the learner’s gestures is also transmitted to this module which

is in charge of detecting the mistakes made by the learner

and the risks taken. The learner’s state is also recognized

thanks to data coming from physiological sensors. From the

learner’s state, his/her mistakes and the risks taken, a decision-

making module activates the right metaphors to guide the

learner. This module can also decide to trigger events. For

example, if the learner does not make mistakes and feels at

ease, the environment is complexified by adding some dangers,

SIMNAVShip controls

GULLIVER

Informed virtual environment

Decision-making module

Detection of user’s activity

User’s state

recognition

Actions, events

Mistakes,

risks

User’s state

User’s state

tracking

Information display metaphors,

event triggering

User’s profile

User’s gestures

tracking

Fig. 3. Model of an adaptive training system: GULLIVER

for instance floating objects to avoid or thick fog. In addition

to the learner’s state, his/her mistakes and the risks taken, the

system also takes the learner’s profile into account: his/her

usage history (previous errors, inefficient aids, etc.) and his/her

level (beginner, experienced, etc.). If the learner is a novice,

the guiding system must adapt to a cognitive speed compatible

with the learner’s perception and comprehension speed to

avoid cognitive overload [19].

IV. APPLICATION OF OUR HETEROGENEOUS DATA FUSION

FOR LEARNER GUIDING

A first set of triggerable elements and of possible incoming

information has been defined to test the decision-making

module. For example, the information I1 ”the learner is

taking a bad way under the next bridge” comes from the

user’s activity detection module with a certainty of 75 %. By

reading an XML tree linking the errors to the correspond-

ing triggerable elements, the decision-making module knows

that the aid A1 ”highlighting of the next bridge traffic-sign

indicating the right way” and the event E1 ”hiding of the

next bridge” can help the learner to correct his/her trajectory.

Therefore, m (utility) = 0.75 and m (ignorance) = 0.25for these two triggerable elements, since the certainty of the

information I1 is 75 %. The other triggerable elements have

m (ignorance) = 1, since the information does not bring

some belief on the utility/inutility of their triggering. With the

information I1, another information I2 is added: ”the learner

is under stress” with a certainty of 60 %. This information

comes from the learner’s state recognition module. For this

kind of information, generic rules are used to assign belief

masses. In this case, the following rule is used: ”if the learner

is under stress, the events which add difficulty must not be

triggered and those which simplify the training scenario are

useful at 50 %”. Therefore, events which add difficulty will

have m (inutility) = 0.6 and m (ignorance) = 0.4, since

the certainty of the information I2 is 60 %. Events which add

simplicity will have m (utility) = 0.3 (50 % of 60 %) and

m (ignorance) = 0.7. The other triggerable elements have

m (ignorance) = 1. After merging the bba of I1 and I2, A1

keeps the same belief masses and E1 has m (utility) = 0.825and m (ignorance) = 0.175 because I2 has influenced E1

(since it is an event which adds simplicity). The pignistic

probability of A1 is 87.5 % and for E1 it is 91.25 %. At

the end, the constraint satisfaction problem is solved and the

decision-making module decides to trigger E1 and not A1,

because they are incompatible (it is useless to highlight the

traffic sign of a bridge which is not visible any more) and E1

has been estimated better than A1 (its pignistic probability is

higher).

This simple example shows the different steps of the

decision-making module. First, uncertain information assigns

belief masses according to a strategy of belief assignment

(which depends on the nature of the information). The belief

assignments have been thought to be intuitive in order to be

defined by an expert in fluvial navigation. At the next step,

the belief masses are merged and the pignistic probability

is computed for each triggerable elements. Finally, the set

of triggerable elements which respects the most constraints

(constraints about human perception, etc.) is chosen to guide

the learner. Such an approach is flexible and allows us to

easily update the system by adding/removing constraints, input

information and triggerable elements.

V. FIRST EVALUATIONS OF HETEROGENEOUS DATA

FUSION FOR LEARNER GUIDING

The decision-making module has been evaluated using gen-

erated data which simulate the possible inputs corresponding

to an observation of a learner (example of inputs: I1 with a cer-

tainty of 65 % and I2 with a certainty of 72 %). The first values

of certainty are randomly chosen and then they change over

time (increase or decrease) with a higher probability to change

in a similar way as their previous trend (Fig. 4). Several sets of

500 values for 13 input data and 24 triggerable elements have

been tested with different parameters: for a beginner learner or

an experienced learner and with different values of the minimal

threshold of BetP (utility). This minimal threshold is used

to determine whether there is enough BetP (utility) for a

triggerable element to consider it as useful to trigger. This is

a strong constraint used in the decision-making module.

Fig. 4. Example of input data resulting on learner’s monitoring

The results (Fig. 5) enabled us to adjust the minimal

threshold of BetP (utility). They are encouraging and show

that the decision-making module gives satisfactory answers.

Fig. 5. Example of results

Further tests on the whole system will be realized with flu-

vial navigation learners, as soon as the system implementation

will be completed. This will allow adjusting more precisely the

different parameters. We intend to demonstrate the relevance

of taking the learner’s state into account and the relevance of

the data fusion with the Dempster-Shafer theory for decision-

making about the best learner guiding.

VI. CONCLUSION

We propose GULLIVER as an informed virtual environment

for fluvial navigation training. This environment automatically

guides learners with multimodal feedbacks adapted to human

perception. The system takes heterogeneous data about the

learner’s performance into account to make a decision about

the best multimodal feedbacks to trigger. The use of the

Dempster-Shafer theory for the fusion of these data brings

a new approach in the informed virtual environments. This

allows a strong coupling between the user and the virtual

environment. Our first results show that heterogeneous data

fusion is a powerful method for adaptive guiding based on user

monitoring. Our approach presents the following advantages:

1) better adaptation to the trainee, 2) possibility to specifically

display multimodal aids according to a decision-making mod-

ule and 3) possibility of adaptive guiding in complex situations

with uncertain data.

Our system, GULLIVER, is not limited to fluvial navigation

training and can be used, for example, for assisting car drivers

in augmented reality.

ACKNOWLEDGMENT

We deeply thank A. Pourplanche and F. Hissel, who work at

the CETMEF, for their enthusiastic support on the applicative

part of our work.

REFERENCES

[1] J. Olive, “Realite virtuelle pour la production de pneumatique,” Ph.D.dissertation, Universite de Technologie de Compiegne, 2010.

[2] M. Vayssade and A. Pourplanche, “A piloting SIMulator for maritimeand fluvial NAVigation: SimNav,” in Proc. Virtual Concept, Biarritz,France, 2003.

[3] D. Mellet d’Huart, “De l’intention a l’attention. Contributions a unedemarche de conception d’environnements virtuels pour apprendre apartir d’un modele de l’(en)action,” Ph.D. dissertation, Universite duMans, 2004.

[4] C. Buche, C. Bossard, R. Querrec, and P. Chevaillier, “PEGASE: AGeneric and Adaptable Intelligent System for Virtual Reality LearningEnvironments,” The International Journal of Virtual Reality, vol. 9,no. 2, pp. 73–85, 2010.

[5] D. Mufti-Alchawafa, “Modelisation et representation de la connaissancepour la conception d’un systeme decisionnel dans un environnement in-formatique d’apprentissage en chirurgie,” Ph.D. dissertation, UniversiteJoseph Fourier - Grenoble I, 2008.

[6] N. El-Kechaı, “Suivi et assistance des apprenants dans les environ-nements virtuels de formation,” Ph.D. dissertation, Universite du Maine,2007.

[7] I. Weevers, J. Kuipers, J. Zwiers, B. Dijk van, and A. Nijholt, “Thevirtual driving instructor: a multi-based system for driving instruction,”CTIT technical reports series, 2003.

[8] M. Lopez-Garate, A. Lozano-Rodero, and L. Matey, “An adaptiveand customizable feedback system for VR-based training simulators,”in Proc. Autonomous agents and multiagent systems, vol. 3, Estoril,Portugal, 2008, pp. 1635–1638.

[9] C. Benton and R. Walker, “Augmented Reality for Maritime Naviga-tion: The Next Generation of Electronic Navigational Aids,” in Proc.

Marine Transportation System Research and Technology Coordination

Conference, Washington, USA, 2004.

[10] O. Bjorneseth, “HOTS (Highway On The Sea), a new approach to low-visibility navigation,” in Proc. Marine Simulation and Ship Maneuvra-

bility, Japan, 2003.

[11] D. Foyle, R. McCann, and S. Shelden, “Attentional issues with su-perimposed symbology: formats for scene-linked displays,” in Proc.

International Symposium on Aviation Psychology, Columbus: Ohio StateUniversity, 1995, pp. 98–103.

[12] O. Hugues, J.-M. Cieutat, and P. Guitton, “An Experimental AugmentedReality Platform Application for Assisted Maritime Navigation: Follow-ing Targets,” in Proc. Virtual Reality International Conference, Laval,France, 2010, pp. 149–154.

[13] L. J. Prinzel III, L. J. Kramer, R. E. Bailey, J. J. Arthur, S. P.Williams, and J. McNabb, “Augmentation of Cognition and PerceptionThrough Advanced Synthetic Vision Technology,” in Proc. International

Conference on Augmented Cognition, Las Vegas, NV, USA, 2005.[14] D. A. Bowman, C. North, J. Chen, N. F. Polys, P. S. Pyla, and

U. Yilmaz, “Information-rich virtual environments: theory, tools, andresearch agenda,” in Proc. Virtual Reality Software and Technology,Osaka, Japan, 2003, pp. 81–90.

[15] B. Frohlich, S. Barrass, B. Zehner, J. Plate, and M. Gobel, “Exploringgeo-scientific data in virtual environments,” in Proc. Visualization, SanFrancisco, California, USA, 1999.

[16] S. Onimaru, T. Uraoka, N. Matsuzaki, and M. Kitazaki, “Cross-modalinformation display to improve driving performance,” in Proc. Virtual

Reality Software and Technology, Bordeaux, France, 2008, pp. 281–282.[17] G. Bouyer, “Rendu multimodal en Realite Virtuelle : Supervision des

interactions au service de la tache,” Ph.D. dissertation, Universite ParisXI, 2007.

[18] S. Onimaru and M. Kitazaki, “Visual and Tactile Information to Im-prove Drivers’ Performance,” in Proc. IEEE Virtual Reality, Waltham,Massachusetts, USA, 2010, pp. 295–296.

[19] S. Bottecchia, “Systeme T.A.C. : Tele-Assistance Collaborative. Realiteaugmentee et NTIC au service des experts et des operateurs dans le cadred’une tache de maintenance industrielle supervisee.” Ph.D. dissertation,Universite de Toulouse III, 2010.

[20] O. Palinko, A. L. Kun, A. Shyrokov, and P. Heeman, “Estimatingcognitive load using remote eye tracking in a driving simulator,” in Proc.

Symposium on Eye-Tracking Research - Applications, Austin, Texas,2010, pp. 141–144.

[21] I. Mouttapa Thouvenin, “Interaction et connaissance : construction d’uneexperience dans le monde virtuel,” Habilitation thesis, Universite deTechnologie de Compiegne, 2009.