Bioinspired Visuovestibular Artificial Perception System for Independent MotionSegmentation

Jorge Lobo, João Filipe Ferreira and Jorge DiasInstitute of Systems and RoboticsUniversity of Coimbra — Polo II

3030-290 Coimbra, Portugal{jlobo,jfilipe,jorge}@isr.uc.pt

Abstract

In vision based systems used in mobile robotics and vir-tual reality systems the perception of self-motion and thestructure of the environment is essential. Inertial and earthfield magnetic pose sensors can provide valuable data aboutcamera ego-motion, as well as absolute references for struc-ture feature orientations. In this article we present severaltechniques running on a biologically inspired artificial sys-tem which attempts to recreate the “hardware” of biolog-ical visuovestibular systems resorting to computer visionand inertial-magnetic devices. More specifically, we ex-plore the fusion of optical flow and stereo techniques withdata from the inertial and magnetic sensors, enabling thedepth flow segmentation of a moving observer. A depth mapregistration and motion segmentation method is proposed,and experimental results of stereo depth flow segmentationobtained from a moving robotic/artificial observer are pre-sented.

1. Introduction

In biological vision systems, inertial cues provided bythe vestibular system play an important role, and are fusedwith vision in the early processing stages of image pro-cessing (e.g, the gravity vertical cue). Artificial perceptionsystems for robotic applications have since recently beentaking advantage from low-cost inertial sensors for comple-menting vision systems, using both static and dynamic cues.

Inertial sensors attached to a camera can provide valu-able data about camera pose and movement. Micromachin-ing enables the development of low-cost single-chip inertialsensors that can be easily incorporated alongside the cam-era’s imaging sensor, thus providing an artificial vestibularsystem. Figure 1 shows a stereo-camera pair with an iner-tial measurement unit (IMU) mounted on a mobile robotic

ground plane

{I }

{W }

movingobserver

verticalfeatures

{C }

gravity field

3D segmented depth map

Figure 1. Stereo vision system with an inertialmeasurement unit used on robotic system,frames of reference and its 3D segmenteddepth map output.

platform. The 3D-structured world is observed by the vi-sual sensor, and its pose and motion are directly measuredby the inertial sensors. These motion parameters can alsobe inferred from the image flow and known scene features.Combining the two sensing modalities simplifies the 3D re-construction of the observed world. Inertial sensors alsoprovide important cues about the observed scene structure,such as vertical and horizontal references. Inertial naviga-tion systems obtain velocity and position by integration, anddo not depend on any external references, except gravity.

The inertial-sensed gravity vector provides a unique ref-erence for image-sensed spatial directions. More specifi-cally, previous work has shown that the use of visual sen-sors together with IMUs can be used to estimate camera fo-cal distance [9] or to perform cross-calibration [3]. Know-ing the vertical-reference and stereo-camera parameters, theground plane can be fully determined. The collineation

between image ground-plane points can be used to speedup ground-plane segmentation and 3D reconstruction. Us-ing the inertial reference, vertical features starting from theground plane can also be segmented and matched acrossthe stereo pair, so that their 3D position is determined. Theinertial vertical reference can also be used after applyingstandard stereo-vision techniques; taking the ground planeas a reference, the fusion of multiple maps reduces to a 2Dtranslation and rotation problem, and dynamic inertial cuesmay be used as a first approximation for this transforma-tion, providing a fast depth-map registration method (Fig-ure 1) [8]. In addition, inertial data can be integrated intooptical flow techniques, through compensating camera ego-motion, improving interest-point selection, matching the in-terest points, and performing subsequent image-motion de-tection and tracking for depth-flow computation. The imagefocus of expansion and centre of rotation are determined bycamera motion and can both be easily found using inertialdata alone, provided that the system has been calibrated.This information can be useful during vision-based naviga-tion tasks.

Three-dimensional scene flow estimation was studied byVedula et al. [15][14]. Several scenarios are presented, andthe tradeoffs between structure knowledge, correspondencematching, number of cameras and computed optical flowexplored. Dense scene flow estimation using only two cam-eras was proposed by Li and Sclaroff by fusing stereo andoptical flow estimation in a single coherent framework [7].Ye Zhang and Kambhamettu computed dense 3D scene flowand structure from multiview image sequences with non-rigid motion in the scene [17]. Stereoscopic MPEG basedvideo compression methods also deal with motion flow seg-mentation, such as the joint motion and disparity fields es-timation method proposed by Yang et al. [16]. A statisticalapproach to background modelling was used for segmenta-tion of video-rate stereo sequences by Eveland et al.[5].

However, when dealing with a free moving stereo cam-era observer, the methods described above are not directlyapplicable. Visual and inertial sensing are two sensorymodalities that can be explored to give robust solutionson image segmentation and recovery of 3D structure fromimages [9]; inertial sensors provide valuable data to dealwith the camera motion [10]. Consequently, artificial sys-tems dealing with motion perception in more complex sit-uations would clearly gain by introducing bionsinspired vi-suovestibular sensing using computer vision and inertial-magnetic devices. In this article, we will present ap-proaches for ego-motion and independent motion percep-tion and segmentation based on these biologically inspiredvisuovestibular artficial systems.

Correlation-based stereo depth maps can be generatedfrom a moving vision system, and rotated to a common lev-elled reference provided by the rotation update from inertial

Biological Perception,Bayesian Model

Artificial Perception,Bayesian Model

Artificial Observer (Vision+Inertial)

SensorReadings

Artificial Perception

Human/BiologicalObserver

(Vision+Vestibular)

Perception Psychophysical Study ⇒ Model Analysis

Artificial & Biological

Model Output Comparison

Ego-MotionIllusions, Conflicts & Ambiguities

Model Re-

evaluation

Model Re-

evaluation

Model

SynthesisM

odel

Synthesis

Sensation

3D Scene

& Moving Objectsw/ Static Objects

Figure 2. Biomimetic artificial perception re-search proposal schematic (human observerimage courtesy of 3DScience.com).

sensed gravity and magnetic sensed bearing. Voxel quanti-zation can then be performed on the resulting maps. Butthere remains a 3D translation in the successive depth mapsdue to the motion, for which the inertial sensors only pro-vide a rough estimate. By tracking some image targets oversuccessive frames, the system translation between framescan be estimated by subtracting their 3D position. Thetranslation can also be estimated from the 3D data alone.For scenes where a base horizontal plane is always visible(eg: the floor or desktop), a histogram in height can be usedto have a common reference along the vertical axis. Thiscan also be performed for the horizontal axis if the orien-tation of visible planes is known or detected by a 2D fit tothe data. The two identified planes provide the translationto merge successive depth maps.

Fully registered depth maps can therefore be obtainedfrom the moving system — our solution for correlation-based stereo depth map registration is presented on section2.

The depth flow that remains in the resultant map is dueto the system covering new scenes, or to moving objectswithin the overlap volume of successive observations. Mis-matches between the depth from stereo and depth from opti-cal flow indicate possible independent motion. This can beused to better segment moving objects in the overlap vol-ume and avoid artifacts from slow moving objects. On sec-tion 3 we describe our approaches for independent motionsegmentation using these registered maps.

Subsequently, we present results on section 4, draw someconclusions on section 5 and finally, on section 6, we dis-cuss the outcome of our studies and propose future work

{W }

{C }|i

{I }|i

{C }| i+1

{I }|i+1

{R }| i+1

{R }|i

Figure 3. Moving observer and world fixedframes of reference.

which will pave way to the next step in this line of research:from bioinspired to biomimetic.

As can be seen on Figure 2, the outcome of this kind ofresearch may become not only a significant advance in ar-tificial sensing, such as in the solution of ego-motion or in-dependent motion segmentation which we propose to tackledirectly, but also a challenge in allowing further insight onaiding human beings to surpass their own perceptive lim-itations, helping disambiguation and coping with illusionsor conflicts arising in extreme conditions where humans areprone to failure, namely: in extreme environments (e.g., inspace exploration), where humans are displaced from nor-mal conditions and factors such as 1G gravity; in perceptivepathologies (i.e. perception-impaired patients).

2. Registering Stereo Depth Maps

Biological visuovestibular systems take into accountego-motion, and deal well with independent motion seg-mentation, while they successfully integrate 3D informa-tion of the surrounding scene as the biological observermoves along its trajectory. Taking this into account, our ap-proaches to ego and independent motion segmentation weredevised so as to take advantage of the output of a precedingstereo depth map registration procedure.

A moving stereo observer of a background static scenewith some moving objects can compute at each instant acorrelation-based dense depth map. The maps will changein time due to both the moving objects and the observerego-motion. A first step to process the incoming data isto register the maps to a common fixed frame of reference{W}, as shown on Figure 3.

The stereo cameras provide intensity images Il(u, v)|iand Ir(u, v)|i, where u and v are pixel coordinates, and

i the frame time index. Having the stereo rig calibrated,depth maps for each frame can be computed. A set of 3Dpoints CP|i is therefore obtained at each frame, given in thecamera frame of reference {C}|i. Each 3D point has a cor-responding intensity gray level c given by the pixel in thereference camera, i.e c = Il(u, v)|i. Each point in the setretains both 3D position and gray level

P (x, y, z, c) ∈ CP|i . (1)

2.1. Rotate to Local Vertical and MagneticNorth

The inertial and magnetic sensors, rigidly fixed to thestereo camera rig, provide a stable camera rotation updateRRC relative to the local gravity vertical and magneticnorth camera frame of reference {R}|i.

Calibration of the rigid body rotation between {I}|i and{C}|i can be performed by having both sensors observinggravity, as vertical vanishing points and sensed acceleration,as described in [11].

The rotated camera frame of reference {R}|i is time-dependent only due to the camera system translation, sincerotation has been compensated for.

2.2. Translation from Image Tracked Target

The translation component can be obtained using a sin-gle fixed target tracked in the scene. The image feature musthave the corresponding 3D point Pt in each depth map, sothat translation can be estimated from

∆~t = Pt|i+1 − Pt|i (2)

with Pt|i+1 ∈ RP|i+1 and Pt|i ∈ RP|i.The fixed target can be an artificial one, or set of sparse

tracked natural 3D features can be used to improve robust-ness, but assumptions have to be made in order to rejectoutliers that occur from tracking features of the moving ob-jects.

2.3. Voxel Quantization

The above equations are provided for discrete sets ofpoints. In order to deal with noise and allow 3D volumeprocessing, a 3D array is built representing 3D space as vox-els. For each stereo frame, the corresponding cubic array ofvoxels Vox|i can be built. For the occupied voxels the corre-sponding gray level can be stored in the array. When two ormore points contribute to the same voxel, the average graylevel is used.

Figure 4. Summary of voxel background sub-traction and optical flow consistency anal-ysis methods for independent motion seg-mentation.

For each P (x, y, z, c) ∈ CP|i, Vox(x, y, z)|i = c if pre-viously empty, or Vox(x, y, z)|i = c̄, where c̄ is the averagegray level of the contributing points.

For a sequence of stereo frames, two cumulative voxelarrays Voxc and Voxv can be built for both gray level andoccupancy statistics over the frames, with

Voxc(x, y, z) = c̄v, Voxv(x, y, z) = v (3)

where v is the number of frames that voted voxel (x, y, z)as occupied, and c̄v the average gray level from the votingframes.

3. Independent Motion Segmentation in FullyRegistered Maps

Having the dense depth maps in a common frame of ref-erence we can proceed to segment the moving objects seenby the moving stereo observer. Biological vision systemsare very successful in movement segmentation since theyefficiently resort to flow analysis and accumulated priorknowledge of the 3D structure of the scene. Artificial per-ception systems may also build 3D structure maps and useoptical flow to provide cues for ego and independent motionsegmentation (see Figure 4). The maps will change in timedue to moving objects, and eventually grow as the artificialobserver covers new scene areas.

3.1. Background Subtraction for Voxel Seg-mentation

Occupancy statistics can be used to segment the set ofvoxels that correspond to the static scene observed by themoving system, and segment the moving objects.

Applying a threshold vback on the accumulated votecount, a binary array of background voxels Voxb can be builtas

Voxb(x, y, z) = 1 when Voxv(x, y, z) > vback . (4)

To improve noise filtering and robustness, a thinning andgrowing transformation is applied, removing isolated vox-els and filling in gaps. The thinning filter takes out voxelswithout a minimum number of neighbours, by performinga convolution with a cubic unit kernel and thresholding theresult back to a binary array. The growing simply performsa convolution with a cubic unit kernel, and rebuilds the bi-nary array with all the non-zero voxels.

For a single frame i, the set of voxels from moving ob-jects will be given by

Voxm|i = Vox|i⋂

Voxb . (5)

To deal with noise, thinning and growth smoothing canalso be applied to Voxm|i, but smearing of the intensity graylevel might not help subsequent 3D intensity based meth-ods.

The underlying assumption is that the moving observerrepeatedly covers the same scene so that background voxelsare seen more times than moving objects. Experimental re-sults show that moving objects are successfully segmentedand that thinning and growth smoothing filter out noise fromthe correlation based stereo depth maps.

3.2. Optical Flow Consistency Segmenta-tion

Optical flow is the apparent motion of brightness patternsin the image. Generally, optical flow corresponds to theprojected motion field, but not always. Shading, changinglighting and some texture patterns might induce an opticalfield different from the motion field. However since whatcan be observed is the optical field, the assumption is madethat optical flow field provides a good estimate for the trueprojected motion field.

Optical flow computation can be made in a dense way,by estimating motion vectors for every image pixel, or fea-ture based, estimating motion parameters only for matchedfeatures.

Representing the 2D velocity of an image pixel u =(u, v)T as du

dt , the brightness constancy constraint says thatthe projection of a world point has a constant intensity overa short interval of time, i.e., assuming that the pixel intensityor brightness is constant during dt, we have

I(u +du

dtdt, v +

dv

dtdt)|t+dt = I(u, v)|t (6)

Figure 5. Experimental setup of 3D scenewith static background and swinging pendu-lum.

If the brightness changes smoothly with u, v and t, we canexpand the left-hand-side by a Taylor series and reject thehigher order terms to obtain

∇I · du

dt+

∂I

∂tdt = 0 (7)

where ∇I is the image gradient at pixel u. These spatialand time derivatives can be estimated using a convolutionkernel on the image frames.

But for each pixel we only have one constraint equation,and two unknowns. Only the normal flow can be deter-mined, i.e., the flow along the direction of image gradient.The flow on the tangent direction of an isointensity contourcannot be estimated. This is the so called aperture prob-lem. Therefore, to determine optical flow uniquely addi-tional constraints are needed.

The problem is that a single pixel cannot be tracked, un-less it has a distinctive brightness with respect to all of itsneighbours. If a local window of pixels is used, a local con-straint can be added, i.e., single pixels will not be tracked,but windows of pixels instead.

Barron et al. [4] present a quantitative evaluation of op-tical flow techniques, including the Lucas-Kanade method,that uses local consistency to overcome the aperture prob-lem [12]. The assumption is made that a constant model canbe used to describe the optical flow in a small window.

When the camera is moving and observing a static scenewith some moving objects, some optical flow will be consis-tent with the camera ego-motion observing the static scene,other might be moving objects. Since the stereo providesa dense depth map, and we reconstruct camera motion, wecan compute the expected projected optical flow in the im-age from the 3D data.

In the perspective camera model, the relationship be-tween a 3D world point x = (X, Y, Z)T and its projectionu = (u, v)T in the 2D image plane is given by

u =P1 (x, y, z, 1)T

P3 (x, y, z, 1)T(8)

Figure 6. Overlaid rotated 3D depth mapsfrom frames 1 and 20 (on the right) showinga clear mismatch, and circled image featuretracked to estimate translation.

Figure 7. Depth maps rotated and translatedto common world fixed frame of reference,for frames 1 and 20 on the left, and for fullset of frames with moving pendulum on theright.

v =P2 (x, y, z, 1)T

P3 (x, y, z, 1)T(9)

where Pj is the jth row of the camera projection matrix P .When the camera moves, the relative motion of the 3D

point dxdt will induce a projected optical flow given by

dui

dt=

δui

δx

dx

dt(10)

where δui

δx is the 2 × 3 Jacobian matrix that represents thedifferential relationship between x and ui, which can beobtained by differentiating (8) and (9).

Image areas where the computed flow is inconsistentwith the expected one indicate moving objects, and the cor-responding voxels can be segmented. This approach doesnot require the occupancy statistics memory, since it’s dif-ferential and can be applied to pairs of successive frames.

Experimental results show that this method works on se-quences with significant optical flow. However, this pro-cedure is noise sensitive and, due to its differential based

Figure 8. 3D volume of all accumulated vox-els in frame sequence on the left, and withvote count above 30 on the right.

Figure 9. Background voxels after thinningfor at least 6 neighbours on the left, and sub-sequent growth with a size 53 kernel on theright.

estimation, it performs poorly at low speeds, where the un-certainties in camera motion and optical flow are higher.

A summarising diagram of the procedures for both inde-pendent motion segmentation methods studied in this workis presented on Figure 4.

4. Results

The hardware system used to acquire data from a movingobserver is shown in fig. 1. The stereo vision is provided bythe Videre MEGA-D Digital Stereo Head [1], and the posefrom the inertial and magnetic sensor package MT9-B fromXsens [2].

To compute range from stereo images we are using theSRI Stereo Engine with the Small Vision System (SVS)Software [6].

A scene was set up with a swinging cylindrical can toprovide motion independent from the observer movement,as shown on Figure 5. The moving observer surveyed thescene performing map registration and subsequent indepen-dent motion segmentation as described below.

Figure 10. Initial voxel set and segmentedmoving object voxels for one frame.

4.1. Moving Depth Map Registration

As described above, the rotation update provided by theinertial and magnetic sensor package is applied to the suc-cessive depth maps. As shown on Figure 6, the depth mapsare correctly rotated, but shifted due to the observer transla-tion.

The translation was estimated by tracking an image fea-ture, and observing the translation between the correspond-ing 3D points in the depth maps. Figure 6 shows data forframes 1 and 20 of a take of 200 frames with a moving ob-server of a static scene with a moving pendulum, for whichthe registration performed well.

The registered depth map can be seen in Figure 7. Thefused map from frames 1 and 20 is shown on the left. On theright the fused map corresponding to the full set of framesis shown with the moving pendulum leaving its trace.

4.2. Background subtraction for Voxel Seg-mentation

The above results are shown with VRML rendering ofthe full set of computed points without voxel quantization.As described above, occupancy statistics can be used toidentify the static scene voxels.

In a new test sequence, a one cubic meter volume of theobserved space was chosen as the working volume, quan-tized to a 100 × 100 × 100 array corresponding to 1 cm3

voxels.Figure 8 shows the 3D volume of all accumulated voxels

for this test sequence with 130 frames, and the ones witha vote count above the empirically chosen threshold of 30.This choice was made based on the following observations:very low thresholds will mark slow objects as background;too high will segment newly observed static background asmoving objects. Frame rate, observer motion and indepen-dent motion velocities are determining factors when choos-ing appropriate thresholds.

Image Optical Flow (LK)

Estimated Optical Flow

Optical Flow Difference

Figure 11. Difference between observed andestimated optical flow indicating areas incon-sistent with static scene after camera motioncompensation.

Figure 12. Output of the two methods for thesame frame, voxel background subtractionon the left, and optical flow consistency onthe right.

The result of thinning and growth filters applied to thebackground voxels is shown in Figure 9. Figure 10 showsinitial voxel set and segmented moving object voxels af-ter thinning and subsequent smoothing, for one frame fromthe above sequence. These results show that moving objectvoxels can be successfully segmented, however the movingobserver has to cover the same scene more than once so thatthe background voxels can be correctly segmented.

4.3. Optical Flow Consistency Segmenta-tion

Figure 11 shows the optical flow at one frame within atest sequence where the pendulum and observer were bothmoving. The image observed optical flow was estimatedwith the Lucas-Kanade [12] method applied to successiveframes. The predicted flow was computed considering the

3D motion of the depth map relative to the camera, and pro-jecting onto the image using (10).

The difference between the observed optical flow andthe predicted flow indicate areas inconsistent with a staticscene. The voxels associated with these image pixels corre-spond to moving objects. A decision threshold is applied tothe optical flow difference to segment the voxels. The resultfor this frame is shown in figure 12 on the right.

The method works, but is clearly sensitive to noise dueto the differential based estimation. In the complete test se-quence there, frames with small camera motion performedpoorly, since the uncertainties in camera motion and opticalflow computation degrade the detection of moving objects.

5. Conclusions

Two methods were presented for motion segmentationfor a moving observer of a background static scene withsome independently moving objects. The moving observerhas stereo vision and inertial and magnetic sensors to pro-vide a rotation update. Having compensated rotation, trans-lation can be obtained from a single tracked image feature.Depth maps from stereo vision can therefore be registeredto a common frame of reference.

Occupancy statistics can then be used to segment thevoxels between the static background scene and moving ob-jects. However, the moving observer has to cover the samescene so that the background voxels can be correctly seg-mented. An alternative method is to check the consistencyof the observed image optical flow. This approach is differ-ential and can be applied to pairs of successive frames, butis more noise sensitive.

Figure 12 shows the output of the two methods for thesame frame. The voxel background subtraction correctlyidentifies the independent motion. Due to this fact, in thiswork it also provided a ground truth to compare the opti-cal flow consistency method. The optical flow consistencymethod also segments the independent motion, but withadded false positives due to uncertainties in the optical flowcomputation and camera motion reconstruction.

On the other hand, voxel background subtraction re-quires a volumetric representation of the whole workspace,and also some past history statistics, which introduces astart-up lag of at least 10 frames, whereas optical flow con-sistency only needs the present and immediately precedingframes to function.

Therefore, a hybrid method can be devised which wouldtake advantage of the strengths of both of these methods byusing a differential approach based on optical flow whilst re-taining a short-term memory of 3D space occupancy, sincethe inertial data allows fast depth map registration. Further-more, this hybrid approach would more closely follow whatindeed happens in biological/human perception systems,

where priors gathered from past states of the workspace be-ing perceived are combined with fast low-level processingof retinal optical flow.

6. Discussion and Future Work

Although the techniques presented in this text are basedon models that assume sensing technology that attempts torecreate the “hardware” of biological visuovestibular sys-tems, no attempt has yet been made to follow the internalbiological models of perception.

The usefulness of introducing models which mimic bio-logical systems of perception and the limitations of biologi-cal perception posed by the physiological characteristics ofbiological motion sensors, which in certain situations yieldpartial or ambiguous information, has been demonstrated inprevious research (see, for example, work by Reymond etal. [13]). Biological visuovestibular systems take into ac-count ego-motion, and deal well with independent motionsegmentation. In spite of this, however robust, biologicalperception estimation processes are prone to suffering fromillusions, conflicts and ambiguities.

We have thus reached a point in which the next step willbe to take artificial perception to the next level: from bioin-spired to biomimetic — see figure 2.

We therefore propose in future work to perform psy-chophysical studies, such as in [13], of human visuovestibu-lar models under a Bayesian framework, to implement thesemodels as closely as possible using the technology pre-sented on [9, 3, 8] in a robotic-based artificial perceptionsystem, to tackle 3D structure perception (specifically in-dependent motion segmentation in the presence of self-motion), and to test the possibilities opened by the robust-ness of artificial sensor technology as opposed to biologicalsensory solutions on extreme perception tasks (see Figure2). In the case of independent motion segmentation, wewill address the use of inertial dynamic data to improve theoptical flow consistency check, without depending on anytracked feature for the translation, and on combining thetwo methods to improve robustness.

Acknowledgements

This publication has been supported by EC-contractnumber FP6-IST-027140, Action line: Cognitive Systems.The contents of this text reflect only the author’s views. TheEuropean Community is not liable for any use that may bemade of the information contained herein.

References

[1] Videre Design. http://www.videredesign.com/.[2] Xsens Technologies. http://www.xsens.com/.[3] J. Alves, J. Lobo, and J. Dias. Camera-Inertial Sensor mod-

eling and alignment for Visual Navigation. In 11th Interna-tional Conference on Advanced Robotics, pages 1693–1698,July 2003.

[4] J. Barron, D. Fleet, and S. Beauchemin. Performance of Op-tical Flow Techniques. International Journal of ComputerVision, 12(1):43–77, February 1994.

[5] C. Eveland, K. Konolige, and R. C. Bolles. Backgroundmodeling for segmentation of video-rate stereo sequences.In Conference on Vision and Pattern Recognition, Santa Bar-bara, CA, USA, June 1998.

[6] K. Konolige. Small vision systems: Hardware and imple-mentation. In Eighth International Symposium on RoboticsResearch, Hayama, Japan, October 1997.

[7] R. Li and S. Sclaroff. Multi-scale 3d scene flow from binoc-ular stereo sequences. In To appear in Proc. IEEE Workshopon Motion and Video Computing, January 2005.

[8] J. Lobo and J. Dias. Inertial Sensed Ego-motion for 3D Vi-sion. In International Conference on Advanced Robotics,pages 1907–1914, July 2003.

[9] J. Lobo and J. Dias. Vision and Inertial Sensor CooperationUsing Gravity as a Vertical Reference. IEEE Transactionson Pattern Analysis and Machine Intelligence, 25(12):1597–1607, December 2003.

[10] J. Lobo and J. Dias. Inertial sensed ego-motion for 3d vision.Journal of Robotic Systems, 21(1):3–12, January 2004.

[11] J. Lobo and J. Dias. Relative pose calibration between visualand inertial sensors. In ICRA 2005 Workshop on Integra-tion of Vision and Inertial Sensors (InerVis2005), Barcelona,Spain, April 2005.

[12] B. D. Lucas and T. Kanade. An Iterative Image RegistrationTechnique with an Application to Stereo Vision . In Pro-ceedings of Imaging Understanding Workshop, pages 674–679, 1981.

[13] G. Reymond, J. Droulez, and A. Kemeny. Visuovestibularperception of self motion modelled as a dynamic optimiza-tion process. Biol. Cybern., 87:301–314, 2002.

[14] S. Vedula, S. Baker, P. Rander, R. Collins, and T. Kanade.Three-dimensional scene flow. In Proceedings of the Sev-enth IEEE International Conference on Computer Vision,volume 2, pages 722–729, 1999.

[15] S. Vedula, P. Rander, R. Collins, and T. Kanade. Three-dimensional scene flow. Pattern Analysis and Machine Intel-ligence, IEEE Transactions on, 27(3):475–480, March 2005.

[16] W. Yang, K. Ngan, J. Lim, and K. Sohn. Joint motion anddisparity fields estimation for stereoscopic video sequences.Signal Processing: Image Communication, 20(3):265–276,March 2005.

[17] Y. Zhang and C. Kambhamettu. On 3-d scene flow andstructure recovery from multiview image sequences. IEEETransactions on Systems, Man and Cybernetics, Part B,33(4):592–606, August 2003.