OBSTACLE DETECTION AND AVOIDANCE ON SIDEWALKS

D. CastellsUniversidad Politecnica de Madrid, Spain

J. M. F. Rodrigues and J. M. H. du BufVision Laboratory, Institute for Systems and Robotics (ISR), University of the Algarve (ISE and FCT)

8005-139 Faro, Portugal

Keywords: Sidewalk border detection, Obstacle avoidance, Path tracking, Visually impaired.

Abstract: We present part of a vision system for blind and visually impaired people. It detects obstacles on sidewalks andprovides guidance to avoid them. Obstacles are trees, light poles, trash cans, holes, branches, stones and otherobjects at a distance of 3 to 5 meters from the camera position. The system first detects the sidewalk borders,using edge information in combination with a tracking mask, to obtain straight lines with their slopes andthe vanishing point. Once the borders are found, a rectangular window is defined within which two obstacledetection methods are applied. The first determines the variation of the maxima and minima of the gray levelsof the pixels. The second uses the binary edge image and searches in the vertical and horizontal histograms fordiscrepancies of the number of edge points. Together, these methods allow to detect possible obstacles withtheir position and size, such that the user can be alerted and informed about the best way to avoid them. Thesystem works in realtime and complements normal navigation with the cane.

1 INTRODUCTION

Every car and bicycle can be equipped with aGPS/GIS-based navigation system that may cost afew hundreds of euros. By contrast, blind and visu-ally impaired persons need to navigate using the stickor, at best, an ultrasonic obstacle detector. This asym-metry needs to be solved, because there are an esti-mated 180 million persons with severe impairmentsof which 40-50 million are completely blind, and ev-ery year 2 million more become blind. The Por-tuguese project “SmartVision: active vision for theblind” aims at developing a portable GIS-based nav-igation aid for the blind, for both outdoor and indoornavigation, with obstacle avoidance and object recog-nition based on active vision modules.

There are a few recent systems for visually im-paired users which may assist them in navigation,with and without obstacle detection and avoidance,e.g., (Lee and Kang, 2008) who developed a systemwhich integrates outdoor navigation and obstacle de-tection. (Kim et al., 2009) presented an electronictravel aid called iSONIC. It complements the conven-tional cane by detecting obstacles at head-height.

The work presented here concerns one of the mod-ules of the SmartVision project. This module servesto detect sidewalk borders and assists the user in cen-tering on the sidewalk, thereby avoiding any obsta-cles. Typical obstacles are light poles, trash cans andtree branches, also imperfections as holes and loosestones, at a distance of 3 to 5 meters from the user.The system automatically adapts to different types ofsidewalks and paths, and it works in realtime on a nor-mal portable computer.

2 SYSTEM SETUP

In the SmartVision project, a stereo camera (Bum-blebee 2 from Point Grey Research Inc.) is fixedto the chest of the blind, at a height of about 1.5mfrom the ground. Results presented here were ob-tained by using only the right-side camera, but thesystem performs equally well using a normal, inex-pensive webcam with about the same resolution. Theresolution must be sufficient to resolve textures of thepavements related to possible obstacles like holes and

235Castells D., M. F. Rodrigues J. and M. H. du Buf J. (2010).OBSTACLE DETECTION AND AVOIDANCE ON SIDEWALKS.In Proceedings of the International Conference on Computer Vision Theory and Applications, pages 235-240DOI: 10.5220/0002816002350240Copyright c© SciTePress

loose stones with a minimum size of about 10 cen-timeters at a distance of 3 to 5 meters from the cam-era (the first meters are not covered because of theheight of the camera; this area is covered by the caneswayed by the user). Figure 1 (top) shows a typicalframe; Fig. 4 shows one of our test sequences.

The system is composed of three processing steps:(1) Sidewalk border detection and the definition of theobstacle detection window (ODW). (2) The detectionof obstacles in the ODW using two complementaryprocesses for tracking irregularities: (i) the numberof local maxima and minima of pixel values, and (ii)histograms of binary edge information. (3) Trackingof obstacles in subsequent frames for alerting the userand obstacle avoidance.

2.1 Sidewalk Border Detection

There are some methods to detect the borders of side-walks (Kayama et al., 2007). Here we detect themby using a simple edge detector in combination witha tracking mask to obtain straight lines, from the bot-tom of each frame to the top, characterized by slope,length and proximity to the left and right boundariesof the frame. The detected borders will define the hor-izontal position and width of the ODW.

We apply the Canny edge algorithm (Heath et al.,1997) with three parameters:σ defines the size of theGaussian filter, andTh and Tl are the high and lowthresholds for hysteresis edge tracking. We alwaysuseσ = 1.5, Th = 0.95 andTl = 0.5. Figure 1 (2ndimage from top) shows edges detected in the case ofthe frame shown above. Other edge detectors mayperform better, see (Rodrigues and du Buf, 2009), butmost require more CPU time which is critical in thisapplication.

In order to detect potential sidewalk borders, sev-eral horizontal test lines are defined in the binary edgeimage,Ii(x,y) with i the frame number. Figure 1 (2ndimage) shows these on the left (in red). On the firsttest line “1TL” there are many edge pixels which maybe part of potential border lines. They are startingpoints which are labelled differently for the testingprocess. A line-tracking mask of size 5x3 (Fig. 1,bottom-right) is applied to track connected pixels up-wards, vertically with an opening angleα ≈ 120deg,from “1TL” to “2TL”, 20 pixels higher, and con-nected pixels are attributed the same label. Then,starting at “2TL”, the process is repeated for findingmore potential border lines, complementing the firstsearch. New labels are generated for pixels on “2TL”which are not connected to pixels on the first test line.This second search continues until the last testing line“last TL”, 200 pixels above “1TL”. Figure 1 (3rd row,

Figure 1: Sidewalk border detection. Top to bottom: aninput frame; edge detection with testing lines indicated (inred) at the left; start of tracking edge pixels at the first testline (1 TL) with label 43 (at left) plus the filling of a smallgap (at right); the three steps a, b and c for coping with veryclose lines; and the line tracking mask with angleα.

at left) shows the mask tracking edge pixels with label43 and (at right) tracked edge pixels with a small gapwhich will be filled.

Occasionally, there are two or more labeled edgepixels which are very close. These could correspond

VISAPP 2010 - International Conference on Computer Vision Theory and Applications

236

to a single or to several border lines. In those cases,when the mask is applied to a new edge pixel, thealready labelled pixels below are checked by usingthe vertically mirrored mask. If there are many pixelswith a label not equal to the label of the mask’s centralpixel, the label of the central pixel changes to the oneof the majority of the pixels in the mask. The 4th and5th rows in Fig. 1 show an example of this processwith steps a, b and c. Final images with connectedand labelled edge pixels are denoted byLi(x,y), butthese still containpotentialborder lines. To be con-sidered sidewalk borders, detected lines must satisfythe following three requirements:

(1) Connected edge pixels must have a minimumlength (MinEL) covering at least 80 vertical positions(MinEL 1TL or MinEL 2TL, depending on the linestarting on 1TL or on 2TL, see Fig. 1, 2nd image fromtop). Shorter series are removed.

(2) Connected edge pixels must be almost lin-ear, i.e., with correlatonr > 0.9, with a slope|b|between 0.5 and 10. The slope also provides infor-mation about the sidewalk’s width: the higher theslope, the narrower the sidewalk. In order to speedup the correlation/slope process, only eight equidis-tant points of each potential line are processed. Thecorrelationr of edge pixels with the same label is

given byr = σ2xy/

√

σ2x ·σ2

y, whereσ2xy is the covari-

anceσ2xy = ∑xy/n− (∑x ·∑y)/n2 andσ2

x andσ2y are

the variances ofx andy

σ2x =

∑x2

n−

(

∑xn

)2

;σ2y =

∑y2

n−

(

∑yn

)2

. (1)

The regression line is given byL = a+bx, with

b=∑x ·∑y−n∑xy

(∑x)2−n∑x2;a=

∑x ·∑xy−∑y ·∑x2

(∑x)2−n∑x2. (2)

(3) Occasionally, more than two lines remain aspotential sidewalk borders, or none at all. Depend-ing on the number of linesnl , the following is done:(i) If no lines are found,nl = 0, the last two bordersfound in a previous frame will be used. (ii) If a sin-gle line is found,nl = 1, the second line will be au-tomatically generated, symmetrically with respect tothe vertical line that passes through the intersectionof the line found and the horizontal line through thevanishing point. The latter is updated dynamically foreach new frame, as explained below. (iii) If two linesare found,nl = 2, they are accepted as sidewalk bor-ders, but with the following exception: If the signs ofthe slopes of the two lines are not different, this meansthat we do not have the right lines. The outermost oneis ignored, the innermost one is used, and a new lineis generated as in casenl = 1. Here, innermost means

closest to the center of the frame and outermost clos-est to the left or right frame borders. (iv) In the caseof more lines,nl > 2, the most symmetrical and innerpair of lines is selected.

Above, the vanishing point is used to generatesymmetrical line pairs. At the start of a sequence offrames, or when no acceptable sidewalk borders canbe detected, the height of the vanishing point will beinitialized at 3/4 of the frame height. Then, whentwo correct sidewalk borders are found, the vanish-ing point is determined by the intersection of the twoborders, and the point is dynamically updated by av-eraging the points of the previous frame and the newframe.

After obtaining two valid sidewalk borders, theobstacle detection window (ODW) is defined for de-tecting and locating possible obstacles. This windowhas predefined upper and a lower limits with a heightof Nv,ODW = 100 pixels (v is vertical); see Section 2.The left and right limits are defined by taking 80% ofthe distance between the two borders found at the up-per limit, which givesNh,ODW pixels (h is horizontal).

3 OBSTACLE DETECTION ANDAVOIDANCE

For obstacle detection, two different methods are ap-plied to the ODW. The first one counts variations ofgray values, i.e., local maxima and minima, on eachhorizontal line inside the ODW. Then, outliers are re-duced by averaging counted variations over groups oflines, and variations over these groups are combinedinto a single value which indicates whether the framehas a possible obstacle or not. Final confirmation isobtained by combining the results of a few subsequentframes. The second method is based on irregulari-ties in vertical and horizontal histograms of the binaryedge imageIi. An obstacle can lead to two differ-ent signatures: if the pavement is smooth, an obstaclemay appear as a local excess of edge points, but ifit has a strong texture there will be a huge amountof edge points and an obstacle may appear as a localdeficiency (lack or gap) of edge points. The secondmethod is used to confirm the result of the first one,but it also serves to detect the size and position of anobstacle in order to guide the user away from it.

3.1 Local Maxima and Minima

(a) A small lowpass filter (averaging block filter ofsize 3x3; LP(x,y)) is applied twice to the graylevelODW of frame i (FODW,i), so high frequencies are

OBSTACLE DETECTION AND AVOIDANCE ON SIDEWALKS

237

suppressed and a less noisy window can be pro-cessed. If∗ denotes convolution, thenFODW,i(x,y) =FODW,i(x,y)∗LP(x,y)∗LP(x,y).(b) Then, the variations of gray values on each hori-zontal line of the window are computed by applyingthe first derivativeF ′

ODW,i(x) = ∂(FODW,i(x))/∂x.(c) In order to keep significant variations, a thresholdTd = ±2 is applied to the derivative. This suppressessmall transitions and maxima and minima can noweasily be found: where the derivative changes its sign(zero crossing orZC), +/− for a maximum and−/+for a minimum.(d) The next step consists of counting on each hor-izontal line y the number of maxima and minimaMM(y) over the ODW window (100 horizontal lines),MMi(y) = ∑ZC[Td[F ′

ODW,i(x)]].(e) The result is stabilized by removing outliers.This is done by taking the average ofMM overtriplets of lines, i.e., over three consecutive horizontallines, which results in only 33 values for each ODW.With k = 0,1,2 and line counting starting aty = 0,MMi(y/3) = ∑MMi(y+ k)/3.(f) For calculating variations over the ODW’slines, the first derivative is applied toMMi(y/3),MM′

i (y/3) = ∂(MMi(y/3)/∂y.(g) The last processing step of the ODW of frameiconsists of determining the maximum value(max) ofthe absolute value ([·]+) of the derivative from step(f), maxi = max[MM′

i (y/3)]+. This value indicates apossible obstacle in the ODW.(h) In order to detect and confirm obstacles, a dy-namic threshold is used to alert the user. The dy-namic threshold is initialized by computing the av-erage of maxi over the first five frames,max=∑5

i=1maxi/5, and the average of the deviation, i.e.,the difference betweenmax and eachmaxi , dev=[

∑5i=1(maxi −max)/5

]+. The first threshold,T6 =

max+ dev, is going to be tested againstmax6 ob-tained from frame 6, after this frame has been pro-cessed from step (a) to (g). The same processing isdone withmaxi , devi andTi . Two conditions can oc-cur for i > 5:(h.1) If maxi does not exceed thresholdTi , a newthreshold for the next frame will be calculated byTi+1 = 4/5 ·Ti +1/5 · (maxi +devi).(h.2) Otherwise, a warning-level counter is activated.If maxi of the next two frames continue exceedingthe threshold, an obstacle warning will be issued.The same happens when, after the warning level hasbeen activated, there is only one frame which doesnot exceed the threshold. If more than two consecu-tive frames do not exceed the threshold, the warningcounter will be reset and the threshold will continueto be adapted dynamically.

The processing described above detects big vari-ations of the number of local maxima and minima,first in the horizontal lines and then over the lines inthe ODW. This allows to detect the appearance andthe disappearance of an obstacle in the window, be-cause these coincide with the first and lastmaxi whichexceed the dynamic threshold. For detecting the po-sition and size of an obstacle, we apply an analysis ofedge histograms.

3.2 Edge Histograms

This method exploits the already available edge mapsIi(x,y), see Section 2.1, but only inside the ODWIODW,i(x,y). Depending on the smoothness (texture)of a sidewalk’s pavement, different characteristics areexpected. Ifrw is the fraction of the number of“white” (edge) pixelsNw,ODW in IODW,i(x,y), withNh,ODW andNv,ODW the window’s dimensions,rw =Nw,ODW/(Nh,ODW×NvODW).Extensive tests with real pavements, but withoutobstacles, revealed thatrw > 0.1 indicates roughsurfaces, for example with cobblestones like inPortuguese-style “calcada,” whereas smaller valuesindicate smooth surfaces. In both cases, vertical andhorizontal edge histograms are computed, i.e., foreach line and for each column in the ODW the num-ber of “white” pixels are summed. Then, the two his-tograms are smoothed by applying a simple 1D aver-aging filter.

Two thresholds,Tc andTl , are computed for thehistograms of the lines and columns. For the columnhistogram,Tc is the ratio between the number of whitepixels in the ODW and the number of columns of thewindow, Tc = (Nw,ODW/Nh,ODW)×K, and, similarly,Tl is the ratio between number of white pixels and thenumber of lines,Tl = (Nw,ODW/Nv,ODW)×K, whereK is a normalization factor.

In the case of a smooth pavement(rw < 0.1), anobstacle will appear as an excess of white points, seeFig. 2, and we applyK = 0.8. An obstacle will bedetected if at least two neighboring values in both theline and column histograms exceed the thresholds,TlandTc, respectively.

In the case of a textured pavement(rw ≥ 0.1), anobstacle will appear as a lack of white points, seeFig. 3, and we applyK = 0.4. Now at least two neigh-boring values in both histograms must be lower thanthe corresponding thresholds.

3.3 Obstacle Avoidance

In order to detect an obstacle, both methods describedabove must detect something, but in order to save

VISAPP 2010 - International Conference on Computer Vision Theory and Applications

238

Figure 2: Smooth pavement with a possible obstacle andcorresponding edge histograms.

CPU time the histogram method is only applied toframes in which the first method has detected some-thing. Once the histogram method has confirmed thedetection, a sound is generated in order to alert theuser. This sound is modulated such that it indicatesthe approximate position and the best way to avoidthe obstacle.

The first method indicates an obstacle somewherein the entire ODW. Often, but not always becauseit depends on the textures of the pavement and theobject and the latter’s size, the histogram methodcan narrow the approximate position to the object’sbounding box. Blind people prefer to walk near wallsor the facades of buildings along streets, swaying thecane in front and keeping contact with the wall orfacade. Since a wall- and facade-detection algorithmhas not yet been implemented, we illustrate walkingclose to the centerline of paths and sidewalks in thispaper. This scenario is also quite realistic, and avail-able information about the path’s borders and the van-ishing point can be used to inform the user about thecenterline. In addition, having the position and di-mensions (in pixels) of an obstacle’s bounding box,and the dimensions (in pixels and in meters) of theODW, see Section 2, it is easy to convert the bound-ing box to meters and provide information about theapproximate position and size of the obstacle. If theobstacle is not centered on the path, the user can beinformed about the left or right side which has thelargest distance between the bounding box and thepath’s left and right borders. In any case, the userwill check the obstacle using the cane.

It should be stressed that the camera is tightlyfixed to the person’s breast such that it points straight

Figure 3: Textured pavement with a possible obstacle andcorresponding edge histograms.

forward, also that the blind have been trained not tosway much with the body while swaying the cane.The obstacle-avoidance module requires initial cali-bration in order to obtain correct distances, for thesedepend on each user’s length and posture. In thefuture, a disparity module which uses both camerasof the stereo camera will be integrated. This mod-ule will complement the obstacle-detection methodsand it will also provide calibrated distance informa-tion, such that the calibration mentioned above willno longer be required.

4 CONCLUSIONS AND RESULTS

Various test sequences have been captured on severalpaths and sidewalks of Gambelas Campus at the Uni-versity of the Algarve, with different pavements andobstacles. Figure 4 shows one sequence with, apartfrom the frame number given in the upper-left cor-ner, the following frame annotation: Type 0 concernthe first 5 initial calibration frames with no obstacles.Type 1 frames are those in which the variation of max-ima and minima in the ODW does not exceed the dy-namic threshold. Type 2 frames do exceed the thresh-old and the first one activates an alert counter whichcounts, from 1 toN, the number of subsequent Type2 frames. The third alert (N = 3) activates “obstaclewarning” which remains until the first new frame ofType 1 is encountered.

The first results are very encouraging. The num-bers of false positives and negatives of the sequences

OBSTACLE DETECTION AND AVOIDANCE ON SIDEWALKS

239

tested were quite small, with more false positives thanfalse negatives. False positives were mainly causedby tree leaves and litter, whereas false negatives weremainly cobblestones pushed up a few centimeters bytree roots, such that the irregularity of the calcada’stexture is not detectable.

On a portable computer with an Intel Pentiumclocked at 1.6 GHz, elapsed time to process individ-ual frames is about 0.5 second. This is already fastenough for realtime application where the user walksat normal speed. By using a new portable with amulti-core processor, more than two frames per sec-ond can be processed, but the disparity module willalso consume CPU time (although disparity process-ing might be limited to the ODW), as will other mod-ules for using GPS in combination with a dedicatedGIS for autonomous navigation.

In general, good results were obtained in the de-tection of the borders of paths and sidewalks, andmany more paths and sidewalks are now being tested.Difficulties mainly arise when the color difference orcontrast of a path’s curbs is small, when the curbsare partly hidden by plants and long grass, or whena path has no curbs but is delimited by grass or lowshrubs. Similar problems arise in the case of obsta-cle detection, when the contrast and the texture of anobstacle and those of the pavement are too similar.However, most obstacles, including missing cobble-stones in Portuguese-style “calcada,” the smallest butmost frequent problem, can be detected, but not yetelevated cobblestones which are being pushed up bylong tree roots. Therefore, detection algorithms mustbe improved, even at the cost of more CPU time.

A specific problem is the detection of single andmultiple steps, for which no dedicated algorithm hasbeen included yet. Occasionally, a wrong obstacle de-tection window is caused by a wrong detection of theborders. However, normally this happens in a sin-gle frame and the problem can be solved by keep-ing the alert counter counting such that at the nextType 2 frame an obstacle warning will be issued.This solution, i.e., tracking information over multipleframes, can be applied in many more cases. For exam-ple, positions of borders detected in previous framescan be extrapolated to new frames in order to narrowthe search area and to confirm the new borders, al-though sudden and unpredictable movements of theuser cannot be excluded unless they are detected bybig changes of the global optical flow of entire frames.Such aspects require more research because of theCPU times which are involved. Also being devel-oped is a dynamic adaptation of the parameters of theCanny edge detector as a function of the type of pave-ment, for resolving finer textures of obstacles like ele-

Figure 4: One test sequence with annotation.

vated cobblestones, but also for the detection of oftenminute differences between textures of horizontal andvertical surfaces of steps when the contrast betweenthem is too low.

ACKNOWLEDGEMENTS

Portuguese Foundation for Science and Technol-ogy (FCT) through the pluri-annual funding ofthe Inst. for Systems and Robotics (ISR/IST), thePOSConhecimento Program with FEDER funds, andFCT project SmartVision (PTDC/EIA/73633/2006).

REFERENCES

Heath, M., Sarkar, S., Sanocki, T., and Bowyer, K. (1997).A robust visual method for assessing the relative per-formance of edge-detection algorithms.IEEE Trans.PAMI, 19(12):1338–1359.

Kayama, K., Yairi, I., and Igi, S. (2007). Detection ofsidewalk border using camera on low-speed buggy.Proc. IASTED Int. Conf. on Artificial Intelligenceand Applications,13-15 February, Innsbruck, Austria,pages 262–267.

Kim, L., Park, S., Lee, S., and Ha, S. (2009). An electronictraveler aid for the blind using multiple range sensors.IEICE Electronics Express, 11(6):794–799.

Lee, S. and Kang, S. (2008). A walking guidance systemfor the visually impaired.Int. J. Pattern Recogn. Artif.Intell., 22(6):1171 – 1186.

Rodrigues, J. and du Buf, J. (2009). Multi-scale lines andedges in V1 and beyond: Brightness, object catego-rization and recognition, and consciousness.BioSys-tems, 95:206–226.

VISAPP 2010 - International Conference on Computer Vision Theory and Applications

240