1

Proper Reference: Bender, P. , and Bone, G. M., "Automated Grasp Planning and Execution

for Real-world Objects using Computer Vision and Tactile Probing", International Journal of

Robotics and Automation, Vol.19, No.1, pp.15-27, 2004.

AUTOMATED GRASP PLANNING AND EXECUTION FOR REAL-WORLD OBJECTS

USING COMPUTER VISION AND TACTILE PROBING

P. A. Bender and G. M. Bone*

Department of Mechanical EngineeringMcMaster University

Hamilton, Ontario, Canada, L8S 4L7*Corresponding author. E-mail: gary@mcmaster.ca

Abstract

This paper describes an automated grasping system suitable for complex 2.5D real-world objects

(i.e. objects with height < width). The system hardware consists of a robotic manipulator, a

three-fingered dexterous hand with a palm mounted CCD camera and a PC. The shape of the

given object, along with its position and orientation, are not known by the system a priori. A 2D

model of the object is first obtained using computer vision. This model, along with height

information obtained using online tactile measurements, forms the input to the grasp planner.

The planner extends our second order limited mobility grasping theory to generate optimal

immobilizing 2D grasps. Two novel quality metrics are employed in the optimization. The grasp

planning theory is applicable to grasps with three or more fingers. The tactile information is used

to extend the grasp to 2.5D objects. We assume that the fingers of the robotic hand will provide

an out-of-plane constraint for the object. Experiments are performed on three complex shaped

automotive parts using a three-fingered, nine-axis, dexterous hand. The total time for object

modeling and grasp planning was under 0.3 s for each part. The parts were successfully grasped

and immobilized in all of the tests performed.

2

Keywords: Grasp planning, grasp analysis, grasping, second order immobilization, second order

mobility, dexterous hands, computer vision, tactile probing.

1. Introduction

An important goal in robotics research today is the ability to grasp objects whose shape is not

known a priori. This has application in both service robotics where many unknown objects are

encountered, and in the growing area of rapid response, small batch manufacturing of customized

products. This requires a method for generating a model of the object online, followed by grasp

planning based on this model. The most common way to gain geometric information about an

unknown object is through the use of a CCD camera.

In recent years, there has been some investigation into the grasping of objects using visual

information. Schrott [1] used visual information to identify the object to be grasped from a

known set of objects. Laugier et al. [2] combined visual data with partially known geometric in-

formation about the object, as well as the workspace information for the robot, to grasp the

objects. Kragic, Miller and Allen [3] started with a CAD model of the object, used computer

vision to find its pose, then planned and executed the grasp online. In all of these papers the

object was at least partially known a priori.

Bendiksen and Hagar [4], as well as Sanz et al. [5] described vision systems which grasp

unknown planar objects with a parallel-jaw gripper. Rodrigues, Li, Lee and Rowland [6,7]

performed the same task using a three-fingered gripper, which acts in a similar manner as a par-

allel-jaw gripper, i.e. it only has two states: open and closed.

3

Other researchers have dealt with unknown three dimensional objects. Taylor et al. [8] and

Trobina and Leonardis [9] presented visually guided systems that grasped arbitrarily shaped 3D

objects from a pile. Davidson and Blake [10] visually planned a 2D “cage” grasp and applied it

to 3D objects. All made use of simple two-fingered grippers. Taylor et al. generated a model by

performing world motions about the object to determine the most ideal grasp location, while

Trobina and Leonardis used range sensors to create the 3D model.

Bard et al.[11], and Stansfield [12] both presented a different approach to the problem. They

made use of a Salisbury hand, which is a three-fingered device, where each finger has three

degrees of freedom. Bard et al. used stereo vision to obtain a 3D model of the unknown part.

However, this model was found to be inadequate for successful grasping. Stansfield used a

structured lighting vision system to obtain a set of object surface points. Stansfield also

employed grasping knowledge obtained from studying the way a human performed the task, in

order to improve the system’s performance.

The systems described above are all used to grasp relatively simple, prismatic objects. In many

applications (e.g. the automotive industry), a wide variety of 2.5D objects (defined as those with

height<width) are encountered, and they tend to have complex contours which are not easily

modeled.

Our objective is to create a system that can grasp real-world 2.5D objects, without any prior

geometric knowledge of the object, with a bias towards complex shaped parts. We employ a

three-fingered dexterous hand [13]. The hand has a CCD camera mounted in its palm that is

used to obtain visual information about the unknown part to be grasped. The goal is to perform

4

the task in the simplest, most effective way possible. Using a simple 2D model of the part,

obtained online from the camera, an optimal immobilizing grasp is determined. The grasp is

then extended to 2.5D using tactile information.

In this paper, our methodologies for automated object modeling and grasp planning are described

first along with some numerical examples. The experimental procedure and results are then

presented, and conclusions drawn.

2. Object Modeling

The 2.5D object is first modeled as a 2D projection using computer vision. This process is

described in section 2.1. The extension of the planned grasp to 2.5D using tactile information is

described in section 3.2.

2.1 2D Modeling using Computer Vision

Our objective is to create a concise 2D model suited for the grasp planning algorithm. The grasp

planner exploits object surface curvature to achieve 2D immobilization with three or more

frictionless contacts. Concave segments are preferable for grasping. This has been proven by

several researchers, for example see Rimon and Burdick [16]. For this reason we extract the set

of concave edges and the set of non-concave edges from the image of the object.

Prior to obtaining the image, it is assumed that the object is stationary, within the field of view,

and lying horizontally on a black surfaced table. The image plane of the camera is set to be

parallel to the table surface, and the camera has been calibrated a priori. The captured image is

processed online to obtain the 2D model.

5

The first step in processing the image consists of thresholding, to separate the object from the

background, followed by binary edge detection. The edge detected image is then searched for

continuous edges. These edges are stored in chain coded form and classified as either inside

edges (i.e. holes) or the outside edge (i.e. the perimeter). The total set of edges, E, is given by:

}...,,2,1{},{ ei nie ==E (1)

where ei is an individual edge, and ne is the number of edges. Each edge consists of a set of

ordered points:

}...,,2,1{)},,({ ijjji npjyxPe == (2)

where xj and yj are the X-axis and Y-axis image coordinates of the point Pj, and npi is the number

of points contained in edge ei.

Having found all continuous edges, the next step is to find the concave and non-concave edge

segments. Referring to Fig. 1, the approximate tangent angle at the jth point is given by:

−+

−+−

−

−=

δδ

δδθjj

jjj xx

yy1tan (3)

where δ is a small positive integer. To determine whether or not a specific point lies within a

concave edge segment, we examine the rate of change of this angle with respect to the change in

position along the edge, ∆s, as follows:

( ) ( )

−+−

−⋅=

∆∆

−+−+

−+

211

211

11

jjjj

jj

j yyxxK

sθθθ , (4)

where

directionsearch theofright isobject theif 1, directionsearch theofleft isobject theif 1,-

=K .

The constant K is used to change the sign depending on whether the object is to the left or right

6

of the search direction. K is easily determined by checking appropriate pixels in the thresholded

image. A concave segment, csk, is identified as a sequential set of points along a given edge, ei,

such that:

}...,,2,1{, km

ncms

=∀>

∆∆ εθ (5)

where nck is the number of points belonging to the segment csk, and ε is a small positive real

constant. The set of concave segments identified from all edges is:

}21{}{ csk n...,,,k,cs= =C (6)

where ncs is the number of concave segments .

Similarly, a non-concave edge segment, fsk, is a set of sequential edge points that satisfy:

}...,,2,1{, km

nfms

=∀≤

∆∆ εθ (7)

The set of non-concave segments from all edges is:

}21{}{ fsk n...,,,k,fs= =F (8)

where nfs is the number of non-concave segments. The magnitude of ε will determine the relative

size of ncs and nfs. A single edge, ei, must contain at least one edge segment that may be either

concave or non-concave. In general there is no relationship between the number of segments and

the number of edges other than: ( ) efscs nnn ≥+ . Also, segments containing a small number of

points will be deleted from C and F in our implementation since they are impractical for

grasping.

Two examples illustrating the application of equations 4, 5 and 7 are shown in Fig. 1. In (a), ∆θ

is positive but since the object is to the left of the search direction K=−1 and the edge segment

7

will be identified as non-concave. In (b), ∆θ is positive, K=1, and the segment will be identified

as concave, assuming that ( ) εθ >∆∆ s for all of the points shown.

3. Grasp Planning

After generating a model of the object, the next task is to plan an immobilizing grasp for it.

Planning the 2D grasp involves three stages. The first stage is the generation of all potential

grasps. All of the potential grasps are then tested for object immobilization. In the final stage,

each immobilizing grasp is evaluated using two quality metrics and an optimal grasp is

determined. The optimal 2D grasp is then extended to 2.5D by performing a tactile probing

operation. The details of these procedures are described in the following.

3.1 Generating Potential 2D Grasps

Since concave edge segments are beneficial for grasping, the potential grasps will be determined

from the concave segments alone if possible. It is assumed that the robotic hand or gripper has at

least three fingers, only one finger can fit on each concave segment, and each finger forms a

frictionless point contact with the object (i.e. the finger radius of curvature < object radius of

curvature at the point of contact). For an nf fingered hand, a potential grasp is then described by

the set:

{ } { } { } nfcsnfknfkki kkknkkkcsccsccsc=pgnf

≠≠≠∈∈∈∈ ... and ...,,2,1,...,,...,,, 212121 21 (9)

where pgi is the ith potential grasp, and cj is the contact point between the jth finger and the

object. If the fingers are identical (as is typical of most dexterous hands), the number of unique

combinations of fingers and segments is given by:

)!(!!

nfnnfn

ncs

csu −= (10)

8

For example, with a three fingered hand and five concave edge segments, the number of

combinations equals ten. Generating these combinations is relatively straightforward and will

not be covered here. It must be noted that each finger/segment combination provides numerous

potential grasps so the total number of potential grasps is often large (i.e. in the thousands for a

typical object). The number of potential grasps will be denoted npg. So far we have assumed that

the nfncs ≥ so that using the non-concave segments is unnecessary. If this assumption is

incorrect then csnnf − non-concave segments must be used with equation 9 to complete the

potential grasps.

3.2 Testing for 2D Immobilization

Having determined all potential grasps, the next step is to determine which of these will provide

object immobilization. Plut and Bone [14,15] developed a second order grasp planning strategy

known as Limited Mobility Grasping (LMG) that is extended in this paper.

Several planar objects in contact with circular fingers are shown in Fig. 2. The fingers are

assumed to be fixed in position and rigid frictionless contacts are assumed. Traditional first

order grasping theories do not consider the local geometry of finger-object contact. The finger is

assumed to have a zero radius, resulting in overly conservative predictions of object

immobilization. For example, first order analysis of Fig. 2(a) would suggest that the object can

rotate slightly about point A, meaning it is not immobilized. However, for a realistic finger with

a non-zero radius, this rotation would require the fingers to penetrate the object so the rotation is

infeasible and the object is immobilized. This was first observed by Rimon and Burdick [16].

We are more interested here in the three finger situation since this requires simpler and less

9

expensive hardware. The square object illustrated in Fig. 2(b) is clearly not immobilized by the

three fingers and is free to translate as shown. Similarly the circular object shown in Fig. 2(c) is

free to rotate about its centre and cannot be immobilized by any number of fingers. We refer to

situations similar to (b) and (c) as exceptional cases. It has been previously mentioned that

concavities are beneficial for grasping. It is not just the object curvature, but also its relative

location that is important. For example, the object shown in Fig. 2(d) is not immobilized even

though one of the finger-object contacts is at a concavity. By relocating one of the other fingers

relative to the concavity, object immobilization may be achieved as shown in Fig. 2(e). A

contact at a concavity provides a tangential constraint that does not exist with a non-concave

edge. Therefore, additional contacts at concavities will improve the ability of the grasped object

to resist external forces. However, it must be noted that an object may also be second order

immobilized using three fingers and non-concave edge segments (see the example shown in Fig.

2(f)), unless it belongs to the exceptional cases previously mentioned.

LMG has the advantage that it is relatively easy to compute, making it suitable for online

planning. When used with concave edge segments, LMG also provides convergence to the

desired contact locations in the presence of initial positioning errors of either the object or the

robot. This convergence is desirable in most assembly tasks. The planning strategy applies to

both inside-out and outside-in grasps. With an inside-out grasp the fingers are moved away from

each other to complete the grasp, while they are moved towards each other with an outside-in

grasp. When the contact points have been reached the fingers are locked in position,

immobilizing the part.

LMG examines the feasibility of relative motion between the object and the fingers. If relative

10

motion is infeasible the object is immobilized. To properly model the motion the finger

geometry must be included. Assuming a circular finger cross-section, the finger path is

equivalent to a zero radius finger tracing the object grown by the finger radius Rf. This idea is

illustrated in Fig. 3. The point corresponding to contact point ci is denoted gi and is termed a

grasp point. Next, the polygon formed by the grasp points is analyzed. For an immobilizing

inside-out grasp the side lengths of the grasp polygon will be locally maximized. This may be

tested by displacing each grasp point by a small distance along its path. Specifically, a necessary

but not sufficient condition for a potential grasp pgi to be an immobilizing inside-out grasp is as

follows:

},,2,1{ and }1,1{],[],[],[],[ nfavggLggvgLggLggvgL kakajaja …∈−∈∀<∆+∨<∆+ (11)

where

=<+

=nfanfaa

j,1

,1,

=>−

=1,

1,1anf

aak , L[p,q] is the Euclidean distance between grasp

points p and q, and ∆g is a small positive displacement of the grasp point along its path. For an

outside-in grasp condition (11) should be used with the inequalities reversed. Note that while

outside-in grasps may be applied to any combination of inside or outside edges, inside-out grasps

cannot be applied if any of the contact points lies on the outside edge.

3.2.1 Special Consideration for Three Finger Grasps

Static equilibrium is a necessary but not sufficient condition for an immobilizing grasp. With

frictionless contacts in 2D the contact forces must act normal to the edges, and usually at least

four fingers are required [16]. However, if the lines of action of the contact forces are concurrent

then equilibrium may be satisfied using only three fingers. This occurs because the concurrency

makes the moment equilibrium equation redundant. Grasps of this type are known as concurrent

grasps. We have developed a simple test for three finger 2D concurrent grasps based on Ceva’s

11

theorem and the law of sines. A three finger 2D grasp is a concurrent grasp if and only if:

( )( )( ) ( )( )( )133221332211 tltltltltltl ⋅⋅⋅=⋅⋅⋅ (12)

where il are vectors tangential to the sides of the grasp polygon, and it are tangent vectors for

each finger location. The vectors may be obtained using:

113

112

111

3

2

1

313

232

121

−+

−+

−+

−≈

−≈−≈

=

==

−=

−=

−=

kk

jj

ii

k

j

i

PPtPPtPPt

cPcPcP

gglgglggl

(13)

Since the edges have been discretized it is unlikely that any potential grasp will satisfy equation

12. In practice some contact friction will always be present so grasps that are close to being

concurrent will also satisfy static equilibrium. For these reasons we relaxed the equality in (12)

in our implementation.

3.2.2 Exceptional Cases

For certain objects, (11) will hold but the grasp will not be truly immobilizing. Examples were

shown in Fig. 2(b)-(d). Bender [17] has extended LMG to properly deal with these exceptional

cases. Firstly, the following necessary but not sufficient condition must be satisfied for all ui∈{-

1,0,1} and i∈{1,2,…,nf}, except the case when u1 = u2 =…= unf = 0:

[ ] [ ]∨<∆+∆+ 212211 g,gLgug,gugL[ ] [ ]∨<∆+∆+ 323322 g,gLgug,gugL

· · ·[ ] [ ]111 ,, ggLguggugL nfnfnf <∆+∆+ (14)

12

where ∆g is a small positive displacement of the contact point gi along the object edge. The

downside of (17) is that nf 3−1 comparisons are required whereas (11) requires only 2nf

comparisons.

Secondly, another set of exceptional cases must be considered to complete the algorithm. These

cases satisfy (11) and (14) but are not valid immobilizing grasps. Davidson and Blake [10] made

a similar observation in their work on two-fingered grasps, and termed these cases “false local

minima”. We have developed a simple test to eliminate these invalid cases for nf-fingered grasps

by exploiting the existing image data. For an inside-out grasp, the distances between successive

contact points are increased by the equivalent of a few image pixels. If these test points lie in the

object (and not in the background) the grasp is a valid inside-out grasp. This approach is

illustrated by two examples in Fig. 4. In (a) all of the test points lie in the object so the grasp is

valid. In (b) one of the test points lies in the background so the grasp is invalid. With an outside-

in grasp the same check may be performed after the distances have been decreased.

3.2.3 Summary of the Algorithm

In summary, the algorithm for testing the potential grasps for immobilization consists of the

following steps:

1. Select the first potential grasp from the set.

2. If condition (11) is satisfied then continue, else go to step 8.

3. If condition (14) is satisfied then continue, else go to step 8.

4. Test for the false local minima case using the method from section 3.2.2. If the grasp

passes the test then continue, else go to step 8.

13

5. If 3≠nf then go to step 7, else continue.

6. If condition (12) is satisfied then continue, else go to step 8.

7. The potential grasp is an immobilizing grasp. Add it to the set of immobilizing grasps.

8. The potential grasp is not an immobilizing grasp. If the potential grasp is not the last

member of the set then select the next potential grasp and go to step 2, else stop.

Steps 2 and 3 of the algorithm warrant further explanation. From the point of view of sufficiency

we could remove step 2 without affecting the algorithm since condition (14) is a more restrictive

version of condition (11). However, this change would require nf 3−1 comparisons to be

performed for every potential grasp rather than 2nf comparisons. Since the majority of potential

grasps will not satisfy condition (11) a relatively small number will reach step 3. So the

combination of steps 2 and 3 is a much more computationally efficient approach. The number of

immobilizing grasps found by this algorithm is denoted nim. Please note that there is no

guarantee that the algorithm will find an immobilizing grasp for any given object.

3.3 Grasp Optimization in 2D

3.3.1 Theory

Many immobilizing grasps typically exist for a given object. It is desirable to be able to

automatically select a single optimal grasp from this set. We have developed two quality metrics

for this purpose: mean length gradient, and area of the grasp polygon.

The first quality metric is related to the lengths of the sides of the polygon joining the grasp

points. These lengths will change when each contact point is moved along its contacting edge.

The length gradient associated with this motion is determined by the local curvature of the

object. Although the fingers are locked in position at the grasp points, they have finite stiffness

14

and will deflect under load. With a larger length gradient the object will deflect less for a given

finger deflection. Therefore the length gradients are a measure of the stiffness of the grasped

object. The gradients may be approximated as follows:

{ }nfig

ggLgggL jijii ...,,2,1

],[],[∈∀

∆−∆+

=λ (18)

{ }nfig

ggLgggL jijinfi ...,,2,1

],[],[∈∀

∆−∆−

=+λ (19)

{ }nfig

ggLgggL kikinfi ...,,2,1

],[],[2 ∈∀

∆−∆+

=+λ (20)

{ }nfig

ggLgggL kikinfi ...,,2,1

],[],[3 ∈∀

∆−∆−

=+λ (21)

where j and k are defined as in (11) with a=i. Applying equations 18-21 will produce a total of

4nf gradients for each grasp. Some of these will be negative and some positive. For an inside-

out grasp, restraint of the deflected object can only result from the negative gradients, so the

positive gradients are not meaningful quality measures (and vice-versa for an outside-in grasp).

A negative length gradient is illustrated in Fig. 5. The mean absolute value of the remaining

gradients is then selected as the first quality metric. For inside-out grasps this is obtained using:

( )∑=

<−=nf

iiimean nf

4

1

041 λλλ (22)

For outside-in grasps this is obtained using:

( )∑=

>=nf

iiimean nf

4

1

041 λλλ (23)

The second quality metric is a measure of the ability of the grasped object to resist both in-plane

and out-of-plane applied moments. For an in-plane applied moment, the wider apart the grasp

points are located the smaller the resulting contact forces and finger deflections. Using this idea

15

a possible quality metric would be the length of the longest side of the grasp polygon. However

this could cause the grasp polygon to take on an elongated shape. In the extreme case, the grasp

points could become collinear and the grasp would be unable to resist an out-of-plane moment

applied about this line. If the points of the grasp polygon are widely spaced, and far from being

collinear, the area of the polygon will be large. The polygon area is therefore a suitable measure

of the grasp’s ability to resist applied moments. Applying the standard area formula derived

from Green’s theorem gives:

( )∑=

++ −=nf

iiiiigp yxyxA

1112

1 (24)

where gpA is the area of the grasp polygon,{ }nfggg ...,,, 21 are the grasp points in

counterclockwise order, ( )ii yx , are the coordinates of gi when nfi ≤ , and

( ) ( )1111 ,, yxyx nfnf =++ .

The remaining issue is how to use these quality metrics to optimize the grasp. First, both metrics

must be calculated for the set of nim immobilizing grasps. Next, the maximum value of each

metric is determining by exhaustive search. We have then chosen to normalize each metric and

combine them with a weighted sum:

( ) ( )gp

gp

mean

meang A

AwwQ

maxmax 21 +=λ

λ(25)

where w1 ≥ 0, w2 ≥ 0, w1 + w2 = 1, ( )⋅max is obtained by exhaustive search, and Qg is the overall

quality metric. Finally, the grasp with the maximum Qg is found by exhaustive search and is

selected as the optimal grasp.

3.3.2 Numerical Examples

In this section some numerical examples are presented illustrating the performance of the grasp

16

planner. The optimal grasps obtained using different combinations of quality metrics are shown

for four objects in Fig. 6. Neglecting the concavities, objects 1-3 are squares with a side length

of 20 mm. The concavities have a radius of 4.5 mm. Object 4 is a right triangle with side

lengths of 23.3 mm, 20 mm and 12 mm. An edge point spacing of 1 mm, a ∆g of 1 mm and a

finger radius of 1.5 mm were used with all objects. The grasp polygons for the grasps optimized

using meanλ alone are indicated with dashed lines, those for grasps optimized using gpA alone are

indicated with dash-dot lines, and those for grasps optimized using meanλ and gpA together are

indicated with dotted lines. The grasp optimization results are listed in Table 1.

The grasps optimized using meanλ alone (cases A, D, G and J) tended to position two of the

fingers close together. This improved the length gradients but also made the grasp points

somewhat collinear making the grasp more susceptible to applied moments. In contrast, the

grasps optimized using gpA alone (cases B, E, H and K) spread out the fingers in a fairly uniform

fashion, avoiding the weakness to applied moments. The grasps optimized using meanλ and gpA

together (cases C, F, I and L) naturally tried to balance out the two quality metrics. Comparing

the grasps for cases J, K and L provides a good example. The role of concavities may be

observed by comparing the optimal grasps for objects 1-3. Comparing cases B and E, fingers 1

and 2 are unchanged, but the edge at finger 3 becomes a concavity in case E. This is reflected by

a 15% increase in meanλ . Comparing cases F and I, locating fingers 1 and 2 at concavities in case

I caused meanλ to increase by 12%. The new finger positions also caused gpA to decrease by 8%

but this was not enough to offset the increase in meanλ .

An overall quality metric (Qg) of 1 indicates that the optimal grasp has maximized all metrics

17

included in Qg. This will always occur if w1 = 1 and w2 = 0 (or vice-versa) since an exhaustive

search is used to obtain Qg. If both w1 and w2 are not equal to zero then Qg may only equal one if

the optimal grasp maximizes both meanλ and gpA . The overall quality metrics (Qg) for cases C,

F, I and L are all approximately 0.9 demonstrating that the meanλ and gpA values for these grasps

are close to their maximum values from the set of immobilizing grasps. The numbers of

immobilizing grasps (nim) for the different objects are not particularly large and are a small

percentage of the numbers of potential grasps (npg). It is possible to increase nim at the cost of

increased computation time by decreasing the size of ∆g. However, in practice only one high

quality grasp is needed for a given object, so a large nim value is usually unnecessary.

Table 1. Grasp optimization results for the numerical examples.

Case Object npg nim w1 w2 meanλ Agp (mm2) Qg

A 1 18,522 28 1 0 0.0824 54.3 1

B 1 18,522 28 0 1 0.0507 139.8 1

C 1 18,522 28 0.5 0.5 0.0599 139.8 0.864

D 2 19,404 50 1 0 0.0824 54.4 1

E 2 19,404 50 0 1 0.0581 115.9 1

F 2 19,404 50 0.5 0.5 0.0720 109.3 0.870

G 3 20,328 50 1 0 0.0824 54.4 1

H 3 20,328 50 0 1 0.0581 115.9 1

I 3 20,328 50 0.5 0.5 0.0804 101.0 0.888

J 4 4,598 53 1 0 0.0945 42.5 1

K 4 4,598 53 0 1 0.0819 53.9 1

L 4 4,598 53 0.5 0.5 0.0882 51.0 0.940

4. Extension of Grasp to 2.5D using Tactile Probing

Up until this point, we have assumed that the object is 2D, or, in other words, the height of all the

18

points is uniform. The grasp will now be extended into the third dimension.

The height of the object near the grasp points is determined by simple probing. The fingers are

first moved over the object surface adjacent to the grasp points. With the fingers positioned in

these Probe positions (position 1 in Fig. 7a), a guarded move is performed in the Z-direction,

perpendicular to the image plane. The finger continues to move until it makes contact with the

object (position 2). Contact is sensed by reading the finger’s position sensor and estimating its

velocity. When the velocity is zero (at the moment of contact) the finger’s position is recorded in

the controller. The finger is then retracted to a safe height (position 3). The Z-coordinates of

position 2 for each finger completes the information required for determining the coordinates of

the remaining finger positions (4≡Drop, 5≡Approach, and 6≡Grasp in Fig. 7) needed to finish the

grasp plan.

The grasp planning algorithm does not specifically check for immobilization in 2.5D. It is

assumed that the fingers of the hand used to execute the grasp are designed to provide a sufficient

out-of-plane constraint (either through a kinematic constraint or through frictional contact) to

ensure 2.5D immobilization. The robotic hand used in our experiments features interchangeable

fingertips. Specially grooved fingertips are used with parts formed from sheet to provide an out-

of-plane kinematic constraint (as shown in Fig. 7c). Plain or knurled cylindrical fingertips are

used with cast or machined parts.

The use of this technique for extending a 2D grasp to 2.5D is not exclusive to grasps planned

using LMG. It is applicable to any 2D grasp planning method, subject to a few hardware

19

constraints. The robotic hand or gripper used must provide an out-of-plane constraint for the

object, and requires fingers that are independently actuated in the vertical direction if the object

height at the grasp points is not uniform (as is the case with the parts used in our experiments in

Section 5).

5. Experimental Verification

5.1 Procedure

The architecture of the hardware and software used for the experiments is shown in Fig. 8. The

robot is a Puma 762 manipulator. The dexterous hand is a custom made device [13]. The hand

has three pneumatically servoed fingers, each with three degrees-of-freedom (X, Y and Z), and a

CCD camera mounted in its palm. Several automated grasping tests were performed on a set of

three parts. Parts 1 and 2 are sheet metal automotive body components, while part 3 is a cast

aluminum water pump housing. The parts are roughly 280 x 180 x 40 mm in size. Images of the

parts obtained with the CCD camera are presented in Fig. 9. An edge point spacing of 0.8

mm, a ∆g of 1.6 mm were used. With parts 1 and 2, grooved fingertips with Rf=3.3 mm were

used, while plain cylindrical fingertips with Rf=6.4 mm were used with part 3. As suggested in

section 3.1, since all of the parts contain more concave edge segments than the number of

fingers, for planning efficiency only concave segments were used in the grasp planning.

Initially, the camera was situated approximately 1.3 m above a black surfaced table, with the

image plane parallel to the plane of the table. The camera was situated at this height so that the

field of view was the same size as the workspace of the hand. An unknown object was randomly

placed on the table at the start of each test within the camera’s field of view. After the camera

20

obtains an image of the part, a grasp was automatically planned and executed online. A manual

inspection of the grasp was then performed, to check it for immobilization.

5.2 Experimental Results and Discussion

The concave edge segments and optimal 2D grasps determined online for the parts are shown in

Fig. 10. The corresponding optimization results are listed in Table 2. Cases A, B and C are

inside-out grasps while the others are outside-in grasps. All of the optimal grasps for parts 1 and

2 located one or more fingers on the edges of holes. This is logical since the edges of the holes

are more highly curved than the concave segments on the outside edge, leading to grasps with

higher stiffness and larger meanλ values. With part 3, the hole diameters were found to be less

than the finger diameter and these concave edge segments were automatically rejected by the

grasp planner. The planned grasps followed a similar pattern to the numerical examples. Those

optimized using meanλ alone (cases A, D and G) tended to place the fingers at strong concavities

regardless of the size of the resulting grasp polygon. Those optimized using gpA alone (cases B,

E and H) spread out the fingers without regards to edge curvature. Cases D and E are good

examples of the different types of grasps that these two strategies tend to produce. These

conclusions are also supported by the meanλ and gpA values listed in Table 2. The optimizations

using meanλ and gpA together (cases C, I and L) produced grasps with both metrics close to their

maximum values as evidenced by their Qg values being close to 0.9. Case F is good example of

the benefits of combining the two metrics when optimizing. It avoided the sensitivity to applied

moments that would have been caused by case D’s relatively close finger spacing, while at the

same time it placed fingers 2 and 3 on strongly concave regions. In Fig. 11 Qg is plotted versus

grasp number for Part 2 and w1=w2=0.5. The point corresponding to case F has been indicated in

21

the figure. The plot contains several local maxima suggesting that the use of exhaustive search

for the optimization was appropriate.

As previously observed with the numerical examples, the nim values are a small percentage of the

npg values. Due to the efficiency of our test for 2D immobilization the computation time required

to evaluate the large number of potential grasps for each part was quite small. On a 1.5 GHz

Pentium 4 PC, the total time to extract the concave segments, perform the LMG grasp planning

and compute the finger positions was less than 0.3 seconds for each part. The task of computing

the finger positions includes determining the positions (Probe, Drop, etc. as shown in Fig. 7),

checking for and avoiding finger-part collisions and fitting the positions to the hand’s workspace.

Table 2. Grasp optimization results for the grasping experiments.

Case Part npg nim w1 w2 meanλ Agp (mm2) Qg

A 1 208,000 44 1 0 0.662 4,810 1

B 1 208,000 44 0 1 0.519 5,680 1

C 1 208,000 44 0.5 0.5 0.622 5,530 0.956

D 2 1,380,000 58 1 0 0.617 392 1

E 2 1,380,000 58 0 1 0.469 4,410 1

F 2 1,380,000 58 0.5 0.5 0.549 4,230 0.924

G 3 390,000 208 1 0 0.641 5,000 1

H 3 390,000 208 0 1 0.582 9,910 1

I 3 390,000 208 0.5 0.5 0.582 9,910 0.953

The execution of a grasping sequence for part 1 is shown as a series of photographs in Fig. 12.

Once the grasp has been planned, the hand is moved to a new position closer to the object as

22

shown in Fig. 12a. At the same time, the fingers are moved to their Probe positions, in

preparation for the tactile probing routine. The part is then probed (Fig. 12b). After grasping,

the hand is moved to a new position above the table for manual inspection (Fig. 12c). Similarly,

parts 2 and 3 are shown immobilized above the table in Figs. 13 and 14, respectively.

Several grasping experiments were performed on each of the parts. A test was deemed successful

if the object was properly grasped and immobilized. All tests were successful. During the

manual inspection, the grasp was also checked to see how the finger contact positions matched

those determined by the grasping algorithm. Although no accurate measurements were taken, the

positions appeared to be closely matched. This was due in part to the ability of the extended

LMG strategy to correct for both positioning (hand and robot) and camera related errors as

mentioned in section 3.2.

6. Conclusions

An automated grasping system for complex 2.5D real-world objects was presented in this paper.

The system hardware consists of a robotic manipulator, a three-fingered dexterous hand with a

palm mounted CCD camera and a PC. The 2D shape, position and orientation of the objects

were obtained online using computer vision. Tactile probing with the dexterous hand was used

to complete the 2.5D model. The grasp planner used this model and an extended version of

limited mobility grasping theory to compute optimal immobilizing grasps. Our grasping theory

is applicable to dexterous hands with three or more fingers, and most 2D objects. It is a second

order theory that recognizes the benefits of object concavities. Two quality metrics were

introduced for use in the grasp optimization. The first is a measure of the stiffness of the grasp.

The second is a measure of the ability of the grasped object to resist in-plane and out-of-plane

23

applied moments. An overall quality metric combining these two metrics was then introduced to

produce grasps with both stiffness and moment resistance. The optimal 2D grasp is the

immobilizing grasp with the maximum overall quality as found by exhaustive search. Several

numerical examples were presented to demonstrate the grasp planning theory. Next, experiments

were performed on three complex shaped automotive parts. The parts were successfully grasped

and immobilized in all of the tests performed. The total computation time (for object modeling

and grasp planning) was under 0.3 s for each part.

There are some important limitations to our approach. First, it is limited to 2.5D objects as has

already been stated. Second, the colour of the object must be light enough that thresholding may

be used to distinguish it from the background. Third, the grasp planning is based on a discretized

model of the object edges. If this discretization is too coarse then the planned grasp may not be

effective. We have not observed this in any of our experiments, but it is still a limitation of our

approach. Decreasing the size of ∆g will mitigate this problem at the cost of increased

computation time. Modeling the edges with continuous functions would be an improvement.

Fourth, only one finger may be placed on each edge segment with our approach. This has the

disadvantage that some immobilizing grasps will be not be found by the grasp planner, and the

advantage that finger-to-finger collisions are avoided. If too few immobilizing grasps are found,

the edge segments that are sufficiently long could be broken into smaller sub-segments to

circumvent this limitation. Fifth, our approach requires that the fingers of the robotic hand

provide an out-of-plane constraint for the object. This will not always be possible, but can often

be achieved by suitably designing the shape and material of the fingers. Sixth, there is no

guarantee that the planner will find an immobilizing grasp for any given object. We have not

observed this in our work, other than the exceptional cases already described, but it is possible

24

that nim will equal zero for certain objects.

References

[1] A. Schrott, Feature-Based Camera-Guided Grasping by an Eye-in Hand Robot, IEEE. Int.

Conf. on Rob. and Aut., 1990, 1832-1837.

[2] C. Laugier, A. Ijel and J. Troccaz, Combining vision based information and partial

geometric models in automatic grasping, IEEE. Int. Conf. on Rob. and Aut., 1990, 676-682.

[3] D. Kragic, A.T. Miller and P.K. Allen, RealTime Tracking Meets Online Grasp Planning,

IEEE Int. Conf. on Rob. and Aut., 2001, 2460-2465.

[4] A. Bendiksen and G. Hager, A Vision-Based Grasping System for Unfamiliar Planar

Objects, IEEE. Int. Conf. on Rob. and Aut., 1994, 2844-2849.

[5] P.J. Sanz, A.P. del Pobil, J.M. Iñesta and G. Recatalá, Vision-Guided Grasping of

Unknown Objects for Service Robots, IEEE. Int. Conf. on Rob. and Aut., 1998, 3018-3025.

[6] Y.F. Li, M.H. Lee, M.A. Rodrigues and J.J. Rowland, A Visually Guided Robot System

For Food Handling Applications, IEEE. Int. Conf. on Rob. and Aut., 1994, 2591-2596.

[7] M.A. Rodrigues, Y.F. Li, M.H. Lee and J.J. Rowland, Robotic grasping of complex shapes:

is full geometrical knowledge of the shape really necessary?, Robotica 13, 1995, 499-506.

[8] M. Taylor, A. Blake and A. Cox, Visually guided grasping in 3-D, IEEE. Int. Conf. on Rob.

and Aut., 1994, 761-766.

[9] M. Trobina and A. Leonardis, Grasping Arbitrarily Shaped 3-D Objects from a Pile, IEEE.

Int. Conf. on Rob. and Aut., 1995, 241-246.

[10] C. Davidson and A. Blake, Error-Tolerant Visual Planning of Planar Grasp, IEEE Int. Conf.

on Comp. Vision., 1998, 911-916.

25

[11] C. Bard, C. Laugier, C.Milési-Bellier, J. Troccaz, B Triggs, G. Vercelli, Achieving

Dextrous Grasping by Integrating Planning and Vision-Based Sensing, Int. Journ. of Rob.

Res., 14(5), 1995, 445-464.

[12] S.A. Stansfield. Robotic Grasping of Unknown Objects: A Knowledge-Based Approach.

Int. Journ. of Rob. Res., 10(4), 1991, 314-326.

[13] R.B. van Varseveld and G.M. Bone, Design and implementation of a lightweight, large

workspace, non-anthropomorphic, dexterous hand, ASME Journ. of Mechanical Design,

121, 1999, 480-484.

[14] W. Plut and G.M. Bone, Limited Mobility Grasps for Fixtureless Assembly, IEEE. Int.

Conf. on Rob. and Aut., 1996, 1465-1470.

[15] W. Plut and G.M. Bone, 3-D Flexible Fixturing Using a Multi-Degree of Freedom Gripper

for Robotic Fixtureless Assembly, IEEE. Int. Conf. on Rob. and Aut., 1997, 379-384.

[16] E. Rimon and J. Burdick, Mobility of Bodies in Contact – I: A New 2nd Order Mobility

Index for Multiple-Finger Grasps, IEEE Int. Conf. on Rob. and Aut., 1994, 2329-2335.

[17] P.A. Bender, Automated grasping: planning, analysis and execution, M. Eng. Thesis,

McMaster University, Hamilton, ON, Canada, 2001.

Authors' Biographies

Peter Bender received the Bachelor of Engineering and Master of Engineering degrees from the

Department of Mechanical Engineering at McMaster University, Hamilton, ON, Canada, in 1997

and 2001, respectively. In 1997 he became a member of Dr. Bone's robotics research group at

McMaster, where his research work was focussed in two main topics: robotic grasping of

unknown objects and automated part inspection. His research interests include grasp planning

and execution, automated assembly, machine vision systems, advanced control systems and

26

automated inspection. Mr. Bender currently holds a position as a Research Engineer with the

McMaster Manufacturing Research Institute.

Gary M. Bone received the B.Sc.(Ap.Sc.) degree from the Department of Mechanical

Engineering, Queen’s University, Kingston, ON., Canada, and the M.Eng. and Ph.D. degrees

from McMaster University, Hamilton, ON, Canada in 1986, 1988, and 1993, respectively. In

1994 he joined the Faculty of Engineering, McMaster University, where he is currently an

Associate Professor in the Department of Mechanical Engineering. His research interests

include: flexible assembly, robotic hands, grasp planning, multisensor fusion, pneumatic servos

and advanced control systems.

27

Figure 1. Determination of edge concavity.

Figure 2. Second order immobilization examples.

P5

P7

P6

P8 Object

θ

P5

P7

P6

P8

Object

θ

(a) (b)

A

Freerotation

Free translation

(d)(a)

(b)

(c)

(e)

(f)

Freetranslation

28

Figure 3. Example of contact points and grasp points for a three finger outside-in grasp.

c2

g2

g1

c3

c1 g3

Object

Object expanded byfinger radius (Rf)

29

Figure 4. Examples of tests for false local minima. (a) Valid inside-out grasp. (b) Invalid inside-out grasp.

Grasp polygon

(a)

Objectexpanded

by Rf

Grasp PointsTest Points

Grasp polygon

(b)

Objectexpanded

by Rf

30

∆g

g3 g2L

L'

Figure 5. Length gradient of g

LL∆−′ )( for an inside-out grasp.

Figure 6. LMG grasp planning examples. The superscripts refer to the cases listed in Table 1.

Object 1

Ag1

Bg2

Ag3

CB gg 11 & CB gg 33 &

Ag2Cg2

Object 3

Gg3

Gg2Hg2

Hg3

Hg1

IG gg 11 &

Ig2

Ig3

Kg2

Jg1Jg3

Kg1

LJ gg 22 &

Kg3

Lg1Lg3

Object 4

Object 2

Eg2Dg2

Fg2

Dg3

Fg1

Eg3 Eg1

Fg3Dg1

31

Figure 7. Definition of finger positions. 1≡Probe, 2≡Contact Made, 3≡Retract, 4≡Drop, 5≡Approach and 6≡Grasp.

32

Figure 8. System architecture.

2.5DObject

Image Capture ImageProcessing

GraspPlanner

RobotController

HandController

GraspExecution

Puma 762 RoboticManipulator

3 Finger, 9 DOF,Dexterous Hand

CCD Camera(Black and White) 2D Object

Model

SystemControllers

SystemWorkspace PC

ControlCommands

33

(a) Part 1

(b) Part 2

(c) Part 3

Figure 9. Parts used in the experiments.

34

Figure 10. Concave edge segments and optimal grasps planned online for the three test parts.The superscripts refer to the cases listed in Table 2.

Ac2

Bc2

Ac1

CB cc 11 &

Cc2

Bc3Cc3Ac3

Part 1

Ec1

Ec2

Fc2

Dc2

Dc1

Fc1

Dc3

Ec3

Fc3

Gc1

Gc2

Gc3

IH cc 11 &

IH cc 22 &

IH cc 33 &

Part 2

Part 3

35

Figure 11. Qg versus grasp number for Part 2 and w1=w2=0.5.

0 10 20 30 40 50 600.3

0.4

0.5

0 60.6

0.7

0.8

0.9

1

Grasp Number

Qg

Case F

36

(a)

(b)

(c)

Figure 12. Execution of a grasping sequence for part 1: (a) fingers in their Probe positions, (b)tactile probing - fingers in contact with part, (c) part has been grasped and lifted above table formanual inspection.

37

Figure 13. Part 2 successfully grasped.

Figure 14. Part 3 successfully grasped.