Camera Basics

Image formation & CameraBasics

Adopted from http://www.cs.wustl.edu/~pless/

Image Formation

• Vision infers world properties from images.• How do images depend on these properties?• Two key elements

– Geometry (shape, size, relative position of figures, and the properties of space)

– Radiometry (techniques for measuring electromagnetic radiation)

– We consider very simple models of these.• If you want to consider really complicated models, try a

graphics class. In vision, the real world does these computations for us.

2

Camera Obscura

"When images of illuminated objects ... penetrate through a small hole into a very dark room ... you will see [on the opposite wall] these objects in their proper form and color, reduced in size ... in a reversed position, owing to the intersection of the rays".

Da Vinci

Camera = Latin for “room”Obscura = Latin for “dark”

3

Pinhole cameras

• Abstract camera model - box with a small hole in it

• Pinhole cameras work in practice

(Forsyth & Ponce) 4

Pinhole cameras

[X,Y,Z]

f 5

“Normalized Camera”

• Pinhole at (0,0,0), looking at points (X,Y,Z)• Virtual film at z = 1 plane.• Point X,Y,Z is imaged at intersection of:

– Line from (0,0,0) to (X,Y,Z), and – the Z = 1 plane

• So intersection point (x,y) has coordinates (X/Z, Y/Z, 1)

• A camera is normalized if the units are chosen so that the focal length is 1.

• Next, a math trick, then some algebra.

Z

Y

X

P

Camera Center of Projection

y

xp

6

Homogeneous Coordinates• Trick. Represent 2d point using 3 numbers,

instead of the usual 2 numbers (objective to represent points at infinity using finite coordinates)

• Mapping 2d 2d homogeneous (simply move all points to a plane in z direction, for example in z=1):

– (x,y) (x,y,1)– (x,y) (sx, sy, s)

• Mapping 2d homogeneous to 2d (simply project all points to one plane in z=1 and remove the third coordinate):

– (x,y,w) (x/w, y/w)

• The coordinates of a 3D point (X,Y,Z) *already is* the 2D (but homogeneous) coordinates of its projection.

• Will make some things simpler and linear-er. 7

… but we’re stuck being computational, so lets measure everything in “pixel widths”.

The units of measuring X,Y,Z don’t matter (because we are going to divide them anyway), so lets make the unit be the size of the pixel on the ccd (the photon detector) chip (5 mm /1000 pixels ) = 5 mm / pixel = 1 “unit”Express the focal length in these “units”, so:x = fX/ZY = fY/Z

8

Intrinsic Camera Parameters

CCD sensor array

The pixels may be rectangular:x = aX/ZY = bY/Z

9


The chip sensor may not be centered, or the origin of the chip may not be in the middle of the chip.x = aX/Z + x0Y = bY/Z + y0

10


x = aX/Z – A cot(q) Y/Z + x0Y = b/sin(q) Y/Z + y0

cheapglue

cheap CMOS chipcheap lens image

cheap camera

11


In homogeneous coordinates

These five parameters are known as intrinsic parameters

Z

Y

X

y

x

y

x

100)sin(

0

)cot(

10

0

Z

Y

X

yf

xsf

y

x

y

x

100

0

10

0

12

Z

Y

X

yf

xsf

y

x

y

x

100

0

10

0

Given enough examples of 3D points and their 2D projections, we can solve for the 5 intrinsic parameters…

Example calibration object…

13

Practical Linear Algebra 1:

• Given many examples of the equation at the right, solve for the matrix of values fx, fy, s, x0, y0

• How can we solve for this?

Z

Y

X

yf

xsf

y

x

y

x

100

0

10

0

14

Z

Y

X

yf

xsf

y

x

y

x

100

0

10

0

Given enough examples of 3D points and their 2D projections, we can solve for the 5 intrinsic parameters…

… and all this assumes that we know X,Y,Z

(0,0,0)

Exactly Known!

15

Z

Y

X

yf

xsf

y

x

y

x

100

0

10

0

More commonly, we can put a collection of points (usually in a grid) in the world with known relative position, but with unknown position relative to the camera.

Then we need to solve for the intrinsic calibration parameters, and the extrinsic parameters (the unknown relative position of the grid).

How do we represent the unknown relative position of the grid?

16

The grid defines its own coordinate system

X

Y

Z

That coordinate system has a Euclidean transformation that relates its coordinates to the camera coordinates… that is, there is some R, T such that:

1...

..

...

grid

grid

grid

z

y

x

grid

grid

grid

Z

Y

X

T

T

T

R

Z

Y

X

T

Z

Y

X

R

Z

Y

X

With enough example points, could we solve for R,T?

17

An aside…Representation of Rotation with a Rotation Matrix

Rotation matrix R has two properties:

Property 1:R is orthonormal, i.e. IRRT In other words, 1 RRT

Property 2: Determinant of R is 1.

Therefore, although the following matrix is orthonormal,it is not a rotation matrix because its determinant is -1.

100

010

001this is a reflection matrix

18

Representation of Rotation -- Euler Angles

• Euler angles • pitch: rotation about x axis : • yaw: rotation about y axis: • roll: rotation about z axis:

X

Y

Z

W

W

W

C

C

C

Z

Y

X

Z

Y

X

cossin0

sincos0

001

cos0sin

01

sin0cos

100

0cossin

0sincos

~

~

~

19

1...

..

...

grid

grid

grid

z

y

x

Z

Y

X

T

T

T

R

Z

Y

X

Z

Y

X

yf

xsf

y

x

y

x

100

0

10

0

+

1...

..

...

100

0

10

0

grid

grid

grid

z

y

x

y

x

Z

Y

X

T

T

T

Ryf

xsf

y

x

P is a 4X3 matrix, that defines projection of points (used in graphics). P has 11 degrees of freedom, (the scale of the matrix doesn’t matter). 5 intrinsic parameters, 3 rotation, 3 translation

11 grid

grid

grid

Z

Y

X

Py

x

Putting it all together…

20

• Given many examples of:– World points (X,Y,Z), and – Their image points (x,y)

• Solve for P.

• Then

11 world

world

world

Z

Y

X

Py

x

z

y

x

T

T

T

KRP

...

..

...

translationRotation + intrinsic, all mixed up.

21

z

y

x

T

T

T

KRP

...

..

...

Then take the QR decomposition of this part of the matrix to get the rotation and the intrinsic parameters. (from mathworld).

22

Special Case: Suppose the world is a plane• Projection from the world to the image:

• Given many examples of:– World points (X,Y), and – Their image points (x,y)

• Solve for P.• Take the QR decomposition of this part of the matrix to

get the rotation and the intrinsic parameters.

11

grid

xgrid

y

gridz

Xx T

Yy KR T

ZT

0

Irrelevant

23

1 1

gridx

y grid

z

xx T

y KR T y

T

P

1 1 111

1 1 112

1 113

221

222

23

331

332

33

1 0 0 0 0 0 0

0 0 0 1 0 0 0

0 0 0 0 0 0 1 1

1

1

grid grid

grid grid

grid grid

ax y x

ax y y

ax y

a x

a y

a

a x

a y

a

Ax=bMatlab: x = A\bThen construct matrix P

24

Given three examples11 12 13

21 22 23

31 32 331 1

grid

grid

xx a a a

y a a a y

a a a

Example 2

Example 3

25

Matlab Code

load my_imageimshow(data)title('click on 3 corner points');p = ginput(3); % Image Pointshold onplot(p(1,1),p(1,2),'marker','o')plot(p(2,1),p(2,2),'marker','o')plot(p(3,1),p(3,2),'marker','o')title('');

pg=[p(1,1) p(1,2);p(1,1)+12 p(1,2);p(1,1)+12 p(1,2)+9.6]; % Word Points

A=[...pg(1,1) pg(1,2) 1 0 0 0 0 0 0;0 0 0 pg(1,1)

pg(1,2) 1 0 0 0;0 0 0 0 0 0 pg(1,1) pg(1,2) 1;...pg(2,1) pg(2,2) 1 0 0 0 0 0 0;0 0 0 pg(2,1)

pg(2,2) 1 0 0 0;0 0 0 0 0 0 pg(2,1) pg(2,2) 1;...pg(3,1) pg(3,2) 1 0 0 0 0 0 0;0 0 0 pg(3,1)

pg(3,2) 1 0 0 0;0 0 0 0 0 0 pg(3,1) pg(3,2) 1];

b=[...p(1,1);p(1,2);1;...p(2,1);p(2,2);1;...p(3,1);p(3,2);1];

x=A\b;

% Parameter MatrixP=[x(1) x(2) x(3);x(4) x(5) x(6);x(7) x(8) x(9)];

% without translationPr=P(1:3,1:2);

% QR decomposition% q = rotation matrix, r intrinsic matrix[q,r]=qr(Pr);

% intrinsic matrix without x0 and y0r=r(1:2,1:2);

% check the resulttitle('click on any point');pt = ginput(1);plot(pt(1,1),pt(1,2),'marker','+')pc=r\[pt(1,1);pt(1,2)];text(pt(1,1),pt(1,2),['(' num2str(pc(1,1)) ','num2str(pc(2,1)) ,')'])title('');

Other common trick to find P, use many planes, then you have to solve for multiple different R,T, one for each plane…

Counting:

For each grid, we have 3 unknowns (translation), 3 unknowns (rotation).

+ 5 extra unknowns defining the intrinsic calibration.

Method. For a collection of planes, calculate the homography (same shapes) between the image and the real world plane.

27

• Let’s consider a special case. Suppose we take two pictures of the world, from a camera at the same location. But the camera has rotated between the two pictures.

29

What happens if the camera moves?

• First image, points in 3D, measured in camera coordinate system

10

0

0

...

..

...

100

0

1

'

'

0

0

grid

grid

grid

y

x

Z

Y

X

Ryf

xsf

y

x

10

0

0

...

..

...

100

0

10

0

grid

grid

grid

y

x

Z

Y

X

Iyf

xsf

y

x

30

• Second image, camera has rotated

– On image 1: p = KPand on image 2: p’ = KRP

p = K((KR)-1)p’= KR-1K-1p’

– Given a few examples of corresponding points p,p’, we can solve for a mapping KR-1K-1 of all points

31

Homography: (x’,y’,1) ~ H (x,y,1)

– Tells you exactly where the point goes.– Point (x,y) in one frame corresponds to point (x’,y’) in the

other frame. If we need to think about multiple points, we may put subscripts on them.

– Being careful about the homogenous coordinate, we write:

1

''

y

x

Hw

wywx

Homography is a “simple” example of a 3D to 2D transformation

It is also the “most complicated” linear 2D to 2D transformation.

What other 2D 2D transformations are there?

33

Homography is most general, encompasses other transformations

1002221

1211

y

x

taa

taa

1002221

1211

y

x

tsrsr

tsrsr

333231

232221

131211

hhh

hhh

hhh

1002221

1211

y

x

trr

trr

Projective8 dof

Affine6 dof

Similarity4 dof

Euclidean3 dof

Views of a plane from different viewpoints, any view of a scene from the same viewpoint.

Images of a “far away” object under any rotation

Camera looking at an assembly line w/ zoom.

Camera looking at an assembly line.

34

Invariants…

1002221

1211

y

x

taa

taa

1002221

1211

y

x

tsrsr

tsrsr

333231

232221

131211

hhh

hhh

hhh

1002221

1211

y

x

trr

trr

Projective8 dof

Affine6 dof

Similarity4 dof

Euclidean3 dof

Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio

Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints).

Ratios of any edge lengths, angles.

Absolute lengths, areas.

35

Image registration

Determining the 2d transformation that brings one image into alignment (registers it) with another.

36

Image Warping

• What kind of warps are these? rotation, translation, zooming, shrinking, …

• How to solve for these mappings?

10010.00011.0

7034.266000.01680.0

2411.341918.06101.0

37

Unwrapping Matrix

w

wywx

y

x

H ''

1

Write out the lines of this matrix equation.

And remember which variables are unknown.

'wxcbyax 'wyfeydx

whygx 1

w

wywx

y

x

hg

fed

cba

''

11

38

000000 ' wxhgfedcbyax

000000 ' wyhgfeydxcba

1000000 whygxfedcba

Looks like another matrix equation:

1

0

0

1

0

0

1

0

0

1

0

0

4

3

2

1000000000

000001000

000000001

0100000000

000001000

000000001

0010000000

000001000

000000001

0001000000

000001000

000000001

'

'

'

'

'

'

'

'

w

w

w

w

h

g

f

e

d

c

b

a

yx

yyx

xyx

yx

yyx

xyx

yx

yyx

xyx

yx

yyx

xyx

i

i

i

i

i

i

i

i

Data from different points

w

wywx

y

x

hg

fed

cba

''

11

39

'

'''1000

''0001

y

x

h

g

f

e

d

c

b

a

yyxyyx

yxxxyx

'''

'''

000

000

iii

iii

yhyygxyfeydxcba

xhyxgxxfedcbyax

And just add two more rows for each corresponding point40

'

'''1000

''0001

y

x

h

g

f

e

d

c

b

a

yyxyyx

yxxxyx

Ax=bMatlab: x = A\bThen make your homography matrix by rearranging x into a 3 x 3 matrix

Size of A? b? 41

42

Matlab Codel% read in file 1[fname pathname] = uigetfile('*.*','C:\');fn1 = [pathname fname];im1 = imread(fn1);% read in file 2[fname pathname] = uigetfile('*.*','C:\');fn2 = [pathname fname];im2 = imread(fn2);

subplot(121); % set up to draw on left of two plotsimshow(im1); % show the imagetitle('click on 4 points'); % tell the user toclick on 4 pointsp1 = ginput(4); % wait while they click 4 pointshold onfor i=1:4, plot(p1(i,1),p1(i,2),'marker','o'); endtitle('');

subplot(122); % set up to draw on right of two plotsimshow(im2);title('click on the same 4 points');p2 = ginput(4);hold onfor i=1:4, plot(p2(i,1),p2(i,2),'marker','o'); endtitle('');

%% now we have corresponding points in p1, p2A = []; % make an empty array

for ix = 1:4r1 = [-p1(ix,1) -p1(ix,2) -1 0 0 0 p1(ix,1)*p2(ix,1) p1(ix,2)*p2(ix,1)];

r2 = [0 0 0 -p1(ix,1) -p1(ix,2) -1 p1(ix,1)*p2(ix,2) p1(ix,2)*p2(ix,2)];

A = [A;r1;r2]; End

% make the Right Hand Side of the Ax = b equationtmp = p2'; b = -tmp(:);

% now solve for the vector of unknowns:x = A\b;

%reconstitute the homography matrix:H = [x(1) x(2) x(3); x(4) x(5) x(6); x(7) x(8) 1];

43

% warp points to see if it works..i=10;while (i>0)

subplot(121);title('click on any point');p1 = ginput(1);plot(p1(1,1),p1(1,2),'marker','+','color',[1 0 0])title('');p2 = H*[p1(1,1) p1(1,2) 1]';p2 = p2./p2(3);subplot(122);plot(p2(1,1),p2(2,1),'marker','+','color',[1 0 0])i=i-1;

end

• Let’s consider a special case. Suppose we take two pictures of the world, from a camera at different locations looking at same point, assuming the world is fixed, (mono vision)

• Or two cameras looking at same point at the same time (stereo vision)

44

What happens if the camera moves?

Epipolar LinesEach point in one image corresponds to a line of possibilities in the other image “Epipolar Line”

Potential 3d points

Epipolar Line

Image of all these points is one point

Image of Potential 3d points in image 2

Camera 1

Camera 2

Image1

Image2

45

Epipolar Geometry• An epipole is the image point of the other camera’s center

• Epipoles lie on the cameras’ baseline.

• All epipolar lines meet at the epipoles

baseline

46

Epipolar Equation

a translation vector defining where one camera is relative to the other

Point in space

Camera 1Camera 2

Direction of the point from Camera 1Direction of the point from

Camera 2

or

47

• All lines in the plane are perpendicular to the normal to the plane

• normal of the plane is perpendicular to both p and t

• As all three vectors in the same plane:

y

x

y

Epipolar Equation

a translation vector defining where one camera is relative to the other

Point in space

Camera 1Camera 2

Direction of the point from Camera 1Direction of the point from

Camera 2 with possible rotation R

Epipolar Equation

Essential Matrix48

Pixel Coordinate & Epipolar Equation

F is 3x3 “fundamental matrix”

K maps normalized coordinatesonto pixel coordinates

Fundamental Equation

49

Finding Fundamental Matrix

• Click on “the same world point” in the left and right image, to get a set of point correspondences: (x,y) that correspond to (x’,y’)

• Use the fundamental equation to find F

• Need at least 8 points (each point gives one constraint, F is 3x3, but scale invariant, so there are 8 degrees of freedom in F)

50

Using Fundamental Matrix

Given a point (x,y) on the left image, F defines the “Epipolar Line” and tells where the corresponding points must lie

51

• Least squares solution using SVD on equations from 8 pairs of correspondences

• Enforce det(F)=0 constraint using SVD on F

53

Matlab Code

% read in file 1[fname pathname] = uigetfile('*.*','C:\');fn1 = [pathname fname];im1 = imread(fn1);% read in file 2[fname pathname] = uigetfile('*.*','C:\');fn2 = [pathname fname];im2 = imread(fn2);

subplot(121); % set up to draw on left of two plotsimshow(im1); % show the imagetitle('click on 8 points'); % tell the user to click on 4 pointsp1 = ginput(8); % wait while they click 4 pointshold onfor i=1:8, plot(p1(i,1),p1(i,2),'marker','o'); endtitle('');

subplot(122); % set up to draw on right of two plotsimshow(im2);title('click on the same 8 points');p2 = ginput(8);hold onfor i=1:4, plot(p2(i,1),p2(i,2),'marker','o'); endtitle('');

% finding A matrix:A=[];for i=1:8

x1 = p1(i,1);y1 = p1(i,2);x2 = p2(i,1);y2 = p2(i,2);A(i,:) = [x1*x2 y1*x2 x2 x1*y2 y1*y2 y2 x1 y1 1];

end

% SVD of A:[U D V] = svd(A);

% Finding Fundamental Matrix F:f = V(:,9);F = [f(1) f(2) f(3); f(4) f(5) f(6); f(7) f(8) f(9)];

% Modify F:[FU FD FV]= svd (F);FDnew = FD;FDnew(3,3) = 0;

FM = FU*FDnew*FV';plot_epipolar;

54

% script plot_epipolar;

% Plotting epipolar line:list =['r' 'b' 'g' 'y' 'm' 'k' 'w' 'c'];

% Clicking a point on the left image:subplot(121);title('click on a point');[left_x left_y] = ginput(1);hold on;plot(left_x,left_y,'r*');title('');

% Finding the epipolar line on the right image:left_P = [left_x; left_y; 1];right_P = FM*left_P;right_epipolar_x=1:length(im2);

% Using the eqn of line: ax+by+c=0; y = (-c-ax)/bright_epipolar_y=(-right_P(3)-

right_P(1)*right_epipolar_x)/right_P(2);subplot(122);hold on;plot(right_epipolar_x,right_epipolar_y,list(mod(i,8)+1));

Recovering the Depth

T

f O1 O2

P

• Assumes (two) cameras

• Known positions and focal lengths

• Recover depth

55

Depth can be recovered by triangulation Prerequisite: Finding Correspondences

• Given an image point in left image, the (corresponding) point in the right image, is the point which is the projection of the same 3-D point

• How can find this correspondence automatically?→ Stereo Matching

56

Stereo Matching

For each scanline , for each pixel in the left image• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost• This will never work, so: improvement match windows

57

=?

f g

Mostpopular

• For each pixel in the first image Find corresponding epipolar line in the right image Examine all pixels on the epipolar line and pick the best match Triangulate the matches to get depth information

Comparing Windows:

58

Special Case: Parallel images

59

• For each pixel x in the first image Find corresponding epipolar line in the right image Examine all pixels on the epipolar line and pick the best match x’ Triangulate the matches to get depth information depth(x) = fB/(x-x’)

(B = Base line, f = focal length), x-x’ is called disparity

60

Matlab Code

my_lec3;plot_epipolar;ssd=[];w=20; % window sizesubplot(121)[winxy1,win1]=find_win([left_x left_y],im1,w);plot([winxy1(:,1);winxy1(1,1)],[winxy1(:,2);winxy1(1,2)],'r');subplot(122)for i=1:s(2) % s(2) number of points of epipolar

[winxy2,win2]=find_win([epipolar_x(i) epipolar_y(i)],im2,w);plot([winxy2(:,1);winxy2(1,1)],[winxy2(:,2);winxy2(1,2)],'b');ssd(i)=sum(sum((win1-win2).^2));

endi=find(ssd==min(ssd));[winxy2,win2]=find_win([epipolar_x(i) epipolar_y(i)],im2,w);plot([winxy2(:,1);winxy2(1,1)],[winxy2(:,2);winxy2(1,2)],'r');

disparity=left_x-epipolar_x(i)

disparity =

-1.5909

• If B and f are known we can calculate the distance

• Y is not included in our calculation because we supposed that images are parallel

• If images are not parallel, a rectificationmethod must be applied to align two images

Camera Basics

Documents

Transcript of Camera Basics