| Operators |
camera_calibration — Determine all camera parameters by a simultaneous minimization process.
camera_calibration( : : NX, NY, NZ, NRow, NCol, StartCamParam, NStartPose, EstimateParams : CameraParam, NFinalPose, Errors)
camera_calibration performs the calibration of a camera. For this, known 3D model points (with coordinates NX, NY, NZ) are projected in the image and the sum of the squared distance between these projections and the corresponding image points (with coordinates NRow, NCol) is minimized.
If the minimization converges, the exact internal (CameraParam) and external (NFinalPose) camera parameters are determined by this minimization algorithm. The parameters StartCamParam and NStartPose are used as initial values for the minimization process. Since this algorithm simultaneously handles correspondences between image and model points from different images, it is also called multi-image calibration.
In general, camera calibration means the exact determination of the parameters that model the (optical) projection of any 3D world point P(w) into a (sub-)pixel [r,c] in the image. This is important, if the original 3D pose of an object has to be computed using an image (e.g., measuring of industrial parts).
The projection consists of multiple steps: First, the point p(w) is transformed from world into camera coordinates (points as homogeneous vectors, compare affine_trans_point_3d):
/ \ / x \ / \ / \ | p(c) | = | y | = | R t | * | p(w) | | | | z | | | | | \ 1 / \ 1 / \ 0 0 1 / \ 1 /
Then, the point is projected into the image plane, i.e., onto the sensor chip.
For the modeling of this projection process that is determined by the used combination of camera, lens, and frame grabber, HALCON provides the following three 3D camera models:
Area scan pinhole camera: The combination of an area scan camera with a lens that effects a perspective projection and that may show radial and decentering distortions.
Area scan telecentric camera: The combination of an area scan camera with a telecentric lens that effects a parallel projection and that may show radial and decentering distortions.
Line scan pinhole camera: The combination of a line scan camera with a lens that effects a perspective projection and that may show radial distortions.
For area scan cameras, the projection of the point p(c) that is given in camera coordinates into a (sub-)pixel [r,c] in the image consists of the following steps: First, the point is projected into the image plane, i.e., onto the sensor chip. If the underlying camera model is an area scan pinhole camera, i.e., if the focal length passed in CameraParam is greater than 0, the projection is described by the following equations:
/ x \
p(c) = | y |
\ z /
u = Focus * x / z
v = Focus * y / z
In contrast, if the focal length is passed as 0 in CameraParam, the camera model of an area scan telecentric camera is used, i.e., it is assumed that the optics of the lens of the camera performs a parallel projection. In this case, the corresponding equations are:
/ x \
p(c) = | y |
\ z /
u = x
v = y
For both types of area scan cameras, the lens distortions can be modeled either by the division model or by the polynomial model. The division model uses one parameter (Kappa) to model the radial distortions.
The following equations transform the distorted image plane coordinates into undistorted image plane coordinates if the division model is used:
u = u' / (1+Kappa*(u'^2+v'^2) v = v' / (1+Kappa*(u'^2+v'^2)
These equations can be inverted analytically, which leads to the following equations that transform undistorted coordinates into distorted coordinates if the division model is used:
u' = (2*u) / (1+sqrt(1-4*Kappa*(u^2+v^2))) v' = (2*v) / (1+sqrt(1-4*Kappa*(u^2+v^2)))
The polynomial model uses three parameters (K1, K2, K3) to model the radial distortions, and two parameters (P1, P2) to model the decentering distortions. The following equations transform the distorted image plane coordinates into undistorted image plane coordinates if the polynomial model is used:
u = u' + u'*(K1*r^2 + K2*r^4 + K3*r^6) +
2*P1*u'*v' + P2*(r^2 + 2*u'^2)
v = v' + v'*(K1*r^2 + K2*r^4 + K3*r^6) +
P1*(r^2 + 2*v'^2) + 2*P2*u'*v'
with: r = sqrt(u'^2+v'^2)
These equations cannot be inverted analytically. Therefore, distorted image plane coordinates must be calculated from undistorted image plane coordinates numerically.
Which of the two distortion models is used, is defined by the number of values passed in the parameter StartCamParam: If StartCamParam contains 8 values, the division model is used. If StartCamParam contains 12 values, the polynomial model is used.
Finally, the point is transformed from the image plane coordinate system into the image coordinate system, i.e., the pixel coordinate system:
c = u' / Sx + Cx
r = v' / Sy + Cy
For line scan cameras, also the relative motion between the camera and the object must be modeled. In HALCON, the following assumptions for this motion are made:
the camera moves with constant velocity along a straight line
the orientation of the camera is constant
the motion is equal for all images
The motion is described by the motion vector V = (Vx,Vy,Vz)' that must be given in [meter/scanline] in the camera coordinate system. The motion vector describes the motion of the camera, assuming a fixed object. In fact, this is equivalent to the assumption of a fixed camera with the object traveling along -V.
The camera coordinate system of line scan cameras is defined as follows: The origin of the coordinate system is the center of projection. The z-axis is identical to the optical axis and directed so that the visible points have positive z coordinates. The y-axis is perpendicular to the sensor line and to the z-axis. It is directed so that the motion vector has a positive y-component. The x-axis is perpendicular to the y- and z-axis, so that the x-, y-, and z-axis form a right-handed coordinate system.
As the camera moves over the object during the image acquisition, also the camera coordinate system moves relatively to the object, i.e., each image line has been imaged from a different position. This means, there would be an individual pose for each image line. To make things easier, in HALCON, all transformations from world coordinates into camera coordinates and vice versa are based on the pose of the first image line only. The motion V is taken into account during the projection of the point p(c) into the image. Consequently, only the pose of the first image line is returned by the operators find_marks_and_pose and camera_calibration.
For line scan pinhole cameras, the projection of the point p(c) that is given in the camera coordinate system into a (sub-)pixel [r,c] in the image is defined as follows:
Assuming
/ x \
p(c) = | y |,
\ z /
the following set of equations must be solved for m, u', and t:
m * D * u' = x - t * Vx
-m * D * pv = y - t * Vy
m * Focus = z - t * Vz
with
1
D = -----------------------
1 + Kappa*(u'*u' + pv*pv)
pv = Sy*Cy
This already includes the compensation for radial distortions. Note that for line scan cameras, only the division model for radial distortions can be used.
Finally, the point is transformed into the image coordinate system, i.e., the pixel coordinate system:
c = u' / Sx + Cx
r = t
The total of 14 or 18 camera parameters for area scan cameras (depending on the selected distortion model) and 17 camera parameters for line scan cameras, respectively, can be divided into the internal and external camera parameters:
These parameters describe the characteristics of the used camera, especially the dimension of the sensor itself and the projection properties of the used combination of lens, camera, and frame grabber.
For area scan cameras, the above described camera model contains the following parameters (8 parameters if the distortions are modeled by the division model and 12 parameters if the polynomial model is used):
Focal length of the lens. 0 for telecentric lenses.
Either:
Distortion coefficient to model the radial lens distortions (omitted if the polynomial model is used).
Or:
Distortion coefficients of the polynomial model to model the radial and decentering lens distortions (omitted if the division model is used).
Scale factor. For pinhole cameras, it corresponds to the horizontal distance between two neighboring cells on the sensor. For telecentric cameras, it represents the horizontal size of a pixel in world coordinates. Attention: This value increases, if the image is subsampled!
Scale factor. For pinhole cameras, it corresponds to the vertical distance between two neighboring cells on the sensor. For telecentric cameras, it represents the vertical size of a pixel in world coordinates. Since in most cases the image signal is sampled line-synchronously, this value is determined by the dimension of the sensor and needn't be estimated for pinhole cameras by the calibration process. Attention: This value increases, if the image is subsampled!
Column coordinate of the image center point (center of the radial distortion).
Row coordinate of the image center point (center of the radial distortion).
Width of the sampled image. Attention: This value decreases, if the image is subsampled!
Height of the sampled image. Attention: This value decreases, if the image is subsampled!
For line scan cameras, the above described camera model contains the following 11 parameters:
Focal length of the lens.
Distortion coefficient of the division model to model the radial lens distortions.
Scale factor, corresponds to the horizontal distance between two neighboring cells on the sensor. Attention: This value increases if the image is subsampled!
Scale factor. During the calibration, it appears only in the form pv = Sy*Cy. pv describes the distance of the image center point from the sensor line in [meters]. Attention: This value increases if the image is subsampled!
Column coordinate of the image center point (center of the radial distortion).
Distance of the image center point (center of the radial distortion) from the sensor line in [scanlines].
Width of the sampled image. Attention: This value decreases if the image is subsampled!
Height of the sampled image. Attention: This value decreases if the image is subsampled!
X-component of the motion vector.
Y-component of the motion vector.
Z-component of the motion vector.
Note that the term focal length is not quite correct and would be appropriate only for an infinite object distance. To simplify matters, always the term focal length is used even if the image distance is meant.
Note that for all operators that use camera parameters as input the parameter values are checked as to whether they fullfill the following restrictions:
Sx > 0
Sy >= 0
Focus >= 0
ImageWidth > 0
ImageHeight > 0
For some operators the restrictions differ slightly. In particular, for operators that do not support telecentric cameras the following restriction applies:
Focus > 0
For operators that do not support line scan cameras the following restrictions apply:
Sy > 0
Vx^2+Vy^2+Vz^2 != 0
These 6 parameters describe the 3D pose, i.e., the position and orientation, of the world coordinate system relative to the camera coordinate system. For line scan cameras, the pose of the world coordinate system refers to the camera coordinate system of the first image line. Three parameters describe the translation, three the rotation. See create_pose for more information about 3D poses. Note that camera_calibration operates with all types of 3D poses for NStartPose.
When using the standard calibration plate, the world coordinate system is defined by the coordinate system of the calibration plate which is located in the middle of the surface of the calibration plate, its z-axis pointing into the calibration plate, its x-axis to the right, and it y-axis downwards.
The use of camera_calibration leads to some questions, which are dealed with in the following sections:
How to generate an appropriate calibration plate?
The simplest method to determine the internal parameters of a camera is the use of the standard calibration plate as generated by the operator gen_caltab. You can obtain high-precision calibration plates in various sizes and materials from your local distributor. In case of small distances between object and lens it may be sufficient to print the calibration object by a laser printer and to mount it on a cardboard. Otherwise -- especially by using a wide-angle lens -- it is possible to print the PostScript file on a large ink-jet printer and to mount it on a aluminum plate. It is very important that the mark coordinates in the calibration plate description file correspond to the real ones on the calibration plate with high accuracy. Thus, the calibration plate description file has to be modified in accordance with the measurement of the calibration plate!
How to take a set of suitable images?
If you use the standard calibration plate, you can proceed in the following way: With the combination of lens (fixed distance!), camera, and frame grabber to be calibrated a set of images of the calibration plate has to be taken, see open_framegrabber and grab_image. The following items have to be considered:
At least a total of 10 to 20 images should be taken into account.
The calibration plate has to be completely visible (incl. border!).
Reflections etc. on the calibration plate should be avoided.
Within the set of images the calibration plate should appear in different positions and orientations: Once left in the image, once right, once (left and right) at the bottom, once (left or right) at the top, from different distances etc. At this, the calibration plate should be rotated around its x- and/or y-axis, so the perspective distortions of the calibration object are clearly visible. Thus, the external camera parameters (camera pose with regard of the calibration plate) should be set to a large variety of different values!
The calibration plate should fill at least a quarter of the whole image to ensure the robust detection of the marks and a robust modelling of radial distortions.
How to extract the calibration marks in the images?
If a standard calibration plate is used, you can use the operators find_caltab and find_marks_and_pose to determine the coordinates of the calibration marks in each image and to compute a rough estimate for the external camera parameters. The concatenation of these values can directly be used as initial values for the external camera parameters (NStartPose) in camera_calibration.
Obviously, images in which the segmentation of the calibration plate (find_caltab) has failed or the calibration marks haven't been determined successfully by find_marks_and_pose should not be used.
Which distortion model should be used?
For area scan cameras, two distortion models can be used: The division model and the polynomial model. The division model uses one parameter to model the radial distortions while the polynomial model uses five parameters to model radial and decentering distortions (see above).
The advantages of the division model are that the distortions can be applied faster, especially the inverse distortions, i.e., if world coordinates are projected into the image plane. Furthermore, if only few calibration images are used or if the field of view is not covered sufficiently, the division model typically yields more stable results than the polynomial model. The main advantage of the polynomial model is that it can model the distortions more accurately because it uses higher order terms to model the radial distortions and because it also models the decentering distortions. Note that the polynomial model cannot be inverted analytically. Therefore, the inverse distortions must be calculated iteratively, which is slower than the calculation of the inverse distortions with the (analytically invertible) division model.
Typically, the division model should be used for the calibration. If the accuracy of the calibration is not high enough, the polynomial model can be used. But note that the calibration sequence used for the polynomial model must provide a complete coverage of the area in which measurements will later be performed. The distortions may be modeled inaccurately outside of the area that was covered by the calibration plate. This holds for the image border as well as for areas inside the field of view that were not covered by the calibration plate.
How to find suitable initial values for the internal camera parameters?
The operators find_marks_and_pose (determination of initial values for the external camera parameters) and camera_calibration require initial values for the internal camera parameters. These parameters can be provided by a appropriate text file (see read_cam_par) which can be generated by write_cam_par or can be edited manually.
For area scan cameras, the following should be considered for the initial values of the single parameters:
The initial value is the nominal focal length of the used lens, e.g., 0.008,m.
Either:
Use 0.0 as initial value (omitted if the polynomial model is used).
Or:
Use the initial value 0.0 for each of the five coefficients (omitted if the division model is used).
The initial value for the horizontal distance between two neighboring cells depends on the dimension of the used chip of the camera (see technical specifications of the camera). Generally, common chips are either 1/3”-Chips (e.g., SONY XC-73, SONY XC-777), 1/2”-Chips (e.g., SONY XC-999, Panasonic WV-CD50), or 2/3”-Chips (e.g., SONY DXC-151, SONY XC-77). Notice: The value of S_{x} increases if the image is subsampled! Appropriate initial values are:
Full image (768*576) Subsampling (384*288)
1/3"-Chip 0.0000055 m 0.0000110 m
1/2"-Chip 0.0000086 m 0.0000172 m
2/3"-Chip 0.0000110 m 0.0000220 m
The value for Sx is calibrated, since the video signal of a camera normally isn't sampled pixel-synchronously.
Since most off-the-shelf cameras have quadratic pixels, the same values for Sy are valid as for Sx. In contrast to Sx the value for Sy will not be calibrated for pinhole cameras, because the video signal of a camera normally is sampled line-synchronously. Thus, the initial value is equal to the final value. Appropriate initial values are:
Full image (768*576) Subsampling (384*288)
1/3"-Chip 0.0000055 m 0.0000110 m
1/2"-Chip 0.0000086 m 0.0000172 m
2/3"-Chip 0.0000110 m 0.0000220 m
Initial values for the coordinates of the image center is the half image width and half image height. Notice: The values of Cx and Cy decrease if the image is subsampled! Appropriate initial values are:
Full image (768*576) Subsampling (384*288)
Cx 384.0 192.0
Cy 288.0 144.0
These two parameters are determined by the used frame grabber and therefore are not calibrated. Appropriate initial values are, for example:
Full image (768*576) Subsampling (384*288)
ImageWidth 768 384
ImageHeight 576 288
For line scan cameras, the following should be considered for the initial values of the single parameters:
The initial value is the nominal focal length of the used lens, e.g., 0.008 m.
Use 0.0 as initial value.
The initial value for the horizontal distance between two neighboring cells can be taken from the technical specifications of the camera. Typical initial values are 7e-6 m, 10e-6 m, and 14e-6 m. Notice: The value of Sx increase, if the image is subsampled!
The initial value for the size of a cell in the direction perpendicular to the sensor line can also be taken from the technical specifications of the camera. Typical initial values are 7e-6 m, 10e-6 m, and 14e-6 m. Notice: The value of Sx increase, if the image is subsampled! In contrast to Sx, the value for Sy will NOT be calibrated for line scan cameras, because it appears only in the form pv = Sy * Cy. Therefore, it cannot be determined separately.
The initial value for the x-coordinate of the image center is the half image width. Notice: The values of Cx decreases if the image is subsampled! Appropriate initial values are:
Image width: 1024 2048 4096 8192
Cx: 512 1024 2048 4096
The initial value for the y-coordinate of the image center can normally be set to 0.
These two parameters are determined by the used frame grabber and therefore are not calibrated.
The initial values for the x-, y-, and z-component of the motion vector depend on the image acquisition setup. Assuming a camera that looks perpendicularly onto a conveyor belt, and that is rotated around its optical axis such that the sensor line is perpendicular to the conveyor belt, i.e., the y-axis of the camera coordinate system is parallel to the conveyor belt, the initial values Vx = Vz = 0. The initial value for Vy can then be determined, e.g., from a line scan image of an object with known size (e.g., calibration plate, ruler):
Vy = LM / LR with:LM = Length of the object in object coordinates [meter] LR = Length of the object in image coordinates [rows]
If, compared to the above setup, the camera is rotated 30 degrees around its optical axis, i.e., around the z-axis of the camera coordinate system, the above determined initial values must be changed as follows:
Vx_rot_z = sin(30) * Vy
Vy_rot_z = cos(30) * Vy
Vz_rot_z = Vz = 0
If, compared to the first setup, the camera is rotated -20 degrees around the x-axis of the camera coordinate system, the following initial values result:
Vx_rot_x = Vx = 0
Vy_rot_x = cos(20) * Vy
Vz_rot_x = sin(20) * Vy
The quality of the initial values for Vx, Vy, and Vz are crucial for the success of the whole calibration. If they are not precise enough, the calibration may fail.
Which camera parameters have to be estimated?
The input parameter EstimateParams is used to select which camera parameters to estimate. Usually this parameter is set to 'all', i.e., all 6 external camera parameters (translation and rotation) and all internal camera parameters are determined. If the internal camera parameters already have been determined (e.g., by a previous call to camera_calibration) it is often desired to only determine the pose of the world coordinate system in camera coordinates (i.e., the external camera parameters). In this case, EstimateParams can be set to 'pose'. This has the same effect as EstimateParams = ['alpha','beta','gamma','transx','transy','transz']. Otherwise, EstimateParams contains a tuple of strings indicating the combination of parameters to estimate. In addition, parameters can be excluded from estimation by using the prefix ~. For example, the values ['pose','~transx'] have the same effect as ['alpha','beta','gamma','transy','transz']. Whereas ['all','~focus'] determines all internal and external parameters except the focus, for instance. The prefix ~ can be used with all parameter values except 'all'.
What is the order within the individual parameters?
The length of the tuple NStartPose corresponds to the number of calibration images, e.g., using 15 images leads to a length of the tuple NStartPose equal to 15*7=105 (15 times the 7 external camera parameters). The first 7 values correspond to the pose of the calibration plate in the first image, the next 7 values to the pose in the second image, etc.
This fixed number of calibration images has to be considered within the tuples with the coordinates of the 3D model marks and the extracted 2D marks. If 15 images are used, the length of the tuples NRow and NCol is 15 times the length of the tuples with the coordinates of the 3D model marks (NX, NY, and NZ). If every image consists 49 marks, the length of the tuples NRow and NCol is 15*49=735, while the length of the tuples NX, NY, and NZ is 49. The order of the values in NRow and NCol is “image after image”, i.e., using 49 marks the first 3D model point corresponds to the 1st, 50th, 99th, 148th, 197th, 246th, etc. extracted 2D mark.
The 3D model points can be read from a calibration plate description file using the operator caltab_points. Initial values for the poses of the calibration plate can be determined by applying find_marks_and_pose for each image. The tuple NStartPose is set by the concatenation of all these poses.
What is the meaning of the output parameters?
If the camera calibration process is finished successfully, i.e., the minimization process has converged, the output parameters CameraParam and NFinalPose contain the computed exact values for the internal and external camera parameters. The length of the tuple NFinalPose corresponds to the length of the tuple NStartPose.
The representation types of NFinalPose correspond to the representation type of the first tuple of NStartPose (see create_pose). You can convert the representation type by convert_pose_type.
The computed average error (Errors) reflects the accuracy of the calibration. The error value (root mean square error of the position) is measured in pixels.
Must I use a planar calibration object?
No. The operator camera_calibration is designed in a way that the input tuples NX, NY, NZ, NRow, and NCol can contain any 3D/2D correspondences, see the above paragraph explaining the order of the single parameters.
Thus, it makes no difference how the required 3D model marks and the corresponding extracted 2D marks are determined. On one hand, it is possible to use a 3D calibration object, on the other hand, you also can use any characteristic points (natural landmarks) with known position in the world. By setting EstimateParams to 'pose', it is thus possible to compute the pose of an object in camera coordinates! For this, at least three 3D/2D-correspondences are necessary as input. NStartPose can, e.g., be generated directly as shown in the program example for create_pose.
The minimization process of the calibration depends on the initial values of the internal (StartCamParam) and external (NStartPose) camera parameters. The computed average errors Errors give an impression of the accuracy of the calibration. The errors (deviations in x and y coordinates) are measured in pixels.
For line scan cameras, it is possible to set the start value for the internal camera parameter Sy to the value 0.0. In this case, it is not possible to determine the position of the principal point in y-direction. Therefore, EstimateParams must contain the term '~cy'. The effective distance of the principle point from the sensor line is then always pv = Sy*Cy = 0.0.
Ordered tuple with all x coordinates of the calibration marks (in meters).
Ordered tuple with all y coordinates of the calibration marks (in meters).
Ordered tuple with all z coordinates of the calibration marks (in meters).
Ordered tuple with all row coordinates of the extracted calibration marks (in pixels).
Ordered tuple with all column coordinates of the extracted calibration marks (in pixels).
Initial values for the internal camera parameters.
Number of elements: StartCamParam == 8 || StartCamParam == 11 || StartCamParam == 12
Ordered tuple with all initial values for the external camera parameters.
Camera parameters to be estimated.
Default value: 'all'
List of values: 'all', 'alpha', 'beta', 'camera', 'cx', 'cy', 'focus', 'gamma', 'kappa', 'poly', 'poly_rad_2', 'poly_rad_4', 'poly_rad_6', 'poly_tan_2', 'pose', 'sx', 'sy', 'transx', 'transy', 'transz', 'vx', 'vy', 'vz'
Internal camera parameters.
Ordered tuple with all external camera parameters.
Average error distance in pixels.
* read calibration images
read_image(Image1, 'calib-01')
read_image(Image2, 'calib-02')
read_image(Image3, 'calib-03')
* find calibration pattern
find_caltab(Image1, Caltab1, 'caltab.descr', 3, 112, 5)
find_caltab(Image2, Caltab2, 'caltab.descr', 3, 112, 5)
find_caltab(Image3, Caltab3, 'caltab.descr', 3, 112, 5)
* find calibration marks and start poses
find_marks_and_pose(Image1, Caltab1, 'caltab.descr', \
[0.008, 0.0, 0.000011, 0.000011, 384, 288, 768, 576], \
128, 10, 18, 0.9, 15.0, 100.0, RCoord1, CCoord1, \
StartPose1)
find_marks_and_pose(Image2, Caltab2, 'caltab.descr', \
[0.008, 0.0, 0.000011, 0.000011, 384, 288, 768, 576], \
128, 10, 18, 0.9, 15.0, 100.0, RCoord2, CCoord2, \
StartPose2)
find_marks_and_pose(Image3, Caltab3, 'caltab.descr', \
[0.008, 0.0, 0.000011, 0.000011, 384, 288, 768, 576], \
128, 10, 18, 0.9, 15.0, 100.0, RCoord3, CCoord3, \
StartPose3)
* read 3D positions of calibration marks
caltab_points('caltab.descr', NX, NY, NZ)
* camera calibration
camera_calibration(NX, NY, NZ, [RCoord1, RCoord2, RCoord3], \
[CCoord1, CCoord2, CCoord3], \
[0.008, 0.0, 0.000011, 0.000011, 384, 288, 768, 576], \
[StartPose1, StartPose2, StartPose3], 'all', \
CameraParam, NFinalPose, Errors)
* write internal camera parameters to file
write_cam_par(CameraParam, 'campar.dat')
camera_calibration returns 2 (H_MSG_TRUE) if all parameter values are correct and the desired camera parameters have been determined by the minimization algorithm. If necessary, an exception is raised.
find_marks_and_pose, caltab_points, read_cam_par
write_pose, pose_to_hom_mat3d, disp_caltab, sim_caltab
find_caltab, find_marks_and_pose, disp_caltab, sim_caltab, write_cam_par, read_cam_par, create_pose, convert_pose_type, write_pose, read_pose, pose_to_hom_mat3d, hom_mat3d_to_pose, caltab_points, gen_caltab, calibrate_cameras
Calibration
| Operators |