Multi-View [HALCON Operator Reference / Version 18.11.3.0]

Multi-View

This chapter describes how to calibrate different multi-view camera setups.

General Objectives

To achieve maximum accuracy of measurement for your camera setup, you have to calibrate it accordingly. Therefore, a camera model is determined, which describes the projection of a 3D world point into a (sub-)pixel in the image.

The following paragraphs state how to perform the calibration of different camera setups. In particular they describe

the needed calibration object,
the individual steps to calibrate the cameras, including
- how to prepare the calibration input data,
- how to perform the actual calibration with calibrate_camerascalibrate_camerasCalibrateCamerasCalibrateCamerasCalibrateCameras,
- how to check the success of the calibration, and
- how to obtain the results of the calibration.
the camera parameters,
additional information about the calibration process,
the available 3D camera models, and
limitations related to specific camera types.

Calibration Object

For a successful calibration of your camera setup, at least one calibration object with accurately known metric properties is needed, e.g., a HALCON calibration plate. Before calling calibrate_camerascalibrate_camerasCalibrateCamerasCalibrateCamerasCalibrateCameras, take a series of images of the calibration object in different orientations and make sure that the whole field of view or measurement volume is covered. The success of the calibration highly depends on the quality of the calibration object and the images. So you might want to exercise special diligence during the acquisition of the calibration images. See the section “How to take a set of suitable images?” for further information.

Preparing the Calibration Input Data

Before calling calibrate_camerascalibrate_camerasCalibrateCamerasCalibrateCamerasCalibrateCameras, you must create and adapt the calibration data model with the following steps:

Create a calibration data model with the operator create_calib_datacreate_calib_dataCreateCalibDataCreateCalibDataCreateCalibData, specifying the number of cameras in the setup and the number of used calibration objects.
Specify the camera type and the initial internal camera parameters for all cameras with the operator set_calib_data_cam_paramset_calib_data_cam_paramSetCalibDataCamParamSetCalibDataCamParamSetCalibDataCamParam. Note that only cameras of the same type can be calibrated in a single setup.
Specify the description of all calibration objects with the operator set_calib_data_calib_objectset_calib_data_calib_objectSetCalibDataCalibObjectSetCalibDataCalibObjectSetCalibDataCalibObject.
Collect observation data with the operators find_calib_objectfind_calib_objectFindCalibObjectFindCalibObjectFindCalibObject or set_calib_data_observ_pointsset_calib_data_observ_pointsSetCalibDataObservPointsSetCalibDataObservPointsSetCalibDataObservPoints, i.e., obtain the image coordinates of the extracted calibration marks of the calibration object and a roughly estimated pose of the calibration object relative to the observing camera.
Configure the calibration process, e.g., specify the reference camera or exclude certain camera parameters from the optimization. You can specify parameters for the complete setup or just configure parameters of individual cameras as well as calibration object poses in the setup with the operator set_calib_dataset_calib_dataSetCalibDataSetCalibDataSetCalibData. For example, if the image sensor cell size of camera 0 is known precisely and only the rest of the parameters need to be calibrated, you call set_calib_data (CalibDataID, 'camera', 0, \ 'excluded_settings', ['sx','sy']).

Performing the Actual Camera Calibration

The calibration performed by calibrate_camerascalibrate_camerasCalibrateCamerasCalibrateCamerasCalibrateCameras depends on the camera types that are involved in the calibration setup. While different camera setups require specific conditions when acquiring images, the basic steps of the calibration procedure for setups including projective and/or telecentric cameras are similar:

Building a chain of observation poses: In the first step, the operator calibrate_camerascalibrate_camerasCalibrateCamerasCalibrateCamerasCalibrateCameras tries to build a valid chain of observation poses, that connects all cameras and calibration object poses to the reference camera. Depending on the setup, the conditions for a valid chain of poses differ. For specific information see the respective paragraphs below. If there is a camera that cannot be reached (i.e., it is not observing any calibration object pose that can be connected in the chain), the calibration process is terminated with an error. Otherwise, the algorithm initializes all calibration items' poses by going down this chain.
First optimization: In this step, calibrate_camerascalibrate_camerasCalibrateCamerasCalibrateCamerasCalibrateCameras performs the actual optimization for all optimization parameters that were not explicitly excluded from the calibration.
Second optimization: Based on the so-far calibrated cameras, the algorithm corrects all observations that contain mark contour information (see find_calib_objectfind_calib_objectFindCalibObjectFindCalibObjectFindCalibObject). Then, the calibration setup is optimized anew for the corrections to take effect. If no contour information was available, this step is skipped.
Compute quality of parameter estimation: In the last step, calibrate_camerascalibrate_camerasCalibrateCamerasCalibrateCamerasCalibrateCameras computes the standard deviations and the covariances of the calibrated internal camera parameters.

Detailed information about the calibration of the mentioned camera setups and additionally about the calibration of line scan cameras can be found in the following paragraphs.

Projective area scan cameras

For a setup with projective area scan cameras ('area_scan_division'"area_scan_division""area_scan_division""area_scan_division""area_scan_division", 'area_scan_polynomial'"area_scan_polynomial""area_scan_polynomial""area_scan_polynomial""area_scan_polynomial", 'area_scan_tilt_division'"area_scan_tilt_division""area_scan_tilt_division""area_scan_tilt_division""area_scan_tilt_division", 'area_scan_tilt_polynomial'"area_scan_tilt_polynomial""area_scan_tilt_polynomial""area_scan_tilt_polynomial""area_scan_tilt_polynomial", 'area_scan_tilt_image_side_telecentric_division', 'area_scan_tilt_image_side_telecentric_polynomial', 'area_scan_hypercentric_division'"area_scan_hypercentric_division""area_scan_hypercentric_division""area_scan_hypercentric_division""area_scan_hypercentric_division", and 'area_scan_hypercentric_polynomial'"area_scan_hypercentric_polynomial""area_scan_hypercentric_polynomial""area_scan_hypercentric_polynomial""area_scan_hypercentric_polynomial"), the calibration is performed in the four steps listed above. The algorithm tries to build a chain of observation poses that connects all cameras and calibration object poses to the reference camera like in the diagram below.


(1)	(2)

(1) All cameras can be connected by a chain of observation poses. (2) The leftmost camera is isolated, because the left calibration plate cannot be seen by any other camera.

Telecentric area scan cameras

For a setup with telecentric area scan cameras ('area_scan_telecentric_division'"area_scan_telecentric_division""area_scan_telecentric_division""area_scan_telecentric_division""area_scan_telecentric_division", 'area_scan_telecentric_polynomial'"area_scan_telecentric_polynomial""area_scan_telecentric_polynomial""area_scan_telecentric_polynomial""area_scan_telecentric_polynomial", 'area_scan_tilt_bilateral_telecentric_division', 'area_scan_tilt_bilateral_telecentric_polynomial', 'area_scan_tilt_object_side_telecentric_division', or 'area_scan_tilt_object_side_telecentric_polynomial'), similar to projective area scan cameras, the same four steps that are listed above are executed. In the first step (building a chain of observation poses that connects all cameras and calibration objects), additional conditions must hold. Since the pose of an object can only be determined up to a translation along the optical axis, each calibration object must be observed by at least two cameras to determine its relative location. Otherwise, its pose is excluded from the calibration. Also, since a planar calibration object appears the same from two different observation angles, the relative pose of the cameras among each other cannot be determined unambiguously. Therefore there are always two valid alternative relative poses. Both alternatives result in a consistent camera setup which can be used for measuring. Since the ambiguity cannot be resolved, the first of the alternatives is returned. Note that, if the returned pose is not the real pose but the alternative one, then this will result in a mirrored reconstruction.

Projective and telecentric area scan cameras

For a mixed setup with projective and telecentric area scan cameras, the algorithm performs the same four steps as enumerated above. Possible ambiguities during the first step (building a chain of observation poses that connects all cameras and calibration objects), as described above for the setup with telecentric cameras, can be resolved as long as there exists a chain of observation poses consisting of all perspective cameras and a sufficient number of calibration objects. Here, sufficient number means that each telecentric camera observes at least two calibration objects of this chain.


(1)	(2)

Mixed calibration setup with perspective (P) and telecentric (T) area scan cameras. (1) All perspective cameras are connected by a chain of observation poses that only contains perspective cameras. (2) The second calibration plate (from the left) is not observed by the rightmost perspective camera. Therefore, the relative pose between both perspective cameras cannot be determined uniquely.

Line scan cameras

For a setup with line scan cameras ('line_scan'"line_scan""line_scan""line_scan""line_scan"), some restrictions exist: First, only one camera can be calibrated and only one calibration object per setup can be used. Furthermore, the calibration does not deliver information about standard deviations and covariances for the estimated parameters.

Finally, for calibration plates with rectangularly arranged marks (see gen_caltabgen_caltabGenCaltabGenCaltabGenCaltab) all observations must contain the projection coordinates of all calibration marks of the calibration object. For calibration plates with hexagonally arranged marks (see create_caltabcreate_caltabCreateCaltabCreateCaltabCreateCaltab) this restriction is not applied. You can find further information about calibration plates and the acquisition of calibration images in the section “Additional information about the calibration process”.

Checking the Success of the Calibration

After a successful calibration, the root mean square error (RMSE) of the back projection of the optimization is returned in ErrorErrorErrorErrorerror (in pixels). The error gives a general indication whether the optimization was successful as it corresponds to the average distance (in pixels) between the backprojected calibration points and their extracted image coordinates.

If only a single camera is calibrated, an ErrorErrorErrorErrorerror in the order of 0.1 pixel (the typical detection error by extraction of the coordinates of the projected calibration markers) is an indication that the optimization fits the observation data well. If ErrorErrorErrorErrorerror strongly differs from 0.1 pixels, the calibration did not perform well. Reasons for this might be, e.g., a poor image quality, an insufficient number of calibration images, or an inaccurate calibration plate. If more than one camera is calibrated simultaneously, the value of ErrorErrorErrorErrorerror is more difficult to judge. As a rule of thumb, ErrorErrorErrorErrorerror should be as small as possible and at least smaller than 1.0, thus indicating that a subpixel precise evaluation of the data is possible with the calibrated parameters. This value might be difficult to reach in particular configurations. For further analysis of the quality of the calibration, refer to the standard deviations and covariances of the estimated parameters (currently for area scan cameras only, see get_calib_dataget_calib_dataGetCalibDataGetCalibDataGetCalibData).

Getting the Calibration Results

The results of the calibration, i.e., internal camera parameters, camera poses (external camera parameters), calibration objects poses etc., can be queried with get_calib_dataget_calib_dataGetCalibDataGetCalibDataGetCalibData.

The poses of telecentric cameras can only be determined up to a displacement along the z-axis of the coordinate system of the respective camera (perpendicular to the image plane). Therefore, all camera poses are moved along this axis until they all lie on a common sphere. The center of the sphere is defined by the pose of the first calibration object. The radius of the sphere depends on the calibration setup. If projective and telecentric area scan cameras are calibrated, the radius is the maximum over all distances from the perspective cameras to the first calibration object. Otherwise, if only telecentric area scan cameras are considered, the radius is equal to 1 m.

Camera Parameters

Regarding camera parameters, you can distinguish between internal and external camera parameters.

Internal camera parameters:

These parameters describe the characteristics of the used camera, especially the dimension of the sensor itself and the projection properties of the used combination of lens, camera, and frame grabber. Below is an overview of all available camera types and their respective parameters CameraParam. In the list, “projective area scan cameras” refers to the property that the lens performs a perspective projection on the object-side of the lens, while “telecentric area scan cameras” refers to the property that the lens performs a telecentric projection on the object-side of the lens.

Area scan cameras

have 9 to 16 internal parameters depending on the camera type.

For reasons explained below, parameters that are marked with an * asterisk are fixed and not estimated by the algorithm.

Area scan cameras with regular lenses

Projective area scan cameras

'area_scan_division'"area_scan_division""area_scan_division""area_scan_division""area_scan_division": ['area_scan_division', Focus, Kappa, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]
'area_scan_polynomial'"area_scan_polynomial""area_scan_polynomial""area_scan_polynomial""area_scan_polynomial": ['area_scan_polynomial', Focus, K1, K2, K3, P1, P2, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]

Telecentric area scan cameras

'area_scan_telecentric_division'"area_scan_telecentric_division""area_scan_telecentric_division""area_scan_telecentric_division""area_scan_telecentric_division": ['area_scan_telecentric_division', Magnification, Kappa, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]
'area_scan_telecentric_polynomial'"area_scan_telecentric_polynomial""area_scan_telecentric_polynomial""area_scan_telecentric_polynomial""area_scan_telecentric_polynomial": ['area_scan_telecentric_polynomial', Magnification, K1, K2, K3, P1, P2, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]

Area scan cameras with tilt lenses

Projective area scan cameras

'area_scan_tilt_division'"area_scan_tilt_division""area_scan_tilt_division""area_scan_tilt_division""area_scan_tilt_division": ['area_scan_tilt_division', Focus, Kappa, ImagePlaneDist, Tilt, Rot, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]
'area_scan_tilt_polynomial'"area_scan_tilt_polynomial""area_scan_tilt_polynomial""area_scan_tilt_polynomial""area_scan_tilt_polynomial": ['area_scan_tilt_polynomial', Focus, K1, K2, K3, P1, P2, ImagePlaneDist, Tilt, Rot, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]
'area_scan_tilt_image_side_telecentric_division': ['area_scan_tilt_image_side_telecentric_division', Focus, Kappa, Tilt, Rot, Sx*, Sy*, Cx, Cy, ImageWidth, ImageHeight]
'area_scan_tilt_image_side_telecentric_polynomial': ['area_scan_tilt_image_side_telecentric_polynomial', Focus, K1, K2, K3, P1, P2, Tilt, Rot, Sx*, Sy*, Cx, Cy, ImageWidth, ImageHeight]

Telecentric area scan cameras

'area_scan_tilt_bilateral_telecentric_division': ['area_scan_tilt_bilateral_telecentric_division', Magnification, Kappa, Tilt, Rot, Sx*, Sy*, Cx, Cy, ImageWidth, ImageHeight]
'area_scan_tilt_bilateral_telecentric_polynomial': ['area_scan_tilt_bilateral_telecentric_polynomial', Magnification, K1, K2, K3, P1, P2, Tilt, Rot, Sx*, Sy*, Cx, Cy, ImageWidth, ImageHeight]
'area_scan_tilt_object_side_telecentric_division': ['area_scan_tilt_object_side_telecentric_division', Magnification, Kappa, ImagePlaneDist, Tilt, Rot, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]
'area_scan_tilt_object_side_telecentric_polynomial': ['area_scan_tilt_object_side_telecentric_polynomial', Magnification, K1, K2, K3, P1, P2, ImagePlaneDist, Tilt, Rot, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]

Area scan cameras with hypercentric lenses

Projective area scan cameras with hypercentric lenses

'area_scan_hypercentric_division'"area_scan_hypercentric_division""area_scan_hypercentric_division""area_scan_hypercentric_division""area_scan_hypercentric_division": ['area_scan_hypercentric_division', Focus, Kappa, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]
'area_scan_hypercentric_polynomial'"area_scan_hypercentric_polynomial""area_scan_hypercentric_polynomial""area_scan_hypercentric_polynomial""area_scan_hypercentric_polynomial": ['area_scan_hypercentric_polynomial', Focus, K1, K2, K3, P1, P2, Sx, Sy*, Cx, Cy, ImageWidth, ImageHeight]

Description of the internal camera parameters:

CameraType:

Type of the camera, as listed above.

Focus:

Focal length of the lens (only for lenses that perform a perspective projection on the object side of the lens).

The initial value is the nominal focal length of the used lens, e.g., 0.008m.

Magnification:

Magnification of the lens (only for lenses that perform a telecentric projection on the object side of the lens).

The initial value is the nominal magnification of the used telecentric lens (the image size divided by the object size), e.g., 0.2.

Kappa :

Distortion coefficient to model the radial lens distortions (only for the division model).

Use 0.0 as initial value.

K1, K2, K3, P1, P2:

Distortion coefficients to model the radial and decentering lens distortions (only for the polynomial model).

Use 0.0 as initial value for all five coefficients.

ImagePlaneDist:

Distance of the exit pupil of the lens to the image plane. The exit pupil is the (virtual) image of the aperture stop (typically the diaphragm), as viewed from the image side of the lens. Typical values are in the order of a few centimeters to very large values if the lens is close to being image-side telecentric.

Tilt, Rot:

The tilt angle describes the angle by which the optical axis is tilted with respect to the normal of the sensor plane (corresponds to a rotation around the x-axis). The rotation angle describes the rotation around the optical axis (z-axis). For a rotation the optical axis gets tilted vertically down with respect to the camera housing, corresponds to the optical axis being tilted horizontally to the left (direction of view along the z-z-axis), corresponds to the optical axis being tilted vertically up, and corresponds to the optical axis being tilted horizontally to the right.

These parameters are only used if a tilt lens is part of the camera setup.

The tilt of the lens is described by the parameters , and ImagePlaneDistImagePlaneDistImagePlaneDistImagePlaneDistimagePlaneDist. describes the orientation of the tilt axis in relation to the x-axis of the sensor and has to be applied first. describes the actual tilt of the lens. ImagePlaneDistImagePlaneDistImagePlaneDistImagePlaneDistimagePlaneDist is the distance of the exit pupil of the lens to the image plane.

These angles are typically roughly known based on the considerations that led to the use of the tilt lens or can be read off from the mechanism by which the lens is tilted.

Sx, Sy:

Scale factors. They corresponds to the horizontal and vertical distance between two neighboring cells on the sensor. Since in most cases the image signal is sampled line-synchronously, is determined by the dimension of the sensor and does not need to be estimated by the calibration process.

The initial values depend on the dimensions of the used chip of the camera. See the technical specification of your camera for the actual values. Attention: These values increase if the image is subsampled!

As projective cameras are described through the pinhole camera model, it is impossible to determine FocusFocusFocusFocusfocus, and simultaneously. Therefore, the algorithm will keep fixed.

For telecentric lenses, it is impossible to determine MagnificationMagnificationMagnificationMagnificationmagnification, , and simultaneously. Therefore, the algorithm will keep fixed.

For image-side telecentric tilt lenses (see chapter “Basics”, section “Camera Model and Parameters” in the “Solution Guide III-C 3D Vision” for an overview of different types of tilt lenses), it is impossible to determine FocusFocusFocusFocusfocus, , , and the tilt parameters and simultaneously. Therefore, additionally to , the algorithm will keep fixed.

For bilateral telecentric tilt lenses, it is impossible to determine MagnificationMagnificationMagnificationMagnificationmagnification, , , and the tilt parameters and simultaneously. Therefore, additionally to , the algorithm will keep fixed.

Cx, Cy:

Column () and row () coordinate of the principal point of the image (center of the radial distortion).

Use the half image width and height as initial values. Attention: These values decrease if the image is subsampled!

ImageWidth, Image Height:

Width and height of the sampled image. Attention: These values decrease if the image is subsampled!

Line scan cameras

Line scan cameras have the following 12 internal parameters: ['line_scan', Focus, Kappa, Sx*, Sy*, Cx, Cy, ImageWidth, ImageHeight, Vx, Vy, Vz]

For reasons explained below, parameters that are marked with an * asterisk are fixed and not estimated by the algorithm.

CameraType:

Type of the camera ('line_scan'"line_scan""line_scan""line_scan""line_scan").

Focus:

Focal length of the lens.

The initial value is the nominal focal length of the used lens, e.g., 0.008m.

Kappa :

Distortion coefficient of the division model to model the radial lens distortions.

Use 0.0 as initial value.

Sx:

Scale factor. Corresponds to the horizontal distance between two neighboring cells on the sensor. Note that Focus and cannot be determined simultaneously. Therefore, is kept fixed in the calibration. The initial value for can be taken from the technical specifications of the camera. Attention: This value increases if the image is subsampled!

Sy:

Scale factor. During the calibration, it appears only in the form . Consequently, and cannot be determined simultaneously. Therefore, in the calibration, is kept fixed. describes the distance of the image center point from the sensor line in meters. The initial value for can be taken from the technical specifications of the camera. Attention: This value increases if the image is subsampled!

Cx:

Column coordinate of the image center point (center of the radial distortions). Use half of the image width as the initial value for . Attention: This value decreases if the image is subsampled!

Cy:

Distance of the image center point (center of the radial distortions) from the sensor line in scanlines. The initial value can normally be set to 0.

ImageWidth, ImageHeight:

Width and height of the sampled image. Attention: These values decrease if the image is subsampled!

Vx, Vy, Vz:

X-, Y-, and Z-component of the motion vector.

The initial values for the x-, y-, and z-component of the motion vector depend on the image acquisition setup. Assuming a camera that looks perpendicularly onto a conveyor belt and that is rotated around its optical axis such that the sensor line is perpendicular to the conveyor belt, i.e., the y-axis of the camera coordinate system is parallel to the conveyor belt, use the initial values . The initial value for can then be determined, e.g., from a line scan image of an object with known size (e.g., calibration plate, ruler):