ClassesClassesClassesClasses | | | | Operators

calibrate_hand_eyeT_calibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye (Operator)

Name

calibrate_hand_eyeT_calibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye — Perform a hand-eye calibration.

Signature

calibrate_hand_eye( : : CalibDataID : Errors)

Herror T_calibrate_hand_eye(const Htuple CalibDataID, Htuple* Errors)

Herror calibrate_hand_eye(const HTuple& CalibDataID, HTuple* Errors)

HTuple HCalibData::CalibrateHandEye() const

void CalibrateHandEye(const HTuple& CalibDataID, HTuple* Errors)

HTuple HCalibData::CalibrateHandEye() const

void HOperatorSetX.CalibrateHandEye(
[in] VARIANT CalibDataID, [out] VARIANT* Errors)

VARIANT HCalibDataX.CalibrateHandEye()

static void HOperatorSet.CalibrateHandEye(HTuple calibDataID, out HTuple errors)

HTuple HCalibData.CalibrateHandEye()

Description

The operator calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye determines the 3D pose of a robot (“hand”) relative to a camera or 3D sensor (“eye”) based on the calibration data model CalibDataIDCalibDataIDCalibDataIDCalibDataIDCalibDataIDcalibDataID. With the determined 3D poses, the poses of the calibration object in the camera coordinate system can be transformed into the coordinate system of the robot which can then, e.g., grasp an inspected part. There are two possible configurations of robot-camera (hand-eye) systems: The camera can be mounted on the robot or be stationary and observe the robot. Note that the term robot is used in place of a mechanism that moves objects. Thus, you can use calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye to calibrate many different systems, from pan-tilt heads to multi-axis manipulators.

In essence, systems suitable for hand-eye calibration are described by a closed chain of four Euclidean transformations. In this chain two non-consecutive transformations are either known from the robot controller or computed from camera data, e.g., calibration object poses observed by a camera. The two unknown constant transformations are computed by the hand-eye calibration procedure.

A hand-eye calibration is performed similarly to the calibration of the external camera parameters (see calibrate_camerascalibrate_camerasCalibrateCamerascalibrate_camerasCalibrateCamerasCalibrateCameras): You acquire a set of poses of a calibration object in the camera coordinate system, and a corresponding set of poses of the tool in robot base coordinates and set them in the calibration data model CalibDataIDCalibDataIDCalibDataIDCalibDataIDCalibDataIDcalibDataID.

In contrast to the camera calibration, the calibration object is not moved manually. This task is delegated to the robot. Basically, two hand-eye calibration scenarios can be distinguished. A robot either moves the camera (moving camera) or it moves the calibration object (stationary camera). The robot's movements are assumed to be known. They are used as an input for the hand-eye calibration and are set in the calibration data model CalibDataIDCalibDataIDCalibDataIDCalibDataIDCalibDataIDcalibDataID using set_calib_dataset_calib_dataSetCalibDataset_calib_dataSetCalibDataSetCalibData.

The results of a hand-eye calibration are two poses: For the moving camera scenario, the 3D pose of the tool in the camera coordinate system ('tool_in_cam_pose') and the 3D pose of the calibration object in the robot base coordinate system ('obj_in_base_pose') are computed. For the stationary camera scenario, the 3D pose of the robot base in the camera coordinate system ('base_in_cam_pose') and the 3D pose of the calibration object in the tool coordinate system ('obj_in_tool_pose') are computed. Their pose type is identical to the pose type of the input poses. If the input poses have different pose types, poses of type 0 are returned.

The two hand-eye calibration scenarios are discussed in more detail below, followed by general information about the data for and the preparation of the calibration data model.

Moving camera (mounted on a robot)

In this configuration, the calibration object remains stationary. The camera is mounted on the robot and is moved to different positions by the robot. The main idea behind the hand-eye calibration is that the information extracted from an observation of the calibration object, i.e., the pose of the calibration object relative to the camera, can be seen as a chain of poses or homogeneous transformation matrices from the calibration object via the base of the robot to its tool (end-effector) and finally to the camera:

  Moving camera:   camera_H_cal  =  camera_H_tool * (base_H_tool)^(-1) * base_H_cal
                          |                |               |                   |
                'obj_in_cam_pose' 'tool_in_cam_pose' 'tool_in_base_pose' 'obj_in_base_pose'

From the set of calibration object poses ('obj_in_cam_pose') and the poses of the tool in the robot base coordinate system ('tool_in_base_pose'), the operator calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye determines the two missing transformations at the ends of the chain, i.e., the pose of the robot tool in the camera coordinate system ( , 'tool_in_cam_pose') and the pose of the calibration object in the robot base coordinate system ( , 'obj_in_base_pose'). These two poses are constant.

In contrast, the transformation in the middle of the chain, , is known but changes for each observation of the calibration object, because it describes the pose of the tool with respect to the robot base coordinate system. In the equation the inverted transformation matrix is used. The inversion is performed internally.

Note that when calibrating SCARA robots, it is not possible to determine the Z translation of 'obj_in_base_pose'. To eliminate this ambiguity the Z translation 'obj_in_base_pose' is internally set to 0.0 and the 'tool_in_cam_pose' is calculated accordingly. It is necessary to determine the true translation in Z after the calibration by moving the robot to a pose of known height in the camera coordinate system. For this, the following approach can be applied: The calibration plate is placed at an arbitrary position. The robot is then moved such that the camera can observe the calibration plate. Now, an image of the calibration plate is acquired and the current robot pose is queried (ToolInBasePose1). From the image, the pose of the calibration plate in the camera coordinate system can be determined (ObjInCamPose1). Afterwards, the tool of the robot is manually moved to the origin of the calibration plate and the robot pose is queried again (ToolInBasePose2). These three poses and the result of the calibration (ToolInCamPose) can be used to fix the Z ambiguity by using the following lines of code:

     pose_invert (ToolInCamPose, CamInToolPose)
     pose_compose (CamInToolPose, ObjInCamPose1, ObjInToolPose1)
     pose_invert (ToolInBasePose1, BaseInToolPose1)
     pose_compose (BaseInToolPose1, ToolInBasePose2, Tool2InTool1Pose)
     ZCorrection := ObjInToolPose1[2]-Tool2InTool1Pose[2]
     set_origin_pose (ToolInCamPose, 0, 0, ZCorrection, ToolInCamPoseFinal)

Stationary camera

In this configuration, the robot grasps the calibration object and moves it in front of the camera. Again, the information extracted from an observation of the calibration object, i.e., the pose of the calibration object in the camera coordinate system (e.g., the external camera parameters), are equal to a chain of poses or homogeneous transformation matrices, this time from the calibration object via the robot's tool to its base and finally to the camera:

  Stationary camera:  camera_H_cal  =  camera_H_base  *  base_H_tool  *  tool_H_cal
                             |                |               |               |
                   'obj_in_cam_pose' 'base_in_cam_pose' 'tool_in_base_pose' 'obj_in_tool_pose'

Analogously to the configuration with a moving camera, the operator calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye determines the two transformations at the ends of the chain, here the pose of the robot base coordinate system in camera coordinates ( , 'base_in_cam_pose') and the pose of the calibration object relative to the robot tool ( , 'obj_in_tool_pose').

The transformation in the middle of the chain, , describes the pose of the tool relative to the robot base coordinate system. The transformation describes the pose of the calibration object relative to the camera coordinate system.

Note that when calibrating SCARA robots, it is not possible to determine the Z translation of 'obj_in_tool_pose'. To eliminate this ambiguity the Z translation of 'obj_in_tool_pose' is internally set to 0.0 and the 'base_in_cam_pose' is calculated accordingly. It is necessary to determine the true translation in Z after the calibration by moving the robot to a pose of known height in the camera coordinate system. For this, the following approach can be applied: A calibration plate (that is not attached to the robot) is placed at an arbitrary position such that it can be observed by the camera. The pose of the calibration plate must then be determined in the camera coordinate system (ObjInCamPose). Afterwards the tool of the robot is manually moved to the origin of the calibration plate and the robot pose is queried (ToolInBasePose). The two poses and the result of the calibration (BaseInCamPose) can be used to fix the Z ambiguity by using the following lines of code:

     pose_invert (BaseInCamPose, CamInBasePose)
     pose_compose (CamInBasePose, ObjInCamPose, ObjInBasePose)
     ZCorrection := ObjInBasePose[2]-ToolInBasePose[2]
     set_origin_pose (BaseInCamPose, 0, 0, ZCorrection, BaseInCamPoseFinal)

Preparing the calibration input data

Before calling calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye, you must create and fill the calibration data model with the following steps:

  1. Create a calibration data model with the operator create_calib_datacreate_calib_dataCreateCalibDatacreate_calib_dataCreateCalibDataCreateCalibData, specifying the number of cameras in the setup and the number of used calibration objects. Depending on your scenario, CalibSetup has to be set to 'hand_eye_moving_camera'"hand_eye_moving_camera""hand_eye_moving_camera""hand_eye_moving_camera""hand_eye_moving_camera""hand_eye_moving_camera", 'hand_eye_stationary_camera'"hand_eye_stationary_camera""hand_eye_stationary_camera""hand_eye_stationary_camera""hand_eye_stationary_camera""hand_eye_stationary_camera", 'hand_eye_scara_moving_camera'"hand_eye_scara_moving_camera""hand_eye_scara_moving_camera""hand_eye_scara_moving_camera""hand_eye_scara_moving_camera""hand_eye_scara_moving_camera", or 'hand_eye_scara_stationary_camera'"hand_eye_scara_stationary_camera""hand_eye_scara_stationary_camera""hand_eye_scara_stationary_camera""hand_eye_scara_stationary_camera""hand_eye_scara_stationary_camera". These four scenarios on the one hand distinguish whether the camera or the calibration object is moved by the robot and on the other hand distinguish whether an articulated robot or a SCARA robot is calibrated. The arm of an articulated robot has three rotary joints typically covering 6 degrees of freedom (3 translations and 3 rotations). SCARA robots have two parallel rotary joints and one parallel prismatic joint covering only 4 degrees of freedom (3 translations and 1 rotation). Loosely speaking, an articulated robot is able to tilt its end effector while a SCARA robot is not.

  2. Specify the optimization method with the operator set_calib_dataset_calib_dataSetCalibDataset_calib_dataSetCalibDataSetCalibData. For the parameter DataName='optimization_method'"optimization_method""optimization_method""optimization_method""optimization_method""optimization_method", two options for DataValue are available, DataValue='nonlinear'"nonlinear""nonlinear""nonlinear""nonlinear""nonlinear" and DataValue='linear'"linear""linear""linear""linear""linear" (see paragraph 'Performing the actual hand-eye calibration').

  3. Specify the poses of the calibration object

    1. For each observation of the calibration object, the 3D pose can be set directly using the operator set_calib_data_observ_poseset_calib_data_observ_poseSetCalibDataObservPoseset_calib_data_observ_poseSetCalibDataObservPoseSetCalibDataObservPose. This operator is intended to be used with generic 3D sensors that observe the calibration object.

    2. The pose of the calibration object can also be estimated using camera images. The calibration object has to be set in the calibration data model CalibDataIDCalibDataIDCalibDataIDCalibDataIDCalibDataIDcalibDataID with the operator set_calib_data_calib_objectset_calib_data_calib_objectSetCalibDataCalibObjectset_calib_data_calib_objectSetCalibDataCalibObjectSetCalibDataCalibObject. Initial camera parameters have to be set with the operator set_calib_data_cam_paramset_calib_data_cam_paramSetCalibDataCamParamset_calib_data_cam_paramSetCalibDataCamParamSetCalibDataCamParam. If a standard HALCON calibration plate is used, the operator find_calib_objectfind_calib_objectFindCalibObjectfind_calib_objectFindCalibObjectFindCalibObject determines the pose of the calibration plate relative to the camera and saves it in the calibration data model CalibDataIDCalibDataIDCalibDataIDCalibDataIDCalibDataIDcalibDataID. The operator calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye for articulated (i.e., non-SCARA) robots in this case calibrates the camera before performing the hand-eye calibration. If the provided camera parameters are already calibrated, the camera calibration can be switched off by calling

           set_calib_data(CalibDataID,'camera','general','excluded_settings','params').
        
      In contrast, for SCARA robots calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye always assumes that the provided camera parameters are already calibrated. Therefore, in this case the internal camera calibration is never performed automatically prior to the hand-eye calibration. This is because the internal camera parameters cannot be calibrated reliably without significantly tilting the calibration plate with respect to the camera. For hand-eye calibration, the calibration plate is often approximately parallel to the image plane. Therefore, for SCARA robots all camera poses are approximately parallel. Therefore, the camera must be calibrated beforehand by using a different set of calibration images.

  4. Specify the poses of the tool in robot base coordinates. For each pose of the calibration object in the camera coordinate system, the corresponding pose of the tool in the robot base coordinate system has to be set with the operator

         set_calib_data(CalibDataID,'tool', PoseNumber, 'tool_in_base_pose', ToolInBasePose)
      

Performing the actual hand-eye calibration

The operator calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye can perform the calibration in two different ways. In both cases, all provided calibration object poses in camera coordinates and the corresponding poses of the tool in robot base coordinates are used for the calibration. The method to be used is specified with set_calib_dataset_calib_dataSetCalibDataset_calib_dataSetCalibDataSetCalibData.

For the parameter combination DataName='optimization_method'"optimization_method""optimization_method""optimization_method""optimization_method""optimization_method" and DataValue='linear'"linear""linear""linear""linear""linear", the calibration is performed using a linear algorithm which is fast but in many practical situations not accurate enough.

For the parameter DataName='optimization_method'"optimization_method""optimization_method""optimization_method""optimization_method""optimization_method" and DataValue='nonlinear'"nonlinear""nonlinear""nonlinear""nonlinear""nonlinear", the calibration is performed using a non-linear algorithm, which results in the most accurately calibrated poses and thus is the method of choice.

Checking the success of the calibration

The operator calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye returns the pose error of the complete chain of transformations in ErrorsErrorsErrorsErrorsErrorserrors. To be more precise, a tuple with four elements is returned, where the first element is the root-mean-square error of the translational part, the second element is the root-mean-square error of the rotational part, the third element is the maximum translational error and the fourth element is the maximum rotational error. Using these error measures, it can be determined, whether the calibration was successful.

The ErrorsErrorsErrorsErrorsErrorserrors are returned in the same units in which the input poses were given, i.e., the translational errors are typically given in meters and the rotational errors are always given in degrees.

Getting the calibration results

The poses that are computed with the operator calibrate_hand_eyecalibrate_hand_eyeCalibrateHandEyecalibrate_hand_eyeCalibrateHandEyeCalibrateHandEye can be queried with get_calib_dataget_calib_dataGetCalibDataget_calib_dataGetCalibDataGetCalibData. For the moving camera scenario, the 3D pose of the tool in the camera coordinate system ('tool_in_cam_pose') and the 3D pose of the calibration object in the robot base coordinate system ('obj_in_base_pose') can be obtained. For the stationary camera scenario, the 3D pose of the robot base in the camera coordinate system ('base_in_cam_pose') and the 3D pose of the calibration object in the coordinate system of the tool ('obj_in_tool_pose') can be obtained.

Querying the input data

If the poses of the calibration object relative to a camera were computed with find_calib_objectfind_calib_objectFindCalibObjectfind_calib_objectFindCalibObjectFindCalibObject, then for articulated (i.e., non-SCARA) robots they are used in an internal camera calibration step preceding the hand-eye calibration and are calibrated as well. The calibrated 3D poses can be queried using get_calib_dataget_calib_dataGetCalibDataget_calib_dataGetCalibDataGetCalibData with the parameter ItemType='calib_obj_pose'"calib_obj_pose""calib_obj_pose""calib_obj_pose""calib_obj_pose""calib_obj_pose". If the poses of the calibration object were observed with a generic 3D sensor, they cannot be calibrated and are set by set_calib_data_observ_poseset_calib_data_observ_poseSetCalibDataObservPoseset_calib_data_observ_poseSetCalibDataObservPoseSetCalibDataObservPose. These raw 3D poses can be queried using get_calib_data_observ_poseget_calib_data_observ_poseGetCalibDataObservPoseget_calib_data_observ_poseGetCalibDataObservPoseGetCalibDataObservPose. The corresponding 3D poses of the tool in the coordinate system of the robot base can be queried using get_calib_dataget_calib_dataGetCalibDataget_calib_dataGetCalibDataGetCalibData.

Acquiring a suitable set of observations

The following conditions, especially if using a standard calibration plate, should be considered:

Obtaining the poses of the robot tool

We recommend to create the robot poses in a separate program and save them in files using write_posewrite_poseWritePosewrite_poseWritePoseWritePose. In the calibration program you can then import them and set them in the calibration data model CalibDataIDCalibDataIDCalibDataIDCalibDataIDCalibDataIDcalibDataID.

Via the Cartesian interface of the robot, you can typically obtain the pose of the tool in robot base coordinates in a notation that corresponds to the pose representations with the codes 0 or 2 (OrderOfRotationOrderOfRotationOrderOfRotationOrderOfRotationOrderOfRotationorderOfRotation = 'gba'"gba""gba""gba""gba""gba" or 'abg'"abg""abg""abg""abg""abg", see create_posecreate_poseCreatePosecreate_poseCreatePoseCreatePose). In this case, you can directly use the pose values obtained from the robot as input for create_posecreate_poseCreatePosecreate_poseCreatePoseCreatePose.

If the Cartesian interface of your robot describes the orientation in a different way, e.g., with the representation ZYZ ( ), you can create the corresponding homogeneous transformation matrix step by step using the operators hom_mat3d_rotatehom_mat3d_rotateHomMat3dRotatehom_mat3d_rotateHomMat3dRotateHomMat3dRotate and hom_mat3d_translatehom_mat3d_translateHomMat3dTranslatehom_mat3d_translateHomMat3dTranslateHomMat3dTranslate and then convert the resulting matrix into a pose using hom_mat3d_to_posehom_mat3d_to_poseHomMat3dToPosehom_mat3d_to_poseHomMat3dToPoseHomMat3dToPose. The following example code creates a pose from the ZYZ representation described above:

  hom_mat3d_identity (HomMat3DIdent)
  hom_mat3d_rotate (HomMat3DIdent, phi3, 'z', 0, 0, 0, HomMat3DRotZ)
  hom_mat3d_rotate (HomMat3DRotZ, phi2, 'y', 0, 0, 0, HomMat3DRotZY)
  hom_mat3d_rotate (HomMat3DRotZY, phi1, 'z', 0, 0, 0, HomMat3DRotZYZ)
  hom_mat3d_translate (HomMat3DRotZYZ, Tx, Ty, Tz, base_H_tool)
  hom_mat3d_to_pose (base_H_tool, RobPose)

Please note that the hand-eye calibration only works if the poses of the tool in robot base coordinates are specified with high accuracy!

Parallelization

This operator modifies the state of the following input parameter:

The value of this parameter may not be shared across multiple threads without external synchronization.

Parameters

CalibDataIDCalibDataIDCalibDataIDCalibDataIDCalibDataIDcalibDataID (input_control, state is modified)  calib_data HCalibData, HTupleHTupleHCalibData, HTupleHCalibDataX, VARIANTHtuple (integer) (IntPtr) (Hlong) (Hlong) (Hlong) (Hlong)

Handle of a calibration data model.

ErrorsErrorsErrorsErrorsErrorserrors (output_control)  number-array HTupleHTupleHTupleVARIANTHtuple (real) (double) (double) (double) (double) (double)

Average residual error of the optimization.

Possible Predecessors

create_calib_datacreate_calib_dataCreateCalibDatacreate_calib_dataCreateCalibDataCreateCalibData, set_calib_data_cam_paramset_calib_data_cam_paramSetCalibDataCamParamset_calib_data_cam_paramSetCalibDataCamParamSetCalibDataCamParam, set_calib_data_calib_objectset_calib_data_calib_objectSetCalibDataCalibObjectset_calib_data_calib_objectSetCalibDataCalibObjectSetCalibDataCalibObject, set_calib_data_observ_poseset_calib_data_observ_poseSetCalibDataObservPoseset_calib_data_observ_poseSetCalibDataObservPoseSetCalibDataObservPose, find_calib_objectfind_calib_objectFindCalibObjectfind_calib_objectFindCalibObjectFindCalibObject, set_calib_dataset_calib_dataSetCalibDataset_calib_dataSetCalibDataSetCalibData, remove_calib_dataremove_calib_dataRemoveCalibDataremove_calib_dataRemoveCalibDataRemoveCalibData, remove_calib_data_observremove_calib_data_observRemoveCalibDataObservremove_calib_data_observRemoveCalibDataObservRemoveCalibDataObserv

Possible Successors

get_calib_dataget_calib_dataGetCalibDataget_calib_dataGetCalibDataGetCalibData

References

K. Daniilidis: “Hand-Eye Calibration Using Dual Quaternions”; International Journal of Robotics Research, Vol. 18, No. 3, pp. 286-298; 1999.
M. Ulrich, C. Steger: “Hand-Eye Calibration of SCARA Robots Using Dual Quaternions”; Pattern Recognition and Image Analysis, Vol. 26, No. 1, pp. 231-239; January 2016.

Module

Calibration


ClassesClassesClassesClasses | | | | Operators