HALCON Reference Manual 10.0.2

Table of Contents / 3D Reconstruction / Binocular Stereo

binocular_distanceT_binocular_distancebinocular_distanceBinocularDistanceBinocularDistance (Operator)

Name

binocular_distanceT_binocular_distancebinocular_distanceBinocularDistanceBinocularDistance — Compute the distance values for a rectified stereo image pair using correlation techniques.

Signature

binocular_distance(Image1, Image2 : Distance, Score : CamParamRect1, CamParamRect2, RelPoseRect, Method, MaskWidth, MaskHeight, TextureThresh, MinDisparity, MaxDisparity, NumLevels, ScoreThresh, Filter, SubDistance : )

binocular_distancebinocular_distancebinocular_distanceBinocularDistanceBinocularDistance computes pixel-wise correspondences between two images of a rectified stereo rig using correlation techniques. Different from binocular_disparitybinocular_disparitybinocular_disparityBinocularDisparityBinocularDisparity this operator transforms these pixel correlations into distances of the corresponding 3D world points to the stereo camera system.

The algorithm requires a reference image Image1Image1Image1Image1image1 and a search image Image2Image2Image2Image2image2 which must be rectified, i.e., corresponding epipolar lines are parallel and lie on identical image rows ( r1=r2 ). In case this assumption is violated the images can be rectified by using the operators calibrate_camerascalibrate_camerascalibrate_camerasCalibrateCamerasCalibrateCameras, gen_binocular_rectification_mapgen_binocular_rectification_mapgen_binocular_rectification_mapGenBinocularRectificationMapGenBinocularRectificationMap and map_imagemap_imagemap_imageMapImageMapImage. Hence, given a pixel in the reference image Image1Image1Image1Image1image1 the homologous pixel in Image2Image2Image2Image2image2 is selected by searching along the corresponding row in Image2Image2Image2Image2image2 and matching a local neighborhood within a rectangular window of size MaskWidthMaskWidthMaskWidthMaskWidthmaskWidth and MaskHeightMaskHeightMaskHeightMaskHeightmaskHeight. For each defined reference pixel the pixel correspondences are transformed into distances of the world points defined by the intersection of the lines of sight of a corresponding pixel pair to the z=0 plane of the rectified stereo system. These distances are returned in the single channel image DistanceDistanceDistanceDistancedistance. For this transformation the rectified internal camera parameters CamParamRect1CamParamRect1CamParamRect1CamParamRect1camParamRect1 of the projective camera 1 and CamParamRect2CamParamRect2CamParamRect2CamParamRect2camParamRect2 of the projective camera 2, and the external parameters RelPoseRectRelPoseRectRelPoseRectRelPoseRectrelPoseRect have to be defined. Latter characterizes the relative pose of both cameras to each other and specifies a point transformation from the rectified camera system 2 to the rectified camera system 1. These parameters can be obtained from the operator calibrate_camerascalibrate_camerascalibrate_camerasCalibrateCamerasCalibrateCameras and gen_binocular_rectification_mapgen_binocular_rectification_mapgen_binocular_rectification_mapGenBinocularRectificationMapGenBinocularRectificationMap. After all, a quality measure for each distance value is returned in ScoreScoreScoreScorescore, containing the best result of the matching function S of a reference pixel. For the matching, the gray values of the original unprocessed images are used.

The used matching function is defined by the parameter MethodMethodMethodMethodmethod allocating three different kinds of correlation:

'sad'"sad""sad""sad""sad": Summed Absolute Differences

       S(r1,c1,d) := SUM(|g1(r1',c1')-g2(r1',c1'+d)|), 0=<=S<=255,

'ssd'"ssd""ssd""ssd""ssd": Summed Squared Differences

       S(r1,c1,d) := SUM((g1(r1',c1')-g2(r1',c1'+d))²), 0<=S<=65025,

'nnc'"nnc""nnc""nnc""nnc": Normalized Cross Correlation

       S(r1,c1,d) :=
         SUM(g1(r1',c1')-gm1(r1,c1))*(g2(r1',c1'+d)-gm2(r1,c1+d)))/
         SQRT(SUM((g1(r1',c1')-gm1(r1,c1))²)*
              SUM((g2(r1',c1+d)-gm2(r1,c1+d))²)),
       -1.0<=S<=1.0,

with

r1, c1, r2, c2: row and column coordinates of the corresponding pixels of the two input images,

g1, g2: gray values of the unprocessed input images,

N=(2m+1)(2n+1): size of correlation window

SUM(x): mean value within the correlation window of width 2m+1 and height 2n+1.

Note that the methods 'sad'"sad""sad""sad""sad" and 'ssd'"ssd""ssd""ssd""ssd" compare the gray values of the pixels within a mask window directly, whereas 'ncc'"ncc""ncc""ncc""ncc" compensates for the mean gray value and its variance within the mask window. Therefore, if the two images differ in brightness and contrast, this method should be preferred. For images with similar brightness and contrast 'sad'"sad""sad""sad""sad" and 'ssd'"ssd""ssd""ssd""ssd" are to be preferred as they are faster because of less complex internal computations.

It should be noted that the quality of correlation for rising S is falling in methods 'sad'"sad""sad""sad""sad" and 'ssd'"ssd""ssd""ssd""ssd" (the best quality value is 0) but rising in method 'ncc'"ncc""ncc""ncc""ncc" (the best quality value is 1.0).

The size of the correlation window has to be odd numbered and is passed in MaskWidthMaskWidthMaskWidthMaskWidthmaskWidth and MaskHeightMaskHeightMaskHeightMaskHeightmaskHeight. The search space is confined by the minimum and maximum disparity value MinDisparityMinDisparityMinDisparityMinDisparityminDisparity and MaxDisparityMaxDisparityMaxDisparityMaxDisparitymaxDisparity. Due to pixel values not defined beyond the image border the resulting domain of DistanceDistanceDistanceDistancedistance and ScoreScoreScoreScorescore is generally not set along the image border within a margin of height MaskHeightMaskHeightMaskHeightMaskHeightmaskHeight/2 at the top and bottom border and of width MaskWidthMaskWidthMaskWidthMaskWidthmaskWidth/2 at the left and right border. For the same reason, the maximum disparity range is reduced at the left and right image border.

Since matching turns out to be highly unreliable when dealing with poorly textured areas, the minimum variance within the correlation window can be defined in TextureThreshTextureThreshTextureThreshTextureThreshtextureThresh. This threshold is applied on both input images Image1Image1Image1Image1image1 and Image2Image2Image2Image2image2. In addition, ScoreThreshScoreThreshScoreThreshScoreThreshscoreThresh guarantees the matching quality and defines the maximum ('sad'"sad""sad""sad""sad",'ssd'"ssd""ssd""ssd""ssd") or, respectively, minimum ('ncc'"ncc""ncc""ncc""ncc") score value of the correlation function. Setting FilterFilterFilterFilterfilter to 'left_right_check'"left_right_check""left_right_check""left_right_check""left_right_check", moreover, increases the robustness of the returned matches, as the result relies on a concurrent direct and reverse match, whereas 'none'"none""none""none""none" switches it off.

The number of pyramid levels used to improve the time response of binocular_distancebinocular_distancebinocular_distanceBinocularDistanceBinocularDistance is determined by NumLevelsNumLevelsNumLevelsNumLevelsnumLevels. Following a coarse-to-fine scheme disparity images of higher levels are computed and segmentated into rectangular subimages to reduce the disparity range on the next lower pyramid level. TextureThreshTextureThreshTextureThreshTextureThreshtextureThresh and ScoreThreshScoreThreshScoreThreshScoreThreshscoreThresh are applied on every level and the returned domain of the DistanceDistanceDistanceDistancedistance and ScoreScoreScoreScorescore images arises from the intersection of the resulting domains of every single level. Generally, pyramid structures are the more advantageous the more the distance image can be segmented into regions of homogeneous distance values and the bigger the disparity range must be specified. As a drawback, coarse pyramid levels might loose important texture information which can result in deficient distance values.

Finally, the value 'interpolation'"interpolation""interpolation""interpolation""interpolation" for parameter SubDistanceSubDistanceSubDistanceSubDistancesubDistance increases the refinement and accuracy of the distance values. It is switched off by setting the parameter to 'none'"none""none""none""none".

Parallelization

Multithreading type: reentrant (runs in parallel with non-exclusive operators).
Multithreading scope: global (may be called from any thread).
Automatically parallelized on domain level.

Parameters

Image1Image1Image1Image1image1 (input_object) (multichannel-)image → object (byte)

Rectified image of camera 1.

Image2Image2Image2Image2image2 (input_object) (multichannel-)image → object (byte)

Rectified image of camera 2.

DistanceDistanceDistanceDistancedistance (output_object) singlechannelimage → object (real)

Distance image.

ScoreScoreScoreScorescore (output_object) singlechannelimage → object (real)

Evaluation of a distance value.

CamParamRect1CamParamRect1CamParamRect1CamParamRect1camParamRect1 (input_control) number-array → (real / integer)

Internal camera parameters of the rectified camera 1.

Number of elements: (CamParamRect1 == 8) || (CamParamRect1 == 12)

CamParamRect2CamParamRect2CamParamRect2CamParamRect2camParamRect2 (input_control) number-array → (real / integer)

Internal camera parameters of the rectified camera 2.

Number of elements: (CamParamRect2 == 8) || (CamParamRect2 == 12)

RelPoseRectRelPoseRectRelPoseRectRelPoseRectrelPoseRect (input_control) pose-array → (real / integer)

Point transformation from rectified camera 2 to rectified camera 1.

Number of elements: 7

MethodMethodMethodMethodmethod (input_control) string → (string)

Matching function.

Default value: 'ncc' "ncc" "ncc" "ncc" "ncc"

List of values: 'sad'"sad""sad""sad""sad", 'ssd'"ssd""ssd""ssd""ssd", 'ncc'"ncc""ncc""ncc""ncc"

MaskWidthMaskWidthMaskWidthMaskWidthmaskWidth (input_control) integer → (integer)

Width of the correlation window.

Default value: 11

Suggested values: 5, 7, 9, 11, 21

Restriction: (3 <= MaskWidth) && odd(MaskWidth)

MaskHeightMaskHeightMaskHeightMaskHeightmaskHeight (input_control) integer → (integer)

Height of the correlation window.

Default value: 11

Suggested values: 5, 7, 9, 11, 21

Restriction: (3 <= MaskHeight) && odd(MaskHeight)

TextureThreshTextureThreshTextureThreshTextureThreshtextureThresh (input_control) real → (real / integer)

Variance threshold of textured image regions.

Default value: 0.0

Suggested values: 0.0, 2.0, 5.0, 10.0

Restriction: 0.0 <= TextureThresh

MinDisparityMinDisparityMinDisparityMinDisparityminDisparity (input_control) integer → (integer)

Minimum of the expected disparities.

Default value: 0

Typical range of values: -32768 ≤ MinDisparity MinDisparity MinDisparity MinDisparity minDisparity ≤ 32767

MaxDisparityMaxDisparityMaxDisparityMaxDisparitymaxDisparity (input_control) integer → (integer)

Maximum of the expected disparities.

Default value: 30

Typical range of values: -32768 ≤ MaxDisparity MaxDisparity MaxDisparity MaxDisparity maxDisparity ≤ 32767

NumLevelsNumLevelsNumLevelsNumLevelsnumLevels (input_control) integer → (integer)

Number of pyramid levels.

Default value: 1

List of values: 1, 2, 3, 4

Restriction: 1 <= NumLevels

ScoreThreshScoreThreshScoreThreshScoreThreshscoreThresh (input_control) real → (real / integer)

Threshold of the correlation function.

Default value: 0.0

List of values: 0.0, 2.0, 5.0, 10.0

FilterFilterFilterFilterfilter (input_control) string(-array) → (string)

Downstream filters.

Default value: 'none' "none" "none" "none" "none"

List of values: 'none'"none""none""none""none", 'left_right_check'"left_right_check""left_right_check""left_right_check""left_right_check"

SubDistanceSubDistanceSubDistanceSubDistancesubDistance (input_control) string(-array) → (string)

Distance interpolation.

Default value: 'none' "none" "none" "none" "none"

List of values: 'none'"none""none""none""none", 'interpolation'"interpolation""interpolation""interpolation""interpolation"

Example (HDevelop)

* read the internal and external stereo parameters
read_cam_par ('cam_left.dat', CamParam1)
read_cam_par ('cam_right.dat', CamParam2)
read_pose ('relpose.dat', RelPose)
* compute the mapping for rectified images
gen_binocular_rectification_map (Map1, Map2, CamParam1, CamParam2, \
                                 RelPose, 1, 'geometric', 'bilinear', \
                                 CamParamRect1, CamParamRect2, \
                                 Cam1PoseRect1, Cam2PoseRect2, RelPoseRect)
* compute the distance values in online images
while (1)
  grab_image_async (Image1, AcqHandle1, -1)
  map_image (Image1, Map1, ImageMapped1)

  grab_image_async (Image2, AcqHandle2, -1)
  map_image (Image2, Map2, ImageMapped2)

  binocular_distance (ImageMapped1, ImageMapped2, Distance, Score, \
                      CamParamRect1, CamParam2, RelPoseRect, 'sad', \
                      11, 11, 20, -40, 20, 2, 25, \
                      'left_right_check', 'interpolation')
endwhile

Example (HDevelop)

* read the internal and external stereo parameters
read_cam_par ('cam_left.dat', CamParam1)
read_cam_par ('cam_right.dat', CamParam2)
read_pose ('relpose.dat', RelPose)
* compute the mapping for rectified images
gen_binocular_rectification_map (Map1, Map2, CamParam1, CamParam2, \
                                 RelPose, 1, 'geometric', 'bilinear', \
                                 CamParamRect1, CamParamRect2, \
                                 Cam1PoseRect1, Cam2PoseRect2, RelPoseRect)
* compute the distance values in online images
while (1)
  grab_image_async (Image1, AcqHandle1, -1)
  map_image (Image1, Map1, ImageMapped1)

  grab_image_async (Image2, AcqHandle2, -1)
  map_image (Image2, Map2, ImageMapped2)

  binocular_distance (ImageMapped1, ImageMapped2, Distance, Score, \
                      CamParamRect1, CamParam2, RelPoseRect, 'sad', \
                      11, 11, 20, -40, 20, 2, 25, \
                      'left_right_check', 'interpolation')
endwhile

Example (C++)

// read the internal and external stereo parameters
read_cam_par("cam_left.dat",CamParam1);
read_cam_par("cam_right.dat",CamParam2);
read_pose("relpose.dat",RelPose);
// compute the mapping for rectified images
gen_binocular_rectification_map(&Map1,&Map2,CamParam1,CamParam2,RelPose,1,
                                "geometric","bilinear",&CamParamRect1,
                                &CamParamRect2,&CamPoseRect1,&CamPoseRect2,
                                &RelPoseRect);
// compute the distance values in online images
while (1)
{
  grab_image_async(&Image1,AcqHandle1,-1);
  map_image(Image1,Map1,&ImageMapped1);

  grab_image_async(&Image2,AcqHandle2,-1);
  map_image(Image2,Map2,&ImageMapped2);

  binocular_distance(ImageMapped1,ImageMapped2,&Distance,&Score,
                     CamParamRect1,CamParamRect2,RelPose,"sad",11,11,
                     20,-40,20,2,25,"left_right_check","interpolation");
}

Example (HDevelop)

* read the internal and external stereo parameters
read_cam_par ('cam_left.dat', CamParam1)
read_cam_par ('cam_right.dat', CamParam2)
read_pose ('relpose.dat', RelPose)
* compute the mapping for rectified images
gen_binocular_rectification_map (Map1, Map2, CamParam1, CamParam2, \
                                 RelPose, 1, 'geometric', 'bilinear', \
                                 CamParamRect1, CamParamRect2, \
                                 Cam1PoseRect1, Cam2PoseRect2, RelPoseRect)
* compute the distance values in online images
while (1)
  grab_image_async (Image1, AcqHandle1, -1)
  map_image (Image1, Map1, ImageMapped1)

  grab_image_async (Image2, AcqHandle2, -1)
  map_image (Image2, Map2, ImageMapped2)

  binocular_distance (ImageMapped1, ImageMapped2, Distance, Score, \
                      CamParamRect1, CamParam2, RelPoseRect, 'sad', \
                      11, 11, 20, -40, 20, 2, 25, \
                      'left_right_check', 'interpolation')
endwhile

Example (HDevelop)

* read the internal and external stereo parameters
read_cam_par ('cam_left.dat', CamParam1)
read_cam_par ('cam_right.dat', CamParam2)
read_pose ('relpose.dat', RelPose)
* compute the mapping for rectified images
gen_binocular_rectification_map (Map1, Map2, CamParam1, CamParam2, \
                                 RelPose, 1, 'geometric', 'bilinear', \
                                 CamParamRect1, CamParamRect2, \
                                 Cam1PoseRect1, Cam2PoseRect2, RelPoseRect)
* compute the distance values in online images
while (1)
  grab_image_async (Image1, AcqHandle1, -1)
  map_image (Image1, Map1, ImageMapped1)

  grab_image_async (Image2, AcqHandle2, -1)
  map_image (Image2, Map2, ImageMapped2)

  binocular_distance (ImageMapped1, ImageMapped2, Distance, Score, \
                      CamParamRect1, CamParam2, RelPoseRect, 'sad', \
                      11, 11, 20, -40, 20, 2, 25, \
                      'left_right_check', 'interpolation')
endwhile