| Operators |
reconstruct_surface_stereo — Reconstruct surface from calibrated multi-view stereo images.
reconstruct_surface_stereo(Images : : StereoModelID : ObjectModel3D)
The operator reconstruct_surface_stereo reconstructs a surface from multiple images Images, taken with a calibrated multi-view setup associated to a stereo model StereoModelID. The reconstructed surface is stored in a HALCON 3D object model and a handle to it is returned in ObjectModel3D.
The underlying surface reconstruction algorithm is based on fusing the information obtained from multiple binocular stereo depth maps. For this, the algorithm internally calls a HALCON disparity operator (currently binocular_disparity) for each image pair from a predefined list of pairs. The disparity information is converted to X, Y, and Z coordinate images in the coordinate system of the from-camera. For this, reconstruct_surface_stereo internally calls disparity_image_to_xyz. Then, for each 3D point, the normal vector is calculated. This is done by fitting a plane through the neighboring 3D points. In contrast to surface_normals_object_model_3d, the neighboring points are not determined in 3D but simply in 2D by using the neighboring points in the X, Y, and Z images. The normal vector of each 3D point is then set to the normal vector of the respective plane. In the next step, the X, Y, and Z images as well as the normal vectors are transformed into the common coordinate system that is specified in the camera setup model associated to the stereo model StereoModelID. Finally, the transformed coordinate images are sub-sampled and stored in a common point cloud together with the points and normals extracted from other image pairs. Thus, the object model returned in ObjectModel3D contains the coordinates and a normal for each reconstructed point.
The so-obtained point cloud can be additionally meshed in a post-processing step. The parameter 'point_meshing' enables it. If the meshing is enabled, the object model returned in ObjectModel3D contains the description of the mesh. Currently, the supported meshing algorithm is based on a Poisson solver (see below). It creates a water-tight mesh, therefore surface regions with missing data are covered by an interpolated mesh.
If only one image pair is processed and no point meshing is enabled, reconstruct_surface_stereo stores a 'xyz_mapping' attribute in ObjectModel3D, which reveals the mapping of the reconstructed 3D points to coordinates of the first image of the pair. This attribute is required by operators like segment_object_model_3d or object_model_3d_to_xyz (with Type='from_xyz_map'). In contrast to the single pair case, if two or more image pairs are processed, reconstruct_surface_stereo does not store the 'xyz_mapping' attribute, since single reconstructed points would originate from different image pairs. The presence of the attribute in the output object model can be verified by calling get_object_model_3d_params with ParamName='has_xyz_mapping'.
A camera setup model is associated with the stereo model StereoModelID upon its creation with create_stereo_model. The camera setup must contain calibrated information about the cameras, with which the images in the image array Images were acquired: the I-th image from the array corresponds to the camera with index I-1 from the camera setup; the number of images in the array must be the same as the number of cameras in the camera setup.
A well calibrated camera setup is the main requirement for a precise surface reconstruction, therefore special attention should be paid to obtaining a precise calibration of the cameras in the multi-view stereo setup used. HALCON provides calibration of a multi-view setup with the operator calibrate_cameras. The resulting calibrated camera setup can be accessed with a successive call to get_calib_data. Alternatively, for camera setups with known parameters a calibrated camera setup can be created with create_camera_setup_model.
A bounding box is associated with the stereo model StereoModelID and for the surface stereo reconstruction it is required that the bounding box is valid (see set_stereo_model_param for further details). The reconstruction algorithm needs the bounding box for two reasons. First, it uses the projection of the bounding box into both images of each image pair in order to estimate the values for MinDisparity and MaxDisparity, which in turn are used in the internal call to binocular_disparity. Secondly, the surface fragments lying outside the bounding box are clipped and are not returned in ObjectModel3D. A too large bounding box results in a large difference between MinDisparity and MaxDisparity and this usually slows down the execution of binocular_disparity and reconstruct_surface_stereo, respectively. A too small bounding box might result in clipping valid surface areas.
The objects in the scene should expose certain surface properties in order to make the scene suitable for the dense surface reconstruction. First, the surface reflectance should exhibit Lambertian properties as closely as possible (i.e., light falling on the surface is scattered such that its apparent brightness is the same regardless of the angle of view). Secondly, the surface should exhibit enough texture, but no repeating patterns.
The proper selection of image pairs (see set_stereo_model_image_pairs) has a important role for the general quality of the surface reconstruction. On the one hand, camera pairs with small base line (small distance between the camera centers) are better suited for the binocular stereo disparity algorithms. Hence, pairs with small base lines should be preferred for acquiring accurate depth information of the scene. On the other hand, the pairs should provide different points of view, such that if one pair does not see a certain area of the surface, it is covered by another pair. Please note that the number of pairs linearly affects the runtime of the operator. Therefore use "as many as needed and just as few as possible" image pairs in order to handle the trade-off between completeness of the surface reconstruction and reconstruction runtime.
Finally, Images should represent a static scene or they must be taken simultaneously, otherwise the reconstruction of the surface might be impossible.
A summary of the preparation of a stereo model for surface reconstruction:
Obtain calibrated camera setup model (use calibrate_cameras or create_camera_setup_model and configure it)
Create a stereo model with create_stereo_model by selecting Method='surface_pairwise'
Configure the bounding box for the system with set_stereo_model_param (parameter ParamName='bounding_box')
(optional) Configure further stereo model parameters with set_stereo_model_param
Configure the image pairs with set_stereo_model_image_pairs
Acquire images with the calibrated cameras setup and collect them in an image array Images
Perform surface reconstruction with reconstruct_surface_stereo
(optional) Steps 6. and 7. can be repeated for further scenes
The proper configuration of a stereo model is not always easy. If the reconstruction results are not satisfactory, please consult the following hints and ideas:
If you enable the 'persistence' mode of stereo model (call set_stereo_model_param with ParamName='persistence') a successive call to reconstruct_surface_stereo will store intermediate iconic results, which provide additional information.
If the camera setup was obtained by calibrate_cameras it stores some quality information about the camera calibration in form of standard deviations of the camera internal parameters. This information is then carried in the camera setup model associated to the stereo model. It can be queried by first calling get_stereo_model_param with ParamName='camera_setup_model' and then inspecting the camera parameter standard deviations by calling get_camera_setup_param with ParamName='params_deviations'. Unusually big standard deviation values might indicate a bad camera calibration.
After setting the stereo model 'persistence' mode (see above), we recommend to inspect the rectified images for each image pair. The rectified images are returned by get_stereo_model_object with a camera index pair [From, To] specifying the pair of interest in the parameter PairIndex and the values 'from_image_rect' and 'to_image_rect' in ObjectName, respectively. If the images are properly rectified, all corresponding image features must appear in the same row in both rectified images. A discrepancy of several rows is a serious indication for a bad camera calibration.
After setting the stereo model 'persistence' mode (see above), inspect the disparity and the score images for each image pair. They are returned by get_stereo_model_object with a camera index pair [From, To] specifying the pair of interest in the parameter PairIndex and the values 'disparity_image' and 'score_image' in ObjectName, respectively. If both images exhibit signifiant inperfection (e.g., the disparity image does not really resemble the shape of the object seen in the image), try to adjust the parameters used for the internal call to binocular_disparity (these are all parameters with a 'binocular_' prefix, which can be modified by set_stereo_model_param) until some improvement is achieved. In some cases, it might be easier to find the proper parameters by testing the affected image pair with binocular_disparity directly.
After setting the stereo model 'persistence' mode (see above), call get_stereo_model_param with ParamName='min_disparity' and ParamName='max_disparity', respectively. The returned tuples have the same order as the image pairs returned by get_stereo_model_image_pairs. Select the minimum and maximum disparity values corresponding to the image pair of interest and check if their difference is unusually large. Compare them with the parameters MinDisparity and MaxDisparity that return good results by using the affected image pair directly in binocular_disparity. The internal computation of those values can be corrected by adjusting the values for the bounding box of the stereo model (see set_stereo_model_param)
As already mentioned above, there is a trade-off between completeness of the surface reconstruction on the one hand and reconstruction runtime on the other. In order to improve this trade off, consider the following hints:
The stereo model parameter 'rectif_sub_sampling' (see set_stereo_model_param) controls the sub-sampling in the rectification step. Setting this factor to a value > 1.0 will reduce the resolution of the rectified images compared to the original images. This factor has a direct impact on the succeeding binocular_disparity performance, but it causes loss of image detail. The parameter 'rectif_interpolation' could have also some impact, but typically not a significant one.
Special attention should be paid to the following disparity parameters used for the internal calls to binocular_disparity :
A too large difference between MinDisparity and MaxDisparity has direct impact on the performance of binocular_disparity. Check their values computed internally by reconstruct_surface_stereo (see above) and try to adapt the stereo model bounding box if needed.
Each of these parameters of binocular_disparity have a corresponding stereo model parameter with the prefix 'binocular_', and have, some more or others less impact on the performance. Adapting them properly could improve the performance.
You can use a larger sub-sampling step for the X,Y,Z data in the last step of the reconstruction algorithm by modifying ParamName='sub_sampling_step' with set_stereo_model_param. Increasing this step could have moderate performance improvement. However, the reconstructed data will be much sparser.
Note that if a 3D object model is no longer needed or should be overwritten, the memory has to be freed first by calling the operator clear_object_model_3d.
An image array acquired by the camera setup associated to the stereo model.
Handle of the stereo model.
Handle to the resulting surface.
create_stereo_model, get_calib_data, set_stereo_model_image_pairs
M. Kazhdan, M. Bolitho, and H. Hoppe: “Poisson Surface Reconstruction.” Symposium on Geometry Processing (June 2006).
3D Metrology
| Operators |