scene_flow_uncalibscene_flow_uncalibSceneFlowUncalibscene_flow_uncalibSceneFlowUncalibSceneFlowUncalib — Compute the uncalibrated scene flow between two stereo image pairs.
scene_flow_uncalibscene_flow_uncalibSceneFlowUncalibscene_flow_uncalibSceneFlowUncalibSceneFlowUncalib computes the uncalibrated scene flow
between two consecutive rectified stereo image pairs. The scene
flow is the three-dimensional position and motion of surface points
in a dynamic scene. The movement in the images can be caused by
objects that move in the world or by a movement of the camera (or
both) between the acquisition of the two image pairs. To calculate
the calibrated scene flow, scene_flow_calibscene_flow_calibSceneFlowCalibscene_flow_calibSceneFlowCalibSceneFlowCalib can be used.
GenParamNameGenParamNameGenParamNameGenParamNameGenParamNamegenParamName = 'warp_zoom_factor'"warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor" can be
used to specify the resolution ratio between two consecutive
warping levels in the coarse-to-fine warping hierarchy.
'warp_zoom_factor'"warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor" must be selected from the open interval
(0,1). For performance reasons, 'warp_zoom_factor'"warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor" is
typically set to 0.5, i.e., the number of pixels is halved in each
direction for each coarser warping level. Values for
'warp_zoom_factor'"warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor" close to 1 can lead to slightly better
results. However, they require a disproportionately larger
computation time.
GenParamNameGenParamNameGenParamNameGenParamNameGenParamNamegenParamName = 'warp_levels'"warp_levels""warp_levels""warp_levels""warp_levels""warp_levels" can be used
to restrict the warping hierarchy to a maximum number of levels.
For 'warp_levels'"warp_levels""warp_levels""warp_levels""warp_levels""warp_levels" = 0, the largest possible number of
levels is used. If the image size does not allow to use the
specified number of levels (taking the resolution ratio
'warp_zoom_factor'"warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor""warp_zoom_factor" into account), the largest possible
number of levels is used. Usually, 'warp_levels'"warp_levels""warp_levels""warp_levels""warp_levels""warp_levels" should
be set to 0.
GenParamNameGenParamNameGenParamNameGenParamNameGenParamNamegenParamName = 'warp_last_level'"warp_last_level""warp_last_level""warp_last_level""warp_last_level""warp_last_level" can be used
to specify the number of warping levels for which the flow
increment should no longer be computed. Usually,
'warp_last_level'"warp_last_level""warp_last_level""warp_last_level""warp_last_level""warp_last_level" is set to 1 or 2, i.e., a flow increment
is computed for each warping level, or the finest warping level is
skipped in the computation. In the latter case, the computation
is performed on an image of half the resolution of the original
image and then interpolated to the full resolution.
GenParamNameGenParamNameGenParamNameGenParamNameGenParamNamegenParamName = 'outer_iter'"outer_iter""outer_iter""outer_iter""outer_iter""outer_iter" can be used to
specify the number of outer iterations in the minimization scheme.
Typically, the larger 'outer_iter'"outer_iter""outer_iter""outer_iter""outer_iter""outer_iter", the more accurate the
numerical results are. Higher values for this parameter lead to
an increase in the computation time. Typically,
'outer_iter'"outer_iter""outer_iter""outer_iter""outer_iter""outer_iter" is set to values between 5 and 10.
GenParamNameGenParamNameGenParamNameGenParamNameGenParamNamegenParamName = 'inner_iter'"inner_iter""inner_iter""inner_iter""inner_iter""inner_iter" can be used to
specify the number of inner iterations in the minimization scheme.
Typically, the larger 'inner_iter'"inner_iter""inner_iter""inner_iter""inner_iter""inner_iter", the more accurate the
numerical results are. Higher values for this parameter lead to
an increase in the computation time. Usually, two inner
iterations are sufficient.
GenParamNameGenParamNameGenParamNameGenParamNameGenParamNamegenParamName = 'sor_iter'"sor_iter""sor_iter""sor_iter""sor_iter""sor_iter" can be used to
specify the number of SOR iterations for solving the linear system
of equations. Typically, the larger 'sor_iter'"sor_iter""sor_iter""sor_iter""sor_iter""sor_iter", the more
accurate the numerical results are. Higher values for this
parameter lead to an increase in the computation time. Usually,
three SOR iterations are sufficient.
GenParamNameGenParamNameGenParamNameGenParamNameGenParamNamegenParamName = 'omega'"omega""omega""omega""omega""omega" can be used to
specify the relaxation factor of the SOR
method. 'omega'"omega""omega""omega""omega""omega" must be selected from the open interval
(1,2). Typically, 'omega'"omega""omega""omega""omega""omega" is set to 1.9.
Algorithm
The scene flow is estimated by minimizing a suitable energy
functional:
where f=(u,v,dc) is the optical flow field and the disparity
change. denotes the data term and
the smoothness (regularization) term. The
algorithm is based on the following assumptions, which lead to the
data and smoothness terms:
Brightness Constancy
It is assumed that the gray value of a
point remains constant in all four input images, resulting in the
following four constraints:
The solution is
assumed to be piecewise smooth. This smoothness is achieved by
penalizing the first derivatives of the flow . The use of a statistically robust (linear) penalty
function with provides the desired preservation of edges in the movement
in the scene flow to be determined.
Because the disparity image d is given, the first constraint
can be omitted. Taking into account all of the
above assumptions, the energy functional can be written as
To calculate large displacements, coarse-to-fine warping strategies
use two concepts that are closely interlocked: The successive
refinement of the problem (coarse-to-fine) and the successive
compensation of the current image pair by already computed
displacements (warping). Algorithmically, such coarse-to-fine
warping strategies can be described as follows:
First, all images are zoomed down to a very coarse resolution
level.
Then, the scene flow is computed on this coarse resolution.
The scene flow is required on the next resolution level: It is
applied there to the second image pair of the image sequence,
i.e., the problem on the finer resolution level is compensated by
the already computed scene flow. This step is also known as
warping.
The modified problem (difference problem) is now solved on the
finer resolution level, i.e., the scene scene flow is computed
there.
The steps 3-4 are repeated until the finest resolution level
is reached.
The final result is computed by adding up the scene flow from
all resolution levels.
This incremental computation of the scene flow has the following
advantage: While the coarse-to-fine strategy ensures that the
displacements on the finest resolution level are very small, the
warping strategy ensures that the displacements remain small for the
incremental displacements (scene flow of the difference problems).
Since small displacements can be computed much more accurately than
larger displacements, the accuracy of the results typically
increases significantly by using such a coarse-to-fine warping
strategy. However, instead of having to solve a single
correspondence problem, an entire hierarchy of these problems must
be solved.
The minimization of functionals is mathematically very closely
related to the minimization of functions: Like the fact that the
zero crossing of the first derivative is a necessary condition for
the minimum of a function, the fulfillment of the so called
Euler-Lagrange equations is a necessary condition for the minimizing
function of a functional (the minimizing function corresponds to the
desired scene flow in this case). The Euler-Lagrange equations are
partial differential equations. By discretizing these
Euler-Lagrange equations using finite differences, large sparse
nonlinear equation systems have to be solved in this algorithm.
For each warping level a single equation system must be solved. The
algorithm uses an iteration scheme consisting of two nested
iterations (called the outer and inner iteration) and the SOR
(Successive Over-Relaxation) method. The outer loop contains the
linearization of the nonlinear terms resulting from the data
constraints. The nonlinearity of is removed by
the inner fixed point iteration scheme. The resulting linear system
of equations can be solved efficiently by the SOR method.
A. Wedel, C. Rabe, T. Vaudrey, T. Brox, U. Franke and D. Cremers:
“Efficient dense scene flow from sparse or dense stereo data”;
In: Proceedings of the 10th European Conference on Computer Vision:
Part I, pages 739-751. Springer-Verlag, 2008.