Robust Detection of Point Correspondences in Stereo Images

. A major challenge in 3D reconstruction is the computation of the fundamental matrix. Automatic computation from uncalibrated image pairs is performed from point correspondences. Due to imprecision and wrong correspondences, only an approximation of the true fundamental matrix can be computed. The quality of the fundamental matrix strongly depends on the location and number of point correspondences. Furthermore, the fundamental matrix is the only geometric constraint between two uncalibrated views, and hence it can be used for the detection of wrong point correspondences. This property is used by current algorithms like RANSAC, which computes the fundamental matrix from a restricted set of point correspondences. In most cases, not only wrong correspondences are disregarded, but also correct ones, which is due to the criterion used to eliminate outliers. In this context, a new criterion preserving a maximum of correct correspondences would be useful. In this paper we introduce a novel criterion for out-lier elimination based on a probabilistic approach. The enhanced set of correspondences may be important for further computation towards a 3D reconstruction of the scene.


Introduction
Modern point matching algorithms like SIFT (see [4], [5]), provide a large number of point matches, even in stereo image pairs with large changes in scale, translation and rotation between the images.The only inconvenience is that some wrong matches are among the correspondences, especially when parts of the first image are not visible in the second.
Algorithms for computing the fundamental matrix can be classified into algorithms sensitive to wrong matches and algorithms detecting and ignoring so-called outliers like RANSAC or Least Median of Squares.In [3] and [7] algorithms of both categories are described and compared.Basically, algorithms of the first category should only be used after preliminary outlier removal.The motivation to use them is that some of them perform better than RANSAC or LMedS (see [3]).Based on [3], we used RANSAC combined with the normalized 8-point algorithm to compute a first estimation of the fundamental matrix in order to perform outlier removal.
An observation is, that using the criterion presented in [7], i.e. the distance of a point to its corresponding epipolar line, some true correspondences, which may be important for an estimation of the epipolar geometry, are also eliminated.As a consequence we developed a new criterion to evaluate correspondences between two images, considering the estimation of the fundamental matrix and its uncertainty.
The paper is structured as follows: In section two, the computation of the covariance matrix of the fundamental matrix using Monte Carlo Simulation is described.Further we describe the computation leading to the new criterion.In section three, we show how the criterion can be used in an effective way for outlier removal and compare our results with those using a conventional criterion.In section four a brief conclusion is given.

Confidence measure for point correspondences
To evaluate the quality of a single point correspondence from a set of point correspondences, we use the only available geometric constraint, the epipolar geometry and its algebraic representation, the fundamental matrix.The computation of the fundamental matrix using the Random Sample Consensus combined with the normalized 8-point algorithm is described in [3] and [7].As the locations of point correspondences are superimposed with noise, the computation of the covariance matrix of the fundamental matrix provides further useful information.

Computing the covariance matrix of the fundamental matrix
To compute the covariance matrix, we assume that the noise on the correspondence locations has normal distribution.We simulate this effect by adding Gaussian noise to the eight correspondences selected by RANSAC during the fundamental matrix computation.We obtain a set of different versions of the same eight point correspondences, and compute the fundamental matrix from each version using the normalized 8-point algorithm.
A large number of different fundamental matrices is obtained, showing how the epipolar geometry varies under slight changes in the point correspondence locations.A good description of this effect is given by the covariance matrix of the fundamental matrix, denoted by Σ F .

Epipolar line and epipolar envelope
For a given point x in the first image, the corresponding epipolar line l in homogeneous coordinates is given by where F is the fundamental matrix.Due to the uncertainty of the fundamental matrix, the point correspondence is more likely to be located within an area around the epipolar line.
From the covariance matrix of the fundamental matrix, we can compute the covariance matrix of the epipolar line with where J is the Jacobian of the mapping The Mahalanobis distance k of a possible epipolar line m from the estimated epipolar line l is then given by Thus we have an approximation of how well a certain epipolar line m matches with the knowledge acquired so far about the epipolar geometry and its uncertainty.For a given value of k 2 , we can compute an envelope of epipolar lines containing all possible epipolar lines having a value less or equal to k 2 .With the assumption that the elements of l have normal distribution, k 2 has a cumulative χ 2  2 distribution and a probability that the true epipolar line is within this envelope can be associated to the k 2 -value, as can be seen in figure 1.The region can be described by a conic C defined in homogeneous coordinates by We see that for this value the conic is a hyperbola with two branches.
In [3] the epipolar envelope is used for guided matching, i.e. searching correspondences within the epipolar band after a first estimation of the fundamental matrix.We observed that if the fundamental matrix is computed from correspondences located only in the foreground, the uncertainty in the background becomes very high (see figure 2).This is the reason why correct matches lying in the corners of the image are often eliminated using a conventional criterion.To prevent this, we have to develop a new criterion being less stringent for matches in regions of high uncertainty.

Minimal Mahalanobis distance for a point correspondence Basic idea
The basic idea for the new criterion is to invert the problem of finding an epipolar band for a given likelihood (i.e. a given k 2 respectively probability α).For a given point x in the second view corresponding to a point x in the first view, we want to find the conic with minimal k 2 comprising the point x .In other terms, we are retrieving the value k 2 in equation ( 4), that provides a hyperbola passing through x .
The considerations behind this idea are very simpleif we know that the epipolar line of a point x in the first image will lie with a certain probability within an epipolar band in the second image, the corresponding point x , which is located somewhere on the epipolar line, will be within the epipolar band with the same probability.

General Problem Statement
The point x belongs to the conic C if the following equation is verified where C is given by: F −1 2 (k 2 ) = α is the probability to find x within C.
We use again the notation m for an epipolar line in the second image.In fact l is here the estimated epipolar line corresponding to the point x in the first image.
It is not possible to retrieve directly the corresponding value k 2 from a point x using the equations above.One possibility would consist in computing a multitude of different conics using a range of different values for k 2 , and localize x in between the conics.An approximation for k 2 would be obtained by interpolation, but we found a closed-form solution to the problem, providing an exact result.

Closed-form solution
Let us assume that the confidence, that any point x in the second image, is corresponding to a point x in the first view is in relation with the probability of the epipolar geometry that would explain the correspondence pair x ↔ x and having at the same time the highest probability.In other terms, for a potential point correspondence x ↔ x , we retrieve the epipolar line m passing through x having maximal probability, i.e. minimal k 2 regarding equation 14.This assumption differs from the assumption made in [1], and we obtain a different criterion, which is more suitable for our purpose.
If we denote with m the unknown epipolar line and with l the estimated line in the second image, the constrained minimization problem can be stated as follows: For g(a, b, c) we have the following derivatives: Thus we have As we are solving a problem with three unknown variables, 3 equations are sufficient.Using the relations from equation (12) and the constraint that the point is on the line m, we obtain the set of equations: where This set of equations can be easily computed and the solution vector (a, b, c) T is the vector m passing through x and having minimal k 2 in terms of equation 7. Finally we obtain coordinates of the line m passing through x and the corresponding value k 2 , that would produce an epipolar band delimited by a hyperbola passing through x .The value k 2 will be of special interest in applications, as we will use it as a quality criterion for point correspondences in our applications.

Example in image pair
To summarize the algorithm, we can explain the different steps using an example in an image pair (see figure 3).We start after the fundamental matrix and its covariance have been computed for the given image pair.In red we see the estimated epipolar line in the second image, which we denoted by m, that corresponds to the point x in the first image which is marked by a green point.Further we computed the epipolar band (delimited by the green hyperbola) for a value k 2 = 5.9915, respectively a probability α = 0.95 that the true epipolar line will be located within this region.
Further, for points marked by a blue circle in the second image, we computed the line passing through the point and having a minimal k 2 in terms of equation 7. The lines are blue and represent the lines before denoted by m.
A further question to answer is the value k 2 of the chosen line m.It can be easily computed using In figure 3  6.07 for the point in the right side of the image.The fact that this value is almost equal to 5.991 and the corresponding line m is tangent to the hyperbola is an interesting observation and will be discussed later.

Properties
We want to briefly show some special cases of lines with minimal k 2 .One of the reasons why we cannot use the criterion from [1] is because it also contains depth information coming from the chosen point correspondences to compute the fundamental matrix and its covariance.In practice, this becomes visible in the property that points lying on the estimated epipolar line (which is of course the most probable epipolar line) do not have the same probability.In figure 4 we see that the line m is identical to the estimated epipolar line l.Thus we have k 2 = 0 for each point on the estimated line.Another effect is shown in figure 5. We computed the line m for a point on the hyperbola delimiting the epipolar band.We see that the line in blue we obtain is tangent to the hyperbola and the value for k 2 is equal to the value we used for k 2 to compute the hyperbola.In other terms, if we want to retrieve the value k 2 for a hyperbola passing through the point x , it is sufficient to compute the line with minimal k 2 as the k 2 obtained for the line will be equal to the value obtained if we retrieved the hyperbola.

Results
We started from the situation that for a point x in a first image a region in the second image, delimited by a hyperbola, was defined.We knew from the epipolar geometry and its uncertainty that its correspondence x would be located within this region with a given probability α, which was derived from the Mahalanobis distance k.
The result is that we can now compute for a pair of corresponding points x ↔ x the corresponding hyperbola, passing through x , and the value k 2 corresponding to the hyperbola.In the old application the probability to find the corresponding point x within the hyperbola was increasing with higher values for k 2 , (the region bounded by the hyperbola increases with higher values of k 2 ), in our application a low value for k 2 is indicating that the probability that x ↔ x are corresponding points is high and vice versa.
The way to find the value for k 2 , which is actually the Mahalanobis distance k between the estimated epipolar line and the line passing through x with minimal k, is of no relevance in applications.This means that in our applications, for a point x , we will only be interested in the Mahalanobis distance of the corresponding line m, not in the line itself.
For the purpose of illustration we generated figure 6, where for each pixel x we computed the Mahalanobis distance k 2 according to the minimization problem from equation 7 and represented it using grayscales.In particular, points in white have values close to k 2 = 0, points in black, a value close to the maximum value on the image.We used to whole range of k 2 values on the second image and divided the range in 256 levels of equal width, each one represented by its relative grayscale level.

Application to outlier removal
The initial aim was to develop a new criterion for robust outlier elimination.Existing algorithms use criteria based on the distance of a point to its corresponding epipolar line.For distance based algorithms, in the background, the distance to the epipolar line will increase as the estimation of the fundamental matrix will be far better for points in the foreground.Using a threshold distance to eliminate outliers will lead to an elimination of points further away from the central points, as we can observe in figure 8 (b).
Similar to other applications, we could consider using the epipolar band to remove outliers.This means that instead of computing the epipolar line and the distance of a point to the epipolar line, we just compute the epipolar band, and remove points located outside the epipolar band.This would resolve the problem of removing points from the background.Although this application seems suitable, two parameters have to be set.First we have to make an assumption about the variance of the point correspondences, and the value for k 2 from where we compute the epipolar band needs to be set.In the following we present an algorithm overcoming the drawbacks of the algorithms mentioned above by using our new criterion.

Algorithm
The values chosen for the standard deviation σ of points to compute the covariance matrix of the fundamental matrix are varying, and thus we first computed the value k 2 of each used point correspondence.If we consider the distribution of the value k 2 for each point in figure 7, we see that a majority of the correspondences is very small, whereas some correspondences have huge k 2 values.A classification of inliers and outliers can be easily performed.
Our proposition is to compute the median value of k 2 .As the RANSAC algorithm does not perform well in the case of more than 50 percent outliers, we only consider the case of having at least 50 percent inlier.As a consequence the median value for k 2 will be the value of an inlier.An observation was that the values for inlier were very similar, whereas the values for oultiers were huge.So in practice we eliminate every point correspondence with a value of more than ten times the median value.Another proportion may be used but in our tests we obtained reasonably good results.

Results
The results presented were obtained using RANSAC for an estimation of the fundamental matrix.Further the covariance matrix of the fundamental matrix was computed by adding noise to the eight point correspondences used to compute the fundamental matrix.In fact we obtained the same results for the outlier elimination using a standard deviation σ in a range of 0.02 to 10 pixels.Therefore no knowledge about the quality of the image and the variance of the correspondences is needed for the outlier removal with the new criterion.The results are presented in figure 8.

Discussion
We have seen that the new criterion can be used for outlier removal without side information about the image quality and brings the advantage of providing a larger set of point correspondences.An observation is that the points preserved by the new algorithm are wider spread throughout the image than the points obtained with a conventional criterion.Hence the initial fundamental matrix estimation rather seems to suit point correspondences in the center of the image.
Our actual research is focused on comparing fundamental matrices computed from the initial set of point correspondences to the ones obtained with the enhanced set, using image pairs with known epipolar geometry, i.e. a comparison to ground truth.Furthermore we try to identify different planes spanned in space by the point correspondences, to examine the influence of depth distribution of the correspondences onto the accuracy of the computed fundamental matrix.

Fig. 1 .
Fig. 1.Envelope of epipolar lines: We computed the epipolar line (in red) and the epipolar band (in green) in the second image.The point in the first image we used for the computation is marked by a green point.The epipolar band shown is representing the envelope of epipolar lines with equal k 2 = 5.9915.The according probability that the true epipolar line lies within this band is α = 0.95.We see that for this value the conic is a hyperbola with two branches.

Fig. 2 .
Fig. 2. High insecurity for points in the background: In this fig-ure the effect of choosing points only in the foreground is demonstrated.We see that the insecurity for points in the background is much higher than for points in the foreground, as the epipolar band is much more extended.INRIASyntim owns the copyright of the image.

Fig. 3 .
Fig. 3. Example of an image pair with points and their respective line with minimal k 2 :In red we see the estimated epipolar line in the second image, l, that corresponds to the point x in the first image which is marked by a green point.The epipolar band for a value k 2 = 5.9915, respectively a probability α = 0.95 is within the green hyperbola.Points marked by a blue circle in the second image are shown together with the lines of minimal k 2 , denoted by m, and represented in blue.

Fig. 4 .Fig. 5 .
Fig. 4. Line with minimal k 2 computed for a point on the estimated epipolar line: We see the epipolar band for a value k 2 = 5.9915, marked by the green hyperbola, corresponding to the point x in the first image which is marked by a green point.Obviously the epipolar line with smallest k 2 for points on the estimated epipolar line is the epipolar line itself.Here we see that these two lines are superposed (blue line over the red line).The corresponding value is k 2 = 0

Fig. 6 .
Fig. 6.Image representing the Mahalanobis distance k 2 over an image: The right image shows the epipolar band for k 2 = 5.991.In the left image we computed for each pixel x the Mahalanobis distance k 2 and represented it using grayscales.In particular, points in white have values close to k 2 = 0, points in black, a value close to the maximum value on the image.We used the whole range of k 2 values and divided the range in 256 levels of equal width, each one represented by its relative grayscale level.

Fig. 7 .
Fig. 7. Distribution of the values of k 2 This figure shows a histogram over the values for k 2 of the point correspondences from figure 8 (a).

Fig. 8 .
Fig. 8. Results: (a) stereo image pair with SIFT correspondences, the 8 points selected by RANSAC are red.remaining point correspondences after outlier removal using the distance from the epipolar line.(c) enhanced point correspondence set using the new criterion.INRIA Syntim owns the copyright of the image pair.