1. A stereo matching algorithm based on pixel similarity, comprising:

acquiring a first matching image and a second matching image;

for any current pixel point to be matched in the first matching image, obtaining a pixel point corresponding to the first matching image in the second matching image according to a preset parallax value;

calculating the similarity between the current pixel point to be matched in the first matching image and the corresponding pixel point in the second matching image according to a preset similarity definition formula;

if the similarity is larger than the threshold value, continuing to calculate the similarity between the peripheral pixel points of the current pixel point to be matched and the corresponding peripheral pixel points of the corresponding pixel points in the second matched image so as to obtain a first matched image similar area and a second matched image similar area; the first matching image similar area is consistent with the second matching image similar area in shape and size; the similarity between the pixel points in the first matching image similar area and the corresponding pixel points in the second matching image similar area is larger than the threshold, and the similarity between the pixel points around the first matching image similar area and the pixel points around the second matching image similar area is smaller than or equal to the threshold;

calculating the matching cost of the pixel points in the first matching image similar area and the corresponding pixel points in the second matching image similar area;

and calculating the parallax between the pixel point in the first matching image similar area and the corresponding pixel point in the second matching image similar area according to the matching cost.

2. The pixel similarity-based stereo matching algorithm according to claim 1, wherein the preset disparity value is iterated from zero to a maximum value.

3. The pixel similarity-based stereo matching algorithm according to claim 1, wherein the preset similarity definition formula comprises:

taking a neighborhood with two consistent sides around a first pixel point and a second pixel point of two pixel points to be calculated, calculating a correlation coefficient of the two pixel points according to the neighborhoods of the two pixel points, and defining the coefficient as an index of similarity of the two pixel points; the neighborhood size is (2M +1) x (2M +1), M is a natural number equal to or greater than 0;

the calculation formula is as follows:

in the formula, P_lIs a first pixel point; x is the number of_pl、y_plRespectively an x coordinate and a y coordinate of the first pixel point;I_lthe color values of the first pixel point and the pixel points in the neighborhood of the first pixel point comprise a gray value and an RGB value; p_rIs a second pixel point; x is the number of_pr、y_prX and y coordinates of the second pixel point respectively; i is_rThe color values of the second pixel and the pixels in the neighborhood thereof.

4. The pixel similarity-based stereo matching algorithm according to claim 3, wherein the values of M include 1, 2 and 3.

5. The pixel similarity-based stereo matching algorithm according to claim 1, wherein the preset similarity definition formula includes SSIM.

6. The pixel similarity-based stereo matching algorithm according to claim 1, wherein the calculating the matching cost of the pixel point in the first matching image similar region and the corresponding pixel point in the second matching image similar region comprises:

matching the pixel points in the first matching image similar area with the corresponding pixel points in the second matching image similar area by adopting ZNCC to obtain matching cost, wherein the calculation formula is as follows:

in the formula, Z_dAre similar regions; p'_l、p′_rRespectively being a pixel point in the first matching image similar area and a corresponding pixel point in the second matching image similar area.

7. The pixel similarity-based stereo matching algorithm according to claim 1, wherein the calculating the matching cost of the pixel point in the first matching image similar region and the corresponding pixel point in the second matching image similar region comprises:

the matching cost is calculated using SAD, SSD or NCC.

8. The pixel similarity-based stereo matching algorithm according to claim 1, wherein the calculation method for calculating the parallax between the pixel point in the first matching image similar region and the corresponding pixel point in the second matching image similar region according to the matching cost includes weighted median filtering, semi-global, minimum spanning tree, and LocalExp.

9. A stereo matching system based on pixel similarity, comprising:

a memory for storing a program;

a processor for implementing the method of any one of claims 1-8 by executing a program stored by the memory.

10. A computer-readable storage medium, characterized in that the medium has stored thereon a program which is executable by a processor to implement the method according to any one of claims 1-8.

Background

In binocular vision stereo matching, a key problem is to find corresponding points in left and right images to obtain a horizontal position difference, also called parallax, of corresponding pixels in the two images, so that the depth of the pixel can be directly calculated according to the parallax, and the parallax and the depth are in one-to-one correspondence.

However, points which are not at the same depth may have the same color, texture, gradient and the like, so that stereo matching is mismatched, parallax calculation is greatly wrong, and the application of binocular vision in depth measurement is greatly influenced.

In order to overcome the disadvantage of being susceptible to noise, light variation and the like, pixels in a peripheral region of a pixel may be used to estimate matching cost and disparity estimation of the pixel, where there are rectangular (square) regions and adaptively grown regions, but these regions have a high probability (high probability) of including pixels with disparity being inconsistent, and therefore, the cost calculation for estimating the pixel including different disparity pixels is also inaccurate. Even if bilateral filtering, minimum spanning tree, etc. are used, this problem cannot be completely solved because bilateral filtering can only be weighted by color difference, and the minimum spanning tree is also weighted by color, and has no direct relation to whether the parallax between the pixels of the region is consistent (or close).

Disclosure of Invention

Aiming at the problem that the matching area may contain pixels with inconsistent parallax, the invention provides a stereo matching algorithm based on similarity.

According to a first aspect, an embodiment provides a stereo matching algorithm based on pixel similarity, including:

acquiring a first matching image and a second matching image;

for any current pixel point to be matched in the first matching image, obtaining a pixel point corresponding to the first matching image in the second matching image according to a preset parallax value;

calculating the similarity between the current pixel point to be matched in the first matching image and the corresponding pixel point in the second matching image according to a preset similarity definition formula;

if the similarity is larger than the threshold value, continuing to calculate the similarity between the peripheral pixel points of the current pixel point to be matched and the corresponding peripheral pixel points of the corresponding pixel points in the second matched image so as to obtain a first matched image similar area and a second matched image similar area; the first matching image similar area is consistent with the second matching image similar area in shape and size; the similarity between the pixel points in the first matching image similar area and the corresponding pixel points in the second matching image similar area is larger than the threshold, and the similarity between the pixel points around the first matching image similar area and the pixel points around the second matching image similar area is smaller than or equal to the threshold;

calculating the matching cost of the pixel points in the first matching image similar area and the corresponding pixel points in the second matching image similar area;

and calculating the parallax between the pixel point in the first matching image similar area and the corresponding pixel point in the second matching image similar area according to the matching cost.

In an embodiment, the preset disparity value is iterated from zero to a maximum value.

In one embodiment, the predetermined similarity definition formula includes:

taking a neighborhood with two consistent sides around a first pixel point and a second pixel point of two pixel points to be calculated, calculating a correlation coefficient of the two pixel points according to the neighborhoods of the two pixel points, and defining the coefficient as an index of similarity of the two pixel points; the neighborhood size is (2M +1) x (2M +1), M is a natural number equal to or greater than 0;

the calculation formula is as follows:

in the formula, P_lIs a first pixel point; x is the number of_pl、y_plRespectively an x coordinate and a y coordinate of the first pixel point; i is_lThe color values of the first pixel point and the pixel points in the neighborhood of the first pixel point comprise a gray value and an RGB value; p_rIs a second pixel point; x is the number of_pr、y_prX and y coordinates of the second pixel point respectively; i is_rThe color values of the second pixel and the pixels in the neighborhood。

In an embodiment, the value of M includes 1, 2, and 3.

In one embodiment, the predetermined similarity definition formula includes SSIM.

In an embodiment, the calculating the matching cost between the pixel point in the first matching image similar region and the corresponding pixel point in the second matching image similar region includes:

matching the pixel points in the first matching image similar area with the corresponding pixel points in the second matching image similar area by adopting ZNCC to obtain matching cost, wherein the calculation formula is as follows:

in the formula, Z_dAre similar regions; p'_l、p′_rRespectively being a pixel point in the first matching image similar area and a corresponding pixel point in the second matching image similar area.

In an embodiment, the calculating the matching cost between the pixel point in the first matching image similar region and the corresponding pixel point in the second matching image similar region includes:

the matching cost is calculated using SAD, SSD or NCC.

In an embodiment, the method for calculating the parallax between the pixel point in the first matching image similar region and the corresponding pixel point in the second matching image similar region according to the matching cost includes weighted median filtering, semi-global, minimum spanning tree, and LocalExp.

According to a second aspect, an embodiment provides a stereo matching system based on pixel similarity, including:

a memory for storing a program;

a processor for implementing the method of any of the embodiments herein by executing a program stored by the memory.

According to a third aspect, an embodiment provides a computer readable storage medium having a program stored thereon, the program being executable by a processor to implement a method as described in any of the embodiments herein.

According to the stereo matching algorithm based on the pixel similarity, firstly, the similarity of the pixels is defined, the similar range of the matching area is defined according to the similarity of the pixels, and then the similar area with similar pixel point parallax is determined, so that more accurate stereo matching is realized.

Drawings

FIG. 1 is a flow chart of an embodiment of a stereo matching algorithm based on pixel similarity;

FIG. 2 is a flow chart of the establishment of the "similarity definition formula" in the stereo matching algorithm based on pixel similarity according to an embodiment;

FIG. 3 is a diagram illustrating a similarity calculation region with a neighborhood size of (5x5) according to an embodiment;

FIG. 4 is a diagram illustrating an embodiment of iteratively obtaining a pixel point corresponding to a second matching image by using a preset disparity value;

FIG. 5(a) is a schematic diagram of a first expansion of similar areas according to an embodiment; FIG. 5(b) is a schematic diagram of a second expansion of similar regions according to an embodiment; FIG. 5(c) is a diagram illustrating the matching result of the second expansion similarity region according to an embodiment; fig. 5(d) is a schematic diagram of the expanded first matching image similar region and the second matching image similar region;

fig. 6 is a schematic structural diagram of a stereo matching system based on pixel similarity according to an embodiment.

Detailed Description

The present invention will be described in further detail with reference to the following detailed description and accompanying drawings. Wherein like elements in different embodiments are numbered with like associated elements. In the following description, numerous details are set forth in order to provide a better understanding of the present application. However, those skilled in the art will readily recognize that some of the features may be omitted or replaced with other elements, materials, methods in different instances. In some instances, certain operations related to the present application have not been shown or described in detail in order to avoid obscuring the core of the present application from excessive description, and it is not necessary for those skilled in the art to describe these operations in detail, so that they may be fully understood from the description in the specification and the general knowledge in the art.

Furthermore, the features, operations, or characteristics described in the specification may be combined in any suitable manner to form various embodiments. Also, the various steps or actions in the method descriptions may be transposed or transposed in order, as will be apparent to one of ordinary skill in the art. Thus, the various sequences in the specification and drawings are for the purpose of describing certain embodiments only and are not intended to imply a required sequence unless otherwise indicated where such sequence must be followed.

The numbering of the components as such, e.g., "first", "second", etc., is used herein only to distinguish the objects as described, and does not have any sequential or technical meaning. The term "connected" and "coupled" when used in this application, unless otherwise indicated, includes both direct and indirect connections (couplings).

The invention discloses a stereo matching algorithm based on pixel similarity, and provides a region extraction method based on pixel similarity. Therefore, in some embodiments of the present application, similarity between pixels is first proposed, and then a similarity range of a region to be matched is defined according to a definition of the similarity of the pixels, so as to determine a similar region, thereby obtaining a more accurate stereo matching algorithm.

The present invention will be explained below with reference to specific examples.

The first embodiment is as follows:

referring to fig. 1, an embodiment of the present invention provides a stereo matching algorithm based on pixel similarity, which includes steps S110 to S160, which are described in detail below.

Step S110: and acquiring a first matching image and a second matching image.

Step S120: and for any current pixel point to be matched in the first matching image, obtaining a pixel point corresponding to the second matching image according to a preset parallax value. In an embodiment, the disparity values are iterated from zero to a maximum value.

Step S130: and calculating the similarity between the current pixel point to be matched in the first matching image and the corresponding pixel point in the second matching image according to a preset similarity definition formula.

One of the core of the present invention is to define the similarity of pixels, which is defined by the similarity of the same neighborhood around two pixels in one embodiment of the present invention. How to establish the similarity definition formula referred to in step S130 is explained below.

Referring to fig. 2, in an embodiment, the establishment of the similarity definition formula includes steps S210-S230, which are described in detail below.

Step S210: and taking a neighborhood which is consistent on two sides and is around the first pixel point and the second pixel point of the two pixel points to be calculated. In one embodiment, the neighborhood size is (2M +1) x (2M +1), M being a natural number equal to or greater than 0.

Step S220: and calculating the correlation coefficient of the two pixel points according to the neighborhoods of the two pixel points.

In one embodiment, the calculation formula is as follows:

in the formula, P_lIs a first pixel point; x is the number of_pl、y_plRespectively an x coordinate and a y coordinate of the first pixel point; i is_lThe color values of the first pixel point and the pixel points in the neighborhood of the first pixel point comprise a gray value and an RGB value; p_rIs a second pixel point; x is the number of_pr、y_prX and y coordinates of the second pixel point respectively; i is_rIs as followsThe color values of the two pixels and the pixels in the neighborhood.

Step S230: and defining the coefficient as an index of similarity of the two pixel points.

Through the above steps S210 to S230, the similarity definition formula (1) is established. Fig. 3 is a schematic diagram of a similarity calculation region with a neighborhood size of (5 × 5) in an embodiment. In one embodiment, the neighborhood size includes (3x3), (5x5), and (7x7), if (3x3) is taken, the similarity of the boundary pixels is more accurate but less robust, and if (7x7) is taken, the similarity is more robust but less accurate, so in general, (5x5) is a more appropriate choice. In this regard, it is obvious to those skilled in the art that the definition of two pixels in the present invention is defined by the similarity (e.g., correlation coefficient) of the same neighborhood around the two pixels, and the definition of the similarity of the same neighborhood is within the scope of the present application regardless of the size of the neighborhood, and the definition of the similarity of the same neighborhood is within the scope of the present application as long as some similarity between the two pixels is defined. For example, in one embodiment, SSIM (structural similarity algorithm) is used to calculate the similarity of pixels, which is as follows:

in the formula, l (p)_l，p_r) As a brightness contrast function, c (p)_l，p_r) As contrast function, s (p)_l，p_r) As a function of structural contrast, mu_plIs the average value of gray levels, mu, of pixels in the neighborhood around the first pixel_prIs the gray average, sigma, of the pixels in the neighborhood around the second pixel_plIs the gray standard deviation, sigma, of the pixels in the neighborhood around the first pixel_prIs the gray scale standard deviation of the pixels in the neighborhood around the second pixel, c₁、c₂Is constant and prevents divide by zero.

The following is a continued description of an embodiment of the present invention that provides a stereo matching algorithm based on pixel similarity.

Step S140: if the similarity is larger than the threshold value, the similarity between the peripheral pixel points of the current pixel point to be matched and the corresponding peripheral pixel points of the corresponding pixel points in the second matching image is continuously calculated, so that a first matching image similar area and a second matching image similar area are obtained.

In an embodiment, if the similarity is smaller than or equal to the threshold, the process returns to step S120, the preset parallax value is iterated to the next numerical value to obtain a new pixel point corresponding to the second matching image, and step S130 is then performed. As shown in fig. 4, a schematic diagram of obtaining a pixel point corresponding to the second matching image through iteration by using a preset disparity value in an embodiment is shown, where the pixel point corresponding to the second matching image when the disparity value is d1 is not greater than the threshold value, the disparity value is iterated to d2, and the similarity is still not greater than the threshold value, and thus until the similarity of the corresponding pixel point in the second matching image when the disparity value is d5 is greater than the threshold value, the similarity between the peripheral pixel point of the current pixel point to be matched and the corresponding peripheral pixel point of the corresponding pixel point in the second matching image is continuously calculated, so as to obtain a first matching image similar region and a second matching image similar region.

In an embodiment, please refer to fig. 5(a) -5(d), which are schematic diagrams illustrating a process of continuously calculating similarities between peripheral pixels of the current pixel to be matched and corresponding peripheral pixels of the corresponding pixel in the second matching image to obtain a first matching image similar area and a second matching image similar area. Fig. 5(a) is a schematic diagram of expanding a similar region for the first time, and the similarity between the peripheral pixel point of the current pixel point to be matched and the corresponding peripheral pixel point of the corresponding pixel point in the second matching image is calculated in the expanded region, for example, the relative positions of the pixel point pairs (a1, B1), (a2, B2) and (A3, B3) in fig. 5(a), the relative positions of the pixel point pairs a1, a2, A3 and the current pixel point to be matched in the first matching image are respectively the same as the relative positions of the pixel point pairs B1, B2, B3 and the corresponding pixel point in the second matching image, and other calculated pixel point pairs also have the characteristic. In an embodiment, if the similarity of corresponding pixel point pairs in the expanded similarity region in fig. 5(a) is greater than the threshold, fig. 5(b) is a schematic diagram of expanding the similarity region for the second time, and the similarity of the corresponding pixel point pairs is respectively calculated, in an embodiment, the matching result of the expanded similarity region for the second time is as shown in fig. 5 (c). In an embodiment, the pixel point pairs with similarity greater than the threshold continue to expand to the periphery until the similarity of all the expanded pixel point pairs is less than or equal to the threshold, for example, fig. 5(d) is a schematic diagram of the expanded first matching image similar area and the expanded second matching image similar area in an embodiment. Thus, in an embodiment, the first matching image similar region is consistent in shape and size with the second matching image similar region; the similarity between the pixel points in the first matching image similar area and the corresponding pixel points in the second matching image similar area is larger than the threshold, and the similarity between the pixel points around the first matching image similar area and the corresponding pixel points around the second matching image similar area is smaller than or equal to the threshold. The threshold is set according to the similarity definition calculation method and the requirement for matching accuracy, for example, in one embodiment, similarity calculation is performed using SSIM, and the threshold is set to 0.9.

Step S150: and calculating the matching cost of the pixel points in the first matching image similar area and the corresponding pixel points in the second matching image similar area. In an embodiment, calculating the matching cost between the pixel point in the first matching image similar region and the corresponding pixel point in the second matching image similar region includes:

matching the pixel points in the first matching image similar area with the corresponding pixel points in the second matching image similar area by adopting ZNCC (zero mean normalized cross correlation coefficient), so as to obtain a matching cost, wherein the calculation formula is as follows:

in the formula, Z_dIs a phase ofSimilar areas; p'_l、p′_rRespectively being a pixel point in the first matching image similar area and a corresponding pixel point in the second matching image similar area.

In one embodiment, SAD, SSD or NCC is used to calculate the matching cost, and accordingly, equation (2) for calculating the matching cost is adaptively modified.

In one embodiment, the matching cost is calculated using SAD (sum of squared differences in gray scale), and the calculation formula is as follows:

in the formula, I () is the gray level of the pixel.

In one embodiment, the matching cost is calculated using SSD (sum of absolute difference in gray scale), and the calculation formula is as follows:

in one embodiment, the matching cost is calculated using NNC (normalized correlation coefficient), which is calculated as follows:

in the formula, H () is a color value of a pixel, including a gray scale or RGB value.

In view of the above, those skilled in the art will have no doubt understand that, the formula for calculating the matching cost in the present invention is within the scope of the present application, regardless of the formula for calculating the matching cost, as long as the calculation region is a region defined (or expanded) by using pixel similarity.

Step S160: and calculating the parallax between the pixel point in the first matching image similar area and the corresponding pixel point in the second matching image similar area according to the matching cost. In one embodiment, the disparity calculation method includes weighted median filtering, semi-global, minimum spanning tree, and LocalExp.

Referring to fig. 6, an embodiment of the invention provides a stereo matching system based on pixel similarity, which includes a memory 10 and a processor 12. The memory 10 is used to store programs. The processor 12 is used to implement the method described in any of the embodiments herein by executing programs stored in the memory 10.

An embodiment of the invention also provides a computer medium having a program stored thereon, the program being executable by a processor to implement a method as described in any of the embodiments herein.

The above are some descriptions of the stereo matching algorithm and system based on pixel similarity according to the present invention. In one embodiment of the present invention, the main contents include:

1) firstly, defining similarity of two pixels in the main image, namely, taking a small neighborhood with two consistent edges around the two pixels, here taking a region N of (2M +1) x (2M +1), and taking a typical value of 5x5(M is 2), and then calculating a correlation coefficient between the two pixels, wherein the coefficient technology defines an index of similarity of the two pixels, namely, the index

Of course, the neighborhood N may be (3x3) or (7x7) — if (3x3) is taken, the similarity accuracy of the boundary pixels is a little higher but the robustness is poor, and if (7x7) is taken, the similarity robustness is a little higher but the similarity accuracy of the boundary pixels is a little worse. Therefore, (5 × 5) is a more appropriate choice in general.

2) After the calculation method of pixel similarity is determined, an adaptive matching region can be defined, namely, the pixel P of any left image_lFor one pixel P of the right picture with disparity preset to d_r,P_r＝P_lD, if the similarity of the two pixels is smaller than a certain threshold Ts, they must not match. (its cost function can be set to infinity, or a relatively large value). If two are providedIf the pixels Pl, Pr are similar (the similarity is greater than a threshold Ts), then a similarity definition formula (1) can be used to expand to P_lSimilar zone Z, i.e. in this zone, P_lAnd P_rThe corresponding pixels at the periphery are similar so that their disparities are also substantially uniform. (Please note that for each preset parallax d, the similar region Z obtained is different, so we can mark it as Z_d) Therefore, the condition that the parallax of the pixels in the matching area is inconsistent can be overcome, and the matching precision is greatly improved.

Assuming that the region expanded according to the definition formula (1) of similarity is Z_dThen the region where matching is performed is Z_dThat is, the region for calculating the matching cost is Z_dThen the (zero mean normalized cross correlation coefficient) ZNCC can be used for matching, i.e.

Of course, other cost functions may be used, but equation (2) for calculating the cost may be modified as appropriate.

3) After the matching cost is solved, disparity estimation can be performed according to various subsequent disparity estimation methods, such as weighted median filtering, semi-global, minimum spanning tree, LocalExp, and the like.

The invention provides a region extraction method based on pixel similarity, and the main idea of the method is that if a certain region is consistent in parallax on the left and right, the region is necessarily similar, but how to define that two regions are similar, whether each pair of pixels are similar needs to be defined. Therefore, the invention firstly proposes the similarity between the pixels, then defines the similar range of the region to be matched according to the similarity of the pixels, and further determines the similar region, thereby obtaining a more accurate stereo matching algorithm.

Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments may be implemented by hardware, or may be implemented by computer programs. When all or part of the functions of the above embodiments are implemented by a computer program, the program may be stored in a computer-readable storage medium, and the storage medium may include: a read only memory, a random access memory, a magnetic disk, an optical disk, a hard disk, etc., and the program is executed by a computer to realize the above functions. For example, the program may be stored in a memory of the device, and when the program in the memory is executed by the processor, all or part of the functions described above may be implemented. In addition, when all or part of the functions in the above embodiments are implemented by a computer program, the program may be stored in a storage medium such as a server, another computer, a magnetic disk, an optical disk, a flash disk, or a removable hard disk, and may be downloaded or copied to a memory of a local device, or may be version-updated in a system of the local device, and when the program in the memory is executed by a processor, all or part of the functions in the above embodiments may be implemented.

The present invention has been described in terms of specific examples, which are provided to aid understanding of the invention and are not intended to be limiting. For a person skilled in the art to which the invention pertains, several simple deductions, modifications or substitutions may be made according to the idea of the invention.