1. A high-speed global chromatic aberration correction method for a real-time video splicing system is characterized by comprising the following steps:

(1) acquiring a camera image sequence, wherein the camera image sequence refers to acquiring a frame of image of all cameras at the same moment in a camera video stream;

(2) for the overlapped area of the camera images during splicing, carrying out pairwise sampling and quantization and multi-image left-right sampling and quantization on the overlapped area simultaneously to obtain a corresponding image matrix;

(3) adding an alpha channel for expressing transparency to the image matrix obtained in the step (2) to mix data, and fusing and calculating two image matrices corresponding to the two graphs containing the overlapped area to obtain a mixed new image matrix;

(4) carrying out averaging calculation on the fused new image matrix;

(5) calculating the chrominance and brightness information of each image to be spliced after the image matrix is equalized and the chrominance and brightness information of an overlapping area between the images, and storing into a color lookup table; when the information is needed, the color checking table is searched for assignment directly;

(6) correcting chromatic aberration according to a color checking table: comparing two images to be spliced, and rapidly positioning and modifying by utilizing a color checking table to enhance and attenuate colors of relevant areas of the images;

(7) and outputting the spliced image after the chromatic aberration correction.

2. The method of claim 1, wherein: the sampling adopts a resampling method based on multiple secondary sampling.

3. The method of claim 1, wherein: the sampling and quantization specifically comprises the steps of respectively sampling and resampling an image overlapping area and each image on the premise of meeting the Nyquist sampling theorem, quantizing information on sampling points into a matrix containing image chromaticity and brightness after sampling, and obtaining the image matrix after quantization as an object of subsequent processing.

4. The method of claim 1, wherein: the image matrix in step (2) is based on a YUV color space.

5. The method of claim 4, wherein: the two image matrices are respectively A: (Ya, Ua, Va, α a), B: (Yb, Ub, Vb, α B), α a and α B respectively represent corresponding alpha channel variables, Ya, Ua and Va represent YUV variables of the image A, Yb, Ub and Vb represent YUV variables of the image B, and a new image matrix obtained by mixing is C: (Yc, Uc, Vc), Yc, Uc, Vc represent YUV variables of image C, where Yc is Ya × α a + Yb × α b; uc ═ Ua × α a + Ub × α b; vc is Va × α a + Vb × α b.

6. The method of claim 1, wherein different image averaging calculation methods are used for different actual scenes, so that color imbalance after channel fusion can be solved by averaging calculation.

7. The method of claim 1, wherein the look-up table is a color space reduction process by: the existing color space value is divided by the predetermined input value to obtain a smaller number of colors.

8. The method according to any of claims 1 to 7, wherein said step (6) further comprises a look-up table update sub-step of: and (4) re-executing the operations of the steps (1) to (5) at preset time intervals to update the color checking table so as to adapt to the color change changing along with the time.

9. The method of claim 8, wherein the enhancing and attenuating includes attenuating areas with high chrominance and luminance and enhancing areas with low chrominance and luminance to make the color of the spliced picture more uniform.

10. The method of claim 1, wherein the chromatic aberration correction process operates in a GPU to improve real-time performance of chromatic aberration correction.

Background

With the development of machine vision, the shooting of panoramic videos based on video splicing is also increasingly prevalent. The video splicing is a technology for splicing a plurality of partially overlapped images from different cameras or lenses and different visual angles into a panoramic or high-resolution video, and is widely applied to various fields such as security monitoring, virtual reality, medicine, remote sensing technology and the like. The video splicing technology is based on image splicing, and the image splicing mainly comprises image preprocessing, feature point extraction, transformation model construction, coordinate transformation and fusion reconstruction. Due to the difference of camera placement angles and ambient illumination, chromatic aberration and vignetting phenomena existing in camera imaging devices, and differences of internal reference matrixes and exposure intensities of different cameras, the spliced images are easy to have obvious chromatic aberration.

Methods of correcting chromatic aberration are largely divided into two. One is local processing, using progressive fusion in the overlap region to mitigate the differences between images, which is computationally inexpensive and fast, but produces significant color bands in the transition regions. The other is global processing, a certain camera is selected as a contrast system, the other images obtain a global correction coefficient according to the channel parameters of the camera image, and then the images are corrected, so that the two images have similar colors, but the overlapped area still has obvious chromatic aberration; the image can be subdivided into a plurality of strips, and the correction coefficients are respectively obtained to correct the whole image, so that a good chromatic aberration correction effect can be obtained, but a large amount of computing resources and computing time are required, and the requirement of real-time video splicing cannot be met.

Therefore, it is necessary to improve the global chromatic aberration correction method to meet the requirement of high-speed and high-quality shooting.

Disclosure of Invention

The invention aims to overcome the defects or the improvement requirements in the prior art, and provides a high-speed global color difference correction method for a real-time video splicing system, which can well eliminate the whole color difference of a spliced image and the color difference of an overlapped area and improve the whole quality of the spliced image.

In order to achieve the above object, the present invention provides a high-speed global color difference correction method for a real-time video splicing system, comprising the following steps:

(1) a sequence of camera images is acquired. The camera image sequence refers to acquiring a frame of image of all cameras at the same time in a camera video stream;

(2) for the overlapped area of the camera images during splicing, carrying out pairwise sampling and quantization and multi-image left-right sampling and quantization on the overlapped area simultaneously to obtain a corresponding image matrix;

(3) adding an alpha channel for expressing transparency to the image matrix obtained in the step (2) to mix data, and fusing and calculating two image matrices corresponding to the two graphs containing the overlapped area to obtain a mixed new image matrix;

(4) carrying out averaging calculation on the fused new image matrix;

(5) calculating the chrominance and brightness information of each image to be spliced after the image matrix is equalized and the chrominance and brightness information of an overlapping area between the images, and storing into a color lookup table; when the information is needed, the color checking table is searched for assignment directly;

(6) correcting chromatic aberration according to a color checking table: comparing two images to be spliced, and rapidly positioning and modifying by utilizing a color checking table to enhance and attenuate colors of relevant areas of the images;

(7) and outputting the spliced image after the chromatic aberration correction.

Furthermore, the sampling adopts a resampling method based on multiple sub-sampling.

The sampling and quantization specifically comprises the steps of respectively sampling and resampling an image overlapping area and each image on the premise of meeting the Nyquist sampling theorem, quantizing information on sampling points into a matrix containing image chromaticity and brightness after sampling, and obtaining the image matrix after quantization as an object of subsequent processing.

Further, the image matrix in step (2) is based on YUV color space.

Further, the two image matrices are respectively a: (Ya, Ua, Va, α a), B: (Yb, Ub, Vb, α B), α a and α B respectively represent corresponding alpha channel variables, Ya, Ua and Va represent YUV variables of the image A, Yb, Ub and Vb represent YUV variables of the image B, and a new image matrix obtained by mixing is C: (Yc, Uc, Vc), Yc, Uc, Vc represent YUV variables of image C, where Yc is Ya × α a + Yb × α b; uc ═ Ua × α a + Ub × α b; vc is Va × α a + Vb × α b.

The invention also adopts different image equalization calculation methods aiming at different actual scenes, and aims to solve the color imbalance after channel fusion through equalization calculation.

Wherein, the look-up table is a color space reduction process, which comprises the following steps: the existing color space value is divided by the predetermined input value to obtain a smaller number of colors.

Further, the step (6) further comprises a color lookup table updating sub-step of: and (4) re-executing the operations of the steps (1) to (5) at preset time intervals to update the color checking table so as to adapt to the color change changing along with the time. The enhancement and attenuation comprise the step of attenuating places with strong chroma and brightness and enhancing places with weak chroma and brightness, so that the color of the spliced picture is uniform.

Compared with the prior art, the scheme of the invention has the following beneficial effects:

1. on the premise of meeting the requirement of multi-camera real-time video splicing, the method creatively utilizes an alpha channel to realize the rapid correction of the image chromatic aberration, improves the correction effect of the global chromatic aberration, and effectively reduces the consumption of computing resources and computing time.

2. Aiming at the change of light caused by factors such as time lapse and the like, the color of the original image acquired by each camera can be obviously changed, and the method also can eliminate or weaken the influence of the condition by regularly calculating and updating the color checking table so as to ensure the color difference correction effect of the image and improve the quality of the spliced image.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the example serve to explain the principles of the invention and not to limit the invention.

Fig. 1 is a schematic flow chart of a chromatic aberration correction algorithm according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a panorama stitching process provided by an embodiment of the present invention;

fig. 3 is a schematic diagram of an image sampling process according to an embodiment of the present invention.

Detailed Description

For the purpose of making the present invention more comprehensible, and for the purpose of making the present application more comprehensible, embodiments and advantages thereof, the present invention will be further described with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.

The invention provides a high-speed global chromatic aberration correction method for a real-time video splicing system, which can effectively fade and eliminate chromatic aberration of an overlapped area of spliced images and integral chromatic aberration among the images and improve the quality of panoramic images.

The image splicing can be realized by various algorithms, the used actual algorithms are different, the consumed computing resources and computing time are different, and the effects are different. The method mainly aims at the fusion reconstruction stage of image splicing, and performs a chromatic aberration correction algorithm on the fusion reconstruction stage, as shown in figure 1. The method comprises the steps of sampling two by two in the overlapping area of the original images of the camera, then sampling multiple images left and right, calculating the balance value by using an alpha channel after sampling to obtain the color checking table of each image, correcting the color difference according to the content of the color checking table during projection, and keeping the color checking table periodically updated, thereby not only improving the quality of the panoramic spliced image, but also effectively reducing the consumption of computing resources on the premise of meeting the real-time requirement of large-size multi-camera real-time video splicing.

As an embodiment, the image arrangement mode used in the image stitching of the present invention is shown in FIG. 2. In fig. 2, 1-5 are 5 images obtained from five cameras, the shadow represents the overlapping area between adjacent images, the imaging of different cameras has chromatic aberration due to the factors of different placement positions of the cameras, different internal reference matrixes, vignetting phenomenon, etc., the chromatic aberration of the overlapping area between the spliced images is very obvious, and the overall quality of the spliced images is poor.

The method can weaken and eliminate the chromatic aberration caused by the factors, emphasizes on the fusion reconstruction stage of image splicing, and specifically comprises the following steps:

(1) a sequence of camera images is acquired. The camera image sequence is to capture a frame of image of all cameras at the same time in a camera video stream, and then the correction method is started to be executed. The acquisition of the image is the basic work of the method, and the accuracy of chromatic aberration correction can be ensured only by ensuring the time synchronization of the image sequence.

(2) And for the overlapping area of every two camera images during splicing, carrying out pairwise sampling and quantization and multi-image left-right sampling and quantization on the overlapping area at the same time.

After the camera original image is acquired, sampling processing is performed on the image to reduce consumption of computing resources and computing time. The method adopts a resampling method, and multiple secondary samplings can be used for resampling. However, if for a high-definition picture, according to the nyquist sampling theorem, the sampling frequency needs the maximum frequency of the image which is greater than or equal to 2 times, which causes very high sampling frequency and resource consumption, the image can be fuzzified by some filtering methods and then sub-sampled. After the image is resampled, the consumption of resources is reduced, and the integrity of the chrominance and luminance information of the image can be ensured.

After the resampling is completed, the interested part of the image is sampled, namely the shaded part shown in fig. 3 and the 1-5 image overall colors, and the sampling method is shown in fig. 3. The sampling method is to divide a continuous image into M x N grids in space, each grid is represented by a certain brightness value, one grid is called a pixel, and the sampling satisfies the Nyquist sampling theorem.

And the quantization is a process of converting the corresponding brightness continuous change interval on the sampling point into a single specific digital code, after sampling, quantizing the information on the sampling point into a matrix containing image chromaticity and brightness, and taking the quantized digital matrix M multiplied by N as an object after the processing in the step (2), so that the processing of pairwise sampling and quantization in an overlapping area and left-right sampling and quantization of multiple images can be simultaneously carried out.

Furthermore, the two-by-two sampling of the overlapped areas of the stitched images means that in a plurality of overlapped areas of the stitched images, the left and right images of each overlapped area are respectively sampled and quantized for a plurality of times to obtain an integer matrix containing the chrominance information, the luminance information and the scale information of the images of the overlapped areas, and the consumption of computing resources and computing time is reduced as much as possible on the premise of not losing the image information of the overlapped areas.

Furthermore, the multi-image left-right sampling refers to that for global information, a spliced image is formed by splicing and merging a plurality of images in a left-right arrangement mode, each image and the left and right images of each image are sampled and quantized for multiple times respectively to obtain an integer matrix containing chromaticity information, brightness information and scale information of each image, and consumption of computing resources and computing time is reduced as much as possible on the premise that original image information is not lost.

(3) And adding an alpha channel to the image matrix after sampling and quantization processing. The color of an image is generally described by three independent attributes, and the three independent variables are combined to form a space coordinate, namely a color space. Common image color spaces include RGB color space, YUV color space, and HSV color space. The method uses the YUV color space which only occupies little bandwidth and is dominant in image transmission. After sampling and quantization processing are carried out, the region of interest of the image is converted into data in a processable YUV color space, and a fourth variable alpha channel is added for data fusion. The alpha channel of the special channel represents transparency, so that a mixed new image matrix is obtained by calculating the two original image matrixes through alpha fusion, and two graphs containing the overlapped area can be fused into one image by adding the alpha channel.

In brief, the color to be combined is formed by combining two original YUV images without alpha component, for example, now there are two images, which are respectively referred to as image a and image B, and the image formed by combining the two images is referred to as C, there are two original quadruples a and B as follows:

A：(Ya，Ua，Va，αa)B：(Yb，Ub，Vb，αb)

and the combined triplet C:

C：(Yc，Uc，Vc)

then there are combination formulas

Yc＝Ya×αa+Yb×αb；

Uc＝Ua×αa+Ub×αb；

Vc＝Va×αa+Vb×αb

The fused color can be obtained.

(4) And carrying out averaging calculation on the fused image matrix. After the interested part of the image is mixed with colors, averaging calculation needs to be carried out on the mixed part, different image averaging calculation methods are adopted according to different actual scenes, and the purpose is to solve the color imbalance phenomenon after the alpha channel fusion is added through the averaging calculation.

Preferably, the fourth transparency component except for the three color components is added to the graph information matrix obtained after the multi-graph left-right sampling, the fourth transparency component is respectively acted on the graph information matrix obtained after the two-two sampling and the graph information matrix obtained after the multi-graph left-right sampling, the image equalization value calculation is carried out, the alpha matrix parameter of the spliced image is obtained, and the image after the equalization processing is approximately uniformly distributed.

As an example, taking a grayscale map as an example: firstly, counting the number N (i) of pixels of each gray level of an original image, wherein the value range of i is [0, max ], and max is the number of all gray levels in the image (the number of gray levels is related to the bit depth of the image, and if the image is 8-bit depth, the total number of gray levels is 2^8 ^ 256); secondly, counting the occurrence probability P (i) ═ N (i)/M of the pixel with the gray level i, wherein M is all the pixel numbers in the image; the cumulative distribution function of p (i) is counted again:

and finally, carrying out averaging calculation on the mixed parts:

f (min) is the minimum value of the cumulative distribution function, and M and N respectively represent the number of the length and width pixels of the image. The method of equalization is used according to the grayscale image described above for the Y, U, V components of the YUV color space of the image, respectively, so that color images can also be processed.

(5) And obtaining a color checking table for the image calculation after the equalization. And aiming at the conditions of large size, more information and the like of the spliced image, reducing the image color space after the equalization value calculation by using a color lookup table method. After the averaging calculation, the chrominance and brightness information of each image to be spliced after the image matrix is averaged and the chrominance and brightness information of the overlapping area between the images can be obtained and stored into a color checking table. At the moment, the corresponding color look-up table is calculated for the image after equalization, the functions of the color look-up table are that the calculation is directly looked up and simplified, and the efficiency of processing the image is greatly improved.

When a picture is processed, namely the whole image data is processed, all the data of the picture is scanned and read, then corresponding processing is carried out, and the operation which consumes the shortest time in computer processing is assignment, so that the color lookup table is used. The color checking table is a color space reduction method, and the method comprises the following steps: dividing the existing color space value by an input value to obtain a smaller number of colors; all possible values are calculated in advance, and when the color checking table is used, the values are directly assigned through the color checking table. For example, color values 0 through 9 may take the new values 0, 10 through 19 may take the new values 10, and so on, may be expressed in the following form:

thus, a simple color space reduction algorithm can consist of the following two steps: firstly, traversing each pixel of an image matrix; secondly, the above formula is applied to the pixels. The method is characterized in that for the used region of interest, all possible values are calculated in advance after averaging calculation and stored as one-dimensional or multi-dimensional arrays, corresponding output values are stored corresponding to different input values, and then when the values are needed, the values are directly assigned by using a lookup table.

(6) Two operations are then performed: on one hand, the camera image is updated, and the color checking table is calculated and updated. The method re-executes the operations (1) - (5) after a period of time, continuously records the chroma and brightness information of the image and periodically updates the color checking table. The period of time can be 1-12 hours.

On one hand, the influence of color change of an original image acquired by the camera due to time lapse and other factors can be eliminated or weakened; on the other hand, the chromatic aberration is corrected according to the content of the color checking table during projection.

And after the color checking table is obtained, enhancing and attenuating the color of each original image. Every two images are compared, the color checking table is utilized to quickly position and modify, places with strong chroma and brightness are attenuated, places with weak chroma and brightness are enhanced, the color of the spliced images is uniform, the effect of correcting chromatic aberration is achieved, and the quality of the panoramic spliced images is improved.

(7) And finally, outputting the spliced image after chromatic aberration correction.

The invention also provides a processor for executing the method.

The invention also provides computer equipment which comprises a memory, the processor and a computer program which is stored on the memory and can run on the processor; wherein the processor implements the above steps when executing the program. Preferably, the chromatic aberration correction process is operated in a GPU to improve the real-time performance of chromatic aberration correction.

In summary, the present invention provides a high-speed global color difference correction method for a real-time video stitching system, which can well fade and eliminate the color difference of the overlapping area of the stitched images and the integral color difference between the images, effectively implement the high-speed global color difference correction of the images in the panoramic video stitching system, improve the quality of the panoramic stitched images, and effectively reduce the consumption of computing resources on the premise of meeting the real-time requirements of large-size multi-camera real-time video stitching.

Although the embodiments of the present invention have been described above, the above description is only for the convenience of understanding the present invention, and is not intended to limit the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the scope of the present invention should be determined by the following claims.