1. A method for repairing a point cloud picture is characterized by comprising the following steps:

acquiring a defect point cloud picture obtained by shooting an object and a digital-analog point cloud picture preset by a system;

converting the digital-analog cloud picture into a coordinate system where the incomplete point cloud picture is located to obtain a conversion point cloud picture;

calculating a displacement deviation vector of the conversion point cloud picture deviating from the defect cloud picture in a coordinate system of the defect cloud picture;

moving the conversion point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector;

judging whether the shifted conversion point cloud picture is overlapped with the incomplete point cloud picture or not;

under the condition that the shifted conversion cloud picture is overlapped with the incomplete point cloud picture, determining incomplete point cloud blocks of the incomplete point cloud picture relative to the conversion cloud picture in the conversion cloud picture;

and filling the incomplete cloud picture by using the cloud point blocks to generate a repaired cloud picture of the object.

2. The method of claim 1, wherein the converting the digital-to-analog cloud map into a coordinate system of the cloud map of the defect points comprises:

performing point cloud registration on the incomplete point cloud picture and the digital-analog point cloud picture to obtain a conversion matrix for converting the digital-analog point cloud picture to a coordinate system where the incomplete point cloud picture is located;

and converting the digital-analog cloud picture into a coordinate system where the incomplete point cloud picture is located according to the conversion matrix to obtain a conversion point cloud picture.

3. The method of claim 1, wherein the calculating a displacement deviation vector of the transformed point cloud image from the incomplete point cloud image comprises:

respectively determining a paired point cloud which is closest to each point cloud in the conversion point cloud picture in Euclidean distance among all point clouds in the incomplete point cloud picture, and pairing each point cloud in the conversion point cloud picture with the corresponding paired point cloud in the incomplete point cloud picture;

according to the matching condition, acquiring first point clouds matched with the point clouds in the conversion point cloud image from all the point clouds in the incomplete point cloud image, and combining the first point clouds into a first point cloud set;

determining the centroid of a point cloud outline formed by the first point cloud set according to the first point cloud set;

acquiring second point clouds closest to the first point cloud in all point clouds of the converted point cloud image, and combining the second point clouds into a second point cloud set;

determining a first vector set consisting of all second point clouds in the second point cloud set and the centroid;

and calculating a displacement deviation vector of the converted point cloud picture deviating from the incomplete point cloud picture according to the first vector set.

4. The method of claim 3, wherein the calculating a displacement deviant vector of the transformed point cloud from the incomplete point cloud from the first set of vectors comprises:

calculating a displacement deviation vector of the conversion point cloud picture deviating from the incomplete point cloud picture according to the first vector set by using the following formula (1):

formula (1)

Wherein the content of the first and second substances,the displacement deviation vector is represented by a vector of displacement deviations,representing an ith first vector in the first set of vectors;representing an ith second point cloud in the second point cloud set;representing a centroid;representing the total number of all first vectors in the first set of vectors.

5. The method of claim 3, wherein the determining whether the shifted cloud of transition points coincides with the cloud of incomplete points comprises:

acquiring a third point cloud closest to the first point cloud in all the point clouds of the moved conversion point cloud picture, and combining the third point clouds into a third point cloud set;

respectively determining Euclidean point cloud distances between all third point clouds in the third point cloud set and the corresponding first point clouds;

determining the average Euclidean point cloud distance of the third point cloud set according to all the Euclidean point cloud distances;

and judging whether the moved converted point cloud picture is overlapped with the incomplete point cloud picture or not by utilizing the average Euclidean point cloud distance and a preset point cloud distance.

6. The method of claim 5, wherein determining an average Euclidean point cloud distance for the third point cloud set from all of the Euclidean point cloud distances comprises:

determining an average Euclidean point cloud distance of the third point cloud set according to all the Euclidean point cloud distances by using the following formula (2):

formula (2)

Wherein the content of the first and second substances,representing an average Euclidean point cloud distance;representing the Euclidean point cloud distance between the ith third point cloud in the third point cloud set and the corresponding first point cloud;representing the ith third point cloud in the third point cloud set;representing a first point cloud corresponding to the ith third point cloud in the third point cloud set;representing the total number of all third point clouds in the third point cloud set.

7. The method of claim 5, wherein after determining whether the shifted cloud images of the transformed points coincide with the cloud images of the incomplete points by using the average Euclidean point cloud distance and a preset point cloud distance, the method further comprises:

after judging that the shifted conversion point cloud picture is not superposed with the incomplete point cloud picture, recalculating a displacement deviation vector of the shifted conversion point cloud picture deviating from the incomplete point cloud picture; continuously moving the conversion point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector; continuously judging whether the moved conversion point cloud picture is overlapped with the incomplete point cloud picture or not; and circulating until the moved conversion point cloud picture is superposed with the incomplete point cloud picture or the moving times reach the preset times.

8. The method of claim 5, wherein determining the cloud blocks of the incomplete cloud image relative to the transformed cloud image in the transformed cloud image comprises:

determining a first point cloud outline formed by all third point clouds in the third point cloud set according to the third point cloud set;

acquiring a point cloud block with a point cloud outline as the first point cloud outline from the moved conversion point cloud image;

and taking the point cloud block with the point cloud outline as the first point cloud outline as the incomplete point cloud block of the incomplete point cloud picture relative to the converted point cloud picture.

9. A point cloud picture repairing apparatus, characterized in that the point cloud picture repairing apparatus comprises:

the acquisition module is used for acquiring a defect cloud picture obtained by shooting an object and a digital-analog point cloud picture preset by the system;

the conversion module is used for converting the digital-analog cloud picture into a coordinate system where the incomplete point cloud picture is located to obtain a conversion point cloud picture;

the calculation module is used for calculating a displacement deviation vector of the conversion point cloud picture deviating from the defect cloud picture in a coordinate system of the defect cloud picture;

the moving module is used for moving the converted point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector;

the judging module is used for judging whether the moved conversion point cloud picture is superposed with the incomplete point cloud picture or not;

a determining module, configured to determine, in the converted cloud image, a cloud point block of the incomplete cloud image relative to the converted cloud image when the shifted converted cloud image is overlapped with the incomplete cloud image;

and the generating module is used for filling the incomplete point cloud picture by using the point cloud blocks and generating a repaired point cloud picture of the object.

10. An electronic device, comprising: a processor and a memory for storing a computer program capable of running on the processor; wherein the content of the first and second substances,

the processor is adapted to perform the steps of the method of any one of claims 1 to 8 when running the computer program.

11. A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the steps of the method of any one of claims 1 to 8.

Background

With the development of science and technology, many three-dimensional cameras based on different principles, such as structured light, line laser, binocular vision, and time of flight (TOF), are emerging on the market. The three-dimensional cameras can obtain three-dimensional information in the visual field of the cameras, so that the practical functions of type recognition, posture estimation, size measurement and the like of the object are realized.

In the industrial production process, a lot of applications for recognizing, positioning and grabbing randomly placed objects based on three-dimensional vision exist, but a three-dimensional camera is difficult to obtain accurate three-dimensional information of objects made of materials, for example, materials such as light absorption, light reflection and transparency are difficult to generate accurate point clouds by using a structured light three-dimensional camera, so that finally obtained object point clouds have cavities at positions made of the materials, and difficulties may be caused to subsequent operations, for example, the real-time planning of a polishing path in the production process may cause the planned polishing path to be interfered because the point clouds have cavities.

Disclosure of Invention

In order to solve the technical problem that a point cloud picture acquired by a structured light three-dimensional camera has a hole, the embodiment of the invention provides a point cloud picture repairing method and device, electronic equipment and a storage medium.

The technical scheme of the embodiment of the invention is realized as follows:

the embodiment of the invention provides a method for repairing a point cloud picture, which comprises the following steps:

acquiring a defect point cloud picture obtained by shooting an object and a digital-analog point cloud picture preset by a system;

converting the digital-to-analog cloud picture into a coordinate system where the incomplete point cloud picture is located to obtain a conversion point cloud picture;

calculating a displacement deviation vector of the transformation point cloud picture deviating from the defect cloud picture in a coordinate system of the defect cloud picture;

moving the transformed point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector;

judging whether the shifted conversion cloud picture is overlapped with the incomplete point cloud picture or not;

under the condition that the moved conversion cloud picture is overlapped with the incomplete point cloud picture, determining incomplete point cloud blocks of the incomplete point cloud picture relative to the conversion cloud picture in the conversion cloud picture;

and filling the incomplete point cloud picture by using the point cloud blocks to generate a restored point cloud picture of the object.

In the above scheme, converting the digital-to-analog cloud picture into a coordinate system in which the incomplete cloud picture is located, and obtaining the converted cloud picture includes:

performing point cloud registration on the incomplete point cloud picture and the digital-analog point cloud picture to obtain a conversion matrix for converting the digital-analog point cloud picture into a coordinate system where the incomplete point cloud picture is located;

and converting the digital-to-analog cloud picture into a coordinate system where the incomplete point cloud picture is located according to the conversion matrix to obtain a conversion point cloud picture.

In the above scheme, calculating the displacement deviation vector of the transformed cloud picture from the incomplete cloud picture includes:

respectively determining a paired point cloud which is closest to each point cloud in the conversion point cloud picture in Euclidean distance among all point clouds in the incomplete point cloud picture for each point cloud in the conversion point cloud picture, and pairing each point cloud in the conversion point cloud picture with the paired point cloud in the corresponding incomplete point cloud picture;

according to the matching condition, acquiring first point clouds matched with the point clouds in the conversion point cloud image from all the point clouds in the incomplete point cloud image, and combining the first point clouds into a first point cloud set;

determining the centroid of the point cloud outline formed by the first point cloud set according to the first point cloud set;

acquiring second point clouds closest to the first point cloud in all point clouds of the converted point cloud picture, and combining the second point clouds into a second point cloud set;

determining a first vector set consisting of all second point clouds in the second point cloud set and the centroid;

and calculating a displacement deviation vector of the converted point cloud picture deviating from the incomplete point cloud picture according to the first vector set.

In the above scheme, calculating the displacement deviation vector of the transformed cloud image from the incomplete cloud image according to the first vector set includes:

calculating a displacement deviation vector of the transformed point cloud picture deviating from the incomplete point cloud picture according to the first vector set by using the following formula (1):

formula (1)

Wherein the content of the first and second substances,the displacement deviation vector is represented by a vector of displacement deviations,representing an ith first vector in the first set of vectors;representing an ith second point cloud in the second point cloud set;representing a centroid;representing the total number of all first vectors in the first set of vectors.

In the above scheme, the determining whether the shifted cloud images of the conversion points coincide with the cloud images of the incomplete points includes:

acquiring a third point cloud closest to the Euclidean distance of the first point cloud from all point clouds of the moved conversion point cloud picture, and combining the third point clouds into a third point cloud set;

respectively determining Euclidean point cloud distances between all third point clouds in the third point cloud set and the corresponding first point clouds;

determining the average Euclidean point cloud distance of a third point cloud set according to all the Euclidean point cloud distances;

and judging whether the moved conversion point cloud picture is overlapped with the incomplete point cloud picture or not by utilizing the average Euclidean point cloud distance and the preset point cloud distance.

In the above scheme, determining the average european point cloud distance of the third point cloud set according to all the european point cloud distances includes:

determining the average Euclidean point cloud distance of the third point cloud set according to all the Euclidean point cloud distances by using the following formula (2):

formula (2)

Wherein the content of the first and second substances,representing an average Euclidean point cloud distance;representing the Euclidean point cloud distance between the ith third point cloud in the third point cloud set and the corresponding first point cloud;representing the ith third point cloud in the third point cloud set;representing a first point cloud corresponding to the ith third point cloud in the third point cloud set;representing the total number of all third point clouds in the third point cloud set.

In the above scheme, after determining whether the moved converted cloud image of the point is overlapped with the cloud image of the incomplete point by using the average european point cloud distance and the preset point cloud distance, the method further comprises:

after judging that the shifted conversion point cloud picture is not superposed with the incomplete point cloud picture, recalculating the displacement deviation vector of the shifted conversion point cloud picture deviating from the incomplete point cloud picture; continuously moving the transformed point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector; and continuously judging whether the moved conversion point cloud picture is overlapped with the incomplete point cloud picture or not; and circulating until the moved conversion cloud picture is superposed with the incomplete cloud picture or until the moving times reach the preset times.

In the above scheme, determining the incomplete point cloud picture relative to the transformed point cloud picture in the transformed point cloud picture includes:

determining a first point cloud outline formed by all third point clouds in the third point cloud set according to the third point cloud set;

acquiring a point cloud block with a point cloud outline as a first point cloud outline from the moved conversion point cloud picture;

and taking the point cloud block with the point cloud outline as the first point cloud outline as the incomplete point cloud block of the incomplete point cloud picture relative to the converted point cloud picture.

The embodiment of the invention also provides a point cloud picture repairing device, which comprises:

the acquisition module is used for acquiring a defect cloud picture obtained by shooting an object and a digital-analog point cloud picture preset by the system;

the conversion module is used for converting the digital-to-analog cloud picture into a coordinate system where the incomplete point cloud picture is located to obtain a conversion point cloud picture;

the calculation module is used for calculating a displacement deviation vector of the conversion point cloud picture deviating from the incomplete point cloud picture in a coordinate system where the incomplete point cloud picture is located;

the moving module is used for moving the converted point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector;

the judging module is used for judging whether the moved conversion point cloud picture is superposed with the incomplete point cloud picture or not;

the determining module is used for determining incomplete point cloud blocks of the incomplete point cloud picture relative to the converted point cloud picture in the converted point cloud picture under the condition that the moved converted point cloud picture is overlapped with the incomplete point cloud picture;

and the generating module is used for filling the incomplete point cloud picture by using the point cloud blocks to generate a repaired point cloud picture of the object.

An embodiment of the present invention further provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor; wherein the content of the first and second substances,

the processor is adapted to perform the steps of any of the methods described above when running the computer program.

The embodiment of the invention also provides a storage medium, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of any one of the methods are realized.

According to the method and the device for repairing the point cloud picture, the electronic equipment and the storage medium, the incomplete point cloud picture obtained by shooting the object and a digital-analog point cloud picture preset by a system are obtained; converting the digital-to-analog cloud picture into a coordinate system where the incomplete point cloud picture is located to obtain a conversion point cloud picture; calculating a displacement deviation vector of the transformation point cloud picture deviating from the defect cloud picture in a coordinate system of the defect cloud picture; moving the transformed point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector; judging whether the shifted conversion cloud picture is overlapped with the incomplete point cloud picture or not; under the condition that the moved conversion cloud picture is overlapped with the incomplete point cloud picture, determining incomplete point cloud blocks of the incomplete point cloud picture relative to the conversion cloud picture in the conversion cloud picture; and filling the incomplete point cloud picture by using the point cloud blocks to generate a restored point cloud picture of the object. The scheme provided by the invention can be used for repairing the incomplete part in the cloud point image of the complex object, has high repairing accuracy and is closer to the appearance of the object.

Drawings

FIG. 1 is a schematic flow chart illustrating a method for repairing a cloud point map according to an embodiment of the present invention;

FIG. 2 is a schematic diagram illustrating a point cloud hole filling process according to an embodiment of the present invention;

FIG. 3 is a schematic diagram illustrating point cloud matching according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a cloud repairing apparatus according to an embodiment of the present invention;

fig. 5 is an internal structural diagram of a computer device according to an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples.

Aiming at the technical problem that a point cloud picture acquired by a structured light three-dimensional camera has a hole, the technical scheme mainly adopted at present comprises the following steps:

(1) and repairing the hole through the overall topological relation of the point cloud.

The method is mainly characterized in that for objects with regular and symmetrical shapes, the point cloud to be filled in the hollow hole is reasonably inferred through the overall structure of the point cloud picture, the curvature and other characteristics around the hollow hole, and therefore repairing is conducted. However, this method is easy to repair the cavity into a completely different shape from the actual object for the object with a complicated shape.

(2) And repairing the hole through a neural network trained through deep learning.

The method is mainly used for repairing the holes in the point cloud image through the neural network of deep learning training, but the method can not be applied to repairing the holes in the point cloud image of the object with complex appearance and is only applied to repairing the holes in the regular object.

Based on this, the embodiment provides a point cloud hole filling method based on an object digital model, which automatically searches for a hole after registering an object digital model point cloud image and a point cloud image of an actual object, and accurately fills a hole in the object point cloud image under the condition of considering the tolerance of the object.

Specifically, an embodiment of the present invention provides a method for repairing a cloud point map, as shown in fig. 1, where the method includes:

step 101: acquiring a defect point cloud picture obtained by shooting an object and a digital-analog point cloud picture preset by a system;

step 102: converting the digital-analog cloud picture into a coordinate system where the incomplete point cloud picture is located to obtain a conversion point cloud picture;

step 103: calculating a displacement deviation vector of the conversion point cloud picture deviating from the defect cloud picture in a coordinate system of the defect cloud picture;

step 104: moving the conversion point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector;

step 105: judging whether the shifted conversion point cloud picture is overlapped with the incomplete point cloud picture or not;

step 106: under the condition that the shifted conversion cloud picture is overlapped with the incomplete point cloud picture, determining incomplete point cloud blocks of the incomplete point cloud picture relative to the conversion cloud picture in the conversion cloud picture;

step 107: and filling the incomplete cloud picture by using the cloud point blocks to generate a repaired cloud picture of the object.

Specifically, the object can be shot by using a structured light three-dimensional camera to obtain a shot incomplete cloud picture. Because the object may include some special materials (such as light absorbing, light reflecting, transparent materials, etc.), the cloud point image obtained by the structured light three-dimensional camera is incomplete, and has defects such as voids, which cannot be used in practical applications.

Further, when manufacturing the object, a general manufacturer may perform manufacturing according to the 3D digital-to-analog file of the object. Therefore, in this embodiment, the point cloud image corresponding to the 3D data file according to which the object is produced and manufactured can be used as the digital-analog point cloud image preset by the system.

In practical application, in an embodiment, the converting the digital-to-analog cloud picture into a coordinate system where the incomplete cloud picture is located to obtain a converted cloud picture includes:

performing point cloud registration on the incomplete point cloud picture and the digital-analog point cloud picture to obtain a conversion matrix for converting the digital-analog point cloud picture to a coordinate system where the incomplete point cloud picture is located;

and converting the digital-analog cloud picture into a coordinate system where the incomplete point cloud picture is located according to the conversion matrix to obtain a conversion point cloud picture.

Here, point cloud registration refers to obtaining a transformation matrix between two point cloud images, by which a source point cloud can be transformed into the same coordinate system as a target point cloud, and the transformation process can be expressed by the following formula (3):

（3）

wherein the content of the first and second substances,a point cloud of a source point is represented,representing the source point cloud changing to the same coordinate system as the target point cloud,a change matrix is represented.

Here, the first and second liquid crystal display panels are,andthe method is characterized in that a pair of corresponding point clouds is formed by a source point cloud and a point cloud which is changed to a target point cloud in the same coordinate system, and the aim of registration is to obtain a transformation matrix in the point cloud.

In practical application, in the process of manufacturing an object according to a 3D digital-analog file, due to the manufacturing process, the manufactured object may have a tolerance, so that even if the incomplete cloud image of the manufactured object and the digital-analog cloud image of the designed 3D digital-analog file are converted into the same coordinate system, the incomplete cloud image of the object and the digital-analog cloud image of the designed 3D digital-analog file may not completely coincide with each other, and a displacement deviation may exist. Therefore, before repairing the incomplete cloud picture, displacement deviation caused by tolerance between the two cloud pictures needs to be eliminated, the two cloud pictures are overlapped, and then the incomplete cloud picture is repaired according to the overlapped conversion cloud picture.

In an embodiment, the calculating a displacement deviation vector of the conversion point cloud image from the incomplete point cloud image includes:

respectively determining a paired point cloud which is closest to each point cloud in the conversion point cloud picture in Euclidean distance among all point clouds in the incomplete point cloud picture, and pairing each point cloud in the conversion point cloud picture with the corresponding paired point cloud in the incomplete point cloud picture;

according to the matching condition, acquiring first point clouds matched with the point clouds in the conversion point cloud image from all the point clouds in the incomplete point cloud image, and combining the first point clouds into a first point cloud set;

determining the centroid of a point cloud outline formed by the first point cloud set according to the first point cloud set;

acquiring second point clouds closest to the first point cloud in all point clouds of the converted point cloud image, and combining the second point clouds into a second point cloud set;

determining a first vector set consisting of all second point clouds in the second point cloud set and the centroid;

and calculating a displacement deviation vector of the converted point cloud picture deviating from the incomplete point cloud picture according to the first vector set.

In practical application, because the cloud point densities of the incomplete point cloud picture and the converted point cloud picture are different, a one-to-many situation occurs in the process of pairing the incomplete point cloud picture and the converted point cloud picture. Because the incomplete point cloud picture is obtained by shooting an object in a real object through the three-dimensional structured light camera, the incomplete point cloud picture has defects such as cavities and the like, and the density of the point cloud is low; the conversion point cloud picture is obtained by performing coordinate conversion on a digital-to-analog point cloud picture preset by a system, and the density of the point cloud is high, so that when the incomplete point cloud picture and the conversion point cloud picture are matched, the condition that one point cloud in the incomplete point cloud picture corresponds to a plurality of point clouds in the conversion point cloud picture can occur.

Specifically, in this embodiment, the manner of determining the first point cloud set is as follows: and for each point cloud in the conversion point cloud picture, searching a matched point cloud which is closest to the point cloud in an Euclidean distance in the incomplete point cloud picture, and matching each point cloud in the conversion point cloud picture with the corresponding matched point cloud. Here, it can be seen from the above analysis that, since the cloud point density of the incomplete cloud image is small and the cloud point density of the converted cloud image is large, one of the incomplete cloud images may be paired with a plurality of cloud points in the converted cloud image. At the moment, point clouds of a plurality of point clouds in the pairing conversion point cloud picture in the incomplete point cloud picture are used as first point clouds, and a set of all first point cloud combinations is used as a first point cloud set. The centroid corresponding to the first point cloud set can be determined by all the first point clouds in the first point cloud set.

After determining the centroid corresponding to the first point cloud set, a second point cloud closest to the first point cloud in the euclidean distance of all point clouds in the converted point cloud image may be obtained, and a set of all second point cloud combinations is used as the second point cloud set. And forming a three-dimensional vector, namely a first vector set, by the centroid and all second point clouds in the second point cloud set, and determining a displacement deviation vector of the converted point cloud image from the incomplete point cloud image according to the first vector set.

Specifically, the displacement deviation vector of the transformed cloud image from the incomplete cloud image can be calculated according to the first vector set by using the following formula (1):

formula (1)

Wherein the content of the first and second substances,the displacement deviation vector is represented by a vector of displacement deviations,representing an ith first vector in the first set of vectors;representing an ith second point cloud in the second point cloud set;representing a centroid;representing the total number of all first vectors in the first set of vectors.

After obtaining the displacement deviation vector of the converted cloud picture deviating from the incomplete cloud picture, each point cloud in the converted cloud picture can be subjected to displacement movement under the coordinate system of the incomplete cloud picture according to the displacement deviation vector, so that the converted cloud picture and the incomplete cloud picture can be superposed.

Further, in an embodiment, the determining whether the shifted cloud cover of the conversion point coincides with the cloud cover of the incomplete point includes:

acquiring a third point cloud closest to the first point cloud in all the point clouds of the moved conversion point cloud picture, and combining the third point clouds into a third point cloud set;

respectively determining Euclidean point cloud distances between all third point clouds in the third point cloud set and the corresponding first point clouds;

determining the average Euclidean point cloud distance of the third point cloud set according to all the Euclidean point cloud distances;

and judging whether the moved converted point cloud picture is overlapped with the incomplete point cloud picture or not by utilizing the average Euclidean point cloud distance and a preset point cloud distance.

In practical application, in this embodiment, an euclidean distance between each third point cloud and a corresponding first point cloud closest to the euclidean distance of the third point cloud is obtained, and then an average euclidean point cloud distance of the third point cloud set is determined by using the euclidean distance corresponding to each third point cloud.

Specifically, if the first point cloud isThe third point cloud corresponding to the first point cloud isThen the Euclidean point cloud distance between the first point cloud and the corresponding third point cloud is。

Further, an average euclidean point cloud distance of the third point cloud set may be determined using the following formula (2) according to all the euclidean point cloud distances:

formula (2)

Wherein the content of the first and second substances,representing an average Euclidean point cloud distance;representing the Euclidean point cloud distance between the ith third point cloud in the third point cloud set and the corresponding first point cloud;representing the ith third point cloud in the third point cloud set;representing a first point cloud corresponding to the ith third point cloud in the third point cloud set;representing the total number of all third point clouds in the third point cloud set.

After the average Euclidean point cloud distance of the third point cloud set is obtained, the average Euclidean point cloud distance of the third point cloud set can be compared with a preset point cloud distance, and whether the moved conversion point cloud picture is overlapped with the incomplete point cloud picture or not is judged. In particular, the amount of the solvent to be used,

judging whether the average Euclidean point cloud distance of the third point cloud set is smaller than a preset point cloud distance or not, and judging that the moved conversion point cloud picture is overlapped with the incomplete point cloud picture when the average Euclidean point cloud distance of the third point cloud set is smaller than the preset point cloud distance; and when the average Euclidean point cloud distance of the third point cloud set is greater than or equal to the preset point cloud distance, judging that the moved conversion point cloud picture is not superposed with the incomplete point cloud picture.

In addition, due to tolerance and other reasons, after the conversion cloud picture is moved once, the moved conversion cloud picture and the incomplete cloud picture cannot be overlapped, and at this time, the conversion cloud picture needs to be moved again.

Based on this, in an embodiment, after determining whether the moved converted cloud image is overlapped with the incomplete cloud image by using the average european point cloud distance and a preset point cloud distance, the method further includes:

after judging that the shifted conversion point cloud picture is not superposed with the incomplete point cloud picture, recalculating a displacement deviation vector of the shifted conversion point cloud picture deviating from the incomplete point cloud picture; continuously moving the conversion point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector; continuously judging whether the moved conversion point cloud picture is overlapped with the incomplete point cloud picture or not; and circulating until the moved conversion point cloud picture is superposed with the incomplete point cloud picture or the moving times reach the preset times.

Here, by setting the preset number of times, the situation that the moved conversion point cloud picture and the incomplete point cloud picture cannot be overlapped for infinite times all the time due to tolerance in some cases is prevented.

Further, in an embodiment, the determining, in the converted point cloud image, the cloud point blocks of the incomplete point cloud image relative to the converted point cloud image includes:

determining a first point cloud outline formed by all third point clouds in the third point cloud set according to the third point cloud set;

acquiring a point cloud block with a point cloud outline as the first point cloud outline from the moved conversion point cloud image;

and taking the point cloud block with the point cloud outline as the first point cloud outline as the incomplete point cloud block of the incomplete point cloud picture relative to the converted point cloud picture.

In practical application, a voxel growing method can be used for obtaining the point cloud blocks with the point cloud outlines being the first point cloud outlines from the shifted conversion point cloud images. Here, the voxel growing method is a conventional technical means, and is not described in detail here.

And fusing the incomplete point cloud picture with the incomplete point cloud picture relative to the transformed point cloud picture to obtain the repaired object point cloud picture.

According to the method and the device for repairing the point cloud picture, the electronic equipment and the storage medium, the incomplete point cloud picture obtained by shooting the object and a digital-analog point cloud picture preset by a system are obtained; converting the digital-analog cloud picture into a coordinate system where the incomplete point cloud picture is located to obtain a conversion point cloud picture; calculating a displacement deviation vector of the conversion point cloud picture deviating from the defect cloud picture in a coordinate system of the defect cloud picture; moving the conversion point cloud picture in a coordinate system where the incomplete point cloud picture is located according to the displacement deviation vector; judging whether the shifted conversion point cloud picture is overlapped with the incomplete point cloud picture or not; under the condition that the shifted conversion cloud picture is overlapped with the incomplete point cloud picture, determining incomplete point cloud blocks of the incomplete point cloud picture relative to the conversion cloud picture in the conversion cloud picture; and filling the incomplete cloud picture by using the cloud point blocks to generate a repaired cloud picture of the object. The scheme provided by the invention can be used for repairing the incomplete part in the cloud point image of the complex object, has high repairing accuracy and is closer to the appearance of the object.

The present invention will be described in further detail with reference to the following application examples.

The application embodiment provides a point cloud hole filling method based on an object digital model. Specifically, referring to fig. 2, the point cloud hole filling method includes the following steps:

step 1: inputting object point clouds needing hole filling; then step 2 is executed;

step 2: registering the digital-analog point cloud corresponding to the object; then executing step 3;

and step 3: converting the digital-analog point cloud and the object point cloud to be coincident through a transformation matrix obtained by registration; then executing step 4;

and 4, step 4: searching a cavity needing to be filled; then step 5 is executed;

and 5: finding a point pair of the object cavity outline point cloud and the digital-analog point cloud; then step 6 is executed;

step 6: calculating and adjusting the attitude of the digital-analog point cloud corresponding to the object point cloud hole; then step 7 is executed;

and 7: judging whether the distance error is smaller than a threshold value or not, or whether the iteration number limit is reached;

if the distance error is judged to be smaller than the threshold value or the iteration number limit is reached, executing the step 8; if not, executing the step 5;

and 8: and outputting the object point cloud after hole filling.

The specific above process may be:

process (1): reading an object point cloud picture S (which may be understood as a defective point cloud picture in the above embodiment) to be subjected to hole filling and a digital-analog point cloud picture T (which may be understood as a digital-analog point cloud picture preset by the system in the above embodiment) corresponding to the object point cloud picture S, and also reading a distance threshold D (which may be understood as a preset point cloud distance in the above embodiment) set by a user and an upper limit of iteration times (which may be understood as preset times in the above embodiment);

here, the digital-analog point cloud picture T refers to a point cloud picture corresponding to a 3D digital-analog file according to which an object is produced and manufactured.

Process (2): performing point cloud registration on the object point cloud picture S and the digital-analog point cloud picture T to obtain a transformation matrix M (which can be understood as a transformation matrix in the above embodiment) for transforming the digital-analog point cloud picture T to a coordinate system where the object point cloud picture S is located;

process (3): by passingObtaining a corresponding digital-to-analog cloud picture T' (which can be understood as a conversion point cloud picture in the above embodiment) in the coordinate system of converting the digital-to-analog cloud picture T to the object point cloud picture S;

process (4): and for each point in the digital-analog point cloud picture T', finding a point which is closest to the Euclidean distance in the object point cloud picture S as a corresponding point pair. Because the point cloud density of the object point cloud picture S obtained by the three-dimensional camera is different from the point cloud density of the digital-analog point cloud picture T ', the situation that the point clouds in a plurality of digital-analog point cloud pictures T' or the point clouds in a plurality of object point cloud pictures S correspond to one same point cloud may exist;

process (5): the factor cloud point map T ' has a complete shape, so that when the digital-analog cloud points are not particularly sparse, part of the digital-analog cloud point map T ' lacks a correct corresponding point cloud in the object cloud point map S, resulting in that the point clouds in the digital-analog cloud point maps T ' correspond to the same point cloud on the cavity contour of the object cloud point map S (for example, as shown in fig. 3). According to the property, the point clouds in the object point cloud picture S can be sorted in the order of the number of the point clouds corresponding to the point clouds from large to small (that is, the number of the point clouds corresponding to the point clouds in the object point cloud picture S is sorted in a descending manner), the point clouds of the plurality of point clouds in the digital-to-analog point cloud picture T' corresponding to the object point cloud picture S are taken as elements in the point set P, and a point set P (which can be understood as a first point set in the above embodiment) of the hollow contour points in the object point cloud picture S is obtained (for example, in the point cloud picture shown in fig. 3, the point set P includes point cloud 1 and point cloud 2);

process (6): calculating the centroid C (X) of the point set P of the hollow contour points in the object point cloud picture S_c，Y_c，Z_c）；

Process (7): for each point P in point set P of hollow hole contour points in object point cloud picture S_iFinding the point p in the digital-analog cloud picture T_iEuclidean distance nearest point q_iStoring a point set Q (which may be understood as a second point cloud set in the above-described embodiment) (for example, in the point cloud diagram shown in fig. 3, the point set Q includes a point cloud 3 and a point cloud 4);

process (8): for each point Q in the point set Q_iCalculating each point q_iThree-dimensional vector between C and centroidThen, the adjustment vector of the current adjustment is calculated and obtained through the formula (1)(it can be understood as a displacement offset vector in the above embodiment);

process (9): by adjusting the vectorMoving the position of each point cloud in the digital-analog point cloud picture T';

process (10): clearing the Point set Q, and counting each point P in the Point set P_iBy adjusting the vectorSearching and each point p in the adjusted digital-analog point cloud picture T_iEuclidean distance nearest point q_iThe point set Q is stored and the average distance d (which can be understood as the average euclidean point cloud distance in the above embodiment) is calculated using the above formula (2). If the average distance D is smaller than a distance threshold value D set by a user or the iteration times reach an upper limit time set by the user, jumping to a process (11), otherwise, jumping to a process (8);

process (11): taking the point set Q as a point cloud boundary for filling a point cloud cavity in the digital-analog point cloud picture T ', obtaining a local point cloud block for filling the cavity by a voxel growing method for the digital-analog point cloud picture T ', and fusing the local point cloud block and the object point cloud picture S to obtain an object point cloud S ' after hole filling;

process (12): and outputting the object point cloud S' after hole filling.

The embodiment provides a point cloud hole filling method based on an object digifax, which is characterized in that a hole is automatically searched after the digifax is registered with the point cloud of an actual object, and the iterative adjustment and optimization are carried out on the attitude position of the point cloud filled in a registered hole area by considering the small difference between the object and the digifax caused by the tolerance generated by the manufacturing process, so that the object point cloud image after the hole is filled is closer to the real appearance of the object.

In order to implement the method according to the embodiment of the present invention, an embodiment of the present invention further provides a cloud point map repairing apparatus, as shown in fig. 4, the cloud point map repairing apparatus 400 includes: an acquisition module 401, a conversion module 402, a calculation module 403, a movement module 404, a judgment module 405, a determination module 406 and a generation module 407; wherein the content of the first and second substances,

the acquisition module 401 is configured to acquire a defect cloud picture obtained by shooting an object and a digital-analog cloud picture preset by a system;

a conversion module 402, configured to convert the digital-to-analog cloud image into a coordinate system in which the incomplete cloud image is located, so as to obtain a conversion cloud image;

a calculating module 403, configured to calculate a displacement deviation vector of the transformed cloud image deviating from the incomplete cloud image in a coordinate system where the incomplete cloud image is located;

a moving module 404, configured to move the transformed cloud image in a coordinate system where the incomplete cloud image is located according to the displacement deviation vector;

a judging module 405, configured to judge whether the shifted cloud image of the conversion point is overlapped with the cloud image of the incomplete point;

a determining module 406, configured to determine, in the converted cloud image, a incomplete cloud point block of the incomplete cloud point image relative to the converted cloud point image when the shifted converted cloud point image is overlapped with the incomplete cloud point image;

and the generating module 407 is configured to fill the incomplete cloud image with the point cloud blocks, and generate a repaired point cloud image of the object.

In actual application, the obtaining module 401, the converting module 402, the calculating module 403, the moving module 404, the judging module 405, the determining module 406, and the generating module 407 may be implemented by a processor in the cloud point image patch apparatus.

It should be noted that: the above-mentioned apparatus provided in the above-mentioned embodiment is only exemplified by the division of the above-mentioned program modules when executing, and in practical application, the above-mentioned processing may be distributed to be completed by different program modules according to needs, that is, the internal structure of the terminal is divided into different program modules to complete all or part of the above-mentioned processing. In addition, the apparatus provided by the above embodiment and the method embodiment belong to the same concept, and the specific implementation process thereof is described in the method embodiment and is not described herein again.

To implement the method of the embodiment of the present invention, the embodiment of the present invention further provides a computer program product, where the computer program product includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the steps of the above-described method.

Based on the hardware implementation of the program module, in order to implement the method according to the embodiment of the present invention, an electronic device (computer device) is also provided in the embodiment of the present invention. Specifically, in one embodiment, the computer device may be a terminal, and its internal structure diagram may be as shown in fig. 5. The computer apparatus includes a processor a01, a network interface a02, a display screen a04, an input device a05, and a memory (not shown in the figure) connected through a system bus. Wherein processor a01 of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises an internal memory a03 and a non-volatile storage medium a 06. The nonvolatile storage medium a06 stores an operating system B01 and a computer program B02. The internal memory a03 provides an environment for the operation of the operating system B01 and the computer program B02 in the nonvolatile storage medium a 06. The network interface a02 of the computer device is used for communication with an external terminal through a network connection. The computer program is executed by the processor a01 to implement the method of any of the above embodiments. The display screen a04 of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device a05 of the computer device may be a touch layer covered on the display screen, a button, a trackball or a touch pad arranged on a casing of the computer device, or an external keyboard, a touch pad or a mouse.

Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and capable of running on the processor, and when the processor executes the program, the method according to any one of the embodiments described above is implemented.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory computer readable media (transmyedia) such as modulated data signals and carrier waves.

It will be appreciated that the memory of embodiments of the invention may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. Among them, the nonvolatile Memory may be a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a magnetic random access Memory (FRAM), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical disk, or a Compact Disc Read-Only Memory (CD-ROM); the magnetic surface storage may be disk storage or tape storage. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced DRAM), Synchronous Dynamic Random Access Memory (SLDRAM), Direct Memory (DRmb Access), and Random Access Memory (DRAM). The described memory for embodiments of the present invention is intended to comprise, without being limited to, these and any other suitable types of memory.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.