1. A junk article image data set construction method based on an image generation technology is characterized by comprising the following steps:

1) collecting and sorting a single-target garbage image data set, wherein the images are classified and labeled;

2) carrying out significance detection by using a U2-Net model to obtain a mask image of a single garbage article;

3) automatically detecting a bounding box of a garbage article in a mask image, matting a single garbage article according to the mask image, and then making a background transparent to form a new single-target garbage image data set (called a Crop data set, wherein the image size is the bounding box size of the garbage article);

4) randomly extracting a plurality of images from the Crop data Set to form an image data Set to be pasted_copy；

5) Considering the hiding threshold, Set_copyThe images in the step (a) are randomly pasted on a large pure-color background image to obtain a multi-target junk object synthetic image Syn_iI represents the ith composite image, and simultaneously records a bounding box and a semantic segmentation label of each garbage object to realize automatic labeling, wherein the label conforms to the labeling format of a COCO data set for use;

6) and (5) repeating the step 4 and the step 5 to generate a plurality of synthetic graphs Syn_iAnd labeling, forming a multi-target junk object composite map data set (called Syn data set);

7) collecting a small amount of multi-target real graphs on a production line garbage sorting platform, and regarding the real graphs as target domains, Syn_iRegarding the source domain, performing style conversion by using a CycleGAN model to enable Syn_iThe style characteristics of the system are more approximate to the real assembly line garbage sorting scene, and the converted Cyc is obtained_iForming a Cyc data set;

8) for Cyc_iOptimizing a Syn_iThe Mask image of each garbage article is amplified according to the central point of each garbage article without changing the overall size of the image to obtain a Mask';

9) performing mean filtering on the Mask 'to obtain a Mask' with an edge feathering effect;

10) cyc extraction by Mask_iThereby eliminating Cyc_iThe "white edge" in (1) makes the transition more natural, and the final synthetic picture (called End) is obtained_i) By End_iAnd forming a garbage object image data set, wherein the data set contains a bounding box and semantic segmentation labels.

2. The method for constructing the junk item image data set based on the image generation technology as claimed in claim 1, wherein the step 4) of randomly extracting comprises the following specific steps:by calculating the proportion w of the number of images of each category in the Crop data set to the total number of images_kAnd k represents the kth category. Randomly selecting Set_copyTotal number of classes and total number of images in (1), and then by probability P_k＝w_kSelecting the category of the image to be pasted, and finally randomly extracting the image from the selected category, wherein at least 1 image in each category in the selected category is ensured to be extracted. The data extraction method considers the quantity distribution characteristics of different types, reduces the possibility that the types with small quantity of partial images are selected repeatedly, and can more diversify the finally extracted images.

3. The method for constructing the junk item image data set based on the image generation technology as claimed in claim 1, wherein the specific steps in step 5) are: from Set_copyExtract a picture Image₀Randomly rotating and then digging out a bounding box area where the foreground (single garbage article) is located, and obtaining Image _ r after the background is transparent₀Randomly selecting a paste point, and applying the Image _ r₀Pasted on a large solid background picture. Then from Set_copyExtract another picture Image_pP represents Set_copyThe same method is used to obtain Image _ r from the p-th Image_pIf Image _ r is present_pAnd IoU of all garbage articles stuck on the background image is smaller than a threshold value (such as 0.2), sticking the garbage articles on the background image, otherwise, abandoning the sticking, continuing to randomly select sticking coordinates, continuing to try to stick, abandoning the sticking of the image if 5 times of sticking still fails, and continuing to extract Set_copyThe remaining images in (1). Repeating the above process until the Set is extracted_copyAll the images in the image table generate a multi-target junk object composite map Syn_i. Syn can be adjusted according to needs_iWhile automatically adjusting the label.

4. The method according to claim 1, wherein the junk item image dataset construction method according to the image generation technology is characterized in that the junk items are enlarged according to the center points of the junk items in step 8)The mask image comprises the following specific steps: using semantic segmentation notation to segment the Syn_iThe Mask image is divided into a plurality of Mask images of single garbage articles (each image is called Mask)_jJ denotes Syn_iMiddle j-th garbage article, Mask_jSize and Syn_iThe same). Calculating Mask by using bounding box label_jCoordinate (x) of center point of unique garbage object in the garbage_j，y_j) Selecting scaling alpha as 1.25, and masking_jThe size of the image is enlarged by a factor of α, and the coordinates of the cutting point are calculated:

O_j＝((α-1)x_j，(α-1)y_j) (1)

wherein (x)_j，y_j) Is the coordinate of the center point of the garbage item, alpha is the scaling, O_jIs the Mask after amplification_jThe coordinates of the shear point in (1). With O_jThe enlarged Mask is the shear point at the upper left corner_jShear to Syn_iImages of the same size, resulting in a Mask image (called Mask ') enlarged by the center point of the trash item'_j) To all Mask'_jAnd obtaining Mask' by solving a union set.

5. The method for constructing the image data set of the junk objects based on the image generation technology as claimed in claim 1, wherein the elimination Cyc in step 10)_iThe specific steps of the 'white edge' in (1) are as follows: traverse all pixels of Mask' if gray value gray at point (x, y)_xyWhen it is 0, then End_iFills in the solid background at point (x, y); if the gray value gray at point (x, y)_xy> 230, then at End_iIs filled with Cyc at the point (x, y) of (A)_iRGB values of the corresponding positions; if 0 < gray_xyIf < 230, Cyc is linearly superimposed according to the gray scale_iCorresponding position RGB value and solid background, at End_iIs filled at point (x, y):

wherein c is_xy，e_xyAre each Cyc_i，End_iThe RGB value at the midpoint (x, y), b is the RGB value of the solid background, gray_xyIs the gray value at the midpoint (x, y) of Mask ".

Background

With the continuous integration of artificial intelligence and the environmental protection industry, the household garbage intelligent sorting system gradually shows huge application value, wherein the garbage article multi-target detection technology based on deep learning is a key perception technology of the system. Deep learning depends on a large-scale data set, the data volume of a small-scale data set is too small, a high-performance deep learning model cannot be trained, and the large-scale data set is not easy to construct.

The existing large-scale Image data sets (such as ImageNet, COCO, Open Image) often adopt a construction method (conventional method) of manually acquiring images and manually labeling. However, there are two problems with constructing a garbage object multi-target detection image dataset using conventional methods: 1) in view of the particularity of the garbage articles, the manual shooting and collection of multi-target garbage images (a plurality of garbage articles are in one image) has higher health risks and is not suitable for manual collection; 2) manual labeling belongs to extreme repetitive labor, wastes time and energy, is low in efficiency, and labeling personnel cannot finish labeling work of large-scale data sets in a short time.

Disclosure of Invention

The invention aims to solve the technical problem of providing a junk article image data set construction method which has an automatic marking function and is based on an image generation technology.

The technical scheme adopted by the invention is as follows: a junk article image data set construction method based on an image generation technology comprises the following steps:

1) collecting and sorting a single-target garbage image data set, wherein the images are classified and labeled;

2) carrying out significance detection by using a U2-Net model to obtain a mask image of a single garbage article;

3) automatically detecting a bounding box of a garbage article in a mask image, matting a single garbage article according to the mask image, and then making a background transparent to form a new single-target garbage image data set (called a Crop data set, wherein the image size is the bounding box size of the garbage article);

4)randomly extracting a plurality of images from the Crop data Set to form an image data Set to be pasted_copy：

5) Considering the hiding threshold, Set_copyThe images in the step (a) are randomly pasted on a large pure-color background image to obtain a multi-target junk object synthetic image Syn_iI represents the ith composite image, and simultaneously records a bounding box and a semantic segmentation label of each garbage object to realize automatic labeling, wherein the label conforms to the labeling format of a COCO data set for use;

6) and (5) repeating the step 4 and the step 5 to generate a plurality of synthetic graphs Syn_iAnd labeling, forming a multi-target junk object composite map data set (called Syn data set);

7) collecting a small amount of multi-target real graphs on a production line garbage sorting platform, and regarding the real graphs as target domains, Syn_iRegarding the source domain, performing style conversion by using a CycleGAN model to enable Syn_iThe style characteristics of the system are more approximate to the real assembly line garbage sorting scene, and the converted Cyc is obtained_iForming a Cyc data set;

8) for Cyc_iOptimizing a Syn_iThe Mask image of each garbage article is amplified according to the central point of each garbage article without changing the overall size of the image to obtain a Mask';

9) performing mean filtering on the Mask 'to obtain a Mask' with an edge feathering effect;

10) cyc extraction by Mask_iThereby eliminating Cyc_iThe "white edge" in (1) makes the transition more natural, and the final synthetic picture (called End) is obtained_i) By End_iAnd forming a garbage object image data set, wherein the data set contains a bounding box and semantic segmentation labels.

The specific steps of random extraction in step 4 are: by calculating the proportion w of the number of images of each category in the Crop data set to the total number of images_kAnd k represents the kth category. Randomly selecting Set_copyTotal number of classes and total number of images in (1), and then by probability P_k＝w_kSelecting the category of the image to be pasted, and finally randomly extracting the image from the selected categoryIt is guaranteed that at least 1 image in each of the selected categories is drawn. The data extraction method considers the quantity distribution characteristics of different types, reduces the possibility that the types with small quantity of partial images are selected repeatedly, and can more diversify the finally extracted images.

The specific steps in the step 5 are as follows: from Set_copyExtract a picture Image₀Randomly rotating and then digging out a bounding box area where the foreground (single garbage article) is located, and obtaining Image _ r after the background is transparent₀Randomly selecting a paste point, and applying the Image _ r₀Pasted on a large solid background picture. Then from Set_copyExtract another picture Image_pP represents Set_copyThe same method is used to obtain Image _ r from the p-th Image_pIf Image _ r is present_pAnd IoU of all garbage articles stuck on the background image is smaller than a threshold value (such as 0.2), sticking the garbage articles on the background image, otherwise, abandoning the sticking, continuing to randomly select sticking coordinates, continuing to try to stick, abandoning the sticking of the image if 5 times of sticking still fails, and continuing to extract Set_copyThe remaining images in (1). Repeating the above process until the Set is extracted_copyAll the images in the image table generate a multi-target junk object composite map Syn_i. Syn can be adjusted according to needs_iWhile automatically adjusting the label.

The specific steps of amplifying the mask image of each garbage article according to the central point of each garbage article in the step 8 are as follows: using semantic segmentation notation to segment the Syn_iThe Mask image is divided into a plurality of Mask images of single garbage articles (each image is called Mask)_jJ denotes Syn_iMiddle j-th garbage article, Mask_jSize and Syn_iThe same). Calculating Mask by using bounding box label_jCoordinate (x) of center point of unique garbage object in the garbage_j，y_j) Selecting scaling alpha as 1.25, and masking_jThe size of the image is enlarged by a factor of α, and the coordinates of the cutting point are calculated:

O_j＝((α-1)x_j，(α-1)y_j) (1)

wherein (x)_j，y_j) Is the coordinate of the center point of the garbage item, alpha is the scaling, O_jIs the Mask after amplification_jThe coordinates of the shear point in (1). With O_jThe enlarged Mask is the shear point at the upper left corner_jShear to Syn_iImages of the same size, resulting in a Mask image (called Mask ') enlarged by the center point of the trash item'_j) To all Mask'_jAnd obtaining Mask' by solving a union set.

Elimination of Cyc as described in step 10_iThe specific steps of the 'white edge' in (1) are as follows: traverse all pixels of Mask' if gray value gray at point (x, y)_xyWhen it is 0, then End_iFills in the solid background at point (x, y); if the gray value gray at point (x, y)_xy> 230, then at End_iIs filled with Cyc at the point (x, y) of (A)_iRGB values of the corresponding positions; if 0 < gray_xyIf < 230, Cyc is linearly superimposed according to the gray scale_iCorresponding position RGB value and solid background, at End_iIs filled at point (x, y):

wherein c is_xy，e_xyAre each Cyc_i，End_iThe RGB value at the midpoint (x, y), b is the RGB value of the solid background, gray_xyIs the gray value at the midpoint (x, y) of Mask ".

According to the junk article image data set construction method based on the image generation technology, complex data acquisition and labeling processes are automatically completed by a computer, the construction efficiency of the data set can be remarkably improved, and the labor cost is reduced. The invention has the following advantages:

1) according to the invention, significance detection is carried out through the U2-Net model, and then matting is carried out according to the sizes of the mask image and the bounding box, so that the bounding box and semantic segmentation labels of a single garbage article can be obtained, and the algorithm can efficiently and accurately carry out End detection on the object_iAll the garbage articles in the garbage can be automatically marked, so that the manual marking cost is greatly reduced;

2) the invention providesGenerating a multi-target image, Syn_iThe total quantity, the category number and the covering degree of the garbage articles are controllable, the position and the rotating angle are not controllable, the determinacy and the randomness are considered, the control problem of the multi-target image generation process is well solved, meanwhile, the method can avoid manual shooting and collecting the multi-target garbage articles, manpower and material resources can be greatly saved, and the hidden health trouble is avoided;

3) the invention provides a Syn pair through a cycleGAN model_iStyle conversion is carried out, and style characteristics of folds, dirt, shadows, light changes and the like of garbage articles in a real scene are introduced into Syn_iIn, make Syn_iThe method is more vivid, and the performance of the target detection model can be improved by the training set which is closer to a real scene, and in addition, the method is also suitable for constructing image data sets in other fields, so the method has wide application prospect in the field of data set construction;

4) the method for magnifying the mask image according to the fixed points does not change the size of the image, only relates to the integral magnification and cutting of the image in the whole process, has simple and efficient algorithm, and is also suitable for the color image, so the method has wide application prospect in the field of image scaling.

Drawings

FIG. 1 is a flow chart of a junk image dataset construction method based on an image generation technique of the present invention;

FIG. 2 is a summary view of a single target spam image dataset gathered and collated in an example of the invention;

FIG. 3 is an input image (single target spam image) in an example of the invention;

FIG. 4 is a diagram of the results of saliency detection for an input image in an example of the present invention;

FIG. 5 is a schematic of a Crop data set in an example of the invention;

FIG. 6 is a schematic diagram of a Syn data set in an example of the present invention;

FIG. 7 is a diagram of a small number of multiple targets actually collected at a production-line garbage sorting platform in an example embodiment of the present invention;

FIG. 8 is a schematic representation of a Cyc data set in an example of the invention;

FIG. 9 is a Syn in an example of the present invention_i；

FIG. 10 shows Cyc in an example of the present invention_i；

FIG. 11 is the final composite map End in an example of the present invention_i；

FIG. 12 is a Syn in an embodiment of the present invention_iThe mask image of (1);

FIG. 13 is a Mask' image in an example of the present invention;

FIG. 14 is a Mask "image in an example of the present invention;

FIG. 15 is a schematic illustration of a constructed garbage item image dataset in an example of the present invention;

FIG. 16 is a schematic diagram of a bounding box and semantic segmentation labels of a constructed garbage item image dataset according to an embodiment of the present invention.

Detailed Description

The following describes a junk image data set construction method based on an image generation technology in detail with reference to examples and the accompanying drawings.

1) Collecting and sorting a single-target garbage image data set, wherein all images are classified and labeled as shown in FIG. 2;

2) reading the single-target garbage image shown in the figure 3, and performing significance detection by using a U2-Net model to obtain a mask image of a single garbage article, as shown in figure 4;

3) automatically detecting a bounding box of a garbage article in a mask image, matting a single garbage article according to the mask image, and then transparentizing a background to form a new single-target garbage image data set (called a Crop data set, wherein the image size is the bounding box size of the garbage article), as shown in fig. 5;

4) randomly extracting a plurality of images from the Crop data Set to form an image data Set to be pasted_copyThe method comprises the following specific steps:

calculating the proportion w of the number of the images of each category in the Crop data set to the total number of the images_kK represents the kth category, and Set is randomly selected_copyTotal number of classes and total number of images in (1), and then by probability P_k＝w_kSelecting the adhesive to be adheredPasting the image categories, and finally randomly drawing the images from the selected categories, wherein at least 1 image in each category of the selected categories is guaranteed to be drawn.

5) Considering the hiding threshold, Set_copyThe images in the step (a) are randomly pasted on a large pure-color background image to obtain a multi-target junk object synthetic image Syn_iAnd i represents the ith composite image, and simultaneously records a bounding box and a semantic segmentation label of each garbage article to realize automatic labeling, wherein the label conforms to the labeling format of a COCO data set for use, and the specific steps are as follows:

from Set_copyExtract a picture Image₀Randomly rotating and then digging out a bounding box area where the foreground (single garbage article) is located, and obtaining Image _ r after the background is transparent₀Randomly selecting a paste point, and applying the Image _ r₀Pasted on a large solid background picture. Then from Set_copyExtract another picture Image_pP represents Set_copyThe same method is used to obtain Image _ r from the p-th Image_pIf Image _ r is present_pAnd IoU of all garbage articles pasted on the background map is smaller than a threshold value (such as 0.2), pasting the garbage articles on the background map and recording bounding box and semantic segmentation labels of the garbage articles, otherwise, abandoning the pasting, continuing to randomly select pasting coordinates, continuing to try pasting, abandoning the pasting of the image if 5 attempts still fail, continuing to extract Set_copyThe remaining images in (1). Repeating the above process until the Set is extracted_copyAll the images in the image table generate a multi-target junk object composite map Syn_i. Syn can be adjusted according to needs_iWhile automatically adjusting the label.

6) And (5) repeating the step 4 and the step 5 to generate a plurality of synthetic graphs Syn_iAnd annotations, forming a multi-target junk item composite map dataset (called Syn dataset), as shown in FIG. 6;

7) a small number of multi-target real maps are collected at the production-line garbage sorting platform, as shown in fig. 7. Considering the real graph as the target domain, Syn_iRegarding the source domain, performing style conversion by using a CycleGAN model to enable Syn_iThe style characteristics ofObtaining the converted Cyc according to the nearly real assembly line garbage sorting scene_iThe Cyc data set is constructed as shown in FIG. 8.

Due to the limitation of manpower and material resources, the quantity of the manually collected real images is small, so that the CycleGAN is overfitted to a certain degree, and the Cyc is caused_iThe hard 'white edge' appears, the authenticity of the synthetic image is seriously reduced, and the 'white edge' is eliminated in the following steps, so that the transition is more natural.

8) For Cyc_iOptimizing a Syn_iThe Mask image of each garbage article is amplified according to the central point of each garbage article without changing the overall size of the image to obtain Mask', and the specific steps are as follows:

for a Syn as shown in FIG. 9_iThe mask image is shown in FIG. 12, and the Cyc image is the CycleGAN-converted image_iAs shown in fig. 10. Using semantic segmentation notation to segment the Syn_iThe Mask image is divided into a plurality of Mask images of single garbage articles (each image is called Mask)_jJ denotes Syn_iMiddle j-th garbage article, Mask_jSize and Syn_iThe same). Calculating Mask by using bounding box label_jCoordinate (x) of center point of unique garbage object in the garbage_j，y_j) Selecting scaling alpha as 1.25, and masking_jThe size of the image is enlarged by a factor of α, and the coordinates of the cutting point are calculated:

O_j＝((α-1)x_j，(α-1)y_j) (1)

wherein (x)_j，y_j) Is the coordinate of the center point of the garbage item, alpha is the scaling, O_jIs the Mask after amplification_jThe coordinates of the shear point in (1). With O_jThe enlarged Mask is the shear point at the upper left corner_jShear to Syn_iImages of the same size, resulting in a Mask image (called Mask ') enlarged by the center point of the trash item'_j) To all Mask'_jThe union is performed to obtain Mask' as shown in FIG. 13.

9) Performing mean filtering on Mask' to obtain a Mask "with edge feathering effect, as shown in fig. 14;

10) cyc extraction by Mask_iThereby eliminating Cyc_iThe "white edge" in (1) makes the transition more natural, and the final synthetic picture (called End) is obtained_i) By End_iForming a garbage article image data set, wherein the data set contains a bounding box and semantic segmentation labels, and the method specifically comprises the following steps:

traverse all pixels of Mask' if gray value gray at point (x, y)_xyWhen it is 0, then End_iFills in the solid background at point (x, y); if the gray value gray at point (x, y)_xy> 230, then at End_iIs filled with Cyc at the point (x, y) of (A)_iRGB values of the corresponding positions; if 0 < gray_xyIf < 230, Cyc is linearly superimposed according to the gray scale_iCorresponding position RGB value and solid background, at End_iIs filled at point (x, y):

wherein c is_xy，e_xyAre each Cyc_i，End_iThe RGB value at the midpoint (x, y), b is the RGB value of the solid background, gray_xyIs the gray value at the midpoint (x, y) of Mask ". The resulting final composite map End_iAs shown in fig. 11.

By End_iThe constructed garbage object image data set is shown in fig. 15, and its bounding box and semantic segmentation labels are shown in fig. 16.