Method for detecting change of remote building by monitoring camera
1. A method for detecting the change of a building with a monitoring camera in a long distance is characterized by comprising the following steps,
building image data of the same place and different time phases are collected by using a monitoring camera, and a building low-resolution image data set is established;
carrying out bilinear interpolation processing on the front time phase image and the rear time phase image to obtain two images with the same resolution;
registering the front time phase image and the rear time phase image in a characteristic point matching mode to obtain two images corresponding to the building position;
performing hyper-resolution reconstruction on the front and rear time phase images by using a generated countermeasure network to generate a high-resolution front time phase image and a high-resolution rear time phase image;
generating a low-resolution variable image by adopting an image difference method aiming at the generated high-resolution front time phase image and high-resolution rear time phase image;
performing hyper-resolution reconstruction on the generated low-resolution change image, and performing secondary enhancement on pixels of an area containing building information to generate a high-resolution change image;
detecting the building in the high-resolution change image by using the fast-rcnn target detection network to obtain a high-resolution building change image;
binarizing the high-resolution front time phase image, and performing morphological post-processing on the high-resolution front time phase image to remove speckle noise to obtain a high-resolution front time phase image binary image;
and performing one-to-one pixel weighted fusion on the high-resolution front time phase image binary image and the high-resolution building change image to obtain a building change area result image.
2. The method for remote building change detection by a monitoring camera of claim 1, wherein in using the monitoring camera to acquire building image data at the same location and different time phases to create the building low resolution image data set, the method further comprises,
and setting N cruising points aiming at the installed monitoring camera, acquiring images at each cruising point at a certain time interval, naming the images in a file format easy to read, and establishing a low-resolution image data set of the building.
3. The method for remote building change detection by a surveillance camera as claimed in claim 1, wherein in the bilinear interpolation of the front-phase image and the rear-phase image to obtain two images with the same resolution, the method further comprises,
the bilinear interpolation adopts a resize function in python, wherein the selected interpolation mode is bilinear interpolation, namely linear interpolation is respectively carried out in two directions.
4. The method for remote building change detection by a surveillance camera as claimed in claim 1, wherein in registering the front-phase image and the back-phase image by feature point matching to obtain two images corresponding to the building location, the method further comprises,
the characteristic point matching mode is to use a sift algorithm to extract key points which are not changed along with the change of the external environment in the image.
5. The method for remote building change detection by a surveillance camera of claim 4, wherein in said feature point matching in a manner of extracting keypoints in the image that do not change with changes in the external environment using a sift algorithm, the method further comprises,
the same scene exists in the two images, respective stable points are extracted, and matching points corresponding to each other exist between the stable points.
6. The method for remote building change detection by a surveillance camera of claim 1, wherein in the generation of the high resolution forward-phase image and the high resolution backward-phase image using the hyper-resolution reconstruction of the forward-phase image and the backward-phase image using the generation countermeasure network, the method further comprises,
processing the input early-phase images into (128, 128, 3) -dimensional pictures by bilinear interpolation, and generating (512, 512, 3) -dimensional images by convolution, up-sampling, standardization and deconvolution;
respectively inputting the generated (512, 512, 3) -dimensional image and the original high-resolution picture into a VGG19 network, extracting the feature vectors of the two pictures, and calculating a loss1 value;
inputting the generated (512, 512, 3) -dimensional image into a discrimination network, outputting a 32x32x1 vector, judging each pixel point one by one, and finally mapping the output to a probability value between 0 and 1 by a sigmoid function;
comparing the output probability value with 1, and calculating a loss2 value;
and training the two loss values to finally obtain the optimal high-resolution picture.
7. The method for remote building change detection by a surveillance camera of claim 1, wherein in detecting the building in the high resolution change image using a fast-rcnn object detection network resulting in a high resolution building change image, the method further comprises,
bilinear interpolation is carried out on the length and the width of the input high-resolution front time phase image to be 600;
extracting buildings in the high-resolution change image by using the regional candidate network to form a plurality of suggestion frames, screening the suggestion frames, and calculating the coincidence degree of the suggestion frames and the real frames;
and finally, carrying out regression classification on the screened suggestion boxes to finally obtain a prediction result and a type.
8. The method for remote building change detection by a monitoring camera of claim 7, wherein in extracting the building in the high resolution change image using the area candidate network, forming a plurality of suggestion boxes, and filtering the suggestion boxes, calculating the degree of coincidence between the suggestion boxes and the real boxes, the method further comprises,
if the coincidence degree is greater than 0.5, the suggested frame can be adjusted to be a real frame, so that the suggested frame is reserved and is regarded as a positive sample; if the coincidence degree is between 0.1 and 0.5, the coincidence degree is regarded as a negative sample; if the degree of coincidence is less than 0.1, the suggestion box is ignored.
9. The method for monitoring camera remote building change detection as claimed in claim 1, wherein in binarizing the high resolution forward time phase image and performing morphological post-processing thereon to remove speckle noise and obtain a high resolution forward time phase image binary image, the method further comprises,
and binarizing the high-resolution front time phase image, performing morphological post-processing on the high-resolution front time phase image, performing morphological closing operation to fill holes in a high-resolution front time phase image binary image region, performing morphological opening operation, and removing the high-resolution front time phase image binary image spot noise to obtain a high-resolution front time phase image binary image.
Background
At present, most of building change detection is carried out based on satellite remote sensing data. However, the remote sensing satellite generally forms an orthoscopic image shadow, the data resolution is not high, the satellite return period is long, and the early warning period for finding the change of the building is long.
In terms of the overlooking angle of the remote sensing image, the monitoring camera is usually deployed on a high tower or a high building and can monitor 5-10 kilometers of natural resource change, but when the monitoring camera is used for observing the change of a building, the building picture under the monitoring camera is small, the resolution ratio is low, and the change area cannot be detected.
Disclosure of Invention
The invention aims to provide a method for detecting the change of a building with a monitoring camera in a long distance, and aims to solve the technical problem that a change area cannot be detected due to the fact that a building picture under the monitoring camera is small and the resolution is low when the monitoring camera is used for observing the change of the building in the prior art.
In order to achieve the above object, the present invention provides a method for detecting building change at a remote distance from a monitoring camera, comprising the steps of,
building image data of the same place and different time phases are collected by using a monitoring camera, and a building low-resolution image data set is established;
carrying out bilinear interpolation processing on the front time phase image and the rear time phase image to obtain two images with the same resolution;
registering the front time phase image and the rear time phase image in a characteristic point matching mode to obtain two images corresponding to the building position;
performing hyper-resolution reconstruction on the front and rear time phase images by using a generated countermeasure network to generate a high-resolution front time phase image and a high-resolution rear time phase image;
generating a low-resolution variable image by adopting an image difference method aiming at the generated high-resolution front time phase image and high-resolution rear time phase image;
performing hyper-resolution reconstruction on the generated low-resolution change image, and performing secondary enhancement on pixels of an area containing building information to generate a high-resolution change image;
detecting the building in the high-resolution change image by using the fast-rcnn target detection network to obtain a high-resolution building change image;
binarizing the high-resolution front time phase image, and performing morphological post-processing on the high-resolution front time phase image to remove speckle noise to obtain a high-resolution front time phase image binary image;
and performing one-to-one pixel weighted fusion on the high-resolution front time phase image binary image and the high-resolution building change image to obtain a building change area result image.
Wherein in the step of acquiring building image data of the same place and different time phases by using the monitoring camera to establish a building low-resolution image data set, the method further comprises the steps of,
and setting N cruising points aiming at the installed monitoring camera, acquiring images at each cruising point at a certain time interval, naming the images in a file format easy to read, and establishing a low-resolution image data set of the building.
Wherein, in the step of carrying out bilinear interpolation processing on the front time phase image and the rear time phase image to obtain two images with the same resolution, the method also comprises the following steps,
the bilinear interpolation adopts a resize function in python, wherein the selected interpolation mode is bilinear interpolation, namely linear interpolation is respectively carried out in two directions.
Wherein, in the step of registering the front time phase image and the rear time phase image in a characteristic point matching mode to obtain two images corresponding to the building position, the method also comprises the steps of,
the characteristic point matching mode is to use a sift algorithm to extract key points which are not changed along with the change of the external environment in the image.
Wherein, in the 'the feature point matching mode is to use sift algorithm to extract the key points which are not changed along with the change of the external environment' in the image, the method also comprises the following steps,
the same scene exists in the two images, respective stable points are extracted, and matching points corresponding to each other exist between the stable points.
Wherein, in the 'super-resolution reconstruction of the front-phase image and the back-phase image using the generation countermeasure network to generate the high-resolution front-phase image and the high-resolution back-phase image', the method further comprises,
processing the input early-phase images into (128, 128, 3) -dimensional pictures by bilinear interpolation, and generating (512, 512, 3) -dimensional images by convolution, up-sampling, standardization and deconvolution;
respectively inputting the generated (512, 512, 3) -dimensional image and the original high-resolution picture into a VGG19 network, extracting the feature vectors of the two pictures, and calculating a loss1 value;
inputting the generated (512, 512, 3) -dimensional image into a discrimination network, outputting a 32x32x1 vector, judging each pixel point one by one, and finally mapping the output to a probability value between 0 and 1 by a sigmoid function;
the output probability value is compared with 1, and loss2 value is calculated.
Wherein in the 'detecting the building in the high-resolution change image by using the fast-rcnn target detection network to obtain the high-resolution building change image', the method further comprises the following steps,
bilinear interpolation is carried out on the length and the width of the input high-resolution front time phase image to be 600;
extracting buildings in the high-resolution change image by using the regional candidate network to form a plurality of suggestion frames, screening the suggestion frames, and calculating the coincidence degree of the suggestion frames and the real frames;
and finally, carrying out regression classification on the screened suggestion boxes to finally obtain a prediction result and a type.
Wherein, in the step of extracting buildings in the high-resolution change image by using the regional candidate network, forming a plurality of suggestion frames, screening the suggestion frames, and calculating the coincidence degree of the suggestion frames and the real frames, the method also comprises the steps of,
if the coincidence degree is greater than 0.5, the suggested frame can be adjusted to be a real frame, so that the suggested frame is reserved and is regarded as a positive sample; if the coincidence degree is between 0.1 and 0.5, the coincidence degree is regarded as a negative sample; if the degree of coincidence is less than 0.1, the suggestion box is ignored.
Wherein, in the 'binarizing the high-resolution front time phase image and performing morphological post-processing on the image to remove speckle noise to obtain a high-resolution front time phase image binary image', the method also comprises the following steps,
and binarizing the high-resolution front time phase image, performing morphological post-processing on the high-resolution front time phase image, performing morphological closing operation to fill holes in a high-resolution front time phase image binary image region, performing morphological opening operation, and removing the high-resolution front time phase image binary image spot noise to obtain a high-resolution front time phase image binary image.
The invention relates to a method for detecting remote building change of a monitoring camera, which comprises the steps of carrying out bilinear interpolation processing on a front time phase image and a rear time phase image, registering the front time phase image and the rear time phase image to generate a high-resolution front time phase image and a high-resolution rear time phase image, detecting a building in the high-resolution change image to obtain a high-resolution building change image, carrying out binarization on the high-resolution front time phase image, and the binary image is processed after morphology to remove speckle noise, a high-resolution front time phase image binary image is obtained, the binary image and the high-resolution building change image are subjected to one-to-one pixel weighting fusion to obtain a building change area result image, the size of the target building is improved by converting the low-resolution image into the high-resolution image, and then the change detection of the building is carried out, so that the change detection of the building under the monitoring camera is realized.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a method for remote building change detection with a surveillance camera according to the present invention.
Fig. 2 is a flow chart of the present invention for performing hyper-resolution reconstruction on the front and rear time phase images using the generated countermeasure network to obtain an optimal high resolution picture.
FIG. 3 is a flow chart of the present invention for detecting buildings in high resolution changing images using a fast-rcnn object detection network.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
In the description of the present invention, it is to be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on the orientations or positional relationships illustrated in the drawings, and are used merely for convenience in describing the present invention and for simplicity in description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed in a particular orientation, and be operated, and thus, are not to be construed as limiting the present invention. Further, in the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
Referring to fig. 1 to 3, the present invention provides a method for detecting a change of a building with a remote monitoring camera, comprising the following steps,
s101: setting N cruise points aiming at an installed monitoring camera, collecting images at each cruise point at intervals, naming the images as a file format which is easy to read, and establishing a low-resolution image data set of a building;
s102: carrying out bilinear interpolation processing on the front time phase image and the rear time phase image, wherein the bilinear interpolation adopts a resize function in python, the selected interpolation mode is bilinear interpolation, namely linear interpolation is respectively carried out in two directions to obtain two images with the same resolution;
s103: registering the front time phase image and the rear time phase image in a characteristic point matching mode, wherein the characteristic point matching mode is to extract key points which do not change along with the change of an external environment in the images by using a sift algorithm to obtain two images corresponding to the position of a building, the two images have the same scenery, respective stable points are extracted, and the stable points have corresponding matching points;
s104: performing hyper-resolution reconstruction on the front and rear time phase images by using a generated countermeasure network to generate a high-resolution front time phase image and a high-resolution rear time phase image;
s1041: processing the input early-phase images into (128, 128, 3) -dimensional pictures by bilinear interpolation, and generating (512, 512, 3) -dimensional images by convolution, up-sampling, standardization and deconvolution;
s1042: respectively inputting the generated (512, 512, 3) -dimensional image and the original high-resolution picture into a VGG19 network, extracting the feature vectors of the two pictures, and calculating a loss1 value;
s1043: inputting the generated (512, 512, 3) -dimensional image into a discrimination network, outputting a 32x32x1 vector, judging each pixel point one by one, and finally mapping the output to a probability value between 0 and 1 by a sigmoid function;
s1044: comparing the output probability value with 1, and calculating a loss2 value;
s1045: training the two loss values to finally obtain an optimal high-resolution picture;
s105: generating a low-resolution variable image by adopting an image difference method aiming at the generated high-resolution front time phase image and high-resolution rear time phase image;
s106: performing hyper-resolution reconstruction on the generated low-resolution change image, and performing secondary enhancement on pixels of an area containing building information to generate a high-resolution change image;
s107: detecting the building in the high-resolution change image by using the fast-rcnn target detection network to obtain a high-resolution building change image;
s1071: bilinear interpolation is carried out on the length and the width of the input high-resolution front time phase image to be 600;
s1072: extracting buildings in the high-resolution change image by using the regional candidate network to form a plurality of suggestion frames, screening the suggestion frames, calculating the coincidence degree of the suggestion frames and the real frames, and if the coincidence degree is greater than 0.5, adjusting the suggestion frames into the real frames, so that the suggestion frames are reserved and are regarded as positive samples; if the coincidence degree is between 0.1 and 0.5, the coincidence degree is regarded as a negative sample; if the coincidence degree is less than 0.1, ignoring the suggestion box;
s1073: finally, carrying out regression classification on the screened suggestion boxes to finally obtain a prediction result and a type;
s108: binarizing the high-resolution front time phase image, performing morphological post-processing on the high-resolution front time phase image, performing morphological closing operation to fill holes in a high-resolution front time phase image binary image region, performing morphological opening operation, and removing high-resolution front time phase image binary image spot noise to obtain a high-resolution front time phase image binary image;
s109: and performing one-to-one pixel weighted fusion on the high-resolution front time phase image binary image and the high-resolution building change image to obtain a building change area result image.
In the embodiment, (1) N cruise points are set for the installed monitoring camera, images are collected at each cruise point every a period of time, the images are named as file formats which are easy to read, and a building low-resolution image data set is established;
(2) carrying out bilinear interpolation processing on the front time phase image and the rear time phase image to keep the resolution of the two images consistent;
the bilinear interpolation processing uses a resize function in python, wherein the selected interpolation mode is bilinear interpolation, namely linear interpolation is respectively carried out in two directions. The prior art is used.
(3) Respectively extracting feature points of the high-resolution front time phase image and the high-resolution rear time phase image by using a scale invariant feature transformation method, and corresponding the feature points to the positions of the buildings;
the scale invariant feature transformation method is characterized in that sift operators are used for extracting key points which are not changed along with changes of external environments such as illumination, brightness and the like in images, the two images have the same scenery, then respective stable points are extracted, and the points have corresponding matching points. Sift is an algorithm.
(4) Performing hyper-resolution reconstruction on the front and rear time phase images by using a generated countermeasure network, improving the resolution of the picture, and generating a high-resolution front time phase image and a high-resolution rear time phase image;
the method comprises the following steps of generating a countermeasure network, generating a high-resolution image, and:
4a) processing the input early-phase images into (128, 128, 3) -dimensional pictures by bilinear interpolation, and generating (512, 512, 3) -dimensional images by convolution, up-sampling, standardization and deconvolution;
4b) inputting the generated (512, 512, 3) -dimensional image and the original high-resolution picture into a VGG19 network respectively, extracting feature vectors of the two pictures, and calculating a loss1 value, wherein the loss1 is shown as the following formula (4):
VGG/Ij, representing the jth convolution kernel at layer I of the VGG19 network;
ILRis a low resolution picture;
IHRis a high resolution picture;
w and H are the width and height of the picture;
Wi；jand Hi；jThe dimensions of the various feature maps within the VGG network are described.
The VGG19 is a deep learning framework, and a one-dimensional feature vector is obtained through convolution, pooling, activation function and full connection.
4c) Inputting the generated (512, 512, 3) -dimensional image into a discrimination network, outputting a 32x32x1 vector, judging each pixel point one by one, and finally mapping the output to a probability value between 0 and 1 by a sigmoid function;
the evaluation is performed on each pixel by 32 × 32 — 1024 judges (how to evaluate is the content inside the algorithm, i are not clear here), and then the evaluation results are fused.
Wherein the sigmoid function is:
4d) comparing the output probability value with 1, and calculating a loss2 value, wherein the formula is represented by a loss2 calculation formula;
is the probability that the reconstructed image is a true high resolution image
ILRIs a low resolution picture
4e) And training the two loss values to finally obtain the optimal high-resolution picture.
The training method is to make the obtained loss value successively iterate to make it converge and become smaller.
Aiming at the generated high-resolution front time phase image and high-resolution rear time phase image, generating a low-resolution change image by adopting an image pixel difference method;
transforming the image into an array by python, then making a difference on the array, and transforming the array into a picture form
(6) Performing secondary enhancement on pixels of an area containing building information by adopting a hyper-resolution reconstruction method on the generated low-resolution change image to generate a high-resolution change image;
(7) detecting the building in the high-resolution change image by using the fast-rcnn target detection network to obtain a high-resolution building change image; the fast-rcnn target detection network detection building comprises the following steps:
7a) bilinear interpolation is carried out on the length and the width of the input high-resolution front time phase image to be 600;
7b) and extracting buildings in the high-resolution change image by using the regional candidate network to form a plurality of suggestion frames, and screening the suggestion frames. Calculating the coincidence degree of the suggestion frame and the real frame, and if the coincidence degree is more than 0.5, the suggestion frame can be adjusted to be the real frame, so that the suggestion frame is reserved and is regarded as a positive sample; if the coincidence degree is between 0.1 and 0.5, the coincidence degree is regarded as a negative sample; if the degree of coincidence is less than 0.1, the suggestion box is ignored.
7c) And finally, carrying out regression classification on the screened suggestion boxes to finally obtain a prediction result and a type.
And binarizing the high-resolution front time phase image, performing morphological post-processing on the high-resolution front time phase image, performing morphological closing operation to fill holes in a high-resolution front time phase image binary image region, performing morphological opening operation, and removing the high-resolution front time phase image binary image spot noise to obtain a high-resolution front time phase image binary image.
Wherein, the open operation is the erosion operation first, then the expansion operation, the close operation is the expansion operation first, then the erosion operation, the expansion (dilate) is the 'field expansion' to the image highlight part, the field expansion, the effect image has the highlight area larger than the original image; corrosion (enode) is when the highlighted area in the original is eaten by silkworm, the area is reduced, and the effect map has a smaller highlighted area than the original.
Building change detection is usually performed more based on satellite remote sensing data. However, the remote sensing satellite generally forms an orthoscopic image like shadow, the data resolution is not high (the highest resolution is 0.10 m internationally, and the highest resolution is 0.80 m in China), the return period of the satellite is long, and the early warning period for finding the change of the building is long. Meanwhile, in the existing building change detection, a semantic segmentation method for remote sensing images is mostly adopted to carry out pixel-level classification on a change area, but the missing detection and the error detection can occur because the building target is too small. Meanwhile, due to the fact that the number of network layers is too deep, an overfitting phenomenon can occur.
In terms of the overlooking of the remote sensing image, the monitoring camera is usually deployed on a high tower or a high building and can monitor the natural resource change of 5-10 kilometers. Meanwhile, the monitoring camera can observe the height change of the building more clearly from the side face of the building. Therefore, the monitoring camera is of practical significance for developing the change of the building.
However, at present, building changes are developed by using a monitoring camera, and the building changes are mainly carried out by using a manual real-time watching mode, so that the intelligent degree is not high. And the change area cannot be detected due to the fact that the building picture under the monitoring camera is small and the resolution ratio is low.
For the above problems, a super-resolution reconstruction method is adopted to convert a low-resolution image into a high-resolution image, so that the size of a target building is increased, and then change detection of the building is performed, so that change detection of the building under a monitoring camera is realized.
The invention has the following advantages:
1. compared with the similar orthographic image formed by the traditional remote sensing building image, the monitoring camera can observe the change of the height of the building more clearly from the side surface of the building, and compared with the remote sensing satellite, the return period is long, the monitoring camera for urban monitoring is close to real-time monitoring, the early warning period for finding the change of the building is short, and the change on the height can be found more easily.
2. Because the building change detection method based on semantic segmentation needs a large amount of training data, and the effect of the model is limited due to small data volume, the invention combines deep learning and machine learning methods, thereby reducing the limitation of the data volume on the effect of the model.
3. The change area cannot be detected due to the fact that the building picture under the monitoring camera is small and the resolution ratio is low. The invention is based on the super-resolution reconstruction method, converts the low-resolution image into the high-resolution image, improves the resolution of the building image, and realizes the change detection of the building under the monitoring camera.
While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
