1. An end-to-end photoelectric detection method based on a micro-optical device is characterized by comprising the following steps:

firstly, optical information is sent out or reflected by a target object (16);

secondly, the micro-optical device (3) performs primary signal processing on the optical information;

and step three, the vision sensor (5) collects the image preliminarily processed by the micro-optical device (3) and directly analyzes to obtain the required photoelectric detection result.

2. A micro-optical device based end-to-end photodetecting method according to claim 1, characterized in that, in step two, in order to allow the micro-optical device (3) to implement more complex optical signal processing, the micro-optical device (3) adopts the following design:

step a, firstly, extracting image features by using a traditional method, and then, confirming and screening by using an expert to obtain an ideal image feature set;

b, giving an optical simulation image;

c, processing the simulation analog image by the signal of the micro-optical device (3);

step d, performing image preprocessing on the simulation image through a vision sensor (5), and obtaining an image feature set of the micro-optical device (3) through an LBP local binary pattern;

step e, calculating the distance between the image characteristic set of the micro-optical device (3) and the ideal image characteristic set to obtain an objective function:wherein, the geometrical Distance is the XYZ Geometric Distance between the image feature set of the micro-optical device (3) and the feature center in the ideal image feature set, and the Brightness Distance is the value of the pixel at the feature center.

3. The method of claim 2, wherein in step a, the conventional method for extracting image features is one of corner detection, edge detection, local binary pattern of LBP, MSER maximum stable outer region, corner detection and line detection of three-dimensional image.

4. The end-to-end photoelectric detection method based on micro-optical devices as claimed in claim 2, wherein a neural network input layer (12), a preprocessing sub-network (13), a detection and classification sub-network (14) and a neural network output layer (15) are arranged in the visual sensor (5), and the neural network input layer (12) is electrically connected with the neural network output layer (15) through the preprocessing sub-network (13) and the detection and classification sub-network (14) in sequence.

5. The end-to-end photoelectric detection method based on micro-optics device as claimed in claim 4, wherein the specific flow of the analysis of the vision sensor (5) to obtain the required photoelectric detection result is as follows:

a, a vision sensor (5) collects an original image;

step B, the collected original image is input through a neural network input layer (12), is processed through a preprocessing sub-network (13), and is detected and analyzed through a detecting and classifying sub-network (14);

and step C, outputting the photoelectric detection analysis result through a neural network output layer (15).

6. An end-to-end photoelectric detection system based on micro-optical devices is characterized by comprising a PCB substrate (1), a supporting tube (2) and the micro-optical devices (3), the PCB substrate (1) is provided with a computer on-chip system (4) and a visual sensor (5), the vision sensor (5) is electrically connected with the computer system-on-chip (4) through the PCB substrate (1), one end of the supporting tube (2) is fixedly connected with the PCB substrate (1), the micro-optical device (3) is arranged at the other end of the support tube (2), the vision sensor (5) is positioned in the support tube (2), the top of little optical device (3) is located outside stay tube (2), the bottom of little optical device (3) is located stay tube (2) and is equipped with filter layer (6), filter layer (6) and little optical device (3) fixed connection.

7. A micro-optics-based end-to-end photoelectric detection system as claimed in claim 6, characterized in that an optical protection screen (7) is arranged on the top of the micro-optics (3), and the optical protection screen (7) is fixedly connected with the micro-optics (3).

8. A micro-optical device-based end-to-end photoelectric detection system as claimed in claim 7, wherein the micro-optical device (3) comprises a glass substrate (8), a micro-optical layer (9) is disposed on one side of the glass substrate (8), the glass substrate (8) is fixedly connected with the optical protection screen (7) through the micro-optical layer (9), and the filter layer (6) is fixedly connected with the other side of the glass substrate (8) corresponding to the filter layer.

9. An end-to-end photoelectric detection system based on micro-optical device as claimed in claim 8, wherein one side of said micro-optical layer (9) is provided with an AR coating (10), said micro-optical layer (9) is fixedly connected with said optical protection screen (7) through the AR coating (10), and the other corresponding side of said micro-optical layer (9) is fixedly connected with said glass substrate (8).

Background

As the computational power of semiconductor processors continues to increase, various computational imaging methods have been proposed in recent years to simplify the structure of vision sensors and add complex imaging functions. One of the methods is to replace a conventional lens consisting of a plurality of lenses with a single Micro Optics Element (MOE) and to combine corresponding calculation methods to reconstruct a 2D or 3D image. The micro-optical device can be a coding aperture, a diffraction optical element, a Fresnel mirror, a micro-lens array, an optical light homogenizing sheet and the like.

The difference between conventional lens-based imaging and micro-optics-based imaging is illustrated in fig. 6 and 7. Macroscopically, the micro-optical device is simple in structure, small in size and light in weight, can greatly simplify the structure of the visual sensor, and has better stability (the influence of temperature change and vibration impact on the optical characteristics of the device is smaller). It should be noted that in an ideal lens imaging vision sensor, a point on the target object corresponds to a point on the vision sensor (the light is refractively focused); in an ideal micro-optical imaging vision sensor, a point on a target object corresponds to a plurality of points (light is scattered or diffracted) on the vision sensor, so that the human eye cannot recognize the specific content of an original image acquired by the micro-optical imaging vision sensor.

Expressed by a simplified model, the optical transfer function (also called point spread function) of the lens imaging vision sensor is approximate to a one-to-one mapping Delta function

The optical transfer function of the micro-optical imaging vision sensor is a complex one-to-many mapping function

The actual imaging system optical transfer function is usually not a discrete function as expressed by equations 1 and 2, and thus the above two equations are approximate descriptions.

The image acquired on the micro-optical imaging vision sensor can be expressed as

Where z1 and z2 are the depth boundaries of the target object, C is the system constant, w (x, y, z) is the surface features (which may include texture/brightness and depth) on the target object at the (x, y, z) location, g (x, y, z) is the optical transfer function in equation (2), n (u, v) is the imaging noise at the vision sensor pixel (u, v), and i (u, v) is the value obtained at the vision sensor pixel (u, v).

If the application scene needs to provide the image acquired by the vision sensor to the human eye or uses a traditional image-based processing method, the image acquired by the micro-optical vision sensor can be reconstructed. The reconstructed image may be further image analyzed and processed by conventional machine vision algorithms to achieve desired results.

In the applications of internet of things and intelligent manufacturing, most of the time, images acquired by a vision sensor are directly seen by a machine (namely, a computer analyzes and decides the images) rather than being seen by a human. Therefore, the end-to-end photoelectric detection system can also utilize the micro-optical device to perform specific processing on optical information reflected or emitted by a target object, directly analyze information required by image extraction acquired by the micro-optical vision sensor, and realize direct extraction of an image analysis result from the sensor without image reconstruction.

Disclosure of Invention

The invention provides an end-to-end photoelectric detection system and method based on micro-optical devices, which can directly extract image analysis results from a visual sensor without image reconstruction, and aims to overcome the defect that the image analysis results can only be obtained by image reconstruction in the prior art.

In order to achieve the purpose, the invention adopts the following technical scheme:

an end-to-end photoelectric detection method based on micro-optical devices comprises the following steps:

firstly, a target object sends out or reflects out optical information;

step two, the micro-optical device performs primary signal processing on the optical information;

and step three, the vision sensor collects the image primarily processed by the micro-optical device and directly analyzes to obtain the required photoelectric detection result.

In an ideal lens imaging vision sensor, a point on the target object corresponds to a point on the vision sensor (light is refracted to focus); in an ideal micro-optic imaging vision sensor, a point on the target object corresponds to several points on the vision sensor (light is scattered and diffracted), so that the original image obtained by the micro-optic vision sensor cannot be recognized by human eyes. As mentioned above, if the application scene needs to provide the image acquired by the vision sensor to the human eye or uses a conventional image-based processing method, the image acquired by the micro-optical vision sensor needs to be reconstructed. The workflow of image reconstruction of the data of the micro-optical vision sensor and then using the conventional image video processing method is shown in fig. 8. The main disadvantage of the data processing mode which is independently carried out in two steps is that the calculation amount of image reconstruction and downstream image and video processing is large, and if the reconstructed image does not need to be processed and judged by human eyes, part of calculation is unnecessary.

As shown in figure 1, the invention adopts an end-to-end processing method, specifically, micro-optics is utilized to carry out certain preliminary signal processing on optical information (starting end) sent or reflected by a target object, an image acquired by a visual sensor and processed by the micro-optics is directly analyzed to obtain a required detection result (ending end), and an image reconstruction link is not needed between the starting end and the ending end, so that the visual sensor has intelligent analysis and processing capability, meanwhile, the calculation amount of photoelectric detection is reduced, and the aim of directly extracting an image analysis result from the visual sensor without image reconstruction is fulfilled.

Preferably, in step two, in order to implement more complicated optical signal processing for the micro-optical device, the micro-optical device adopts the following design:

step a, firstly, extracting image features by using a traditional method, and then, confirming and screening by using an expert to obtain an ideal image feature set;

b, giving an optical simulation image;

c, processing the analog image by the signal of the micro-optical device;

step d, performing image preprocessing on the simulation image through a vision sensor, and obtaining a micro-optical device image feature set through an LBP local binary pattern;

e, calculating the distance between the micro-optical device image characteristic set and the ideal image characteristic set to obtain an objective function:

and obj (feature set) ═ α × Geometric Distance + β × bright Distance, where Geometric Distance is the XYZ Geometric Distance between the set of micro-optics image features and the center of the features in the set of ideal image features, and bright Distance is the value of the pixel at the center of the features.

Micro-optics that implement the above-described end-to-end photodetection can be implemented by the most basic Inverse Fourier Transform (IFFT). The advantage of this design approach is that it is simple to implement, but the resulting micro-optics has relatively limited optical signal processing capabilities, and thus the resulting micro-optics essentially performs a fourier transform on the optical signal emitted or reflected by the target object. Therefore, the invention adopts the closed-loop design method of FIG. 2 to design the micro-optical device, so that the micro-optical device can realize more complex optical signal processing, such as specific feature extraction. Therefore, the objective function of weighting and combining various indexes is realized by combining optical simulation with manual or expert sample labeling, and the designed micro-optical device has the required optical signal processing capacity by optimizing the objective function.

Preferably, in step a, the conventional method for extracting image features is one of corner detection, edge detection, LBP local binary pattern, MSER maximum stable outer region of the two-dimensional image, corner detection and line detection of the three-dimensional image. Because the output of these feature extraction methods is heavily dependent on parameters, the ideal image feature set can be obtained by confirming and screening the output manually or by experts.

Preferably, a neural network input layer, a preprocessing sub-network, a detection and classification sub-network and a neural network output layer are arranged in the visual sensor, and the neural network input layer is electrically connected with the neural network output layer sequentially through the preprocessing sub-network, the detection and classification sub-network and the neural network output layer. The direct extraction of photoelectric detection results from micro-optical vision sensor images is realized through a multilayer neural network.

Preferably, the specific process of analyzing and acquiring the required photoelectric detection result by the vision sensor is as follows: a, a vision sensor collects an original image;

step B, inputting the acquired original image through a neural network input layer, processing the original image through a preprocessing sub-network, and then performing detection analysis through a detection and classification sub-network;

and step C, outputting the photoelectric detection analysis result through a neural network output layer.

The original image obtained on the visual sensor is processed by the specific optical information of the micro-optical device, and the direct extraction of the photoelectric detection result from the visual sensor image can be further realized by the multilayer neural network for the image.

The invention also provides an end-to-end photoelectric detection system based on the micro-optical device, which comprises a PCB substrate, a supporting tube and the micro-optical device, wherein a computer on-chip system and a visual sensor are arranged on the PCB substrate, the visual sensor is electrically connected with the computer on-chip system through the PCB substrate, one end of the supporting tube is fixedly connected with the PCB substrate, the micro-optical device is arranged at the other end of the supporting tube, the visual sensor is positioned in the supporting tube, the top of the micro-optical device is positioned outside the supporting tube, the bottom of the micro-optical device is positioned in the supporting tube and is provided with a filter layer, and the filter layer is fixedly connected with the micro-optical device. The micro-optical device carries out certain preliminary signal processing on optical information (starting end) sent or reflected by a target object, a specific wavelength is gated or shielded through the filter layer, specific image characteristics are extracted, an image which is acquired by the vision sensor and processed by the micro-optical device is directly analyzed under the control of a system on a computer chip to obtain a required detection result (ending end), the link of image reconstruction is not needed between the starting end and the ending end, and the aim of directly extracting an image analysis result from the vision sensor without image reconstruction is fulfilled.

Preferably, an optical protection screen is arranged at the top of the micro-optical device, and the optical protection screen is fixedly connected with the micro-optical device. The optical protective screen is beneficial to protecting the optical device from being damaged.

Preferably, the micro-optical device comprises a glass substrate, a micro-optical layer is arranged on one side of the glass substrate, the glass substrate is fixedly connected with the optical protection screen through the micro-optical layer, and the filter layer is fixedly connected with the other side, corresponding to the glass substrate. More specifically, the optical protective screen is beneficial to protecting the micro-optical layer on the surface of the glass substrate from being damaged; the micro-optical layer adopts a micro-optical structure, so that certain preliminary signal processing can be performed on optical information emitted or reflected by a target object.

Preferably, an AR coating is disposed on one side of the micro-optical layer, the micro-optical layer is fixedly connected with the optical protection screen through the AR coating, and the other side of the micro-optical layer corresponding to the micro-optical layer is fixedly connected with the glass substrate. The AR coating is beneficial to the surface of the micro-optical device to have lower reflectance.

The invention has the beneficial effects that: the vision sensor has intelligent analysis processing capability, and meanwhile, the calculated amount of photoelectric detection is reduced, so that the aim of directly extracting an image analysis result from the vision sensor without image reconstruction is fulfilled; the design method of the micro-optical device for realizing the specific optical signal processing realizes the target function of weighting and combining various indexes by combining optical simulation with manual or expert sample labeling, and ensures that the designed micro-optical device has the required optical signal processing capacity by optimizing the target function; the photoelectric detection result is directly extracted from the visual sensor image through a multilayer neural network; the micro-optical layer on the surface of the glass substrate is protected from being damaged; the micro-optical device surface has lower reflectance.

Drawings

FIG. 1 is a flow chart of the photodetection of a micro-optical device according to the present invention;

FIG. 2 is a flow chart of a closed loop design of a micro-optical device;

FIG. 3 is a vision sensor end-to-end image analysis and processing flow diagram;

FIG. 4 is a system architecture diagram for the photodetection of micro-optics;

FIG. 5 is a schematic diagram of a micro-optical device structure;

FIG. 6 is a system architecture diagram based on lens imaging;

FIG. 7 is a system architecture diagram based on micro-optics imaging;

fig. 8 is a flowchart of image reconstruction of the data of the vision sensor prior to using a conventional image video processing method.

In the figure: the system comprises a PCB substrate, a support tube, a micro-optical device, a computer system-on-chip, a vision sensor, a filter layer, a light protection screen, a glass substrate, a micro-optical layer, a film coating, a lens, a neural network input layer, a preprocessing sub-network, a detection and classification sub-network, a neural network output layer and a target object, wherein the micro-optical device comprises 1, the PCB substrate, 2, the support tube, 3, the micro-optical device, 4, the computer system-on-chip, 5, the vision sensor, 6, the filter layer, 7, the light protection screen, 8, the glass substrate, 9, the micro-optical layer, 10, the AR coating film, 11, the lens, 12, the neural network input layer, 13, the preprocessing sub-network, 14, the detection and classification sub-network, 15, the neural network output layer and 16.

Detailed Description

The invention is further described with reference to the following figures and detailed description.

In the embodiment shown in fig. 1, an end-to-end photoelectric detection method based on micro-optics comprises the following steps:

firstly, optical information is sent out or reflected by a target object 16;

step two, the micro-optical device 3 performs primary signal processing on the optical information;

and step three, the vision sensor 5 collects the image primarily processed by the micro-optical device 3 and directly analyzes to obtain the required photoelectric detection result.

As shown in fig. 2, in step two, in order to implement more complicated optical signal processing for the micro-optical device 3, the micro-optical device 3 adopts the following design:

step a, firstly, extracting image features by using a traditional method, and then, confirming and screening by using an expert to obtain an ideal image feature set;

b, giving an optical simulation image;

step c, the micro-optical device 3 processes the simulation image by signals;

step d, performing image preprocessing on the simulation image through a vision sensor 5, and obtaining an image feature set of the micro-optical device 3 through an LBP local binary pattern;

step e, calculating the distance between the image characteristic set of the micro-optical device 3 and the ideal image characteristic set to obtain an objective function:

obj (feature set) ═ α × Geometric Distance + β × bright Distance, where Geometric Distance is the XYZ Geometric Distance of the image feature set of the micro-optics 3 from the feature center in the ideal image feature set, and bright Distance is the value of the pixel at the feature center.

In step a, the conventional method for extracting image features is one of corner detection, edge detection, LBP local binary pattern, MSER maximum stable outer region, corner detection and line detection of a two-dimensional image.

As shown in fig. 3, a neural network input layer 12, a preprocessing sub-network 13, a detection and classification sub-network 14, and a neural network output layer 15 are provided in the visual sensor 5, and the neural network input layer 12 is electrically connected to the neural network output layer 15 through the preprocessing sub-network 13 and the detection and classification sub-network 14 in this order.

As shown in fig. 3, the specific process of analyzing and acquiring the required photoelectric detection result by the vision sensor 5 is as follows

Step A, a vision sensor 5 collects an original image;

step B, inputting the acquired original image through a neural network input layer 12, processing the original image through a preprocessing sub-network 13, and then performing detection analysis through a detection and classification sub-network 14;

and step C, outputting the photoelectric detection analysis result through the neural network output layer 15.

As shown in fig. 4, the invention further provides an end-to-end photoelectric detection system based on micro-optical devices, which comprises a PCB substrate 1, a supporting tube 2 and micro-optical devices 3, wherein a computer system-on-chip 4 and a vision sensor 5 are arranged on the PCB substrate 1, the vision sensor 5 is electrically connected with the computer system-on-chip 4 through the PCB substrate 1, one end of the supporting tube 2 is fixedly connected with the PCB substrate 1, the micro-optical devices 3 are arranged at the other end of the supporting tube 2, the vision sensor 5 is positioned in the supporting tube 2, the top of the micro-optical devices 3 is positioned outside the supporting tube 2, the bottom of the micro-optical devices 3 is positioned in the supporting tube 2 and is provided with a filter layer 6, and the filter layer 6 is fixedly connected with the micro-optical devices 3. The top of the micro-optical device 3 is provided with an optical protection screen 7, and the optical protection screen 7 is fixedly connected with the micro-optical device 3.

As shown in fig. 4 and 5, the micro-optical device 3 includes a glass substrate 8, a micro-optical layer 9 is disposed on one side of the glass substrate 8, the glass substrate 8 is fixedly connected to the optical protection screen 7 through the micro-optical layer 9, and the filter layer 6 is fixedly connected to the other side of the glass substrate 8 corresponding to the filter layer.

As shown in fig. 4 and 5, an AR coating film 10 is disposed on one side of the micro-optical layer 9, the micro-optical layer 9 is fixedly connected to the optical protection screen 7 through the AR coating film 10, and the other side of the micro-optical layer 9 corresponding to the optical protection screen is fixedly connected to the glass substrate 8.

In an ideal lens imaging vision sensor 5, a point on the target object corresponds to a point on the vision sensor 5 (light is refractively focused); in the ideal micro-optical imaging vision sensor 5, a point on the target object corresponds to a plurality of points on the vision sensor 5 (light is scattered and diffracted), so that the human eye can not recognize the specific content of the original image acquired by the micro-optical vision sensor 5. As mentioned above, if the application scene needs to provide the image acquired by the vision sensor 5 to the human eye or uses a conventional image-based processing method, the image acquired by the micro-optical vision sensor 5 needs to be reconstructed. The workflow of image reconstruction of the data of the micro-optical vision sensor 5 and then using the conventional image video processing method is shown in fig. 8. The main disadvantage of the data processing mode which is independently carried out in two steps is that the calculation amount of image reconstruction and downstream image and video processing is large, and if the reconstructed image does not need to be processed and judged by human eyes, part of calculation is unnecessary.

As shown in fig. 1, the present invention adopts an end-to-end processing method, specifically, a micro-optical device 3 is used to perform a certain preliminary signal processing on optical information (start end) emitted or reflected by a target object, an image acquired by a vision sensor 5 and processed by the micro-optical device 3 is directly analyzed to obtain a required detection result (end), and there is no image reconstruction link between the start end and the end, so that the vision sensor 5 has an intelligent analysis processing capability, and simultaneously, the calculation amount of photoelectric detection is reduced, and the purpose of directly extracting an image analysis result from the vision sensor 5 without image reconstruction is achieved.

The micro-optics 3 that implement the end-to-end photodetection described above can be implemented by the most basic Inverse Fourier Transform (IFFT). This design method has the advantage of simple implementation, but the resulting micro-optics 3 have relatively limited optical signal processing capabilities, and the resulting micro-optics 3 essentially fourier transform the optical signal emitted or reflected by the target object. Therefore, the present invention adopts the design method shown in fig. 2 to design the micro-optical device 3, so that the micro-optical device 3 can realize more complex optical signal processing, such as specific feature extraction. Therefore, the objective function of weighting and combining various indexes is realized by combining optical simulation with manual or expert sample labeling, and the designed micro-optical device 3 has the required optical signal processing capacity by optimizing the objective function.

As shown in fig. 3, the original image obtained on the vision sensor 5 has been subjected to specific optical information processing by the micro-optical device 3, such image is further input through the neural network input layer 12, then processed through the preprocessing sub-network 13, then detected and analyzed through the detecting and classifying sub-network 14, and finally the result of photoelectric detection and analysis is output through the neural network output layer 15, so that the direct extraction of the photoelectric detection result from the image of the vision sensor 5 is realized.