1. A visual imaging system based on P300 and associated imaging, comprising:

the active random light source is used for generating dynamic random speckles to irradiate the surface of an object to be imaged, exposing the surface of the object to be imaged and controlling reflected photons of the exposed object to be imaged to enter human eyes within a time interval for generating random speckle modulation so as to generate optical images, wherein each random speckle corresponds to one optical image;

the electroencephalogram equipment is used for acquiring a P300 signal generated by the human brain according to the optical image;

the data acquisition card is used for acquiring and storing the P300 signal corresponding to each random speckle according to time sequence;

and the computing device is used for performing cross-correlation operation on the random speckles and the P300 signal to obtain correlated imaging.

2. The P300 and linked imaging based vision imaging system of claim 1, wherein the optical image is expressed by the following calculation:

P(x,y,t)＝P₀(x,y,t)+N(x,y,t)＝S(x,y,t)×O(x,y)+N(x,y,t)；

wherein P (x, y, t) is the optical image, S (x, y, t) is dynamic random speckle, P₀(x, y, t) is an optical image obtained by illumination of dynamic random speckles S (x, y, t), O (x, y) is an image of an object to be imaged, N (x, y, t) is noise light, wherein x and y are respectively the horizontal and vertical coordinates of the optical image, and t is the time for generating the random speckles.

3. The P300 and correlated imaging based vision imaging system of claim 2, wherein said computing means cross-correlates random speckle with P300 signal by the following formula:

I(x,y)＝<S(x,y,t)D(t)>；

where < > denotes a cross-correlation operation, I (x, y) denotes a correlation image, S (x, y, t) denotes dynamic random speckle, and d (t) denotes a discrimination signal of the P300 signal, and when the amplitude of the P300 signal is greater than a threshold value, d (t) is 1, otherwise d (t) is 0.

4. The P300 and correlation imaging based visual imaging system of claim 3, wherein the active random light source generates dynamic random speckles as scattered speckles in a sparse matrix.

5. The P300 and correlation imaging based visual imaging system of any one of claims 1-4, wherein the active random light source comprises: the laser comprises a laser and rotatable ground glass, wherein laser emitted by the laser penetrates through the rotatable ground glass to generate pseudo-thermal light.

6. The P300 and correlation imaging based visual imaging system of any one of claims 1 to 4, wherein the active random light source is a projector.

7. A visual imaging method based on P300 and associated imaging is characterized by comprising the following steps:

s1: the method comprises the steps that dynamic random speckles generated by an active random light source are irradiated on the surface of an object to be imaged to be exposed, reflected photons of the exposed object to be imaged enter human eyes within a time interval for generating random speckle modulation to generate optical images, and each random speckle corresponds to one optical image;

s2: acquiring a P300 signal generated by the human brain according to the optical image by adopting electroencephalogram equipment;

s3: collecting and storing a P300 signal corresponding to each random speckle by a data acquisition card according to a time sequence;

s4: and the computing device performs cross-correlation operation on the speckles and the P300 signal to obtain associated imaging.

8. The visual imaging method based on P300 and correlated imaging according to claim 7, wherein in said step S1, the optical image is expressed by the following calculation mode:

P(x,y,t)＝P₀(x,y,t)+N(x,y,t)＝S(x,y,t)×O(x,y)+N(x,y,t)；

wherein P (x, y, t) is the optical image, S (x, y, t) is dynamic random speckle, P₀(x, y, t) is an optical image obtained by illumination of dynamic random speckles S (x, y, t), O (x, y) is an image of an object to be imaged, N (x, y, t) is noise light, wherein x and y are respectively the horizontal and vertical coordinates of the optical image, and t is the time for generating the random speckles.

9. The visual imaging method based on P300 and correlated imaging according to claim 8, wherein said cross-correlation operation of step S4 is:

I(x,y)＝<S(x,y,t)D(t)>；

where < > denotes a cross-correlation operation, I (x, y) denotes a correlation image, S (x, y, t) denotes dynamic random speckle, and d (t) denotes a discrimination signal of the P300 signal, and when the amplitude of the P300 signal is greater than a threshold value, d (t) is 1, otherwise d (t) is 0.

10. The P300 and correlation imaging based visual imaging method of claim 9, wherein the active random light source generates dynamic random speckles as scattered speckles in a sparse matrix.

Background

The brain is the central nervous system of higher nervous activities in the human body, has hundreds of millions of neurons, and transmits and processes human body information by interconnecting. Electroencephalogram signals can be divided into: evoked brain electrical signals and spontaneous brain electrical signals. The induced brain electrical signal is brain electrical activity formed by the potential change of the brain through certain external stimulation; the spontaneous brain electrical signals refer to the brain electrical activity generated by the brain spontaneously without external special stimulation. The P300 event-related potential is a kind of evoked brain signals, and a positive peak (wave presenting an upward trend relative to the baseline) appears in the range of about 300 milliseconds after a small probability stimulation. The occurrence of P300 also varies due to inter-individual variability, and fig. 1 shows the P300 waveform around 450 ms after stimulation occurs. The P300 potential is used as an endogenous component, is not influenced by physical characteristics of stimulation, is related to perceptual or cognitive psychological activities, and is closely related to processes of attention, memory, intelligence and the like. The brain-computer interface based on the P300 has the advantages that a user can obtain higher identification accuracy without complex training, and the brain-computer interface has stable locking time and high time precision.

The method for acquiring the visual information captured by the human eyes has positive effects in scientific research, medical treatment and life. In scientific research, the method can be used as a novel technical means for researching the visual working process of human eyes, researching the information communication mode between human eyes and brain and the like, and has remarkable positive effect on the biological development of the human vision at the front. The method can be used for the research of human retina, the diagnosis of visual pathway and the like, and has auxiliary effect on the diagnosis of brain diseases. In life, the visual enhancement device can be assisted in use, user experience is enhanced, a human-computer interaction effect is improved, and even the experience and effect of a user are directly enhanced. And can further assist the disabled in daily life.

However, in scientific research, medical treatment and life, the acquisition of visual information captured by human eyes is often difficult to realize, that is, the imaging representation of the visual information of human eyes cannot be realized.

Disclosure of Invention

The invention provides a visual imaging system based on P300 and associated imaging, which solves the problem that the existing imaging technology is difficult to realize imaging representation on visual information of human eyes.

The invention relates to a visual imaging system based on P300 and associated imaging, which comprises:

the active random light source is used for generating dynamic random speckles to irradiate the surface of an object to be imaged, exposing the surface of the object to be imaged and controlling reflected photons of the exposed object to be imaged to enter human eyes within a time interval for generating random speckle modulation so as to generate optical images, wherein each random speckle corresponds to one optical image;

the electroencephalogram equipment is used for acquiring a P300 signal generated by the human brain according to the optical image;

the data acquisition card is used for acquiring and storing the P300 signal corresponding to each random speckle according to time sequence;

and the computing device is used for performing cross-correlation operation on the random speckles and the P300 signal to obtain correlated imaging.

Wherein the optical image is expressed by the following calculation:

P(x,y,t)＝P₀(x,y,t)+N(x,y,t)＝S(x,y,t)×O(x,y)+N(x,y,t)；

wherein P (x, y, t) is the optical image, S (x, y, t) is dynamic random speckle, P₀(x, y, t) is an optical image obtained by illumination of dynamic random speckles S (x, y, t), O (x, y) is an image of an object to be imaged, N (x, y, t) is noise light, wherein x and y are respectively the horizontal and vertical coordinates of the optical image, and t is the time for generating the random speckles.

Wherein, the computing device performs cross-correlation operation on the random speckles and the P300 signal by the following formula:

I(x,y)＝<S(x,y,t)D(t)>；

where < > denotes a cross-correlation operation, I (x, y) denotes a correlation image, S (x, y, t) denotes dynamic random speckle, and d (t) denotes a discrimination signal of the P300 signal, and when the amplitude of the P300 signal is greater than a threshold value, d (t) is 1, otherwise d (t) is 0.

The active random light source generates dynamic random speckles which are distributed according to a sparse matrix.

Wherein the active random light source comprises: the laser comprises a laser and rotatable ground glass, wherein laser emitted by the laser penetrates through the rotatable ground glass to generate pseudo-thermal light.

Wherein the active random light source is a projector.

The invention also provides a visual imaging method based on the P300 and the associated imaging, which comprises the following steps:

s1: the method comprises the steps that dynamic random speckles generated by an active random light source are irradiated on the surface of an object to be imaged to be exposed, reflected photons of the exposed object to be imaged enter human eyes within a time interval for generating random speckle modulation to generate optical images, and each random speckle corresponds to one optical image;

s2: acquiring a P300 signal generated by the human brain according to the optical image by adopting electroencephalogram equipment;

s3: collecting and storing a P300 signal corresponding to each random speckle by a data acquisition card according to a time sequence;

s4: and the computing device performs cross-correlation operation on the speckles and the P300 signal to obtain associated imaging.

In step S1, the optical image is expressed by the following calculation method:

P(x,y,t)＝P₀(x,y,t)+N(x,y,t)＝S(x,y,t)×O(x,y)+N(x,y,t)；

wherein P (x, y, t) is the optical image, S (x, y, t) is dynamic random speckle, P₀(x, y, t) is an optical image obtained by illumination of dynamic random speckles S (x, y, t), O (x, y) is an image of an object to be imaged, N (x, y, t) is noise light, wherein x and y are respectively the horizontal and vertical coordinates of the optical image, and t is the time for generating the random speckles.

Wherein, the cross-correlation operation of step S4 is:

I(x,y)＝<S(x,y,t)D(t)>；

where < > denotes a cross-correlation operation, I (x, y) denotes a correlation image, S (x, y, t) denotes dynamic random speckle, and d (t) denotes a discrimination signal of the P300 signal, and when the amplitude of the P300 signal is greater than a threshold value, d (t) is 1, otherwise d (t) is 0.

The active random light source generates dynamic random speckles which are distributed according to a sparse matrix.

In the visual imaging system and method based on the P300 and the associated imaging, the active random light source generates dynamic random speckles to irradiate on the surface of an object to be imaged, the electroencephalogram equipment acquires P300 signals generated after the human brain is stimulated by optical images of the object to be imaged irradiated by the random speckles, the acquisition card acquires the P300 signals corresponding to each speckle according to time sequence, the computing device performs associated operation on the speckles and the P300 signals to obtain the associated imaging, and the associated imaging is the imaging representation of human eye visual information, so that the imaging representation of the human eye visual information is realized.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

Fig. 1 is a P300 waveform of the human brain around 450 milliseconds after stimulation occurs.

FIG. 2 is a schematic block diagram of a P300 and related imaging-based visual imaging system according to an embodiment of the present invention;

FIG. 3 is an original image of an object to be imaged according to an embodiment of the present invention;

FIG. 4 is an image of an object to be imaged under a single speckle in an embodiment of the present invention;

FIG. 5 is a schematic diagram of an imaging range of a human eye according to an embodiment of the invention;

FIG. 6 is a graph of simulation results for imaging on a retina of a human eye in an embodiment of the invention;

fig. 7 is a diagram of simulation results of correlated imaging based on P300 of an object to be imaged in an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The visual imaging system based on P300 and correlated imaging of the present embodiment is shown in fig. 2, and includes: the device comprises an active random light source 1, an electroencephalogram device 3, a data acquisition card 4 and a computing device 5. The active random light source 1 generates dynamic random speckles S (x, y, t) to irradiate on the surface of an object 2 to be imaged, the image of the object 2 to be imaged is O (x, y), the surface of the object 2 to be imaged is exposed, reflected photons of the exposed object 2 to be imaged enter human eyes within a time interval for generating random speckle modulation so as to generate optical images P (x, y, t), each random speckle corresponds to one optical image, and the optical images are optical images for imaging human eyes retina. The brain electrical equipment 3 is used for acquiring a P300 signal generated by the human brain according to the optical image P (x, y, t). The data acquisition card 4 acquires and stores the P300 signal corresponding to each random speckle according to time sequence, and the computing device 5 performs correlation operation on the speckles and the P300 signal to obtain correlated imaging.

When the system works, the active random light source 1 can realize that the reflected photons enter human eyes in the time interval for generating random speckle modulation (namely, the exposure time window is smaller than the time interval for dynamic random speckle modulation) by controlling the duration time of random speckle illumination. The active random light source 1 performs multiple exposures, and the time window of each exposure is smaller than the time interval of dynamic random speckle modulation, so that the imaging result of the changed speckles is avoided, and a plurality of different speckles are superposed. The data acquisition card 4 acquires and stores the P300 signal for multiple times and sends the P300 signal to the computing device 5, and then the computing device 5 is adopted to perform correlation operation on the speckle and the P300 signal to obtain correlation imaging. The electroencephalogram device 3 can be specifically a brain-computer interface device, such as: emotiv EPOC X, used for gathering the brain electrical signal, the above-mentioned data acquisition card 4 of the port equipment of the brain, the computing device 5 can be the computer or one-chip computer, etc..

In the visual imaging system based on the P300 and the correlated imaging of the embodiment, the active random light source 1 generates dynamic random speckles to irradiate on the surface of the object to be imaged 2, the electroencephalogram device 3 collects P300 signals generated after the human brain is stimulated by optical images of the object to be imaged irradiated by the random speckles, the data acquisition card 4 collects the P300 signals corresponding to each speckle according to a time sequence, the computing device 5 performs correlated operation on the speckles and the P300 signals to obtain the correlated imaging, and the correlated imaging is the imaging representation of the human eye visual information, so that the imaging representation of the human eye visual information is realized.

Specifically, the optical image P (x, y, t) is expressed by the following calculation:

P(x,y,t)＝P₀(x,y,t)+N(x,y,t)＝S(x,y,t)×O(x,y)+N(x,y,t)；

wherein S (x, y, t) is dynamic random speckle, P₀(x, y, t) is an optical image obtained by illumination of dynamic random speckles S (x, y, t), O (x, y) is an image of an object to be imaged, N (x, y, t) is noise light, wherein x and y are respectively the horizontal and vertical coordinates of the optical image, and t is the time for generating the random speckles.

The calculating device 5 performs a cross-correlation operation on the random speckles and the P300 signal by the following formula:

I(x,y)＝<S(x,y,t)D(t)>；

where < > denotes a cross-correlation operation, I (x, y) denotes a correlation image, S (x, y, t) denotes dynamic random speckle, and d (t) denotes a discrimination signal of the P300 signal, and when the amplitude of the P300 signal is greater than a threshold value, d (t) is 1, otherwise d (t) is 0. The threshold value may be different from person to person, and the specific value of the threshold value may be different for different persons, and in this embodiment, the average value +2 × the standard deviation of the electroencephalogram signal when a person is not stimulated may be selected. The specific operation mode is as follows: and multiplying and integrating S (x, y, t) and D (t). The random speckles have different shapes, each speckle is applied to an object to cause the total reflected light intensity to be different, the reflected light intensity signal can be used as a stimulation signal for generating a P300 signal, when the total light intensity of P (x, y, t) is greater than the light intensity of the 80 th percentile of the statistical distribution of the total light intensity, the P300 signal with a larger amplitude can be generated, if the total light intensity of P (x, y, t) exceeds the threshold value, D (t) is 1, otherwise D (t) is 0, and the difference of the change of the total light intensity is more obvious, the P300 signal can be excited, and the generated P300 signal is stronger.

Specifically, the data acquisition card 4 usually adopts an algorithm (e.g., a support vector machine, a linear discrimination model, a long-short term time sequence network, etc.) to discriminate whether a P300 signal exists in the signal, and if the discrimination result is that the P300 signal exists (i.e., the amplitude of the P300 signal is greater than a threshold), the output is 1, and if the discrimination result is that the P300 signal does not exist, the output is 0.

The active random light source 1 generates dynamic random speckles which are distributed according to a sparse matrix. The more sparse the speckles are, the more scattered the speckle spots are spatially distributed, so that the spots hitting the object sometimes do not exist, and the probability that the total intensity distribution of the light intensity of the above-mentioned P (x, y, t) is satisfied, so as to stimulate the brain to generate a P300 signal.

The active random light source 1 includes: the laser device comprises a laser device and rotatable ground glass, wherein laser emitted by the laser device penetrates through the rotatable ground glass to generate pseudo-thermo-light, and the pseudo-thermo-light is random speckles. The active random light source 1 may also be a projector, using which random speckles are projected.

The invention also provides a visual imaging method based on the P300 and the associated imaging, which comprises the following steps:

step S1: the active random light source 1 generates dynamic random speckles to irradiate the surface of an object to be imaged 2 for exposure, reflected photons of the exposed object to be imaged enter human eyes within a time interval for generating random speckle modulation to generate optical images, and each random speckle corresponds to one optical image.

Step S2: and acquiring a P300 signal generated by the human brain according to the optical image by adopting electroencephalogram equipment.

Step S3: collecting and storing the P300 signal corresponding to each random speckle by a data acquisition card 4 according to a time sequence; and repeating the exposure and the acquisition for multiple times, namely repeating the steps S1-S3, wherein the time window of the exposure is smaller than the time interval of the dynamic random speckle modulation.

Step S4: and performing cross-correlation operation on the speckles and the P300 signal by using a computing device 5 to obtain correlated imaging.

In step S1, an optical image P (x, y, t) is obtained through the retina of the human eye, i.e. the calculation between the random speckle S (x, y, t) and the image O (x, y) of the object 2 to be imaged is realized, and the optical image P (x, y, t) is expressed by the following calculation method:

P(x,y,t)＝P₀(x,y,t)+N(x,y,t)＝S(x,y,t)*O(x,y)+N(x,y,t)；

in the above formula, S (x, y, t) is dynamic random speckle, P₀(x, y, t) is an optical image obtained under the illumination of the dynamic random speckle S (x, y, t), O (x, y) is an image of the object 2 to be imaged, and N (x, y, t) is noise light.

The cross-correlation operation is: i (x, y) < S (x, y, t) d (t) >, d (t) is a discrimination signal of the P300 signal, I (x, y) is a correlation image, and < > represents a cross-correlation operation, the specific operation method is as follows: and multiplying and integrating S (x, y, t) and D (t). The random speckles have different shapes, each speckle is applied on the object, the total reflected light intensity is different, the reflected light intensity signal can be used as a stimulation signal for generating a P300 signal, when the amplitude of the P300 signal is greater than a threshold value, d (t) is 1, otherwise d (t) is 0, specifically, when the total light intensity of P (x, y, t) is greater than the light intensity of the 80 th percentile of the statistical distribution of the total light intensity, a P300 signal with a larger amplitude is generated, if the threshold value is exceeded, d (t) is 1, otherwise d (t) is 0, and the more obvious the difference of the change of the total light intensity is, the more capable of exciting the P300 signal, and the stronger the generated P300 signal is. Usually, the data acquisition card 4 adopts an algorithm (e.g., a support vector machine, a linear identification model, a long-short time sequence network, etc.) to identify whether a P300 signal exists in the signal, and if the identification result is that the P300 signal exists, the output is 1, and if the identification result is that no P300 signal exists, the output is 0.

Referring to fig. 3, for an original image of an object 2 to be imaged, referring to fig. 4, it can be seen that an image of the object 2 to be imaged under random speckles, that is, a single optical image obtained by dynamic random speckles S (x, y, t), is partially illuminated under random speckle illumination, and it is preferable to use random speckles distributed sparsely, where the sparser the speckles are, the scattered the speckle light spots are distributed spatially, so that the light spots striking the object are sometimes absent, and the general probability satisfies the above-mentioned condition of total intensity distribution of light intensity of P (x, y, t), so that the brain generates a P300 signal. According to the concept of sparse matrix, random speckles are distributed according to sparse matrix when the ratio of the area occupied by the light spot of the random speckles on the object to the total area of the surface of the object is less than or equal to 0.05.

Referring to fig. 5-7, fig. 5 is a view of the area on the object 2 to be imaged as seen by the human eye, which will be imaged on the aligned portion, as shown in fig. 6, when illuminated with uniform illumination, is a view simulating the image of the object as seen by the human eye, i.e., the image on the retina of the human eye. Fig. 7 is an imaging representation of human vision.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.