1. An object recognition method, comprising:

s101: acquiring an event data stream acquired by a visual sensor, wherein the event data stream comprises event data of at least one pixel event, and the event data comprises time information and spatial information of the pixel event;

s102: determining a spatiotemporal association relationship between the pixel events according to the temporal information and the spatial information;

s103: based on the space-time incidence relation, carrying out event data denoising processing on the event data stream to obtain a processed event data stream, wherein the processed event data stream comprises event data of denoised pixel events;

s104: calculating the distance characteristic of the denoised pixel event according to the spatial information of the denoised pixel event;

s105: clustering the denoised pixel events based on the distance characteristics to obtain a clustering result;

s106: and training the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the pulse neural network, so as to recognize the object corresponding to the clustering result through the trained object recognition model.

2. The object recognition method of claim 1, wherein determining the spatiotemporal correlation between the pixel events according to the temporal information and the spatial information comprises:

generating an initial correlation mask of the event data stream based on parameter information of the vision sensor, the initial correlation mask comprising at least one information unit for recording time information of pixel events;

performing information compression on the spatial information of the pixel event to obtain compressed spatial information;

determining a target information unit corresponding to the pixel event according to the compressed spatial information;

and according to the time information of the pixel events, updating the information of the target information unit to generate an updated association mask, wherein the updated association mask represents the space-time association relation between the pixel events.

3. The object recognition method of claim 2, wherein updating the target information unit according to the temporal information of the pixel event comprises:

determining a time threshold parameter required for denoising the event data of the event data stream;

according to the time threshold parameter, performing information compression on the time information to obtain compressed time information;

and updating the information of the target information unit based on the compressed time information.

4. The object recognition method of claim 3, wherein compressing the time information according to the time threshold parameter to obtain compressed time information comprises:

determining a shifting parameter of the time information according to the time threshold parameter;

performing a bit operation on the time information based on the shift parameter to perform information compression on the time information;

based on the operation result, determining compressed time information of the pixel event.

5. The object recognition method of claim 2, wherein compressing the spatial information of the pixel event to obtain compressed spatial information comprises:

performing bit operation on the spatial information of the pixel event to perform information compression on the spatial information of the pixel event;

based on the operation result, compressed spatial information of the pixel event is determined.

6. The object recognition method of claim 1, wherein performing event data denoising processing on the event data stream based on the spatiotemporal correlation relationship comprises:

determining an association mask representing the time-space association relation and a time threshold parameter required for denoising the event data of the event data stream, wherein the association mask comprises at least one information unit;

determining a correlation information unit of the pixel event from the correlation mask based on the spatial information of the pixel event, wherein the correlation information unit has a spatial correlation relationship with the pixel event;

verifying the time information of the pixel event based on the time threshold parameter and the time information recorded by the associated information unit;

and according to the check result, carrying out event data denoising processing on the pixel event so as to carry out event data denoising processing on the event data stream.

7. The object recognition method of claim 1, wherein before the event data de-noising processing is performed on the event data stream based on the spatiotemporal correlation relationship, the method further comprises:

generating an initial filter mask for the event data stream based on parameter information of the vision sensor;

updating the initial filter mask based on the time information and the spatial information of the pixel event to obtain an updated filter mask;

performing event data filtering processing on the event data stream through the updated filtering mask to obtain a filtered event data stream;

the event data denoising processing on the event data stream based on the time-space incidence relation comprises the following steps: and carrying out event data denoising processing on the filtered event data stream based on the space-time incidence relation.

8. The object recognition method of claim 1, wherein calculating the distance feature of the denoised pixel event according to the spatial information of the denoised pixel event comprises:

determining an initial cluster set of the denoised pixel events, the initial cluster set comprising at least one initial cluster, the initial cluster being generated based on historical pixel events;

acquiring clustering characteristic information of the initial clustering;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the spatial information and the cluster characteristic information of the denoised pixel event.

9. The object recognition method of claim 8, wherein the clustering feature information includes clustering features of the initial clusters in at least one feature dimension;

calculating the distance feature between the denoised pixel event and the initial cluster according to the spatial information and the cluster feature information of the denoised pixel event, wherein the distance feature comprises the following steps:

determining a target characteristic dimension to be adjusted and an adjustment parameter of the target characteristic dimension from at least one characteristic dimension of the initial cluster;

based on the adjustment parameters, carrying out information conversion on the spatial information of the denoised pixel event and the clustering characteristic information;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted cluster characteristic information.

10. The object recognition method of claim 8, wherein calculating the distance feature between the denoised pixel event and the initial cluster according to the spatial information of the denoised pixel event and the cluster feature information comprises:

determining coordinate axes to be zoomed in the reference coordinate system of the initial clustering and zooming parameters of the coordinate axes;

based on the scaling parameter, carrying out information conversion on the spatial information of the denoised pixel event and the clustering characteristic information;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted cluster characteristic information.

11. The object recognition method of claim 8, wherein the clustering feature information includes clustering features of the initial clusters in at least one feature dimension;

calculating the distance feature between the denoised pixel event and the initial cluster according to the spatial information and the cluster feature information of the denoised pixel event, wherein the distance feature comprises the following steps:

determining target clustering characteristics to be adjusted and adjustment parameters of the target clustering characteristics from the clustering characteristics;

based on the adjustment parameters, carrying out information conversion on the spatial information of the denoised pixel event and the target clustering characteristics;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted target cluster characteristic.

12. The object recognition method of claim 8, wherein calculating the distance feature between the denoised pixel event and the initial cluster according to the spatial information of the denoised pixel event and the cluster feature information comprises:

determining the rotation angle characteristics of the initial clusters and the adjustment parameters of the rotation angle characteristics;

performing information conversion on the spatial information of the denoised pixel event and the rotation angle characteristic based on the adjusting parameter;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted rotation angle characteristic.

13. The object recognition method according to claim 8, wherein obtaining the cluster feature information of the initial cluster comprises:

determining a cluster shape corresponding to the initial cluster;

and acquiring clustering feature information of the initial clustering based on the clustering shape.

14. The object recognition method of claim 1, wherein clustering the denoised pixel events based on the distance features to obtain a clustering result comprises:

determining a target cluster of the denoised pixel event from an initial cluster set of the denoised pixel event based on the distance features, the initial cluster set comprising at least one initial cluster, the initial cluster being generated based on historical pixel events;

updating the target cluster according to the spatial information of the denoised pixel event to obtain an updated target cluster;

and updating the initial cluster set based on the updated target cluster so as to determine a cluster result according to an update result of the initial cluster set.

15. The object recognition method of claim 14, wherein updating the initial cluster set based on the updated target cluster to determine a cluster result according to an update result of the initial cluster set comprises:

updating the initial cluster set based on the updated target cluster to obtain an updated cluster set;

determining active clusters in the updated cluster set, and calculating cluster distances among the active clusters;

if the clustering distance meets a preset distance condition, clustering and merging the active clusters to obtain a processed cluster set;

and determining a clustering result according to the processed clustering set.

16. The object recognition method of claim 1, wherein training the neural network-based object recognition model according to the clustering result to obtain a trained object recognition model based on a spiking neural network comprises:

generating sample data required by model training according to the clustering result;

training an object recognition model based on a neural network through the sample data to obtain a trained model corresponding to the neural network;

and generating a trained object recognition model based on the impulse neural network based on the trained model corresponding to the neural network.

17. An object recognition apparatus, comprising:

the device comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring an event data stream acquired by a visual sensor, the event data stream comprises event data of at least one pixel event, and the event data comprises time information and space information of the pixel event;

the determining unit is used for determining the space-time association relation between the pixel events according to the time information and the space information;

a denoising unit, configured to perform event data denoising on the event data stream based on the temporal-spatial correlation relationship to obtain a processed event data stream, where the processed event data stream includes event data of denoised pixel events;

the calculating unit is used for calculating the distance characteristic of the denoised pixel event according to the spatial information of the denoised pixel event;

the clustering unit is used for clustering the denoised pixel events based on the distance characteristics to obtain a clustering result;

and the training unit is used for training the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the pulse neural network so as to recognize the object corresponding to the clustering result through the trained object recognition model.

18. A chip comprising an object recognition apparatus according to claim 17 or adapted to perform an object recognition method according to any one of claims 1 to 16.

19. An electronic device comprising a response module and a processing module, and an object recognition apparatus according to claim 17, wherein the object recognition apparatus recognizes an event collected by a visual sensor and transmits the recognition result to the processing module, and the processing module sends a control command to the response module.

Background

The object recognition has great significance in the field of visual processing, and can be used for judging whether certain types of objects are contained in visual information and determining specific information of the objects, such as object positions, object sizes and the like, so that the object recognition has wide application scenes in practical application.

In the course of research and practice on related technologies, the inventors of the present application found that object recognition can be currently performed by applying an Artificial Neural Network (ANN) model, which has the following disadvantages:

in order to identify an object in real time and at high speed, a deep learning model with a very large scale is needed, which requires a large amount of hardware resource consumption, such as GPU;

the traditional frame-based camera collects a large amount of data in real time, and if too many collection points exist, the requirement on bandwidth is extremely high; the frame rate is usually lowered for load reduction, which is not applicable to real-time high-speed object recognition;

the problem of high power consumption: a large number of high-power-consumption acquisition terminals based on frame data, a large number of GPUs for training and reasoning, energy waste caused by the need of online identification no matter whether a target exists or not, and the like.

The inventors of the present application found that the current object recognition method has the above disadvantages, for example, a large amount of hardware resources, network resources, storage resources, etc. are consumed, so that the current object recognition method is yet to be improved.

Disclosure of Invention

The embodiment of the application provides an object identification method and device, a chip and electronic equipment, and the object identification efficiency can be improved.

An object recognition method, comprising:

acquiring an event data stream acquired by a visual sensor, wherein the event data stream comprises event data of at least one pixel event, and the event data comprises time information and spatial information of the pixel event;

determining a spatiotemporal association relationship between the pixel events according to the temporal information and the spatial information;

based on the space-time incidence relation, carrying out event data denoising processing on the event data stream to obtain a processed event data stream, wherein the processed event data stream comprises event data of denoised pixel events;

calculating the distance characteristic of the denoised pixel event according to the spatial information of the denoised pixel event;

clustering the denoised pixel events based on the distance characteristics to obtain a clustering result;

and training the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the pulse neural network, so as to recognize the object corresponding to the clustering result through the trained object recognition model.

In some embodiments, determining the spatio-temporal association relationship between the pixel events according to the temporal information and the spatial information comprises:

generating an initial correlation mask of the event data stream based on parameter information of the vision sensor, the initial correlation mask comprising at least one information unit for recording time information of pixel events;

performing information compression on the spatial information of the pixel event to obtain compressed spatial information;

determining a target information unit corresponding to the pixel event according to the compressed spatial information;

and according to the time information of the pixel events, updating the information of the target information unit to generate an updated association mask, wherein the updated association mask represents the space-time association relation between the pixel events.

In some kind of embodiments, updating information of the target information unit according to the time information of the pixel event includes:

determining a time threshold parameter required for denoising the event data of the event data stream;

according to the time threshold parameter, performing information compression on the time information to obtain compressed time information;

and updating the information of the target information unit based on the compressed time information.

In a certain embodiment, compressing the time information according to the time threshold parameter to obtain compressed time information includes:

determining a shifting parameter of the time information according to the time threshold parameter;

performing a bit operation on the time information based on the shift parameter to perform information compression on the time information;

based on the operation result, determining compressed time information of the pixel event.

In a certain embodiment, the information compression of the spatial information of the pixel event to obtain compressed spatial information includes:

performing bit operation on the spatial information of the pixel event to perform information compression on the spatial information of the pixel event;

based on the operation result, compressed spatial information of the pixel event is determined.

In some kind of embodiments, performing event data denoising processing on the event data stream based on the spatio-temporal correlation relationship includes:

determining an association mask representing the time-space association relation and a time threshold parameter required for denoising the event data of the event data stream, wherein the association mask comprises at least one information unit;

determining a correlation information unit of the pixel event from the correlation mask based on the spatial information of the pixel event, wherein the correlation information unit has a spatial correlation relationship with the pixel event;

verifying the time information of the pixel event based on the time threshold parameter and the time information recorded by the associated information unit;

and according to the check result, carrying out event data denoising processing on the pixel event so as to carry out event data denoising processing on the event data stream.

In some class of embodiments, before the denoising the event data of the event data stream based on the spatio-temporal correlation relationship, the method further includes:

generating an initial filter mask for the event data stream based on parameter information of the vision sensor;

updating the initial filter mask based on the time information and the spatial information of the pixel event to obtain an updated filter mask;

performing event data filtering processing on the event data stream through the updated filtering mask to obtain a filtered event data stream;

the event data denoising processing on the event data stream based on the time-space incidence relation comprises the following steps: and carrying out event data denoising processing on the filtered event data stream based on the space-time incidence relation.

In some kind of embodiments, calculating a distance feature of the denoised pixel event according to the spatial information of the denoised pixel event includes:

determining an initial cluster set of the denoised pixel events, the initial cluster set comprising at least one initial cluster, the initial cluster being generated based on historical pixel events;

acquiring clustering characteristic information of the initial clustering;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the spatial information and the cluster characteristic information of the denoised pixel event.

In some class of embodiments, the cluster feature information includes a cluster feature of the initial cluster in at least one feature dimension;

calculating the distance feature between the denoised pixel event and the initial cluster according to the spatial information and the cluster feature information of the denoised pixel event, wherein the distance feature comprises the following steps:

determining a target characteristic dimension to be adjusted and an adjustment parameter of the target characteristic dimension from at least one characteristic dimension of the initial cluster;

based on the adjustment parameters, carrying out information conversion on the spatial information of the denoised pixel event and the clustering characteristic information;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted cluster characteristic information.

In some embodiments, calculating a distance feature between the denoised pixel event and the initial cluster according to the spatial information of the denoised pixel event and the cluster feature information includes:

determining coordinate axes to be zoomed in the reference coordinate system of the initial clustering and zooming parameters of the coordinate axes;

based on the scaling parameter, carrying out information conversion on the spatial information of the denoised pixel event and the clustering characteristic information;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted cluster characteristic information.

In some class of embodiments, the cluster feature information includes a cluster feature of the initial cluster in at least one feature dimension;

calculating the distance feature between the denoised pixel event and the initial cluster according to the spatial information and the cluster feature information of the denoised pixel event, wherein the distance feature comprises the following steps:

determining target clustering characteristics to be adjusted and adjustment parameters of the target clustering characteristics from the clustering characteristics;

based on the adjustment parameters, carrying out information conversion on the spatial information of the denoised pixel event and the target clustering characteristics;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted target cluster characteristic.

In some embodiments, calculating a distance feature between the denoised pixel event and the initial cluster according to the spatial information of the denoised pixel event and the cluster feature information includes:

determining the rotation angle characteristics of the initial clusters and the adjustment parameters of the rotation angle characteristics;

performing information conversion on the spatial information of the denoised pixel event and the rotation angle characteristic based on the adjusting parameter;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted rotation angle characteristic.

In a certain embodiment, the obtaining of the clustering feature information of the initial cluster includes:

determining a cluster shape corresponding to the initial cluster;

and acquiring clustering feature information of the initial clustering based on the clustering shape.

In a certain embodiment, clustering the denoised pixel events based on the distance features to obtain a clustering result, including:

determining a target cluster of the denoised pixel event from an initial cluster set of the denoised pixel event based on the distance features, the initial cluster set comprising at least one initial cluster, the initial cluster being generated based on historical pixel events;

updating the target cluster according to the spatial information of the denoised pixel event to obtain an updated target cluster;

and updating the initial cluster set based on the updated target cluster so as to determine a cluster result according to an update result of the initial cluster set.

In a certain embodiment, updating the initial cluster set based on the updated target cluster to determine a cluster result according to an update result of the initial cluster set includes:

updating the initial cluster set based on the updated target cluster to obtain an updated cluster set;

determining active clusters in the updated cluster set, and calculating cluster distances among the active clusters;

if the clustering distance meets a preset distance condition, clustering and merging the active clusters to obtain a processed cluster set;

and determining a clustering result according to the processed clustering set.

In a certain class of embodiments, training an object recognition model based on a neural network according to the clustering result to obtain a trained object recognition model based on a pulse neural network, includes:

generating sample data required by model training according to the clustering result;

training an object recognition model based on a neural network through the sample data to obtain a trained model corresponding to the neural network;

and generating a trained object recognition model based on the impulse neural network based on the trained model corresponding to the neural network.

An object recognition device, comprising:

the device comprises an acquisition unit, a processing unit and a display unit, wherein the acquisition unit is used for acquiring an event data stream acquired by a visual sensor, the event data stream comprises event data of at least one pixel event, and the event data comprises time information and space information of the pixel event;

the determining unit is used for determining the space-time association relation between the pixel events according to the time information and the space information;

a denoising unit, configured to perform event data denoising on the event data stream based on the temporal-spatial correlation relationship to obtain a processed event data stream, where the processed event data stream includes event data of denoised pixel events;

the calculating unit is used for calculating the distance characteristic of the denoised pixel event according to the spatial information of the denoised pixel event;

the clustering unit is used for clustering the denoised pixel events based on the distance characteristics to obtain a clustering result;

and the training unit is used for training the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the pulse neural network so as to recognize the object corresponding to the clustering result through the trained object recognition model.

A chip comprising an object recognition arrangement as described above or for performing an object recognition method as described above.

An electronic device comprises a response module, a processing module and the object recognition device, wherein the object recognition device recognizes an event collected by a visual sensor and transmits a recognition result to the processing module, and the processing module sends a control instruction to the response module.

The method and the device for processing the pixel events can acquire an event data stream acquired by a visual sensor, wherein the event data stream comprises event data of at least one pixel event, and the event data comprises time information and space information of the pixel event; determining a spatiotemporal association relationship between the pixel events according to the temporal information and the spatial information; based on the space-time incidence relation, carrying out event data denoising processing on the event data stream to obtain a processed event data stream, wherein the processed event data stream comprises event data of denoised pixel events; calculating the distance characteristic of the denoised pixel event according to the spatial information of the denoised pixel event; clustering the denoised pixel events based on the distance characteristics to obtain a clustering result; and training the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the pulse neural network, so as to recognize the object corresponding to the clustering result through the trained object recognition model.

The scheme realizes object recognition based on the event data stream collected by the visual sensor, and compared with a method for realizing object recognition based on image frames, the method can greatly save required resources. Specifically, the image frame includes all visual information, specifically includes data related to the object recognition task and data unrelated to the object recognition task, and thus the object recognition method based on the image frame has a problem of large data redundancy. As such, the image frame based object recognition scheme consumes a large amount of network bandwidth when acquiring data from an edge device (e.g., a camera) and transmitting predicted values; in addition, the object identification method based on the image frame has huge requirements on storage resources; furthermore, the conventional method can generally support only a lower frame rate in consideration of the time consumed for processing frame data. This also makes it more difficult to identify fast moving objects in the scene when deployed online.

The scheme realizes object identification based on discrete event data, so that the scheme can effectively solve the problem of data redundancy, and thus, the total amount of data is reduced, the network bandwidth required by data transmission is saved, and the requirement on storage resources is reduced; in addition, the scheme is essentially driven by pixel events to perform object recognition, so that the data frames do not need to be stored or collected when the scheme is deployed online, and the scheme can be operated efficiently in real time. Therefore, when the object is identified, the method and the device can achieve more efficient effects in the aspects of resources and power consumption, so that the efficiency of object identification is improved.

Drawings

Fig. 1 is a scene schematic diagram of an object identification method provided in an embodiment of the present application;

fig. 2 is a flowchart of an object identification method provided in an embodiment of the present application;

fig. 3 is a schematic diagram of a pixel distribution of an object identification method according to an embodiment of the present application;

FIG. 4 is a neighborhood diagram of an object recognition method provided in an embodiment of the present application;

fig. 5 is a schematic diagram of event data denoising processing in the object identification method according to the embodiment of the present application;

fig. 6 is a schematic diagram of event data filtering processing of an object identification method according to an embodiment of the present application;

FIG. 7 is a schematic diagram illustrating a distortion of a coordinate system of an object recognition method according to an embodiment of the present application;

fig. 8 is an angle rotation diagram of an object recognition method according to an embodiment of the present disclosure;

fig. 9 is a schematic diagram of a clustering process of an object identification method according to an embodiment of the present application;

fig. 10 is a sample data schematic diagram of an object identification method according to an embodiment of the present application;

fig. 11 is another schematic flowchart of an object identification method according to an embodiment of the present application;

fig. 12 is a schematic identification flow chart of an object identification method according to an embodiment of the present application;

FIG. 13 is a schematic diagram of parallel processing of an object recognition method according to an embodiment of the present application;

fig. 14 is a schematic structural diagram of an object recognition apparatus according to an embodiment of the present application;

fig. 15 is a schematic structural diagram of a computer device provided in an embodiment of the present application.

The meaning of the reference numbers or symbols in the specification are explained as follows:

px: a pixel;

e1: pixel event E1;

n: domain size;

1001: the spatial range of the time information to be viewed by the filter;

DVS: a dynamic vision sensor;

e (x, y, t, p): pixel event E, where x represents the abscissa of the pixel, y represents the ordinate of the pixel, t represents temporal information, and p represents visual information captured by the pixel;

filter [ x, y ]: time information recorded in an associated information unit of the pixel event E, wherein x represents the pixel event E;

filter (t): a variable for temporarily storing the value of filter [ x, y ];

e (t): time information corresponding to when the pixel generates the pixel event E;

e (x, y): generating spatial information for a pixel of pixel event E;

e (y): generating an abscissa of a pixel of pixel event E;

e (x): generating an ordinate of a pixel of pixel event E;

T_thr : a transition state threshold;

t_c: the current time;

y _ size: the major radius of the elliptical cluster;

x _ size: short radius of ellipse clustering;

e2: denoised pixel event E2;

e2 (x): generating the abscissa of the pixel of pixel event E2;

e2 (y): generating an ordinate of the pixel of pixel event E2;

e2': adjusted denoised pixel event E2';

e2' (x): generating the abscissa of the pixel of pixel event E2';

e2' (y): generating the ordinate of the pixel of pixel event E2';

SNN: a spiking neural network.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

Neurons in Spiking Neural networks (Spiking Neural networks) are a type of simulation of biological neurons. Inspired by biological neurons, some concepts related to biological neurons, such as synapses, membrane voltages, post-synaptic currents, post-synaptic potentials, etc., are also referred to using the same terminology when referring to neuron-related concepts in a spiking neural network, according to expressions that are custom defined in the art. In the brain-like chip, a circuit for simulating neurons and a circuit for simulating synapses are designed. That is, chips, etc., and these "biological" concepts in the hardware field are referred to as corresponding analog circuits according to common conventions in the field. Unless specifically indicated otherwise, references to concepts such as those similar to the biological layer described above in this disclosure refer to the corresponding concepts in the spiking neural network rather than the actual biological cell layer.

The embodiment of the application provides an object identification method and device, a chip and electronic equipment. The object recognition device may be specifically integrated in various electronic devices, and the electronic devices may be terminals and other devices.

Referring to fig. 1, the electronic device may acquire an event data stream collected by a vision sensor, the event data stream including event data of at least one pixel event, the event data including temporal information and spatial information of the pixel event; determining a spatio-temporal association relation between pixel events according to the time information and the space information; based on the space-time incidence relation, carrying out event data denoising processing on the event data stream to obtain a processed event data stream, wherein the processed event data stream comprises event data of denoised pixel events; calculating the distance characteristic of the denoised pixel event according to the spatial information of the denoised pixel event; clustering the denoised pixel events based on the distance characteristics to obtain a clustering result; and training the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the impulse neural network, so as to recognize the object corresponding to the clustering result through the trained object recognition model.

The following are detailed below. It should be noted that the following description of the embodiments is not intended to limit the preferred order of the embodiments.

In the object identification method provided in the embodiment of the present application, as shown in fig. 2, a specific process of the object identification method may be as follows:

s101, acquiring an event data stream acquired by a visual sensor, wherein the event data stream comprises event data of at least one pixel event, and the event data comprises time information and spatial information of the pixel event.

The visual sensor is an instrument for acquiring visual information of an external environment by using an optical element and an imaging device, and the performance of the visual sensor can be described by using resolution. The accuracy of the vision sensor is not only related to the resolution, but also to the detection distance of the object to be measured. The further the object to be measured is, the poorer the absolute positional accuracy. For example, a vision sensor may collect visual information at a fixed frequency; as another example, a visual sensor may collect visual information based on Address-Event Representation (AER).

As an example, the Vision Sensor may include a Dynamic Vision Sensor, and particularly, the Vision Sensor in the present application may be an AER-based Dynamic Vision Sensor (DVS). Compared with the traditional visual image acquisition mode which is based on the frame acquired by fixed frequency, the dynamic visual sensor simulates the working mechanism of biological vision and has the defects of high redundancy, high delay, high noise, low dynamic range, high data volume and the like. The dynamic vision sensor works asynchronously for pixels, only outputs addresses and information of pixels with changed light intensity, but passively reads out information of each pixel in a frame in sequence, eliminates redundant data from a source, has the characteristics of real-time dynamic response of scene change, ultra-sparse representation of images, asynchronous output of events and the like, and can be widely applied to the fields of high-speed target tracking, real-time monitoring, industrial automation, robot vision and the like.

The pixel event is an event generated by the visual sensor based on visual information collected by the pixel, where the visual information may include light change information, such as light intensity or brightness change information. For example, when a visual sensor detects that visual information received by a pixel changes, a corresponding pixel event may be generated; for example, when a dynamic vision sensor detects a change in the brightness received by a pixel, a corresponding pixel event may be generated.

Where a pixel event may have corresponding event data, for example, a dynamic vision sensor may generate a pixel event upon detecting a change in brightness received by a pixel, which may include the following event data: temporal information, spatial information, and visual information, among others.

The time information is related information describing the time of the pixel event generation, for example, the time information may be specifically presented in the form of a timestamp, and for example, the time information may be presented in the form of a time difference from a target pixel event, and the like.

The spatial information is position information describing a pixel corresponding to the generated pixel event, and may be specifically presented in the form of pixel coordinates, for example.

Where the visual information is the visual information captured by the pixel event, for example, the visual information may include brightness change information captured by the pixel event.

As an example, the event data of the pixel events may be represented based on AER, for example, the event information of each pixel event may specifically be a quadruple E (x, y, t, p), where x represents the abscissa of the pixel, y represents the ordinate of the pixel, t represents the time information, and p represents the visual information captured by the pixel. For example, t may be embodied as a time stamp. For example, when p takes +1, it may indicate that brightness is detected to be bright, and when p takes-1, it may indicate that brightness is detected to be dark.

Therefore, in the present application, spatial information of a pixel generating the pixel event E may be represented by E (x, y), specifically, E (x) may represent an abscissa of the pixel generating the pixel event E, and E (y) may represent an ordinate of the pixel generating the pixel event E, and further, E (t) may represent corresponding time information when the pixel generates the pixel event E, and E (p) may represent visual information captured when the pixel generates the pixel event E, which is not described in detail later. Alternatively, in practical applications, x, y may be referred to as a spatial channel of pixel events, t as a temporal channel of pixel events, and p as a polar channel of pixel events.

Thus, in the present application, an event data stream is a data stream comprising event data of at least one pixel event.

There are various ways to obtain the event data stream collected by the visual sensor, for example, the visual sensor may be integrated in a terminal, and the event data stream collected by the visual sensor may be transmitted by the terminal.

It is noted that when the visual sensor is a dynamic visual sensor, for the dynamic visual sensor, it generates a corresponding pixel event only when the brightness received by the pixel changes, and propagates the pixel event. In this way, it is ensured that the received data is sparse and, in the present application, the received data is all related to the task of object recognition, so the amount of data that needs to be processed in the present application is very small compared to a frame-based vision scheme, and is therefore more efficient to implement.

102. And determining the space-time correlation relationship between the pixel events according to the time information and the space information.

The spatio-temporal association relationship is used for describing the association relationship of the pixel events in a time dimension and a space dimension. Specifically, in the present application, taking a visual sensor as an example of a DVS, pixel events triggered and generated by a target object are considered to be correlated in time and space, for example, event data of at least one pixel event triggered and generated by the target object is included in an acquired event data stream, pixels corresponding to the pixel events are adjacent pixels, and the pixel events are generated by the pixels within a preset time interval. Thus, in the present application, object recognition may be achieved based on spatiotemporal correlations between pixel events.

In the present application, there are various ways to determine the spatio-temporal association relationship between pixel events, for example, a preset data structure may be used as a container for recording and updating the spatio-temporal association relationship between pixels, so that, in practical applications, the spatio-temporal association relationship between pixel events can be quantitatively determined according to the time information and the spatial information of the pixel events, and the related steps required for subsequently implementing object identification can be conveniently implemented.

In an embodiment, the mask may be utilized to record and update the spatio-temporal correlation between the pixels, and in particular, the step of "determining the spatio-temporal correlation between pixel events according to the temporal information and the spatial information" may include:

generating an initial correlation mask of the event data stream based on parameter information of the vision sensor, wherein the initial correlation mask comprises at least one information unit, and the information unit is used for recording time information of pixel events;

compressing the spatial information of the pixel event to obtain compressed spatial information;

determining a target information unit corresponding to the pixel event according to the compressed spatial information;

and according to the time information of the pixel events, information updating is carried out on the target information unit so as to generate an updated association mask, and the updated association mask represents the space-time association relation between the pixel events.

The mask (mask) is a template for processing the pixel event, for example, the mask may be a template for filtering the pixel event. The concept of a mask in the present application may be used for reference to the concept of a mask in digital image processing, and specifically, in digital image processing, the image mask is a mask that uses a selected image, graphic or object to completely or partially block a processed image so as to control an image processing area or processing process.

In the present application, the form of the mask may be various, for example, the mask may be presented in the form of a matrix, for example, the mask may be a two-dimensional matrix M having several elements, each of which may serve as an information unit for recording related information.

It should be noted that, in the present application, when taking the two-dimensional matrix M as an example of a mask, each element in the mask may be represented in the form of M (x, y), where x represents a row number of the element in the two-dimensional matrix M, and y represents a column number of the element in the two-dimensional matrix M, that is, the function M () may be used to determine a target element in the mask M, and details thereof are not described later.

In this application, the correlation mask is a mask for recording and updating spatio-temporal correlation relationships between pixel events. For example, the correlation mask may be in the form of a two-dimensional matrix M having several elements, each of which may serve as an information unit for recording time information of a pixel event. In the present application, the correlation mask may be used to de-noise pixel events in the event data stream to remove target noise in the event data stream. For example, the association mask may be used as a container for recording and updating the spatio-temporal association relationship between the pixel events, and in this case, the information of the association mask may be updated, so that the updated association mask represents the spatio-temporal association relationship between the pixel events.

The parameter information of the vision sensor may include resolution, signal-to-noise ratio, and the like.

The initial correlation mask may be generated in various ways based on the parameter information of the visual sensor, for example, the parameter information of the visual sensor may be resolution, and the correlation mask may be in the form of a matrix, so that the size of the initial correlation mask to be generated may be determined according to the resolution of the visual sensor, so as to initialize the required initial correlation mask, and further generate the initial correlation mask of the size, where the initial correlation mask includes at least one information unit, for example, each information unit may be specifically an element in the matrix, where time information of a pixel event may be recorded.

The information compression of the spatial information of the pixel event refers to the compression from a spatial dimension so as to reduce the storage resource required for recording the time information of the pixel event. For example, the visual sensor may be a dynamic visual sensor, the correlation mask may be in the form of a matrix, and it may be set that if corresponding pixels of pixel events on the visual sensor meet a preset spatial distribution requirement, the pixel events share one information unit in the correlation mask, and the information unit is used for recording time information of the pixel events.

As an example, the visual sensor may be a DVS, the association mask may be in a matrix form, and it may be specified that for a pixel on the DVS, if it meets the spatial distribution requirement shown in fig. 3, the pixel events corresponding to the 4 pixels will share one information unit in the association mask for recording the time information of the pixel event. It should be noted that fig. 3 is only an example of the requirement of spatial distribution, and in practical applications, the requirement of spatial distribution of pixels on the visual sensor may be set according to requirements.

The method for determining the shared information unit corresponding to the pixel event may be various, for example, the method may be implemented by performing bit operation on spatial information of the pixel event, and specifically, the step "performing information compression on the spatial information of the pixel event to obtain compressed spatial information" may include:

performing bit operation on the spatial information of the pixel event to perform information compression on the spatial information of the pixel event;

based on the operation result, compressed spatial information of the pixel event is determined.

All the numbers in the program are stored in the form of binary system in the computer memory, and the bit operation is to directly operate the binary bit of the integer in the memory. The bit operation may include a variety of operations, such as an and operation, an or operation, an xor operation, an inversion operation, a shift operation, and the like. The shift operation may include a left shift operation and a right shift operation.

In practical applications, the spatial information of the pixel event may be compressed by selecting different operation manners or combining a plurality of operation manners to perform bit operation on the spatial information of the pixel event.

In the foregoing example, the further explanation may be made by taking the example of a bit operation, specifically a shift operation, for example, a right shift operation. The resolution of the DVS may be 128 × 128 pixels, and for the pixel event E1 (3, 3, t1, p 1), the spatial information of the pixel event E1 is the pixel coordinate (3, 3), and the spatial information of the pixel event E1 may be subjected to bit operation, for example, right shift operation, to perform information compression. Specifically, 3 is 00000011 in binary, and 00000001, that is, 1 in decimal, can be obtained by right-shifting the binary by one bit, that is, after the spatial information (3, 3) is compressed, the compressed spatial information is (1, 1).

Further, a target information element of the pixel event in the initial correlation mask may be determined based on the compressed spatial information. Specifically, in the above example, the spatial information of the pixel event is (3, 3), and the corresponding compressed spatial information is (1, 1), so that the target information unit of the pixel event in the initial correlation mask may be determined to be M (1, 1).

In this way, the target information unit may be updated according to the temporal information of the pixel event to generate an updated correlation mask. Specifically, in this example, M (1, 1) may be subjected to information update according to the time information of the pixel event E1, that is, M (1, 1) = t 1.

In the above example, taking the specific pixel event E1 as an example, the information update of the initial correlation mask M is explained based on the spatial information and the temporal information of the pixel event E1. Similarly, the initial correlation mask may be updated based on other pixel events collected by the vision sensor to generate an updated correlation mask, where the updated correlation mask characterizes a spatio-temporal correlation between the pixel events because time information of the pixel events is recorded in an information unit in the updated correlation mask, and the pixel events are determined based on a mapping relationship of spatial information compression when determining a target information unit.

In an embodiment, considering that, in addition to information compression from a spatial dimension to reduce the storage resources required for recording the time information of the pixel event, information compression may also be performed from a temporal dimension to further save the required storage resources, specifically, the step "performing information update on the target information unit according to the time information of the pixel event" may include:

determining a time threshold parameter required for denoising event data of the event data stream;

according to the time threshold parameter, performing information compression on the time information to obtain compressed time information;

and updating the information of the target information unit based on the compressed time information.

It is considered that in practical applications, due to hardware reasons of the vision sensor, pixel events may be generated that are not related to the visual information captured by the pixels, in particular, for DVS, the pixels therein may generate pixel events that are not related to brightness variations, which may cause noise, for example, when the pixel events are plotted in the form of an image histogram, i.e., appear as image noise. For the noise generated by such activities, the object identification is affected, for example, the accuracy of identifying small objects is reduced, and therefore, the noise generated by such activities can be filtered out by performing event data denoising processing on the event data stream.

In the application, event data denoising processing can be realized based on a spatio-temporal correlation relationship between pixel events, specifically, if the spatio-temporal correlation between a pixel event and a previous pixel event is higher than a preset threshold, the pixel event is not filtered, otherwise, the pixel event can be considered as noise and needs to be filtered.

In practical application, a relevant filter can be designed to implement event data denoising processing, and specifically, the filter can implement event data denoising processing by using an updated correlation mask of an event data stream. And, the filter can pass the parameter n and the parameter dt_thrTo determine the spatiotemporal correlation between the computed pixel events.

In particular, the parameter n specifies a neighborhood size that is used for the filter to determine, in the updated correlation mask, a range of information elements to be compared with the temporal information of the pixel event. As an example, if the latest pixel event received by the correlation mask is E1, and after the spatial information of the pixel event E1 is compressed, it is determined that the corresponding target information unit of the pixel event E1 in the correlation mask is as shown in fig. 4, when the parameter n takes 2, the filter will look at the time information in the spatial range shown by 1001, that is, the parameter n is used for the filter to calibrate and determine the spatial range of the spatial correlation of the pixel event, and in practical applications, the parameter n may also be referred to as a division factor.

In particular, parameter dt_thrHow long the next pixel event in a given neighborhood is considered to have a temporal correlation with the pixel event in the neighborhood, i.e. the parameter dt_thrIs a time threshold parameter for filter calibration to determine the time correlation of pixel events.

The information compression of the time information of the pixel event refers to the compression from the time dimension to reduce the storage resource required for recording the time information of the pixel event, and for example, the compression of the time information of the pixel event can be realized by a shift operation. Specifically, the step of performing information compression on the time information according to the time threshold parameter to obtain the compressed time information may include:

determining a shifting parameter of the time information according to the time threshold parameter;

performing bit operation on the time information based on the shifting parameter so as to perform information compression on the time information;

based on the operation result, post-compression time information of the pixel event is determined.

The shift parameter is used to determine the number of bits that the time information needs to be shifted when the time information of the pixel event is shifted, for example, if the shift parameter of the time information is 5, the time information can be shifted by shifting the binary number corresponding to the time information by 5 bits to the right or left.

In an embodiment, the time information of the pixel event may be specifically a time stamp, and in practical applications, the resolution of the time stamp may be specifically 32 bits or 64 bits, and the like. For example, the time threshold parameter dt_thrCan be set to a value on the order of milliseconds, based on the time threshold parameter dt_thrWhen the time correlation is determined for the time stamp of the pixel event, part of the bits of the time stamp are not necessary. As an example, the time threshold parameter dt_thrCan be set to 10000 mus, which can be used with 13 bits (pairs) in hardwareCorresponding to 8192 mus) or 14 bits (corresponding to 16384 mus), so that when the time stamp is judged to be time-dependent, the number of bits can be thrown out of the least significant bits of the time stamp, i.e. the number is a shift parameter, and based on the shift parameter, the time stamp can be right-shifted to achieve information compression of the time stamp, so that the entire number of bits of the time stamp does not need to be stored, but only a part of the significant bits of the time stamp, where the part of the significant bits is based on dt (dt) passing through the time threshold parameter_thrThe time information of the pixel event is determined by judging the time correlation. That is, the present application enables information compression of temporal information of pixel events according to a temporal threshold parameter. In practical applications, the least significant digit to be thrown out in the time information determined according to the time threshold parameter may also be referred to as a shift factor.

Further, the target information unit is updated based on the compressed time information, and specifically, the compressed time information may be recorded in the target information unit to realize the information update of the target information unit.

In the above example, if the resolution of the timestamp is 32 bits, and the time threshold parameter dt_thrIs set to 10000And the time stamp can be represented by using 14 bits in hardware, so that the time stamp can be compressed according to the time threshold parameter, the 14 least significant bits of the time stamp can be discarded, the remaining 18 bits are used for representing the time stamp, and the time stamp represented by 18 bits is the compressed time stamp. Further, the compressed timestamp may be recorded in a target information unit corresponding to the pixel event, so as to update information of the target information unit.

In the application, resources required by information storage can be effectively saved through space compression and time compression. As an example, the visual sensor may be a DVS with a resolution of 128 × 128 pixels, and the resolution of the timestamp may specifically be 32 bits, and before the spatial compression and temporal compression method described in this application is not applied, the time information in the pixel event acquired by the visual sensor is stored, and the required memory resource is 128 × 128 × 32. After the spatial compression and temporal compression method described in the present application is applied, for example, after the division factor takes 2 and the shift factor takes 14, the temporal information in the pixel event collected by the visual sensor is stored, and the required memory resource is 64 × 64 × 18, which is reduced by 14% compared with that before compression is not applied, so that the storage resource required by information storage can be effectively saved.

103. And based on the space-time incidence relation, carrying out event data denoising processing on the event data stream to obtain a processed event data stream, wherein the processed event data stream comprises event data of denoised pixel events.

Based on the spatio-temporal correlation relationship between the pixel events, there may be various implementation manners for performing event data denoising processing on the event data stream, for example, in the present application, an association mask representing the spatio-temporal correlation relationship may be generated, so that the association mask may be utilized to perform event data denoising processing on the event data stream, specifically, the step "performing event data denoising processing on the event data stream based on the spatio-temporal correlation relationship" may include:

determining an association mask representing a time-space association relation and a time threshold parameter required for denoising event data of an event data stream, wherein the association mask comprises at least one information unit;

determining a correlation information unit of the pixel event from a correlation mask based on the spatial information of the pixel event, wherein the correlation information unit and the pixel event have a spatial correlation relationship;

verifying the time information of the pixel event based on the time threshold parameter and the time information recorded by the associated information unit;

and according to the verification result, carrying out event data denoising processing on the pixel event so as to carry out event data denoising processing on the event data stream.

The definition and examples of the correlation mask, the event data denoising process, the time threshold parameter, and the information unit are already explained in step 102, and are not described herein again.

The associated information unit of the pixel event refers to an information unit in the associated mask, which has a spatial association relationship with the pixel event.

In the present application, there are various ways to determine the related information unit of the pixel event from the related mask based on the spatial information of the pixel event, for example, the spatial information of the pixel event may be compressed to determine the target information unit of the pixel event in the related mask, and the information unit is used as the related information unit of the pixel event; for another example, after spatial information of a pixel event is compressed to determine a target information unit of the pixel event in an associated mask, an information unit in the associated mask, which has a neighborhood relationship with the target information unit, is determined as an associated information unit of the pixel event; and so on.

Further, the time information of the pixel event may be checked based on the time threshold parameter and the time information recorded in the associated information unit, specifically, as an example, referring to fig. 5, the time information of the pixel event may be compared with the time information recorded in the associated information unit to determine an event data denoising processing result of the pixel event, for example, if a difference between the time information of the pixel event and the time information of the associated information unit is smaller than the time threshold parameter, the event data denoising processing result of the pixel event is a pass, that is, the event data of the pixel event is not filtered as noise; otherwise, the event data denoising result of the pixel event is filtering, that is, the event data of the pixel event is filtered from the event data stream. In addition, as can be seen from fig. 5, the associated mask may also be updated according to the time information of the pixel event, and specifically, the time information recorded in the associated information unit of the pixel event in the associated mask may be updated according to the time information of the pixel event, so as to update the associated mask.

In practical applications, it is considered that the visual sensor exhibits a hot pixel phenomenon due to aging of the visual sensor or other device reasons, and in particular, the hot pixels have a reduced sensitivity to the brightness variations they observe and generate more pixel events than other pixels in the same time.

Due to the existence of the hot pixel phenomenon, the event data corresponding to the noise event generated by the hot pixel can be included in the event data stream collected by the visual sensor, so that before the background activity filtering is performed on the event data stream, the event data corresponding to the noise event in the event data stream can be filtered to obtain the filtered event data stream, so that the event data denoising processing can be performed on the filtered event data stream based on the spatio-temporal correlation relationship between the pixel events, and the background activity filtering can be performed on the filtered event data stream. Specifically, before the step "denoising event data from an event data stream based on a spatiotemporal correlation," the object identification method may further include:

generating an initial filter mask of the event data stream based on parameter information of the vision sensor;

updating the initial filter mask based on the time information and the spatial information of the pixel event to obtain an updated filter mask;

performing event data filtering processing on the event data stream through the updated filtering mask to obtain a filtered event data stream;

in this case, the step "performing event data denoising processing on the event data stream based on the spatiotemporal correlation relationship" may include: and based on the space-time incidence relation, carrying out event data denoising processing on the filtered event data stream.

The event data filtering process is a process of filtering event data corresponding to a pixel event generated by a hot pixel from an event data stream, and thus, noise caused by the hot pixel can be filtered by performing the event data filtering process on the event data stream.

Here, the filter mask is a mask for performing event data filtering processing on the event data stream, and therefore, the filter mask may be referred to as a hot pixel filter. Similar to the association mask, the filter mask may include at least one information element for storing information relevant to performing the event data filtering process. Similar to the correlation mask, the form of the filter mask may be various, for example, the filter mask may be in the form of a matrix; as another example, the filter mask may be in the form of a set; and so on.

The initial filter mask may be generated in various ways based on parameter information of the vision sensor, for example, the size of the initial filter mask may be determined based on the resolution of the vision sensor, and the initial filter mask may be generated. For example, the visual sensor may specifically be DVS, the form of the filter mask may be a matrix M, the size of the matrix M may be determined based on the resolution of the DVS, and then the initial filter mask may be generated.

In the application, the initial filtering mask can be updated to obtain the updated filtering mask, so that the event data filtering processing can be executed on the event data stream by using the updated filtering mask.

In the present application, there are various ways to update the initial filter mask based on the temporal information and the spatial information of the pixel event, for example, referring to fig. 6, the event data filtering process may include an observation phase and a filtering phase, wherein the observation state is used to update the initial filter mask, and the filtering state is used to perform the event data filtering process on the event data stream through the updated filter mask. Specifically, when the current time t is detected_cEqual to the transition state threshold T_thrThen the state will change from the observation state to the filtering state.

In the observation state, the initial filter mask may be updated based on the temporal and spatial information of each pixel event. As an example, the vision sensor may be DVS, the initial filter mask may be M, and M and DVS have the same resolution, that is, the initial filter mask M may specifically be in the form of a two-dimensional matrix, and the updating of the initial filter mask M based on the temporal information and the spatial information of the pixel event may be implemented with reference to equation (1):

（1）

wherein, T_thrFor the transition state threshold, since event information of E can be represented by a quadruple E (x, y, t, p) for a pixel event E, where x represents an abscissa of the pixel, y represents an ordinate of the pixel, t represents time information, and p represents visual information captured by the pixel, E (x) is an abscissa of the pixel generating the pixel event E, E (y) represents an ordinate of the pixel generating the pixel event E, and E (t) represents corresponding time information when the pixel generates the pixel event E. In addition, the function M () is used to determine the target element in the initial filter mask M.

Thus, in this example, the information elements in the filter mask are used to record the event count of the pixel event. It is noted that in practical applications, the observation phase should be initiated prior to the motion within the field of view of the dynamic vision sensor in order to perform proper calibration.

Further, when the detected time is equal to the transition state threshold T_thrThe observation state can be changed into the transition state, and M at the time can be used as an updated filtering mask, so that event data filtering processing on the event data stream can be realized through the updated filtering mask in the transition state.

In this application, there may be a variety of ways to perform event data filtering processing on an event data stream through the updated filter mask, for example, a target information unit may be determined from the updated filter mask according to spatial information of a pixel event, and information stored in the target information unit is verified, if the verification passes, it is determined that the pixel event is not filtered, otherwise, the pixel event is filtered, so as to obtain a filtered event data stream.

As an example, the visual sensor may be DVS, the initial filtering mask may be a matrix M, and M has the same resolution as DVS, and an information unit in M is used to record an event count of a pixel event, then in the filtering stage, the updated filtering mask may be implemented with reference to equation 2, and the event data filtering process is performed on the event data stream:

（2）

wherein, C_thrFor the event count threshold, since event information of E can be represented by a quadruple E (x, y, t, p) for a pixel event E, where x represents the abscissa of the pixel, y represents the ordinate of the pixel, t represents time information, and p represents visual information captured by the pixel, E (x) is the abscissa of the pixel generating the pixel event E, and E (y) represents the ordinate of the pixel generating the pixel event E. In addition, the function M () is used to determine the target element in the initial filtering mask M, pass represents that the pixel event E passes through, and filter represents that the pixel event E is filtered.

That is, in the filtering stage, if the number of pixel events triggered by a pixel in the DVS exceeds the event count threshold C_thrThen the pixel event is filtered (corresponding to the filter in equation (2)), otherwise the pixel event passes through the hot pixel filter (corresponding to the pass in equation (2)). Thus, in this example, the hot pixel filter is a low pass filter based on event counts.

And after the event data stream is subjected to data filtering processing through the updated filtering mask to obtain the filtered event data stream, further performing event data denoising processing on the filtered event data stream based on the space-time incidence relation between the pixel events. In the present application, the step "denoising the event data from the filtered event data stream based on the temporal-spatial association relationship" may refer to the step "denoising the event data from the event data stream based on the temporal-spatial association relationship", which is not described herein again.

104. And calculating the distance characteristic of the denoised pixel event according to the spatial information of the denoised pixel event.

Since clustering is the process of dividing a set of physical or abstract objects into classes composed of similar objects, and the clusters (which may also be referred to as clusters) generated after clustering are a set of data objects that are similar to objects in the same cluster and different from objects in other clusters. Therefore, in the present application, the clustering process is performed on the denoised pixel events, which essentially is to group the denoised pixel events and determine the clustering process to which the denoised pixel events belong. The mode of determining the cluster to which the denoised pixel event belongs is determined by calculating the distance between the denoised pixel event and each cluster, so that the distance between the denoised pixel event and each cluster is the distance characteristic of the denoised pixel event.

Since the distance characteristic of the denoised pixel event represents the distance between the denoised pixel event and each cluster, when the distance characteristic of the denoised pixel event is calculated, in addition to the spatial information of the denoised pixel event, the cluster characteristic information of each cluster, such as the cluster center information of the cluster, the cluster shape information, and the like, needs to be taken into account. Specifically, the step of calculating the distance feature of the denoised pixel event according to the spatial information of the denoised pixel event may include:

determining an initial cluster set of the denoised pixel events, wherein the initial cluster set comprises at least one initial cluster, and the initial cluster is generated based on the historical pixel events;

acquiring clustering characteristic information of initial clustering;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the spatial information and the cluster characteristic information of the denoised pixel event.

The historical pixel event of the denoised pixel event refers to a pixel event generated before the denoised pixel event in the time dimension.

The initial cluster set of the denoised pixel event is a cluster set generated based on a historical pixel event of the denoised pixel event, and specifically, the historical pixel event can be clustered to obtain at least one initial cluster, so that the initial cluster set is generated.

In the present application, there are various ways to generate the initial cluster based on the historical pixel event, for example, the initial cluster may be generated based on a mean shift algorithm, where the mean shift algorithm is a nonparametric method based on density gradient rise, and the target position may be found through iterative operation to realize target tracking.

For another example, a new feature may be added on the basis of the mean shift algorithm to obtain an improved algorithm, and the clustering process may be performed on the historical pixel events based on the improved algorithm to obtain an initial cluster set. As an example, in the mean shift algorithm, the shape of each cluster is specified as a circle, which makes the component performing the clustering process a circle tracker in the process of performing object recognition on pixel events. In the present application, considering that a circular tracker may affect the clustering accuracy and further affect the accuracy of object identification, when performing clustering processing on historical pixel events, in addition to the circular tracker, the present application also proposes to add trackers of various shapes, such as an elliptical tracker and a rectangular tracker, so as to more accurately perform clustering processing on the historical pixel events to generate an initial clustering set of historical pixel times.

After determining the initial cluster set of the denoised pixel event, further, the cluster feature information of the initial clusters in the initial cluster set may be obtained.

The clustering feature information of the initial clustering is related information describing the clustering features of the initial clustering, and the clustering features may include clustering position, clustering shape, clustering size, clustering rotation angle, clustering speed, and the like.

In this application, considering that the shape of a cluster is not limited to a circle, but may include multiple situations, therefore, when calculating the distance feature between a denoised pixel event and an initial cluster, the required cluster feature information may also be determined based on different situations of the cluster shape, specifically, the step "obtaining the cluster feature information of the initial cluster" may include:

determining a cluster shape corresponding to the initial cluster;

and acquiring clustering characteristic information of the initial clustering based on the clustering shape.

As an example, when calculating the distance feature between the denoised pixel event and the initial cluster, if the cluster shape corresponding to the initial cluster is an ellipse, equation (3) may be referred to:

（3）

wherein, in formula (3), C_lRefers to the position of the ellipse cluster, C_sRefers to the size of the ellipse cluster, and SM may be set to C_sE (x) is the abscissa of the pixel generating the pixel event E, and E (y) is the ordinate of the pixel generating the pixel event E, since the event information of E can be represented by a quadruple E (x, y, t, p) where x represents the abscissa of the pixel, y represents the ordinate of the pixel, t represents the time information, and p represents the visual information captured by the pixel. It is known that the cluster feature information to be obtained may include position information, size information, and the like of the elliptical cluster.

If the cluster shape corresponding to the initial cluster is a rectangle, the following equation (4) may be referred to:

（4）

wherein, in formula (4), C_lRefers to the position of the rectangular cluster, C_sReferring to the size of the rectangular cluster, and SM is a multiple of the parameter size, since for a pixel event E, event information of E can be represented by a quadruple E (x, y, t, p), where x represents the abscissa of the pixel, y represents the ordinate of the pixel, t represents time information, and p represents visual information captured by the pixel, E (x) is the abscissa of the pixel generating the pixel event E, and E (y) represents the ordinate of the pixel generating the pixel event E. It is known that the cluster feature information to be obtained may include position information, size information, and the like of the rectangular cluster.

It should be noted that, in practical applications, the cluster shape may also include multiple situations, and then when the distance feature between the denoised pixel event and the initial cluster is calculated, the feature information of the initial cluster to be acquired may be changed correspondingly. Alternatively, the clustering rotation angle, the clustering speed, and the like may also be taken into consideration.

Further, the distance characteristic between the denoised pixel event and the initial cluster can be calculated according to the spatial information and the cluster characteristic information of the denoised pixel event.

As an example, equation (3) describes calculating the distance between the denoised pixel event and the elliptical cluster by the left part of the inequality, and equation (3) describes determining whether the pixel event belongs to the elliptical cluster by determining whether the pixel event is located in a search distance of an elliptical cluster around the center of the cluster.

Similarly, equation (4) describes determining the distance between the denoised pixel event and the rectangular cluster by the left part of the inequality, and equation (4) describes determining the search distance of whether the pixel event is in a rectangular cluster by determining whether the pixel event is located in a rectangular region around the center of the cluster to determine whether the pixel event belongs to the rectangular cluster.

In addition, compared with a single circular cluster and a simple distance calculation mode, the method and the device have the advantages that not only are clusters of more shapes provided, but also the distance between the pixel events and the clusters is calculated more efficiently, and an improved mode is provided. For example, considering that the cluster feature information of the initial cluster may include a cluster feature of the initial cluster in at least one feature dimension, the feature dimension may be adjusted, the cluster feature may be adjusted, and the like, so that the distance feature between the denoised pixel event and the initial cluster may be calculated based on the adjustment result, which will be described below.

In an embodiment, the step of calculating a distance feature between the denoised pixel event and the initial cluster according to the spatial information and the cluster feature information of the denoised pixel event may include:

determining a target feature dimension to be adjusted and an adjustment parameter of the target feature dimension from at least one of the initial clusters;

based on the adjustment parameters, carrying out information conversion on the spatial information and the clustering characteristic information of the denoised pixel event;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted cluster characteristic information.

Wherein the characteristic dimension of the initial cluster is a dimension related to the cluster feature of the initial cluster, as an example, the cluster feature of the initial cluster may include a cluster position, for example, the cluster position of the initial cluster may be determined using coordinate information of a cluster center of the initial cluster in a reference coordinate system, such as (x, y), where x may represent position information of the cluster center corresponding on an x-axis, and y may represent position information of the cluster center corresponding on a y-axis, and thus, in this example, this cluster feature has two related characteristic dimensions for the cluster position.

As another example, the cluster characteristics of the initial clusters may include a cluster angle, e.g., if the cluster shape of the initial clusters is symmetric (e.g., rectangular, elliptical, etc.) along the cluster center, the cluster angle of the initial clustersSatisfies the following conditions:thus, in this example, for a cluster angle this cluster feature has an associated feature dimension.

In the present application, a feature dimension related to the calculation of the distance feature may be taken as a target feature dimension. For example, referring to equation (3), when calculating the distance feature between the denoised pixel event and the elliptical initial cluster, the feature dimensions involved include a feature dimension related to the position information and a dimension related to the cluster size, and these can be used as the target feature dimensions.

The adjustment parameter of the target feature dimension is a relevant parameter required for adjusting the target feature dimension, and there are various ways for adjusting the target feature dimension, for example, the target feature dimension may be adjusted by adjusting the scale of the target feature dimension, and the adjustment parameter may be a scale parameter for scale adjustment. For example, if the target feature dimension is a distance-related feature dimension, such as representing position information on a coordinate axis, the target feature dimension may be adjusted by enlarging or reducing the scale of the coordinate axis. For another example, the target feature dimension may be adjusted by mapping the target feature dimension to another feature dimension with reference to an embedding method in deep learning, so that the clustering feature in the target feature dimension may be correspondingly converted into a clustering feature in another feature dimension, such as a vector, in this example, the adjustment parameter may be a relevant parameter required for implementing mapping, for example, a matrix.

The adjustment parameters for determining the target feature dimension may be set manually or according to the cluster feature information, for example, may be determined according to the cluster shape of the initial cluster.

After the target characteristic dimension and the adjustment parameter are determined, the spatial information and the clustering characteristic information of the denoised pixel event can be subjected to information conversion based on the adjustment parameter. Specifically, since the target feature dimension is a feature dimension related to the calculated distance feature, and when the distance feature between the denoised pixel event and the initial cluster is calculated, the spatial information of the denoised pixel event needs to be taken into account, the spatial information of the denoised pixel event and the cluster feature related to the cluster feature and the calculated distance feature can be adaptively subjected to information conversion with reference to a manner of adjusting the target feature dimension based on the adjustment parameter. For example, referring to FIG. 3, the coordinate information (including E (x) and E (y)) of the denoised pixel event and the cluster feature information (including the position information: C) of the initial cluster can be adaptively adjusted according to the adjustment of the target feature dimension based on the adjustment parameter_l(x) And C_l(y); size information: c_s(x) And C_s(y); and other parameter information: size of parameterThe multiple SM) is adjusted.

Further, the distance characteristic between the denoised pixel event and the initial cluster can be calculated according to the converted space information and the converted cluster characteristic information. The distance feature may be calculated, for example, with reference to the left part of the inequality in equation (3).

In another embodiment, taking a target feature dimension, specifically a dimension related to a reference coordinate system of an initial cluster as an example, specifically, the step "calculating a distance feature between a denoised pixel event and the initial cluster according to spatial information and cluster feature information of the denoised pixel event" may include:

determining coordinate axes to be zoomed in a reference coordinate system of the initial clustering and zoom parameters of the coordinate axes;

based on the scaling parameters, carrying out information conversion on the spatial information and the clustering characteristic information of the denoised pixel event;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted cluster characteristic information.

The coordinate axes to be scaled in the reference coordinate system of the initial cluster may be determined in various ways, for example, the coordinate axes to be scaled may be determined based on the cluster shape of the initial cluster. Taking the cluster shape of the initial cluster as an ellipse as an example, if the long axis of the initial cluster is parallel to the x axis, the coordinate axis to be zoomed can be determined as the x axis; if the long axis of the initial cluster is parallel to the y-axis, the coordinate axis to be zoomed can be determined as the y-axis.

The manner of determining the scaling parameter for the coordinate axis may be various, for example, the scaling parameter may be determined according to the cluster size of the initial cluster. Taking the cluster shape of the initial cluster as an ellipse as an example, the ratio of the minor axis to the major axis of the initial cluster can be used as a scaling ratio.

After the coordinate axis to be zoomed and the zooming parameter of the coordinate axis are determined, the spatial information and the clustering characteristic information of the pixel event after denoising can be subjected to information conversion based on the zooming parameter.

Taking the shape of the initial cluster as an ellipse as an exampleReferring to FIG. 7, for denoised pixel event E2, the distance between denoised pixel event E2 and the elliptical cluster may be calculated according to parameter C characterizing the cluster size_sThe coordinate system is scaled to convert the calculation of the distance between denoised pixel event E2 and the elliptical cluster to a calculation of the distance between denoised pixel event E and the circular cluster. In particular, since the major axis of the ellipse cluster is parallel to the x-axis, the x-axis can be taken as the coordinate axis to be scaled, and the ratio of the minor axis to the major axis of the ellipse cluster, i.e., C, can be taken_s(y)/ C_s(x) As a scaling parameter. The x-axis may then be scaled according to the scaling parameter. And, the information conversion can be performed on the clustering feature information of the denoised pixel event E2 and the elliptical cluster adaptively based on the scaling parameter, so that in the scaled coordinate system, the elliptical cluster is converted into a circular cluster, and the denoised pixel event is also converted into the adjusted denoised pixel event E2' from the denoised pixel event E2. In this way, whether the denoised pixel event falls into the original cluster can be determined by judging whether the adjusted denoised pixel event E2' falls into the area around the radius of the circular cluster.

In the application, besides the characteristic dimension can be adjusted, the clustering characteristic can be adjusted to calculate the distance characteristic between the denoised pixel event and the initial cluster based on the adjustment result. Specifically, the step of calculating the distance characteristic between the denoised pixel event and the initial cluster according to the spatial information and the cluster characteristic information of the denoised pixel event may include:

determining target clustering characteristics to be adjusted and adjustment parameters of the target clustering characteristics from the clustering characteristics;

based on the adjustment parameters, carrying out information conversion on the spatial information and the target clustering characteristics of the denoised pixel event;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted target cluster characteristic.

In the present application, the cluster feature related to the calculated distance feature in the cluster features of the initial cluster may be used as the target cluster feature to be adjusted, for example, the target cluster feature may include a cluster position, a cluster shape, a cluster size, a cluster angle, and a cluster speed.

The adjustment parameters of the target clustering features are related parameters required for adjusting the target clustering features, and the adjustment mode of the target clustering features can be changed correspondingly based on different target clustering features. For example, the cluster position may be adjusted by changing the cluster position, and the adjustment parameter may be position information of the target position, or incremental information of the target position compared with the current position; for another example, the cluster angle may be adjusted by changing the cluster angle, and the adjustment parameter may be angle information of the target angle, or incremental information of the target angle compared to the current angle.

After the target clustering features to be adjusted and the adjustment parameters of the target clustering features are determined, the target clustering features can be further adjusted based on the adjustment parameters to realize information conversion of the target clustering features. And the information of the spatial information of the denoised pixel event can be correspondingly converted based on the adjustment parameters, so that the distance characteristic between the denoised pixel event and the initial cluster can be calculated according to the converted spatial information and the converted cluster characteristic information.

In another embodiment, taking a target clustering characteristic, specifically a clustering angle characteristic as an example, specifically, the step "calculating a distance characteristic between a denoised pixel event and an initial cluster according to spatial information and clustering characteristic information of the denoised pixel event" may include:

determining the rotation angle characteristics of the initial clustering and the adjustment parameters of the rotation angle characteristics;

performing information conversion on the spatial information and the rotation angle characteristic of the denoised pixel event based on the adjustment parameter;

and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted rotation angle characteristic.

Because the clustering of different shapes is realized on the basis of the circular clustering, the method and the device also add the characteristics of the rotation angle for each cluster in order to improve the accuracy of clustering processing, so that the clusters of various shapes can rotate, and the real shapes of the recognized denoised pixel event sets can be better fitted. Furthermore, the rotation angle of the initial cluster can be adjusted, and the spatial information of the denoised pixel event can be adaptively adjusted, so that the distance characteristic between the denoised pixel event and the initial cluster can be calculated by calculating the distance characteristic between the adjusted denoised pixel event and the adjusted initial cluster.

Taking elliptical clustering as an example, the left graph in fig. 8 shows elliptical clustering with rotation angle characteristics. Further, on the basis that the rotation angle feature is added to the cluster, in order to facilitate calculation of the distance feature of the denoised pixel event, the elliptical cluster may be rotated, specifically, taking the elliptical cluster as an example, referring to fig. 8, the rotation angle of the elliptical cluster may be obtained, and the adjustment parameter of the rotation angle feature may be determined based on the rotation angle, so that the elliptical cluster is rotated based on the adjustment parameter, the long axis of the rotated elliptical cluster may be parallel to the coordinate axis, and the coordinate information of the denoised pixel event E2 may be correspondingly adjusted based on the adjustment parameter of the elliptical cluster, so that the adjusted denoised pixel event E2' is as shown in the right diagram in fig. 8.

Further, the distance feature between the denoised pixel event and the initial cluster can be calculated according to the converted spatial information and the converted rotation angle feature. In the above example, that is, the distance feature between the adjusted denoised pixel event E1 and the transformed elliptical cluster is calculated according to the coordinate information of the adjusted denoised pixel event E1 and the cluster feature information of the transformed elliptical cluster, for example, the distance feature may be calculated by referring to the left part of the inequality in equation (3), so that the amount of calculation required for calculating the distance feature may be reduced compared to before rotation, thereby improving the calculation efficiency.

105. And clustering the denoised pixel events based on the distance characteristics to obtain a clustering result.

After the distance between the denoised pixel event and each initial cluster is obtained through calculation, the distance characteristic of the denoised pixel event can be obtained, and further, the denoised pixel event can be subjected to clustering processing based on the distance characteristic so as to determine the target cluster to which the denoised pixel event belongs.

In the application, besides clustering the denoised pixel events based on the distance characteristics and determining the target cluster to which the denoised pixel events belong, the target cluster can be further updated, and then the initial cluster set is updated to determine the clustering result corresponding to the acquired event data stream.

Specifically, the step of "performing clustering processing on the denoised pixel events based on the distance features to obtain a clustering result" may include:

determining a target cluster of the denoised pixel event from an initial cluster set of the denoised pixel event based on the distance characteristics, wherein the initial cluster set comprises at least one initial cluster, and the initial cluster is generated based on the historical pixel event;

updating the target cluster according to the spatial information of the denoised pixel event to obtain an updated target cluster;

and updating the initial cluster set based on the updated target cluster so as to determine a cluster result according to the update result of the initial cluster set.

In the present application, the distance feature of the denoised pixel event represents the distance between the denoised pixel event and each initial cluster, and therefore, based on the distance feature, there are various ways for determining the target cluster of the denoised pixel event from the initial cluster set of the denoised pixel event. For example, the initial cluster with the smallest distance to the denoised pixel event may be used as the target cluster of the denoised pixel event.

For another example, the shape and cluster size of the initial cluster may be taken into consideration, and specifically, since the initial cluster is a cluster having a certain shape and size, if a denoised pixel event falls within a cluster range of the initial cluster, the initial cluster may be used as a candidate cluster of the denoised pixel event, further, a target cluster of the denoised pixel event may be determined from the candidate clusters of the denoised pixel event, for example, a candidate cluster having a minimum distance to the denoised pixel event may be used as a target cluster of the denoised pixel event.

Specifically, if the target cluster of the denoised pixel event is determined from the initial cluster set of the denoised pixel event based on the distance feature, a new cluster may be generated based on the pixel event, and the generated cluster may be used as the target cluster of the pixel event. As an example, referring to fig. 9, a new cluster with the denoised pixel event as the cluster center may be generated and the new cluster may be used as the target cluster of the denoised pixel event.

After the target cluster of the denoised pixel event is determined, the target cluster can be updated according to the spatial information of the denoised pixel event. For example, the clustering characteristics such as the clustering position, the clustering size, the clustering rotation angle, the number of pixel events in the cluster, the clustering activity, and the clustering speed of the target cluster may be updated according to the spatial information of the denoised pixel events, so as to obtain an updated target cluster.

Further, the updating the initial cluster set based on the updated target cluster to obtain a cluster result of the event data stream, and specifically, the updating the initial cluster set based on the updated target cluster to determine the cluster result according to the update result of the initial cluster set may include:

updating the initial cluster set based on the updated target cluster to obtain an updated cluster set；

Determining active clusters in the updated cluster set, and calculating cluster distances among the active clusters;

if the clustering distance meets the preset distance condition, clustering and merging the active clusters to obtain a processed cluster set;

and determining a clustering result according to the processed clustering set.

In the present application, based on the updated target cluster, there are various ways to update the initial cluster set, for example, it may be determined whether cluster merging needs to be performed on clusters in the initial cluster set based on the updated target cluster. Specifically, it may be specified that, if a clustering distance between two active clusters meets a preset distance condition, the two active clusters are merged into one cluster, where a clustering feature of the merged cluster may be determined based on the clustering features of the two active clusters, for example, the clustering feature may include a clustering position, a clustering rotation angle, and the like. As an example, if both cluster 1 and cluster 2 are active clusters and the clustering distance between cluster 1 and cluster 2 satisfies the preset distance condition, cluster 1 and cluster 2 may be merged into cluster 3, where the clustering characteristic of cluster 3 is obtained by calculating the arithmetic mean of the clustering characteristics of cluster 1 and cluster 2.

As another example, it may be specified that clusters having no liveness (i.e., no pixel events received within a preset time interval) will be removed from the initial cluster set after the updated target clusters are obtained.

In the present application, after the initial cluster set is updated based on the updated target cluster, the cluster result of the event data stream may be determined according to the update result of the initial cluster set, for example, the cluster to which each pixel event in the event data stream belongs is determined.

106. And training the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the impulse neural network, so as to recognize the object corresponding to the clustering result through the trained object recognition model.

Wherein, the impulse neural network is often honored as the third generation artificial neural network. The first generation neural networks are perceptrons, and the second generation neural networks comprise a wide range of BP neural networks with more applications. But essentially these neural networks are encoded based on the frequency of the neural impulses. The impulse neural network, which simulates neurons closer to reality, takes into account the influence of time information. The idea is that a neuron in a dynamic neural network is not activated in each iteration propagation (as in a typical multi-layered perceptron network), but only when its membrane potential reaches a certain value. When a neuron is activated, it generates a signal that is transmitted to other neurons to raise or lower its membrane potential.

In a spiking neural network, the current activation level of a neuron (modeled as some sort of differential equation) is generally considered to be the current state, and an input pulse causes the current value to rise, last for a period of time, and then gradually decay. Many coding schemes have emerged that interpret these output pulse sequences as a practical number, taking into account both pulse frequency and pulse interval time.

In this application, the method may further include identifying an object corresponding to the clustering result through a trained object identification model based on the spiking neural network, specifically, the step "training the object identification model based on the neural network according to the clustering result to obtain the trained object identification model based on the spiking neural network" may include:

generating sample data required by model training according to the clustering result;

training an object recognition model based on a neural network through sample data to obtain a trained model corresponding to the neural network;

and generating a trained object recognition model based on the impulse neural network based on the trained model corresponding to the neural network.

The sample data may be in various forms, and may be specifically determined based on a model type of an object recognition model to be trained. For example, the object recognition model to be trained may be a Convolutional Neural Networks (CNN) model, and the sample data may be image type data; for another example, the object recognition model to be trained may be a neural network that processes time series data, and the sample data may be time series type data; and so on.

As an example, the object recognition model to be trained may be a CNN model, and the sample data may be image type data, and specifically, event data of pixel events belonging to the same cluster may be obtained according to a clustering result, and a corresponding sample image is generated according to the event data, where fig. 10 is an example of generating a sample image. In practical applications, a sufficient number and a sufficient variety of sample data may be obtained by using a data expansion technique, for example, the generated sample image may be subjected to image processing operations such as translation or rotation to obtain processed sample data, so as to expand the sample data set.

Further, the object recognition model based on the neural network can be trained through the sample data, and a trained model corresponding to the neural network is obtained. For example, an object recognition model to be trained may be constructed based on the CNN, and the object recognition model may be trained through a sample image to obtain a trained CNN model. For another example, an object recognition model to be trained may be constructed based on the SNN, and the object recognition model may be trained through sample data to obtain a trained SNN model.

Further, a trained object recognition model based on the spiking neural network may be generated based on the trained model corresponding to the neural network. For example, if the trained model is a CNN model, the CNN model may be converted into an SNN model to obtain a trained object recognition model based on the spiking neural network. For another example, if the trained model is an SNN model, the trained model may be used as a trained object recognition model based on a spiking neural network.

As can be seen from the above, the present embodiment may acquire an event data stream acquired by a visual sensor, where the event data stream includes event data of at least one pixel event, and the event data includes temporal information and spatial information of the pixel event; determining a spatio-temporal association relation between pixel events according to the time information and the space information; based on the space-time incidence relation, carrying out event data denoising processing on the event data stream to obtain a processed event data stream, wherein the processed event data stream comprises event data of denoised pixel events; calculating the distance characteristic of the denoised pixel event according to the spatial information of the denoised pixel event; clustering the denoised pixel events based on the distance characteristics to obtain a clustering result; and training the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the impulse neural network, so as to recognize the object corresponding to the clustering result through the trained object recognition model.

The scheme realizes object recognition based on the event data stream collected by the visual sensor, and compared with a method for realizing object recognition based on image frames, the method can greatly save required resources. Specifically, the image frame includes all visual information, specifically includes data related to the object recognition task and data unrelated to the object recognition task, and thus the object recognition method based on the image frame has a problem of large data redundancy. As such, the image frame based object recognition scheme consumes a large amount of network bandwidth when acquiring data from an edge device (e.g., a camera) and transmitting predicted values; in addition, the object identification method based on the image frame has huge requirements on storage resources; furthermore, the conventional method can generally support only a lower frame rate in consideration of the time consumed for processing frame data. This also makes it more difficult to identify fast moving objects in the scene when deployed online.

The scheme realizes object identification based on discrete event data, so that the scheme can effectively solve the problem of data redundancy, and thus, the total amount of data is reduced, the network bandwidth required by data transmission is saved, and the requirement on storage resources is reduced; in addition, the scheme is essentially driven by pixel events to perform object recognition, so that the data frames do not need to be stored or collected when the scheme is deployed online, and the scheme can be operated efficiently in real time. Therefore, when the object is identified, the method and the device can achieve more efficient effects in the aspects of resources and power consumption, so that the efficiency of object identification is improved.

In addition, the scheme also provides various compression methods for the event data denoising processing so as to reduce storage resources required by a data preprocessing link before object identification is realized, and also provides various improvement measures for a clustering processing mode, such as generating various clustering trackers so as to better fit a clustering result and improve clustering accuracy, and also provides a clustering set updating method comprising clustering combination and clustering removal and timely updating the clustering result of an event data stream so as to more accurately realize the object quilt based on the clustering result.

In addition, the scheme can use a dynamic vision sensor which can operate on compact and low-power consumption neural form hardware or/and a Field Programmable Gate Array (FPGA), and the scheme carries out object recognition based on pixel events, so that the problems of target detection and recognition can be solved, and more efficient effects in terms of resources and power consumption can be achieved.

The method described in the above examples is further described in detail below by way of example.

As shown in fig. 11, an object identification method specifically includes the following steps:

s201, a visual sensor collects an event data stream, wherein the event data stream comprises event data of at least one pixel event, and the event data comprises time information and space information of the pixel event.

202. And determining the space-time correlation relationship between the pixel events according to the time information and the space information.

203. And based on the space-time incidence relation, carrying out event data denoising processing on the event data stream to obtain a processed event data stream, wherein the processed event data stream comprises event data of denoised pixel events.

204. And calculating the distance characteristic of the denoised pixel event according to the spatial information of the denoised pixel event.

205. And clustering the denoised pixel events based on the distance characteristics to obtain a clustering result.

206. And training the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the impulse neural network, so as to recognize the object corresponding to the clustering result through the trained object recognition model.

In practical application, if the vision sensor is specifically a DVS, referring to fig. 12, it can be known that the method for identifying an object according to the embodiment of the present application may be applied to specifically decouple the flow of the method for identifying an object into the following modules: a DVS sensor, a noise filter, a cluster tracker, and an SNN classifier. The DVS sensor can be used for collecting event data streams, the noise filter can be used for denoising the collected event data streams, the clustering tracker can be used for clustering denoised pixel events, and the SNN classifier can be used for identifying objects based on clustering results.

Optionally, the object identification method described in the present application may be further improved with reference to the flow shown in fig. 13 to implement parallel processing of data, so as to further improve the efficiency and accuracy of object identification.

As can be seen from the above, the object identification is realized based on the discrete event data in the embodiment of the present application, so that the problem of data redundancy can be effectively solved, and as the total amount of data is reduced, the network bandwidth required during data transmission is also saved, and the requirement on storage resources is reduced; in addition, the scheme is essentially driven by pixel events to perform object recognition, so that the data frames do not need to be stored or collected when the scheme is deployed online, and the scheme can be operated efficiently in real time. Therefore, when the object is identified, the method and the device can achieve more efficient effects in the aspects of resources and power consumption, so that the efficiency of object identification is improved. In addition, the embodiment of the application also provides a processing mode of data parallel processing, and the efficiency and the accuracy of object identification can be further improved.

In order to better implement the method, correspondingly, the embodiment of the application also provides an object recognition device, wherein the object recognition device can be integrated in the terminal.

For example, as shown in fig. 14, the object recognition apparatus may include an obtaining unit 301, a determining unit 302, a denoising unit 303, a calculating unit 304, a clustering unit 305, and a training unit 306, as follows:

an obtaining unit 301, configured to obtain an event data stream acquired by a visual sensor, where the event data stream includes event data of at least one pixel event, and the event data includes temporal information and spatial information of the pixel event;

a determining unit 302, configured to determine a spatio-temporal association relationship between the pixel events according to the temporal information and the spatial information;

the denoising unit 303 may be configured to perform event data denoising on the event data stream based on the temporal-spatial association relationship, to obtain a processed event data stream, where the processed event data stream includes event data of a denoised pixel event;

a calculating unit 304, configured to calculate a distance feature of the denoised pixel event according to the spatial information of the denoised pixel event;

the clustering unit 305 may be configured to perform clustering processing on the denoised pixel event based on the distance feature to obtain a clustering result;

the training unit 306 may be configured to train an object recognition model based on a neural network according to the clustering result to obtain a trained object recognition model based on a pulse neural network, so as to recognize an object corresponding to the clustering result through the trained object recognition model.

In an embodiment, the determining unit 302 may include:

a mask generation subunit operable to generate an initial correlation mask for the event data stream based on parameter information of the vision sensor, the initial correlation mask including at least one information element for recording temporal information of pixel events;

the spatial compression subunit is configured to perform information compression on the spatial information of the pixel event to obtain compressed spatial information;

a first determining subunit, configured to determine, according to the compressed spatial information, a target information unit corresponding to the pixel event;

and the information updating subunit is used for updating the information of the target information unit according to the time information of the pixel events to generate an updated association mask, and the updated association mask represents a spatio-temporal association relation between the pixel events.

In an embodiment, the information updating subunit may be configured to:

determining a time threshold parameter required for denoising the event data of the event data stream; according to the time threshold parameter, performing information compression on the time information to obtain compressed time information; and updating the information of the target information unit based on the compressed time information.

In an embodiment, the information updating subunit may be specifically configured to:

determining a shifting parameter of the time information according to the time threshold parameter; performing a bit operation on the time information based on the shift parameter to perform information compression on the time information; based on the operation result, determining compressed time information of the pixel event.

In an embodiment, the spatial compression subunit may be configured to:

performing bit operation on the spatial information of the pixel event to perform information compression on the spatial information of the pixel event; based on the operation result, compressed spatial information of the pixel event is determined.

In an embodiment, the denoising unit may include:

a second determining subunit, configured to determine an association mask characterizing the spatio-temporal association relationship, and a time threshold parameter required for performing event data denoising on the event data stream, where the association mask includes at least one information unit;

a third determining subunit, configured to determine, from the correlation mask, a correlation information unit of the pixel event based on spatial information of the pixel event, where the correlation information unit has a spatial correlation with the pixel event;

the verifying subunit is configured to verify the time information of the pixel event based on the time threshold parameter and the time information recorded by the association information unit;

and the denoising subunit is configured to perform event data denoising on the pixel event according to the verification result, so as to perform event data denoising on the event data stream.

In an embodiment, before the event data denoising processing is performed on the event data stream based on the spatiotemporal correlation relationship, the object recognition apparatus may further include:

a generating unit operable to generate an initial filter mask for the event data stream based on parameter information of the vision sensor;

an updating unit, configured to update the initial filter mask based on the temporal information and the spatial information of the pixel event to obtain an updated filter mask;

the filtering unit may be configured to perform event data filtering processing on the event data stream through the updated filtering mask to obtain a filtered event data stream;

the denoising unit may be configured to: and carrying out event data denoising processing on the filtered event data stream based on the space-time incidence relation.

In an embodiment, the calculating unit 304 may include:

a fourth determining subunit operable to determine an initial cluster set of the denoised pixel events, the initial cluster set comprising at least one initial cluster, the initial cluster being generated based on historical pixel events;

an obtaining subunit, configured to obtain clustering feature information of the initial cluster;

and the feature calculating subunit may be configured to calculate, according to the spatial information of the denoised pixel event and the cluster feature information, a distance feature between the denoised pixel event and the initial cluster.

In an embodiment, the cluster feature information includes a cluster feature of the initial cluster in at least one feature dimension; the feature calculation subunit may be configured to:

determining a target characteristic dimension to be adjusted and an adjustment parameter of the target characteristic dimension from at least one characteristic dimension of the initial cluster; based on the adjustment parameters, carrying out information conversion on the spatial information of the denoised pixel event and the clustering characteristic information; and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted cluster characteristic information.

In an embodiment, the feature calculating subunit may be configured to:

determining coordinate axes to be zoomed in the reference coordinate system of the initial clustering and zooming parameters of the coordinate axes; based on the scaling parameter, carrying out information conversion on the spatial information of the denoised pixel event and the clustering characteristic information; and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted cluster characteristic information.

In an embodiment, the cluster feature information includes a cluster feature of the initial cluster in at least one feature dimension; the feature calculation subunit may be configured to:

determining target clustering characteristics to be adjusted and adjustment parameters of the target clustering characteristics from the clustering characteristics; based on the adjustment parameters, carrying out information conversion on the spatial information of the denoised pixel event and the target clustering characteristics; and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted target cluster characteristic.

In an embodiment, the feature calculating subunit may be configured to:

determining the rotation angle characteristics of the initial clusters and the adjustment parameters of the rotation angle characteristics; performing information conversion on the spatial information of the denoised pixel event and the rotation angle characteristic based on the adjusting parameter; and calculating the distance characteristic between the denoised pixel event and the initial cluster according to the converted spatial information and the converted rotation angle characteristic.

In an embodiment, the obtaining subunit may be configured to:

determining a cluster shape corresponding to the initial cluster; and acquiring clustering feature information of the initial clustering based on the clustering shape.

In an embodiment, the clustering unit may include:

a fifth determining subunit, configured to determine, based on the distance feature, a target cluster of the denoised pixel event from an initial cluster set of the denoised pixel event, where the initial cluster set includes at least one initial cluster, and the initial cluster is generated based on historical pixel events;

the cluster updating subunit is configured to update the target cluster according to the spatial information of the denoised pixel event, so as to obtain an updated target cluster;

and the set updating subunit may be configured to update the initial cluster set based on the updated target cluster, so as to determine a clustering result according to an update result of the initial cluster set.

In an embodiment, the set update subunit may be configured to:

updating the initial cluster set based on the updated target cluster to obtain an updated cluster set; determining active clusters in the updated cluster set, and calculating cluster distances among the active clusters; if the clustering distance meets a preset distance condition, clustering and merging the active clusters to obtain a processed cluster set; and determining a clustering result according to the processed clustering set.

In an embodiment, the training unit 306 may include:

the sample generating subunit is used for generating sample data required by model training according to the clustering result;

the model training subunit is used for training an object recognition model based on a neural network through the sample data to obtain a trained model corresponding to the neural network;

and the model generation subunit is used for generating a trained object recognition model based on the impulse neural network based on the trained model corresponding to the neural network.

In a specific implementation, the above units may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities, and the specific implementation of the above units may refer to the foregoing method embodiments, which are not described herein again.

As can be seen from the above, in the object recognition apparatus of the present embodiment, the obtaining unit 301 obtains an event data stream collected by the vision sensor, where the event data stream includes event data of at least one pixel event, and the event data includes temporal information and spatial information of the pixel event; determining, by the determining unit 302, a spatiotemporal correlation between the pixel events according to the temporal information and the spatial information; based on the space-time association relationship, a denoising unit 303 performs event data denoising on the event data stream to obtain a processed event data stream, where the processed event data stream includes event data of denoised pixel events; calculating, by the calculating unit 304, a distance feature of the denoised pixel event according to the spatial information of the denoised pixel event; clustering the denoised pixel events by the clustering unit 305 based on the distance characteristics to obtain a clustering result; the training unit 306 trains the object recognition model based on the neural network according to the clustering result to obtain a trained object recognition model based on the impulse neural network, so as to recognize the object corresponding to the clustering result through the trained object recognition model.

The scheme realizes object identification based on discrete event data, so that the scheme can effectively solve the problem of data redundancy, and thus, the total amount of data is reduced, the network bandwidth required by data transmission is saved, and the requirement on storage resources is reduced; in addition, the scheme is essentially driven by pixel events to perform object recognition, so that the data frames do not need to be stored or collected when the scheme is deployed online, and the scheme can be operated efficiently in real time. Therefore, when the object is identified, the method and the device can achieve more efficient effects in the aspects of resources and power consumption, so that the efficiency of object identification is improved.

In addition, the embodiment of the application also provides electronic equipment, and the electronic equipment can be equipment such as a terminal. As shown in fig. 15, a schematic structural diagram of an electronic device according to an embodiment of the present application is shown, specifically: the electronic device 400 comprises an object recognition arrangement 401. In an embodiment, the object recognition apparatus 401 may be implemented as a chip, and specifically may be a pseudonymous chip (brain-like chip). The object recognition apparatus 401 is coupled with a processing module 403 (such as MCU) of the electronic device 400 through an interface module 402 (such as a wired interface circuit for communication, a bluetooth, ZigBee, UWB, etc. wireless transmission module). The object recognition means 401 performs object recognition on the event data stream and transmits the result to the processing module 403 of the electronic device through the interface module 402, and the processing module 403 controls the response module 404 based on the result fed back by the object recognition means 401. The response module 404 may be a variety of known response modes, for example, output information on a display screen, an alarm, a voice signal output, a motion of a mechanical device (e.g., an intelligent curtain scene), a control of a physical quantity such as a voltage and a current of an electrical device, a switching (e.g., an intelligent lamp), and the like. Some or all of the response module 404, the processing module 403, and the object recognition apparatus 401 may be physically separate apparatuses, which in their entirety constitute the electronic device 400.

As can be seen from the above, the electronic device 400 can implement object recognition based on discrete event data, and therefore, the solution can effectively solve the problem of data redundancy, and thus, since the total amount of data is reduced, the network bandwidth required for data transmission is also saved, and the requirement for storage resources is reduced; in addition, the scheme is essentially driven by pixel events to perform object recognition, so that the data frames do not need to be stored or collected when the scheme is deployed online, and the scheme can be operated efficiently in real time. Therefore, when the object is identified, the method and the device can achieve more efficient effects in the aspects of resources and power consumption, so that the efficiency of object identification is improved.

The object identification method, the device, the chip and the electronic device provided by the embodiment of the present application are introduced in detail, a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiment is only used to help understanding the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.