1. A method of data mapping based on image recognition, the method comprising:

respectively acquiring dynamic image data of a target object by using a plurality of image acquisition devices, and respectively converting the dynamic image data acquired by each image acquisition device into data files, thereby obtaining a plurality of data files associated with the target object;

carrying out image recognition on the characteristic points in each data file so as to obtain a plurality of characteristic points for each data file, and carrying out characteristic point labeling on each data file based on the obtained plurality of characteristic points;

dividing a plurality of characteristic points marked in each data file into characteristic points of a first type and characteristic points of a second type, and merging the characteristic points of the first type of each of the plurality of data files into a target file associated with a target object;

normalizing the plurality of feature points in the target file to obtain a plurality of general feature points, and determining the motion attribute of the target object based on the change angle and/or the movement data of at least one general feature point in the target file; and

and mapping the operation attribute of the target object into a data file with a preset format according to a predefined data mapping rule.

2. The method according to claim 1, wherein the acquiring the dynamic image data of the target object with the plurality of image acquisition devices respectively comprises:

the plurality of image acquisition devices are respectively placed at different positions in advance, so that the plurality of image acquisition devices can respectively acquire the dynamic image data of the target object at different positions.

3. The method according to claim 1, wherein the acquiring the dynamic image data of the target object with the plurality of image acquisition devices respectively comprises:

the plurality of image acquisition devices respectively acquire dynamic image data of the target object at different positions and from different angles.

4. The method of any of claims 1-3, each of the plurality of image acquisition devices being movable.

5. The method of claim 1, wherein the image recognizing the feature points in each data file to obtain a plurality of feature points for each data file comprises:

determining a target object and an auxiliary object in each data file;

and performing image recognition on the target object and the auxiliary object based on the gesture recognition network, and determining a plurality of feature points associated with the target object and the auxiliary object so as to obtain a plurality of feature points for each data file.

6. A method of data mapping based on image recognition, the method comprising:

acquiring a plurality of static images of a target object by using an image acquisition device;

performing image recognition on each static image to obtain a plurality of feature points in each static image;

calculating a parameter value of the target object in each static image based on the plurality of feature points, thereby obtaining a plurality of parameter values of the target object;

determining a property value of the target object based on the plurality of parameter values; and

and mapping the attribute values of the target object into a data file with a preset format according to a predefined data mapping rule.

7. A system for data mapping based on image recognition, the system comprising:

the acquisition device is used for respectively acquiring dynamic image data of a target object by utilizing a plurality of image acquisition devices and respectively converting the dynamic image data acquired by each image acquisition device into a data file, so that a plurality of data files related to the target object are obtained;

the identification device is used for carrying out image identification on the characteristic points in each data file so as to obtain a plurality of characteristic points for each data file, and carrying out characteristic point labeling on each data file based on the obtained plurality of characteristic points;

the fusion device is used for dividing the characteristic points marked in each data file into characteristic points of a first type and characteristic points of a second type, and merging the characteristic points of the first type of each of the data files into the same characteristic points, so that the data files are fused into a target file associated with a target object;

the processing device is used for carrying out normalization processing on the plurality of characteristic points in the target file so as to obtain a plurality of general characteristic points, and determining the motion attribute of the target object based on the change angle and/or the movement data of at least one general characteristic point in the target file; and

and the mapping device is used for mapping the operation attribute of the target object into a data file with a preset format according to a predefined data mapping rule.

8. A system for data mapping based on image recognition, the system comprising:

acquiring means for acquiring a plurality of still images of the target object with the image acquiring apparatus;

the identification device is used for carrying out image identification on each static image so as to obtain a plurality of characteristic points in each static image;

calculating means for calculating a parameter value of the target object in each of the still images based on the plurality of feature points, thereby obtaining a plurality of parameter values of the target object;

determining means for determining a value of an attribute of the target object based on the plurality of parameter values; and

and the mapping device is used for mapping the attribute values of the target object into a data file with a preset format according to a predefined data mapping rule.

9. A computer-readable storage medium, characterized in that the storage medium stores a computer program for performing the method of any of the preceding claims 1-6.

10. An electronic device, characterized in that the electronic device comprises:

a processor;

a memory for storing the processor-executable instructions;

the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-6.

Background

Most countries around the world are rapidly stepping into aging society. From the aspect of chronic disease prevalence of old people, the prevalence of old people is as high as 64.5%, and most diseases have long treatment course, poor prognosis and high cost. As is well known, rehabilitation can effectively reduce the recurrence probability of chronic diseases and improve the healing effect. Taking the cerebrovascular disease with high morbidity, high mortality, high disability rate and high recurrence rate as an example, the active rehabilitation treatment can ensure that 90 percent of patients can regain the walking and life self-care ability, and 30 percent of patients recover the work. The percentage of recovery in both of these aspects was only 6% and 5% without rehabilitation. The rehabilitation evaluation is an objective, qualitative and/or quantitative description of the process, explains effective influence factors of the result, and provides effective scientific basis for judging whether the rehabilitation process is continued later, whether the rehabilitation process can be returned to families and society or further rehabilitation treatment, whether an original rehabilitation plan is modified and the like. .

However, in the fields of nursing and rehabilitation, doctors and staff manually record data and conduct subjective analysis, manual errors are prone to occur, working efficiency is low, and results are not accurate. In the prior art, there is no related technology for performing active continuous data acquisition based on image recognition, feature analysis based on the acquired data, and result display based on the feature analysis for the care, rehabilitation, and nursing processes.

Disclosure of Invention

In order to solve the problems in the prior art, the invention provides a method and a system for mapping data based on image recognition, a computer-readable storage medium and an electronic device. The present invention can perform image recognition processing on a moving image or a still image, thereby determining feature points in the moving image or the still image based on the result of the image recognition processing. The technical scheme of the invention can improve the efficiency and accuracy of image recognition.

According to an aspect of the present invention, there is provided a method for data mapping based on image recognition, the method comprising:

respectively acquiring dynamic image data of a target object by using a plurality of image acquisition devices, and respectively converting the dynamic image data acquired by each image acquisition device into data files, thereby obtaining a plurality of data files associated with the target object;

carrying out image recognition on the characteristic points in each data file so as to obtain a plurality of characteristic points for each data file, and carrying out characteristic point labeling on each data file based on the obtained plurality of characteristic points;

dividing a plurality of characteristic points marked in each data file into characteristic points of a first type and characteristic points of a second type, and merging the characteristic points of the first type of each of the plurality of data files into a target file associated with a target object;

normalizing the plurality of feature points in the target file to obtain a plurality of general feature points, and determining the motion attribute of the target object based on the change angle and/or the movement data of at least one general feature point in the target file; and

and mapping the operation attribute of the target object into a data file with a preset format according to a predefined data mapping rule.

The acquiring of the moving image data of the target object with the plurality of image acquisition devices, respectively, includes: the plurality of image acquisition devices are respectively placed at different positions in advance, so that the plurality of image acquisition devices can respectively acquire the dynamic image data of the target object at different positions.

The acquiring of the moving image data of the target object with the plurality of image acquisition devices, respectively, includes: the plurality of image acquisition devices respectively acquire dynamic image data of the target object at different positions and from different angles. Each of the plurality of image capturing devices is movable. The data file is a video file.

The image recognition of the feature points in each data file, so as to obtain a plurality of feature points for each data file, comprises: determining a target object and an auxiliary object in each data file;

and performing image recognition on the target object and the auxiliary object based on the gesture recognition network, and determining a plurality of feature points associated with the target object and the auxiliary object so as to obtain a plurality of feature points for each data file.

Each image acquisition device also has an infrared thermal imaging function; the method further comprises the steps of determining whether each feature point in the plurality of feature points belongs to the target object and/or the auxiliary object through an infrared thermal imaging function, protecting privacy data of the target object, and identifying and fusing physical sign parameters of the target object.

The coordinates of the world coordinate system and the image coordinates of each image acquisition device are determined by the following formulas:

wherein Zc is the distance from the optical center to the image plane; u and v are coordinates of a coordinate system of the pixel points; dx and dy are the physical lengths corresponding to the pixels; u. of₀，v₀Pixel coordinates that are the center of the image; f is the focal length of the lens; r and T are rotation and translation matrixes of the image plane under a world coordinate system; x_w，Y_w，Z_wCoordinates of the object in a world coordinate system; and K and M are respectively an internal reference matrix and an external reference matrix of the image acquisition equipment.

And calibrating the internal parameter matrix K and the external parameter matrix M of each image acquisition device.

The feature points of the first type are feature points associated with a target object, and feature points that do not satisfy a preset rule among a plurality of feature points of the first type are deleted. The feature points of the second type are feature points associated with an auxiliary object, and the feature points of the second type are used to correct the feature points of the first type. The merging of the same feature points of the first type of each of the plurality of data files includes: and merging the same characteristic points of the first type of the respective characteristic points of the plurality of data files based on the time consistency and the characteristic point attributes. Merging the same feature points of the first type of feature points of each of the plurality of data files, thereby merging the plurality of data files into a target file associated with a target object, comprises: determining the same characteristic point in the first type of characteristic points of each of the plurality of data files; combining the feature points of the first type of each of the plurality of data files by taking the same feature point as a coincident common feature point;

and fusing the plurality of data files into a target file associated with the target object based on the merged feature points of the first type.

Before normalization processing is carried out on a plurality of feature points in the target file, the method further comprises the following steps:

determining three-dimensional coordinates for each feature point in the target file according to the following formula:

wherein Z is_c1And Z_c2Is the distance from the optical center to the image plane; u. of₁，v₁，u₂，v₂Coordinates of a coordinate system of the pixel points; k₁And M₁Obtaining an internal reference matrix and an external reference matrix of the first image acquisition device; k₂And M₂Obtaining an internal reference matrix and an external reference matrix of the second image acquisition device; x_w，Y_w，Z_wIs the three-dimensional coordinates of the feature points.

The normalizing the plurality of feature points in the target file to obtain a plurality of general feature points comprises: calculating a node angle feature based on at least three feature points of the plurality of feature points:

wherein, ja_t(j1, j2, j3) is an angular feature; j1, j2, j3 is a feature point, P_t(j1, j2) represents a bagA vector comprising feature points j1 and j 2; p_t(j2, j3) represents vectors including feature points j2 and j 3;

wherein

P_t(j1，j2)＝(p_t(j1，x)-p_t(j2，x))+(p_t(j1，y)-p_t(j2，y)) +(p_t(j1，z)-p_t(j2，z))

Wherein pt represents the node coordinates;

and carrying out normalization processing on the relative track, wherein the normalization processing comprises the following steps: determining the relative distance between two feature points:

wherein nrt_t(b, s) is the relative distance between two feature points; b and s are characteristic points; rt is an integer of_t(b, s) is the relative track of the characteristic points b and s at the t-th frame, and t is a frame index; rt is an integer of₁(b, s) is the relative trajectory of the feature points of the first frame of the video;

wherein:

wherein pt represents the node coordinates; b, s and c are feature points;

and carrying out normalization processing on the speed, wherein the normalization processing comprises the following steps:

wherein nsp_t(j) The normalized speed of the characteristic point j in time 0-t is obtained; whereinIs the average speed of the time interval from 0 to t,is the maximum speed in the time interval of 0-t;

wherein sp_t(j) For the characteristic point moving speed, the calculation formula is as follows:

sp_t(j)＝F*(rt_t(b，j)-rt_(t-1)(b，j))，ift＞1.

＝0otherwise

wherein rt is_t(b, j) is the relative track of the characteristic points b and j at the time frame t; rt is an integer of_t-1(b, j) is the relative track of the characteristic points b and j at the time frame t-1; f is the sampling rate;

the acceleration is subjected to normalization processing, and the normalization processing comprises the following steps: dividing the average jerk from 0 to t by the maximum jerk

Wherein, njk_t(j) Normalized jerk of feature point j at t frames; whereinIs the average acceleration in the time interval of 0-t,the maximum acceleration within the time interval of 0-t; jk_t(j) The jerk at the feature point j;

wherein:

jk_t(j)＝F*(ac_t(b，j)-ac_t-1(b，j))，ift＞1.

＝0otherwise

wherein ac_tAcceleration for time t:

ac_t(j)＝F*(sp_t(j)-sp_(t-1)(j))，ift＞1.

＝0otherwise

the merging of the same feature points of the first type of the respective feature points of the plurality of data files based on the time consistency and the feature point attributes includes:

in a plurality of data files, merging the characteristic points of the first type of each data file in a time consistency frame synchronization mode by taking the same characteristic point as a merging point.

The determining the motion attribute of the target object based on the change angle and/or the movement data of the at least one common feature point in the target file comprises: selecting at least one common feature point associated with attribute identification from the plurality of common feature points based on the positions of the common feature points; the motion attributes of the target object are determined based on the changing angle and/or movement data of the at least one generic feature point associated with the attribute identification in the target file. The variation angle is used for representing the action completion degree of at least one universal characteristic point. The movement data is used for characterizing the fluency of motion of at least one universal feature point.

The method comprises the steps of pre-defining a data mapping rule for mapping the operation attribute into a data file with a preset format before respectively acquiring the dynamic image data of the target object by utilizing a plurality of image acquisition devices;

the data mapping rule includes a plurality of mapping entries, each mapping entry including < motion attribute, action result >.

According to another aspect of the present invention, there is provided a method for data mapping based on image recognition, the method comprising: acquiring a plurality of static images of a target object by using an image acquisition device; performing image recognition on each static image to obtain a plurality of feature points in each static image; calculating a parameter value of the target object in each static image based on the plurality of feature points, thereby obtaining a plurality of parameter values of the target object; determining a property value of the target object based on the plurality of parameter values; and mapping the attribute values of the target object into a data file with a predetermined format according to a predefined data mapping rule. According to another aspect of the present invention, there is provided a system for data mapping based on image recognition, the system comprising:

the acquisition device is used for respectively acquiring dynamic image data of a target object by utilizing a plurality of image acquisition devices and respectively converting the dynamic image data acquired by each image acquisition device into a data file, so that a plurality of data files related to the target object are obtained;

the identification device is used for carrying out image identification on the characteristic points in each data file so as to obtain a plurality of characteristic points for each data file, and carrying out characteristic point labeling on each data file based on the obtained plurality of characteristic points;

the fusion device is used for dividing the characteristic points marked in each data file into a first type of characteristic points and a second type of characteristic points, merging the same characteristic points of the first type of characteristic points of the data files, and fusing the data files into a target file associated with a target object;

the processing device is used for carrying out normalization processing on the plurality of characteristic points in the target file so as to obtain a plurality of general characteristic points, and determining the motion attribute of the target object based on the change angle and/or the movement data of at least one general characteristic point in the target file; and

and the mapping device is used for mapping the operation attribute of the target object into a data file with a preset format according to a predefined data mapping rule.

The device further comprises an initialization device used for respectively placing the plurality of image acquisition devices at different positions in advance, so that the plurality of image acquisition devices can respectively acquire the dynamic image data of the target object at different positions.

The plurality of image acquisition devices respectively acquire dynamic image data of the target object at different positions and from different angles. Each of the plurality of image capturing devices is movable. The data file is a video file. The identification device is specifically configured to: determining a target object and an auxiliary object in each data file; and performing image recognition on the target object and the auxiliary object based on the gesture recognition network, and determining a plurality of feature points associated with the target object and the auxiliary object so as to obtain a plurality of feature points for each data file. Each image acquisition device also has an infrared thermal imaging function; the removing device is used for determining whether each feature point in the plurality of feature points belongs to the target object and/or the auxiliary object through the thermal infrared imaging function, protecting privacy data of the target object, and identifying and fusing sign parameters of the target object.

Further comprising computing means for determining the coordinates of the world coordinate system and the image coordinates of each image acquisition device by the following formulas:

wherein Zc is the distance from the optical center to the image plane; u and v are coordinates of a coordinate system of the pixel points; dx and dy are the physical lengths corresponding to the pixels; u. of₀，v₀Pixel coordinates that are the center of the image; f is the focal length of the lens; r and T are rotation and translation matrixes of the image plane under a world coordinate system; x_w，Y_w，Z_wCoordinates of the object in a world coordinate system; and K and M are respectively an internal reference matrix and an external reference matrix of the image acquisition equipment.

And calibrating the internal parameter matrix K and the external parameter matrix M of each image acquisition device.

The feature points of the first type are feature points associated with a target object, and feature points that do not satisfy a preset rule among a plurality of feature points of the first type are deleted. The feature points of the second type are feature points associated with an auxiliary object, and the feature points of the second type are used to correct the feature points of the first type.

The fusion device is specifically configured to: and merging the same characteristic points of the first type of the respective characteristic points of the plurality of data files based on the time consistency and the characteristic point attributes.

The fusion device is specifically configured to: determining the same characteristic point in the first type of characteristic points of each of the plurality of data files; combining the feature points of the first type of each of the plurality of data files by taking the same feature point as a coincident common feature point; and fusing the plurality of data files into a target file associated with the target object based on the merged feature points of the first type.

Further comprising computing means for determining three-dimensional coordinates for each feature point in the target file according to the following formula:

wherein Z is_c1And Z_c2Is the distance from the optical center to the image plane; u. of₁，v₁，u₂，v₂Coordinates of a coordinate system of the pixel points; k₁And M₁Obtaining an internal reference matrix and an external reference matrix of the first image acquisition device; k₂And M₂Obtaining an internal reference matrix and an external reference matrix of the second image acquisition device; x_w，Y_w，Z_wIs the three-dimensional coordinates of the feature points.

The processing device is specifically configured to: calculating a node angle feature based on at least three feature points of the plurality of feature points:

wherein, ja_t(j1, j2, j3) is an angular feature; j1, j2, j3 is a feature point, P_t(j1, j2) represents a vector comprising feature points j1 and j 2; p_t(j2, j3) represents vectors including feature points j2 and j 3;

wherein

P_t(j1，j2)＝(p_t(j1，x)-p_t(j2，x))+(p_t(j1，y)-p_t(j2，y)) +(p_t(j1，z)-p_t(j2，z))

Wherein pt represents the node coordinates;

and carrying out normalization processing on the relative track, wherein the normalization processing comprises the following steps: determining the relative distance between two feature points:

wherein nrt_t(b, s) is between two feature pointsThe relative distance of (d); b and s are characteristic points; rt is an integer of_t(b, s) is the relative track of the characteristic points b and s at the t-th frame, and t is a frame index; rt is an integer of₁(b, s) is the relative trajectory of the feature points of the first frame of the video;

wherein:

wherein pt represents the node coordinates; b, s and c are feature points;

and carrying out normalization processing on the speed, wherein the normalization processing comprises the following steps:

wherein nsp_t(j) The normalized speed of the characteristic point j in time 0-t is obtained; whereinIs the average speed of the time interval from 0 to t,is the maximum speed in the time interval of 0-t;

wherein sp_t(j) For the characteristic point moving speed, the calculation formula is as follows:

sp_t(j)＝F*(rt_t(b，j)-rt_(t-1)(b，j))，ift＞1.

＝0otherwise

wherein rt is_t(b, j) is the relative track of the characteristic points b and j at the time frame t; rt is an integer of_t-1(b, j) is the relative track of the characteristic points b and j at the time frame t-1; f is the sampling rate;

the acceleration is subjected to normalization processing, and the normalization processing comprises the following steps: dividing the average jerk from 0 to t by the maximum jerk

Wherein, njk_t(j) Normalized jerk of feature point j at t frames; whereinIs the average acceleration in the time interval of 0-t,the maximum acceleration within the time interval of 0-t; jk_t(j) The jerk at the feature point j;

wherein:

jk_t(j)＝F*(ac_t(b，j)-ac_t-1(b，j))，ift＞1.

＝0otherwise

wherein ac_tAcceleration for time t:

ac_t(j)＝F*(sp_t(j)-sp_(t-1)(j))，ift＞1.

＝0otherwise

the processing device is specifically configured to: in a plurality of data files, merging the characteristic points of the first type of each data file in a time consistency frame synchronization mode by taking the same characteristic point as a merging point.

The processing device is specifically configured to: selecting at least one common feature point associated with attribute identification from the plurality of common feature points based on the location of the common feature point; the motion attributes of the target object are determined based on the changing angle and/or movement data of the at least one generic feature point associated with the attribute identification in the target file. The variation angle is used for representing the action completion degree of at least one universal characteristic point. The movement data is used for characterizing the fluency of motion of at least one universal feature point.

The device also comprises an initialization device which is used for predefining a data mapping rule for mapping the operation attribute into a data file with a preset format; the data mapping rule includes a plurality of mapping entries, each mapping entry including < motion attribute, action result >.

According to another aspect of the present invention, there is provided a system for data mapping based on image recognition, the system comprising: acquiring means for acquiring a plurality of still images of the target object by using the image acquiring apparatus; the identification device is used for carrying out image identification on each static image so as to obtain a plurality of characteristic points in each static image; calculating means for calculating a parameter value of the target object in each of the still images based on the plurality of feature points, thereby obtaining a plurality of parameter values of the target object; determining means for determining a value of an attribute of the target object based on the plurality of parameter values; and mapping means for mapping the attribute values of the target object to a data file of a predetermined format according to a predefined data mapping rule.

According to another aspect of the present invention, there is provided a computer-readable storage medium, characterized in that the storage medium stores a computer program for executing the method described above.

According to another aspect of the present invention, there is provided an electronic apparatus, comprising:

a processor;

a memory for storing the processor-executable instructions;

the processor is configured to read the executable instructions from the memory and execute the instructions to implement any of the methods described above.

Drawings

A more complete understanding of exemplary embodiments of the present invention may be had by reference to the following drawings in which:

FIG. 1 is a flow diagram of a method for data mapping based on image recognition in accordance with an embodiment of the present invention;

FIG. 2 is a flow diagram of a method for data mapping based on image recognition according to another embodiment of the present invention;

FIG. 3 is a schematic diagram of feature points obtained according to another embodiment of the present invention;

FIG. 4 is a schematic diagram of two-dimensional human feature points obtained according to an embodiment of the invention;

FIG. 5 is a schematic diagram of two-dimensional human feature points obtained according to another embodiment of the present invention;

FIG. 6 is a schematic three-dimensional coordinate diagram of feature points according to an embodiment of the invention;

FIG. 7 is a schematic diagram of a system for mapping data based on image recognition according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a system for mapping data based on image recognition according to another embodiment of the present invention.

Detailed Description

FIG. 1 is a flow diagram of a method 100 for data mapping based on image recognition, according to an embodiment of the present invention. The method 100 begins at step 101.

In step 101, a plurality of image capturing devices are used to respectively capture dynamic image data of a target object, and the dynamic image data captured by each image capturing device is respectively converted into a data file, so as to obtain a plurality of data files associated with the target object.

The acquiring of the moving image data of the target object with the plurality of image acquisition devices, respectively, includes: the plurality of image acquisition devices are respectively placed at different positions in advance, so that the plurality of image acquisition devices can respectively acquire the dynamic image data of the target object at different positions.

The acquiring of the moving image data of the target object with the plurality of image acquiring apparatuses respectively includes: the plurality of image acquisition devices respectively acquire dynamic image data of the target object at different positions and from different angles. Preferably, each of the plurality of image capturing devices is movable. The data file is a video file.

Further, in order to enable each image acquisition device to accurately acquire image data or dynamic image data, the coordinates of the world coordinate system and the image coordinates of each image acquisition device are determined by the following formulas:

wherein Zc is the distance from the optical center to the image plane; u and v are coordinates of a coordinate system of the pixel points; dx and dy are the physical lengths corresponding to the pixels; u. of₀，v₀Pixel coordinates that are the center of the image; f is the focal length of the lens; r and T are rotation and translation matrixes of the image plane under a world coordinate system; x_w，Y_w，Z_wCoordinates of the object in a world coordinate system; and K and M are respectively an internal reference matrix and an external reference matrix of the image acquisition equipment. In addition, calibrating the internal reference matrix K and the external reference matrix M of each image acquisition device.

Fig. 3 is a schematic diagram of acquiring feature points according to an embodiment of the present invention. As shown in FIG. 3, for example, the present invention simultaneously uses two fixed/mobile cameras to acquire real-time training images of a patient. For example, the two fixed/movable cameras include a camera 1 and a camera 2. Wherein, camera 1 and camera 2 acquire image 1 and image 2 respectively. And then, respectively identifying key points of the human body on the two images by means of a posture identification network to obtain two-dimensional human body characteristic points at the same moment. For example, two dimensions are map feature points 1 and two-dimensional human feature points 2. Internal and external parameters are calibrated for two-dimensional such as a map feature point 1 and a two-dimensional human body feature point 2 according to coordinates and image coordinates of a world coordinate system of each image acquisition device (such as a camera or a camera), and then, a three-dimensional feature point schematic diagram is acquired, and finally, feature points of a person to be rehabilitated (a person or a target object) are extracted.

Specific examples refer to fig. 4 and 5. Fig. 4 is a schematic diagram of obtaining feature points according to another embodiment of the present invention. In fig. 4, two-dimensional human feature points are obtained for the camera 1. Fig. 5 is a schematic diagram of two-dimensional human feature points obtained according to an embodiment of the present invention. In fig. 5, two-dimensional human feature points are obtained for the camera 2. Since the human body has body temperature, whether the characteristic point is on the human body is judged through infrared thermal imaging. If the characteristic points are not on the human body, the abnormal image is considered, and subsequent calculation is not carried out.

In step 102, image recognition is performed on the feature points in each data file, so as to obtain a plurality of feature points for each data file, and feature point labeling is performed on each data file based on the obtained plurality of feature points.

Performing image recognition on the feature points in each data file, thereby obtaining a plurality of feature points for each data file comprises: determining a target object and an auxiliary object in each data file; and performing image recognition on the target object and the auxiliary object based on the gesture recognition network, and determining a plurality of feature points associated with the target object and the auxiliary object so as to obtain a plurality of feature points for each data file.

Each image acquisition device also has an infrared thermal imaging function; the method further comprises the steps of determining whether each feature point in the plurality of feature points belongs to the target object and/or the auxiliary object through an infrared thermal imaging function, protecting privacy data of the target object, and identifying and fusing sign parameters of the target object.

In step 103, the plurality of feature points labeled in each data file are divided into feature points of a first type and feature points of a second type, and the feature points of the first type of each of the plurality of data files are merged into the same feature points, so that the plurality of data files are merged into a target file associated with the target object.

The feature points of the first type are feature points associated with a target object, and feature points that do not satisfy a preset rule among a plurality of feature points of the first type are deleted. The feature points of the second type are feature points associated with an auxiliary object, and the feature points of the second type are used to correct the feature points of the first type.

The merging of the same feature points of the first type of each of the plurality of data files comprises: and merging the same characteristic points of the first type of the respective characteristic points of the plurality of data files based on the time consistency and the characteristic point attributes.

Merging the same feature points of a first type of each of the plurality of data files, thereby merging the plurality of data files into a target file associated with the target object, comprises: determining the same characteristic points in the first type of characteristic points of each of the plurality of data files; combining the feature points of the first type of each of the plurality of data files by taking the same feature point as a coincident common feature point;

and fusing the plurality of data files into a target file associated with the target object based on the merged feature points of the first type.

In step 104, a plurality of feature points in the target file are normalized to obtain a plurality of general feature points, and the motion attribute of the target object is determined based on the change angle and/or the movement data of at least one general feature point in the target file.

Fig. 6 is a schematic three-dimensional coordinate diagram of a feature point according to an embodiment of the present invention. As shown in fig. 6

Before normalization processing is carried out on a plurality of feature points in the target file, the method further comprises the following steps: determining three-dimensional coordinates for each feature point in the target file according to the following formula:

wherein Z is_c1And Z_c2Is the distance from the optical center to the image plane; u. of₁，v₁，u₂，v₂Coordinates of a coordinate system of the pixel points; k₁And M₁Obtaining an internal reference matrix and an external reference matrix of the first image acquisition device; k₂And M₂Obtaining an internal reference matrix and an external reference matrix of the second image acquisition device; x_w，Y_w，Z_wIs the three-dimensional coordinates of the feature points.

The normalizing the plurality of feature points in the target file to obtain a plurality of general feature points comprises:

calculating a node angle feature based on at least three feature points of the plurality of feature points:

wherein the content of the first and second substances,ja_t(j1, j2, j3) is an angular feature; j1, j2, j3 is a feature point, P_t(j1, j2) represents a vector comprising feature points j1 and j 2; p_t(j2, j3) represents vectors including feature points j2 and j 3;

wherein

P_t(j1，j2)＝(p_t(j1，x)-p_t(j2，x))+(p_t(j1，y)-p_t(j2，y)) +(p_t(j1，z)-p_t(j2，z))

Wherein pt represents the node coordinates;

and carrying out normalization processing on the relative track, wherein the normalization processing comprises the following steps: determining the relative distance between two feature points:

wherein nrt_t(b, s) is the relative distance between two feature points; b and s are characteristic points; rt is an integer of_t(b, s) is the relative track of the characteristic points b and s at the t-th frame, and t is a frame index; rt is an integer of₁(b, s) is the relative trajectory of the feature points of the first frame of the video;

wherein:

wherein pt represents the node coordinates; b, s and c are feature points;

and carrying out normalization processing on the speed, wherein the normalization processing comprises the following steps:

wherein nsp_t(j) The normalized speed of the characteristic point j in time 0-t is obtained; whereinIs the average speed of the time interval from 0 to t,is the maximum speed in the time interval of 0-t;

wherein sp_t(j) For the characteristic point moving speed, the calculation formula is as follows:

sp_t(j)＝F*(rt_t(b，j)-rt_(t-1)(b，j))，ift＞1.

＝0otherwise

wherein rt is_t(b, j) is the relative track of the characteristic points b and j at the time frame t; rt is an integer of_t-1(b, j) is the relative track of the characteristic points b and j at the time frame t-1; f is the sampling rate;

the acceleration is subjected to normalization processing, and the normalization processing comprises the following steps: dividing the average jerk from 0 to t by the maximum jerk

Wherein, njk_t(j) Normalized jerk of feature point j at t frames; whereinIs the average acceleration in the time interval of 0-t,the maximum acceleration within the time interval of 0-t; jk_t(j) The jerk at the feature point j;

wherein:

jk_t(j)＝F*(ac_t(b，j)-ac_t-1(b，j))，ift＞1.

＝0otherwise

wherein ac_tAcceleration for time t:

ac_t(j)＝F*(sp_t(j)-sp_(t-1)(j))，ift＞1.

＝0otherwise

the merging of the same feature points of the first type of the respective feature points of the plurality of data files based on the time consistency and the feature point attributes includes: in a plurality of data files, merging the characteristic points of the first type of each data file in a time consistency frame synchronization mode by taking the same characteristic point as a merging point.

In step 105, the operational attributes of the target object are mapped into a data file of a predetermined format according to predefined data mapping rules. The determining the motion attribute of the target object based on the change angle and/or the movement data of the at least one common feature point in the target file comprises: selecting at least one generic feature point associated with attribute identification from the plurality of generic feature points based on the location of the generic feature point; the motion attributes of the target object are determined based on the changing angle and/or movement data of the at least one generic feature point associated with the attribute identification in the target file.

The variation angle is used for representing the action completion degree of at least one universal characteristic point. The movement data is used to characterize fluency of motion of at least one generic feature point. The method comprises the steps of defining a data mapping rule for mapping the operation attribute into a data file with a preset format in advance before respectively acquiring the dynamic image data of the target object by utilizing a plurality of image acquisition devices; the data mapping rule includes a plurality of mapping entries, each mapping entry including < motion attribute, action result >.

For example, an example flow may include the following steps:

step one, extracting posture characteristic points of a person to be recovered through binocular vision and priori knowledge.

1. Meanwhile, two fixed cameras are used for collecting real-time training images of a patient, and the two images are respectively identified by means of a posture identification network to obtain two-dimensional human body characteristic points at the same moment.

2. The person has body temperature, so whether the characteristic points are on the human body is judged through infrared thermal imaging. If the characteristic point is not in the human body, the abnormal image is considered, and subsequent calculation is not carried out.

3. Three steps are needed to obtain the three-dimensional coordinates of the human body feature points, namely 1 camera calibration, 2 feature point matching and 3 three-dimensional reconstruction.

3.1, calibrating a camera:

for each camera, its coordinates in the world coordinate system and image coordinates can be represented by the following formulas.

Wherein Zc is the distance from the optical center to the image plane; f is the focal length of the lens; u and v are pixel coordinate system coordinates; r and T are rotation and translation matrixes of the image plane under a world coordinate system; dx and dy are the corresponding physical lengths of the pixels; xw, Yw and Zw are coordinates of the object in a world coordinate system; u0, v0 are pixel coordinates of the center of the image; and K and M are respectively an internal reference matrix and an external reference matrix of the camera.

In order to finally obtain accurate three-dimensional position information from the image information, the internal and external matrices K, M of the camera need to be calibrated. The two cameras shoot a plurality of standard black-and-white calibration chessboard with different postures, and the parameters can be obtained by applying the Zhang Zhengyou camera calibration method.

3.2 feature point matching

The feature points on the two images are obtained in the last step, and can be directly used as the feature points, so that the method is more accurate and stable compared with a common algorithm.

3.3, stereo reconstruction

The reconstruction method adopts a least square method for solving.

In the formula, an internal parameter matrix and an external parameter matrix of two cameras K1M1 and K2M2 are obtained; u1, v1, u2 and v2 pixel point coordinates; the remaining Zc1, Zc2, Xw, Yw, Zw 5 unknown parameters. And 6 equations are used for solving 5 unknown numbers, and the optimal solution (Xw, Yw and Zw) can be obtained by adopting a least square method, namely the three-dimensional point coordinates. And performing three-dimensional reconstruction on each key point to obtain the three-dimensional coordinates of the human body characteristic points.

4. The priori knowledge comprises two aspects, on one hand, the relative positions of a person to be recovered, a coach and an object are fixed, so that the characteristic points of the coach and the object can be deleted. And the other is that the human body feature points must meet the structural features of the human body, if the human body feature points do not meet the structural features of the human body, the human body feature points are regarded as abnormal images, and subsequent calculation is not carried out.

And 2, converting the human body feature points into the universal features.

The purpose is as follows: the direct use of the body feature points to predict the activities of the patients to be rehabilitated is inaccurate because each person has different heights and different lengths of the limbs. Therefore, the characteristics are normalized, the individual differences of people are removed, and the characteristics are converted into the universal characteristics.

The method comprises the following steps: 1 normalization of features:

joint angle characteristics: the joint angle needs 3 characteristic points to be calculated, and the formula is as follows:

j1, j2, j3 represents the feature point, and Pt (j1, j2) represents the vectors of the two joints, and the formula is as follows:

P_t(j1，j2)＝(p_t(j1，x)-p_t(j2，x))+(p_t(j1，y)-p_t(j2，y)) +(p_t(j1，z)-p_t(j2，z))

where pt represents a certain joint point coordinate.

1.1 normalized relative trajectory: relative distance between two feature points

Where rt (b, s) is the relative trajectory, the formula is as follows:

where pt represents a certain joint point coordinate.

1.2 normalized velocity: dividing the average speed from 0 to t by the maximum speed

Where sp is velocity, the formula is as follows:

sp_t(j)＝F*(rt_t(b，j)-rt_(t-1)(b，j))，ift＞1.

＝0otherwise

rt is the track in 1.2, velocity is the difference between the current frame and the previous frame, and F is the sampling rate, here 30.

1.3 normalized jerk: dividing the average jerk from 0 to t by the maximum jerk

jk is jerk, and the formula is as follows:

jk_t(j)＝F*(ac_t(b，j)-ac_t-1(b，j))，ift＞1.

＝0otherwise

ac is the acceleration, and the formula is as follows

ac_t(j)＝F*(sp_t(j)-sp_(t-1)(j))，ift＞1.

＝0otherwise

Step 3, converting the general characteristics into the description characteristics of the scale

Through the description of different scales, the description is abstracted into two types, namely the completion degree of the action and the fluency of the action.

1. Degree of completion of movement

1.1, degree

The three states are calculated by joint angle characteristics according to three degrees (failure, partial and complete), if the error of the three states with a standard video is more than 80%, the three states are considered to be failure, the error is 10% -80%, the error is considered to be partial, and the error is less than 10% and the error is considered to be complete.

1.2 detailed description of actions

For example, the knee joint can bend from a slightly extended position, but not more than 90 degrees, and the knee joint cannot bend when the hip joint is extended, and the specific motion description also needs joint angle characteristics to be calculated.

2. Fluency of motion

Including non-stop completion, hyperreflexia, tremor. These subjective descriptions need to be represented by a combination of normalized features.

2.1, non-stop completion

Non-stop completion-normalized velocity 0.7+ normalized jerk 0.3

2.2 hyperreflexia

Hyperfiltration (normalized velocity 0.3+ normalized relative trace 0.7

2.3, tremor

Jerk 0.8+ normalized jerk 0.2 + normalized relative trajectory

FIG. 2 is a flow diagram of a method 200 for data mapping based on image recognition according to another embodiment of the present invention. The method 200 begins at step 201.

In step 201, a plurality of still images of a target object are acquired with an image acquisition device. In step 202, image recognition is performed on each still image to obtain a plurality of feature points in each still image. In step 203, a parameter value of the target object in each still image is calculated based on the plurality of feature points, thereby obtaining a plurality of parameter values of the target object. At step 204, attribute values of the target object are determined based on the plurality of parameter values. In step 205, the attribute values of the target object are mapped to a data file of a predetermined format according to predefined data mapping rules. For example, the invention can measure angles from a back picture. From the measurements, the Cobb angle (a standard angle that is a common measure of scoliosis) can be calculated. The present application determines scoliosis by acquiring two static pictures.

The first step is as follows: based on the existing data sample library, a pix2pix enhanced GAN network is trained, and the generation of X-ray film data through RGB data automatic learning is realized. (currently, approximately 100-300 samples are collected per month, and have been pre-processed, such as image cropping, contrast adjustment, rotation enhancement, etc.)

The second step is that: and (3) marking on an X-ray film, performing supervised learning, training a Labelmarker network, and automatically monitoring the maximum bending angle point in the previous step.

The third step: and (6) connecting lines and calculating a Cobb angle. The invention can also measure the angle according to the front stoop picture. The angle of the vertical line is measured and is used to measure the degree of convexity of the lateral surface (since the spine of the human body is a double S structure in three dimensions). From similar triangles it can be obtained that both angles are identical, i.e. the vertical tilt angle, i.e. the FK, flash reference angle, can be directly obtained by measuring the horizontal tilt angle. Specifically, a static image is acquired first, then background removal is continued, binarization processing is performed on the image from which the background is removed, then Hough transformation is performed, and finally the slope is calculated to obtain the FK angle.

Fig. 7 is a schematic structural diagram of a system 700 for mapping data based on image recognition according to an embodiment of the present invention. The system 700 includes: acquisition means 701, recognition means 702, fusion means 703, processing means 704, mapping means 705, calculation means 706 and initialization means 707.

An acquiring device 701, configured to acquire moving image data of a target object by using a plurality of image acquiring apparatuses, respectively, and convert the moving image data acquired by each image acquiring apparatus into a data file, respectively, so as to obtain a plurality of data files associated with the target object. The plurality of image acquisition devices respectively acquire dynamic image data of the target object at different positions and from different angles. Each of the plurality of image capturing devices is movable. The data file is a video file.

The identifying device 702 is configured to perform image identification on the feature points in each data file, so as to obtain a plurality of feature points for each data file, and perform feature point labeling for each data file based on the obtained plurality of feature points. The identifying means 702 is specifically configured to: determining a target object and an auxiliary object in each data file; and performing image recognition on the target object and the auxiliary object based on the gesture recognition network, and determining a plurality of feature points associated with the target object and the auxiliary object so as to obtain a plurality of feature points for each data file.

A merging device 703, configured to divide the multiple feature points labeled in each data file into feature points of a first type and feature points of a second type, and merge the feature points of the first type of each of the multiple data files into a target file associated with the target object by using the same feature points. The feature points of the first type are feature points associated with a target object, and feature points that do not satisfy a preset rule among a plurality of feature points of the first type are deleted. The feature points of the second type are feature points associated with an auxiliary object, and the feature points of the second type are used to correct the feature points of the first type.

The fusion device 703 is specifically configured to: and merging the same characteristic points of the first type of the respective characteristic points of the plurality of data files based on the time consistency and the characteristic point attributes. The fusion device 703 is specifically configured to: determining the same characteristic point in the first type of characteristic points of each of the plurality of data files; combining the feature points of the first type of each of the plurality of data files by taking the same feature point as a coincident common feature point; and merging the plurality of data files into a target file associated with the target object based on the merged feature points of the first type.

And the processing device 704 is used for performing normalization processing on the plurality of feature points in the target file to obtain a plurality of general feature points, and determining the motion attribute of the target object based on the change angle and/or the movement data of at least one general feature point in the target file.

The mapping device 705 is used for mapping the operation attribute of the target object to a data file with a predetermined format according to a predefined data mapping rule. The processing device 704 is specifically configured to:

calculating a node angle feature based on at least three feature points of the plurality of feature points:

wherein, ja_t(j1, j2, j3) is an angular feature; j1, j2, j3 is a feature point, P_t(j1, j2) represents a vector comprising feature points j1 and j 2; p_t(j2, j3) represents vectors including feature points j2 and j 3;

wherein

P_t(j1，j2)＝(p_t(j1,x)-p_t(j2，x))+(p_t(j1，y)-p_t(j2，y)) +(p_t(j1，z)-p_t(j2，z))

Wherein pt represents the node coordinates;

and carrying out normalization processing on the relative track, wherein the normalization processing comprises the following steps: determining the relative distance between two feature points:

wherein nrt_t(b, s) is the relative distance between two feature points; b and s are characteristic points; rt is an integer of_t(b, s) is the relative track of the characteristic points b and s at the t-th frame, and t is a frame index; rt is an integer of₁(b, s) is the relative trajectory of the feature points of the first frame of the video;

wherein:

wherein pt represents the node coordinates; b, s and c are feature points;

and carrying out normalization processing on the speed, wherein the normalization processing comprises the following steps:

wherein nsp_t(j) The normalized speed of the characteristic point j in time 0-t is obtained; whereinIs the average speed of the time interval from 0 to t,is the maximum speed in the time interval of 0-t;

wherein sp_t(j) For the characteristic point moving speed, the calculation formula is as follows:

sp_t(j)＝F*(rt_t(b，j)-rt_(t-1)(b，j))，ift＞1.

＝0otherwise

wherein rt is_t(b, j) is the relative track of the characteristic points b and j at the time frame t; rt is an integer of_t-1(b, j) is the relative track of the characteristic points b and j at the time frame t-1; f is the sampling rate;

the acceleration is subjected to normalization processing, and the normalization processing comprises the following steps: dividing the average jerk from 0 to t by the maximum jerk

Wherein, njk_t(j) Normalized jerk of feature point j at t frames; whereinAverage acceleration in the time interval of 0-tThe degree of the magnetic field is measured,the maximum acceleration within the time interval of 0-t; jk_t(j) The jerk at the feature point j;

wherein:

jk_t(j)＝F*(ac_t(b，j)-ac_t-1(b，j))，ift＞1.

＝0otherwise

wherein ac_tAcceleration for time t:

ac_t(j)＝F*(sp_t(j)-sp_(t-1)(j))，ift＞1.

＝0otherwise

the processing device 704 is specifically configured to: in a plurality of data files, merging the characteristic points of the first type of each data file in a time consistency frame synchronization mode by taking the same characteristic point as a merging point. The processing device 704 is specifically configured to: selecting at least one common feature point associated with attribute identification from the plurality of common feature points based on the location of the common feature point; the motion attributes of the target object are determined based on the changing angle and/or movement data of the at least one generic feature point associated with the attribute identification in the target file. The variation angle is used for representing the action completion degree of at least one common characteristic point. The movement data is used for characterizing the fluency of motion of at least one universal feature point.

Computing means 706 for determining coordinates of the world coordinate system and image coordinates of each image acquisition device by the following formulas:

wherein Zc is the distance from the optical center to the image plane; u and v are coordinates of a coordinate system of the pixel points; dx and dy are the physical lengths corresponding to the pixels; u. of₀，v₀Pixel coordinates that are the center of the image; f is the focal length of the lens; r and T are rotation and translation of image plane under world coordinate systemA matrix; x_w，Y_w，Z_wCoordinates of the object in a world coordinate system; and K and M are respectively an internal reference matrix and an external reference matrix of the image acquisition equipment.

And calibrating the internal parameter matrix K and the external parameter matrix M of each image acquisition device.

Computing means 706 for determining three-dimensional coordinates for each feature point in the target file according to the following formula:

wherein Z is_c1And Z_c2Is the distance from the optical center to the image plane; u. of₁，v₁，u₂，v₂Coordinates of a coordinate system of the pixel points; k₁And M₁Obtaining an internal reference matrix and an external reference matrix of the first image acquisition device; k₂And M₂Obtaining an internal reference matrix and an external reference matrix of the second image acquisition device; x_w，Y_w，Z_wIs the three-dimensional coordinates of the feature points.

An initialization device 707, configured to place the plurality of image acquisition devices at different positions in advance, respectively, so that the plurality of image acquisition devices can acquire moving image data of the target object at the different positions, respectively. Each image acquisition device also has an infrared thermal imaging function; the initialization device 706 is used for determining whether each feature point in the plurality of feature points belongs to the target subject and/or the auxiliary subject through a thermal infrared imaging function, protecting privacy data of the target subject, and identifying and fusing sign parameters of the target subject.

An initialization device 707 for defining in advance a data mapping rule for mapping the operation attribute to a data file of a predetermined format; the data mapping rule includes a plurality of mapping entries, each mapping entry including < motion attribute, action result >.

Fig. 8 is a schematic structural diagram of a system 800 for mapping data based on image recognition according to another embodiment of the present invention. The system 800 includes:

an acquisition means 801 for acquiring a plurality of still images of a target object with an image acquisition apparatus;

an identifying device 802, configured to perform image identification on each still image to obtain a plurality of feature points in each still image;

a calculating means 803 for calculating a parameter value of the target object in each still image based on the plurality of feature points, thereby obtaining a plurality of parameter values of the target object;

a determining means 804 for determining a property value of the target object based on the plurality of parameter values; and

mapping means 805 for mapping the attribute values of the target object into a data file in a predetermined format according to a predefined data mapping rule.