1. A sign language inter-translation method is characterized in that: the method comprises a sign language forward translation method and a sign language reverse translation method; the sign language forward translation method comprises the following steps:

receiving gesture action data and converting the gesture action data into first character information; wherein the gesture motion data comprises finger bend data and gesture pose angle data;

performing voice synthesis processing on the current first character information to obtain sign language translation voice information;

the sign language reverse translation method comprises the following steps:

receiving voice data and converting the voice data into second text information;

receiving second character information or externally input third character information, and analyzing the current second character information or third character information to obtain second character information after word segmentation or third character information after word segmentation;

and receiving and processing the second character information after word segmentation or the third character information after word segmentation to obtain sign language animation information matched with the second character information after word segmentation or the third character information after word segmentation.

2. The sign language interpretive method according to claim 1, wherein: the gesture motion data is acquired through a data glove, and the data glove comprises a bending sensor for acquiring finger bending data and a main control module connected with the bending sensor;

before the sign language forward translation method, the sign language inter-translation method further comprises a finger bending data calibration method, wherein the finger bending data calibration method is realized based on a main control module of a data glove; the finger bending data calibration method comprises the following steps:

calibrating the bending sensor to obtain the maximum angle data and the minimum angle data of the current bending sensor;

setting a standard measuring range, wherein the range of the standard measuring range is [ a, b ], wherein a is the minimum value of the standard measuring range, and b is the maximum value of the standard measuring range;

acquiring the finger bending data output by the current bending sensor in real time, and then carrying out normalization processing on the finger bending data output by the current bending sensor in real time according to the maximum angle data and the minimum angle data of the current bending sensor and the minimum value and the maximum value of the standard range to obtain a calibration value.

3. The sign language interpretive method according to claim 2, wherein: when the calibration operation is carried out on the bending sensor, the method specifically comprises the following steps:

triggering a calibration instruction of the bending sensor, and entering a calibration state of the bending sensor;

receiving and processing finger bending data output by the current bending sensor in different bending states to obtain angle data of the current bending sensor in different bending states;

traversing angle data of the current bending sensor in different bending states to obtain maximum angle data and minimum angle data of the current bending sensor;

and storing the maximum angle data and the minimum angle data of the current bending sensor to finish the calibration operation of the current bending sensor.

4. A sign language interpretive method according to claim 3, wherein: triggering a calibration instruction of the bending sensor, and after entering a calibration state of the bending sensor, further comprising the following steps:

starting a timer;

after obtaining the angle data of the current bending sensor in different bending states, the method also comprises the following steps:

judging whether the timing of the timer is greater than the calibration time length or not; if the current bending sensor is in the different bending states, traversing the angle data of the current bending sensor in the different bending states, and obtaining the maximum angle data and the minimum angle data of the current bending sensor, otherwise, re-receiving and processing the finger bending data output by the current bending sensor in the different bending states.

5. The sign language interpretive method according to claim 2, wherein: the calibration values are:

wherein X is the finger bending data output by the current bending sensor in real time, D_maxFor maximum angle data of the current bending sensor, D_minAnd the minimum angle data of the current bending sensor is a, the minimum value of the standard measuring range is a, and the maximum value of the standard measuring range is b.

6. The sign language interpretive method according to claim 1, wherein: the gesture motion data is acquired through a data glove, and the data glove comprises a motion processing assembly and a main control module, wherein the motion processing assembly is used for acquiring gesture angle data, and the main control module is connected with the motion processing assembly;

the sign language inter-translation method further comprises a visual angle and direction calibration method, the visual angle and direction calibration method is realized based on a main control module, and the visual angle and direction calibration method comprises the following steps:

course angle offset set for data glove_yawIs zero;

acquiring a Quaternion of the initial operation of the motion processing component in real time, wherein the Quaternion is { q0, q1, q2 and q3 };

converting the quaternion into an Euler angle according to a conversion relation between a direction cosine array of the quaternion and the Euler angle, wherein the Euler angle Eular is { roll, pitch, yaw }, the roll is a roll angle of the data glove, the pitch is a pitch angle of the data glove, and the yaw is a course angle of the data glove;

marking the current course angle output value as yaw' and judging whether a starting gesture triggered by the data glove is detected or not, if so, resetting course angle offset of the data glove_yawIf not, no action is performed;

setting calibrated course angle yaw-offset_yaw。

7. A sign language interpretive system, comprising: the sign language inter-translation system is used for realizing the sign language inter-translation method of any one of claims 1 to 6 and comprises a translation terminal; the translation terminal comprises a sign language translation module, a voice recognition module, a Chinese word segmentation module, a voice synthesis module and a sign language animation synthesis module;

the sign language translation module is used for receiving the gesture action data, converting the gesture action data into first character information and then sending the first character information to the voice synthesis module;

the voice synthesis module is used for receiving the current first character information and carrying out voice synthesis processing on the current first character information to obtain sign language translation voice information;

the voice recognition module is used for receiving the voice data, converting the voice data into second character information and then sending the second character information to the Chinese word segmentation module;

the Chinese word segmentation module is used for receiving second character information or externally input third character information, analyzing the current second character information or the current third character information to obtain second character information after word segmentation or third character information after word segmentation, and then sending the second character information after word segmentation or the third character information after word segmentation to the sign language animation synthesis module;

and the sign language animation synthesis module is used for receiving and processing the second character information after word segmentation or the third character information after word segmentation to obtain sign language animation information matched with the second character information after word segmentation or the third character information after word segmentation.

8. The sign language interpretive system of claim 7, wherein: the sign language intertranslation system also comprises a data glove; the data glove comprises a bending sensor, a motion processing assembly and a main control module;

the bending sensor is used for acquiring finger bending data and then sending the finger bending data to the main control module;

the motion processing assembly is used for acquiring gesture attitude angle data and then sending the gesture attitude angle data to the main control module;

the main control module is used for acquiring gesture action data and then sending the gesture action data to the sign language translation module; wherein the gesture motion data comprises finger bend data and gesture pose angle data.

9. The sign language interpretive system of claim 7, wherein: the sign language translation module comprises a feature extraction unit, a daily sign language template library, a user-defined sign language template library and a sign language identification unit;

the feature extraction unit is used for receiving the gesture action data, extracting a key frame set from the gesture action data and then sending the key frame set to the sign language recognition unit;

the sign language recognition unit is used for receiving the key frame set sent by the feature extraction unit, performing feature matching on the key frame set and sign language template data in a daily sign language template library and/or a user-defined sign language template library to obtain first character information of a gesture template corresponding to the key frame set, and then sending the first character information to the voice synthesis module.

Background

Sign language is a visual and vivid language, and a hearing impaired person can naturally express thought contents by using the sign language. However, in the education of language and text, the great difference exists between sign language and written language, so that students using sign language have difficulty in learning written language, most of them often have difficulty in learning the logical thinking characteristics of language and text, reading and understanding and expression level of written language have certain difficulty, and thought contents expressed by written language are easy to misunderstand or not to understand^[1]([1]Wang flying snow, longevity, talking about hearing impaired people student's written language learning [ J]The special news of Liaoning teachers: society science edition 2014(3) 121-. Combining the related characteristics of the first language learning and the characteristics of the language learning of the hearing impaired people, the natural sign language is generally regarded as the first language of the hearing impaired people^[2]([2]Thinking about Chinese learning property of auditory handicapped [ J]2016(10) modern specialty education. Therefore, the hearing-strengthening person can communicate with the hearing-impaired person in a text mode, certain communication obstacles exist, and the hearing-impaired person can communicate with the hearing-impaired person in a sign language mode, so that the communication method is a relatively friendly communication method.

At present, systems for synthesizing sign language animations through text information exist in the market, and the systems can translate the meanings of characters through a mode of demonstrating sign language by a little in virtual animations according to input character information, so that hearing-impaired people can intuitively and vividly understand the character information. However, such systems cannot interpret the sign language content of the hearing-impaired person, and therefore the hearing-impaired person cannot communicate with the system in two directions through the sign language of the hearing-impaired person, and such systems are only suitable for the situation of sign language-related teaching.

Disclosure of Invention

The present invention is directed to solve the above technical problems to at least some extent, and the present invention provides a sign language inter-translation method and system.

The technical scheme adopted by the invention is as follows:

a sign language inter-translation method comprises a sign language forward translation method and a sign language reverse translation method; the sign language forward translation method comprises the following steps:

receiving gesture action data and converting the gesture action data into first character information; wherein the gesture motion data comprises finger bend data and gesture pose angle data;

performing voice synthesis processing on the current first character information to obtain sign language translation voice information;

the sign language reverse translation method comprises the following steps:

receiving voice data and converting the voice data into second text information;

receiving second character information or externally input third character information, and analyzing the current second character information or third character information to obtain second character information after word segmentation or third character information after word segmentation;

and receiving and processing the second character information after word segmentation or the third character information after word segmentation to obtain sign language animation information matched with the second character information after word segmentation or the third character information after word segmentation.

Preferably, the gesture motion data is acquired through a data glove, and the data glove comprises a bending sensor for acquiring finger bending data and a main control module connected with the bending sensor;

before the sign language forward translation method, the sign language inter-translation method further comprises a finger bending data calibration method, wherein the finger bending data calibration method is realized based on a main control module of a data glove; the finger bending data calibration method comprises the following steps:

calibrating the bending sensor to obtain the maximum angle data and the minimum angle data of the current bending sensor;

setting a standard measuring range, wherein the range of the standard measuring range is [ a, b ], wherein a is the minimum value of the standard measuring range, and b is the maximum value of the standard measuring range;

acquiring the finger bending data output by the current bending sensor in real time, and then carrying out normalization processing on the finger bending data output by the current bending sensor in real time according to the maximum angle data and the minimum angle data of the current bending sensor and the minimum value and the maximum value of the standard range to obtain a calibration value.

Further, when the calibration operation is performed on the bending sensor, the method specifically comprises the following steps:

triggering a calibration instruction of the bending sensor, and entering a calibration state of the bending sensor;

receiving and processing finger bending data output by the current bending sensor in different bending states to obtain angle data of the current bending sensor in different bending states;

traversing angle data of the current bending sensor in different bending states to obtain maximum angle data and minimum angle data of the current bending sensor;

and storing the maximum angle data and the minimum angle data of the current bending sensor to finish the calibration operation of the current bending sensor.

Further, triggering a bending sensor calibration instruction, and after entering a bending sensor calibration state, the method further comprises the following steps:

starting a timer;

after obtaining the angle data of the current bending sensor in different bending states, the method also comprises the following steps:

judging whether the timing of the timer is greater than the calibration time length or not; if the current bending sensor is in the different bending states, traversing the angle data of the current bending sensor in the different bending states, and obtaining the maximum angle data and the minimum angle data of the current bending sensor, otherwise, re-receiving and processing the finger bending data output by the current bending sensor in the different bending states.

Preferably, the calibration values are:

wherein X is the finger bending data output by the current bending sensor in real time, D_maxFor maximum angle data of the current bending sensor, D_minAnd the minimum angle data of the current bending sensor is a, the minimum value of the standard measuring range is a, and the maximum value of the standard measuring range is b.

Preferably, the gesture motion data is acquired through a data glove, and the data glove comprises a motion processing assembly for acquiring gesture attitude angle data and a main control module connected with the motion processing assembly;

the sign language inter-translation method further comprises a visual angle and direction calibration method, the visual angle and direction calibration method is realized based on a main control module, and the visual angle and direction calibration method comprises the following steps:

course angle offset set for data glove_yawIs zero;

acquiring a Quaternion of the initial operation of the motion processing component in real time, wherein the Quaternion is { q0, q1, q2 and q3 };

converting the quaternion into an Euler angle according to a conversion relation between a direction cosine array of the quaternion and the Euler angle, wherein the Euler angle Eular is { roll, pitch, yaw }, the roll is a roll angle of the data glove, the pitch is a pitch angle of the data glove, and the yaw is a course angle of the data glove;

marking the current course angle output value as yaw' and judging whether a starting gesture triggered by the data glove is detected or not, if so, resetting course angle offset of the data glove_yawIf not, no action is performed;

setting calibrated course angle yaw-offset_yaw。

A sign language intertranslation system is used for realizing any one of the above sign language intertranslation methods, and comprises a translation terminal; the translation terminal comprises a sign language translation module, a voice recognition module, a Chinese word segmentation module, a voice synthesis module and a sign language animation synthesis module;

the sign language translation module is used for receiving the gesture action data, converting the gesture action data into first character information and then sending the first character information to the voice synthesis module;

the voice synthesis module is used for receiving the current first character information and carrying out voice synthesis processing on the current first character information to obtain sign language translation voice information;

the voice recognition module is used for receiving the voice data, converting the voice data into second character information and then sending the second character information to the Chinese word segmentation module;

the Chinese word segmentation module is used for receiving second character information or externally input third character information, analyzing the current second character information or the current third character information to obtain second character information after word segmentation or third character information after word segmentation, and then sending the second character information after word segmentation or the third character information after word segmentation to the sign language animation synthesis module;

and the sign language animation synthesis module is used for receiving and processing the second character information after word segmentation or the third character information after word segmentation to obtain sign language animation information matched with the second character information after word segmentation or the third character information after word segmentation.

Preferably, the sign language intertranslation system further comprises a data glove; the data glove comprises a bending sensor, a motion processing assembly and a main control module;

the bending sensor is used for acquiring finger bending data and then sending the finger bending data to the main control module;

the motion processing assembly is used for acquiring gesture attitude angle data and then sending the gesture attitude angle data to the main control module;

the main control module is used for acquiring gesture action data and then sending the gesture action data to the sign language translation module; wherein the gesture motion data comprises finger bend data and gesture pose angle data.

Preferably, the sign language translation module comprises a feature extraction unit, a daily sign language template library, a user-defined sign language template library and a sign language identification unit;

the feature extraction unit is used for receiving the gesture action data, extracting a key frame set from the gesture action data and then sending the key frame set to the sign language recognition unit;

the sign language recognition unit is used for receiving the key frame set sent by the feature extraction unit, performing feature matching on the key frame set and sign language template data in a daily sign language template library and/or a user-defined sign language template library to obtain first character information of a gesture template corresponding to the key frame set, and then sending the first character information to the voice synthesis module.

The invention has the beneficial effects that:

1) the system has the function of bidirectional real-time communication, can forward translate the sign language of the hearing-impaired person in a mode of voice information or character information, and can also reverse translate the voice or characters of the hearing-impaired person in a mode of sign language animation, so that the hearing-impaired person and the hearing-impaired person can communicate without barriers. Specifically, the gesture action data of the hearing-impaired person can be converted into character information and voice information based on a sign language forward translation method, so that the hearing-impaired person can conveniently understand the sign language; when the hearing-impaired person carries out voice or text conversation with the hearing-impaired person, the invention can convert voice or text information into sign language animation information which is convenient for the hearing-impaired person to understand based on the sign language reverse translation method, so that the hearing-impaired person can understand the voice or text information output by the hearing-impaired person, and barrier-free communication between the hearing-impaired person and the hearing-impaired person is realized;

2) the calibration operation is carried out on the bending sensor, the standard range is set after the calibration operation, and the finger bending data output by the current bending sensor in real time is normalized to obtain a calibration value. The glove can be conveniently used by users with different using habits, the problem that output information of the bending sensor is inaccurate due to mechanical difference and installation position deviation of the bending sensor is avoided, and the gesture action recognition accuracy can be effectively improved;

3) by the visual angle and direction calibration method, the effect of eliminating error accumulation of the gesture attitude angle data can be realized, so that each set of actions expressed by the hearing-impaired person are established on the calibrated visual angle and direction, and the gesture attitude angle data can reflect thought meanings expressed by sign language of the hearing-impaired person.

Drawings

FIG. 1 is a flow chart of a sign language forward translation method according to the present invention;

FIG. 2 is a flow chart of a sign language reverse translation method according to the present invention;

FIG. 3 is a flow chart of a method of calibrating finger bending data according to the present invention;

FIG. 4 is a flow chart of the present invention during calibration of the bending sensor;

FIG. 5 is a block diagram showing the construction of a translation terminal according to the present invention;

fig. 6 is a block diagram of the sign language translation module according to the present invention.

Detailed Description

The invention is further described with reference to the following figures and specific embodiments.

It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

It should be understood that, for the term "and/or" as may appear herein, it is merely an associative relationship that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, B exists alone, and A and B exist at the same time.

It should also be noted that, in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may, in fact, be executed substantially concurrently, or the figures may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

Example 1:

the embodiment provides a sign language inter-translation method, which comprises a sign language forward translation method and a sign language reverse translation method, wherein the sign language forward translation method and the sign language reverse translation method are realized based on a translation terminal;

as shown in fig. 1, the sign language forward translation method includes the following steps:

A1. receiving gesture action data and converting the gesture action data into first character information; wherein the gesture motion data comprises finger bend data and gesture pose angle data;

A2. performing voice synthesis processing on the current first character information to obtain sign language translation voice information;

as shown in fig. 2, the sign language translation method includes the following steps:

B1. receiving voice data and converting the voice data into second text information;

B2. receiving second character information or externally input third character information, and analyzing the current second character information or third character information to obtain second character information after word segmentation or third character information after word segmentation;

B3. and receiving and processing the second character information after word segmentation or the third character information after word segmentation to obtain sign language animation information matched with the second character information after word segmentation or the third character information after word segmentation. It should be understood that after the corresponding sign language animation information is obtained, the translation terminal can render and play the sign language animation information.

Specifically, the sign language translation method comprises the following steps:

B101. receiving voice data and converting the voice data into second text information;

B201. receiving second character information, and analyzing the current second character information to obtain second character information after word segmentation;

B301. receiving and processing the second word information after word segmentation to obtain sign language animation information matched with the second word information after word segmentation;

or, the sign language reverse translation method comprises the following steps:

B102. receiving externally input third character information, and analyzing the current third character information to obtain the third character information after word segmentation;

B202. and receiving and processing the third character information after word segmentation to obtain sign language animation information matched with the third character information after word segmentation.

The embodiment has the function of bidirectional real-time communication, can forward translate the sign language of the hearing-impaired person in a mode of voice information or character information, and can also reverse translate the voice or characters of the hearing-impaired person in a mode of sign language animation, so that the hearing-impaired person and the hearing-impaired person can communicate without barriers. Specifically, the gesture action data of the hearing-impaired person can be converted into text information and voice information based on the sign language forward translation method, so that the hearing-impaired person can understand the sign language conveniently; when the hearing-impaired person has a voice or text conversation with the hearing-impaired person, the embodiment can convert the voice or text information into the sign language animation information which is convenient for the hearing-impaired person to understand based on the sign language translation method, so that the hearing-impaired person can understand the voice or text information output by the hearing-impaired person, and barrier-free communication between the hearing-impaired person and the hearing-impaired person is realized.

In this embodiment, the gesture motion data is collected through a data glove, and the data glove includes a bending sensor for collecting finger bending data and a main control module connected with the bending sensor;

before the sign language forward translation method, the sign language inter-translation method further comprises a finger bending data calibration method, wherein the finger bending data calibration method is realized based on a main control module of a data glove;

as shown in fig. 3, the finger bending data calibration method includes the following steps:

C1. calibrating the bending sensor to obtain the maximum angle data D of the current bending sensor_maxAnd minimum angle data D_min(ii) a Wherein, the current bending sensor is arranged corresponding to the appointed finger in the data glove;

C2. setting a standard measuring range, wherein the range of the standard measuring range is [ a, b ], wherein a is the minimum value of the standard measuring range and represents that the finger is in a straight state, and b is the maximum value of the standard measuring range and represents that the finger is in a maximum bending state;

C3. obtaining a current bend sensorThe finger bending data is output in real time and then the maximum angle data D of the current bending sensor is obtained_maxAnd minimum angle data D_minAnd normalizing the finger bending data output by the current bending sensor in real time to obtain a calibration value.

It should be noted that, the bending sensor in this embodiment is used in cooperation with a data glove, the bending sensor is used for sensing the bending degree of the finger of the user and outputting the bending data of the finger, and a plurality of bending sensors may be disposed on the same glove, so that specific information corresponding to a gesture can be acquired according to the bending data of the finger output by different bending sensors on the same glove.

Specifically, in the embodiment, a calibration operation is performed on the bending sensor, a standard range is set after the calibration operation, and normalization processing is performed on the finger bending data output by the current bending sensor in real time, so as to obtain a calibration value. The glove can be used by users with different using habits conveniently, the problem that the output information of the bending sensor is inaccurate due to the mechanical difference and the installation position deviation of the bending sensor is avoided, and the recognition accuracy of gesture actions can be effectively improved.

In this embodiment, when the calibration operation is performed on the bending sensor, as shown in fig. 4, the method specifically includes the following steps:

C101. triggering a calibration instruction of the bending sensor, entering a calibration state of the bending sensor, and starting a timer; it should be understood that the bending sensor calibration command may be triggered, but is not limited to, when the operation is first initiated.

C101a, starting a timer;

C102. receiving and processing finger bending data output by the current bending sensor in different bending states to obtain angle data of the current bending sensor in different bending states;

c102a, judging whether the timing of the timer is greater than the calibration time length; if yes, entering step C103, otherwise, re-receiving and processing finger bending data output by the current bending sensor in different bending states; the maximum angle data of the current bending sensor represents the finger bending data output by the bending sensor when the target finger is in the maximum bending state, and the minimum angle data of the current bending sensor represents the finger bending data output by the bending sensor when the target finger is in the straight state;

C103. traversing the angle data of the current bending sensor in different bending states to obtain the maximum angle data D of the current bending sensor_maxAnd minimum angle data D_min；

C104. Storing maximum angle data D of a current bending sensor_maxAnd minimum angle data D_minAnd finishing the calibration operation of the current bending sensor. By default, the data processing system for the data glove synchronously calibrates the bending sensors corresponding to the plurality of fingers and simultaneously stores the maximum angle data D of the plurality of bending sensors_maxAnd minimum angle data D_minTherefore, the efficiency of the calibration operation of the bending sensor can be improved.

In this embodiment, each finger is correspondingly provided with a bending sensor, and after the bending sensor corresponding to each finger is calibrated, a unique D is generated_maxAnd D_minAnd D between each finger_maxAnd D_minHas mutual independence and is not applicable.

It should be noted that, after the angle data of the current bending sensor in different bending states are obtained, if the finger changes from a fist-making state to a straightening state and corresponds to the angle data of the bending sensor, the current bending sensor completes one bending and straightening change, and then the maximum angle data D of the current bending sensor can be directly obtained according to the current action_maxAnd minimum angle data D_min(ii) a In this case, the bending sensor is subjected to a change only once, and thus a problem that the bent state of the finger cannot be sufficiently expressed is likely to occur. In order to solve the technical problem, the present embodiment further sets the calibration duration, the user can perform bending and straightening of the finger for multiple times within the calibration duration, and the maximum angle data D of the current bending sensor_maxTo minimum angle data D_minThe preset standard range of the hand can be fully displayedRefers to the characteristic change from a flat state to a maximum curved state.

Specifically, when the bending sensor calibration operation is performed, the system allocates a certain calibration time (i.e., a calibration duration) to the user to complete the calibration operation, such as 10 seconds. When the user wears the data gloves and carries out the demarcation operation of crooked sensor, can straighten the finger, bend the finger again and make a fist. The user can repeatedly straighten the fingers and then make a fist within the calibrated duration, and the bending sensor can measure and output the finger bending data of the finger joints in different bending states in real time.

Specifically, the calibration values are:

wherein X is the finger bending data output by the current bending sensor in real time, D_maxFor maximum angle data of the current bending sensor, D_minAnd the minimum angle data of the current bending sensor is a, the minimum value of the standard measuring range is a, and the maximum value of the standard measuring range is b.

It should be noted that, in step C3, after the finger bending data output by the current bending sensor in real time is obtained, the finger bending data output by the current bending sensor in real time may also be subjected to nonlinear calibration to obtain a nonlinear calibration value; the nonlinear calibration values are:

wherein X is the finger bending data output by the current bending sensor in real time, D_maxFor maximum angle data of the current bending sensor, D_minAnd the minimum angle data of the current bending sensor is a, the minimum value of the standard measuring range is a, and the maximum value of the standard measuring range is b.

It should be noted that, by using the processing method of nonlinear calibration to obtain the nonlinear calibration value, the calibration value can be more fit to the characteristics of the bending sensor in the data glove, so that the recognition accuracy of the gesture motion can be further improved, and the gesture determination result is more reliable.

In this embodiment, the gesture motion data is collected through a data glove, and the data glove includes a motion processing component for collecting gesture attitude angle data and a main control module connected with the motion processing component;

it should be understood that the data glove and the translation terminal can be connected by, but not limited to, bluetooth connection, USB serial connection or WiFi connection, and the like, and is not limited herein.

In the prior art, in the process that the data glove expresses sign language actions along with a user, the main control module can sample the Euler angle of the data glove rotating around a world coordinate system in real time through the motion processing assembly, and the Euler angles on the axes of the three world coordinate systems of the data glove are divided into a roll angle roll, a pitch angle pitch and a course angle yaw. However, the data glove often involves a random problem of the initial viewing angle orientation during the use process due to the hand posture and orientation information, wherein the heading angle yaw is used for measuring the change of the user hand in the geographic orientation and is an important measurement parameter in the hand motion. In the using process of the data glove, the course angle yaw can change along with the change of the geographic position of the user, so that when the user uses the data glove to express sign language, the starting position of the front visual angle of the data glove has randomness due to the fact that the standing position has randomness, and the expressed sign language action signal characteristics are completely different in the visual angle position due to the fact that the geographic visual angle position of the user is inconsistent when the user performs the same sign language action. Therefore, the same gesture language action in different directions cannot be explained by using the same action signal characteristic. In order to solve the above technical problem, in this embodiment, the sign language inter-translation method further includes a viewing angle and orientation calibration method, where the viewing angle and orientation calibration method is implemented based on a main control module, and the viewing angle and orientation calibration method includes the following steps:

D1. course angle offset set for data glove_yawIs zero;

D2. acquiring a Quaternion of the initial operation of the motion processing component in real time, wherein the Quaternion is { q0, q1, q2 and q3 };

D3. converting the quaternion into an Euler angle according to a conversion relation between a direction cosine array of the quaternion and the Euler angle, wherein the Euler angle Eular is { roll, pitch, yaw }, the roll is a roll angle of the data glove, the pitch is a pitch angle of the data glove, and the yaw is a course angle of the data glove;

respectively performing first-order smoothing filtering processing on angle data roll, pitch and yaw in the Euler angle based on a filtering formula to obtain processed angle data; wherein, the filtering formula is: y is_n＝0.7X_n+0.3Y_n-1. Wherein, Y_nIs the output value of the target angle, X_nThis sampled value, Y, for the target angle_n-1The last filtering output value of the target angle is obtained, and the filtering coefficient is 0.7;

D4. marking the current course angle output value as yaw' and judging whether a starting gesture triggered by the data glove is detected or not, if so, resetting course angle offset of the data glove_yawIf not, no action is performed; the starting gesture is triggered by the user wearing the data rear sleeve, and is a gesture of tightly holding fingers when the arm naturally drops;

D5. setting calibrated course angle yaw-offset_yawAt offset_yawCalibrating the output course angle yaw to further finish the visual angle and azimuth calibration of the data glove;

by the visual angle and direction calibration method, the effect of eliminating error accumulation of the gesture attitude angle data can be realized, so that each set of actions expressed by the hearing-impaired person are established on the calibrated visual angle and direction, and the gesture attitude angle data can reflect thought meanings expressed by sign language of the hearing-impaired person. Specifically, in the implementation process, when the user is about to start expressing sign language, the main control module may define the viewing angle data in the direction of the heading angle yaw' after the deviation as the initial viewing angle data, and provide the deviation amount for the viewing angle azimuth calibration in the heading angle direction of the subsequent action. In the process, the main control module can calibrate the heading angle data and calculate the angle values of the roll angle and the pitch angle pitch. Therefore, by periodically and cyclically executing the steps of the perspective and orientation calibration method, the main control module can obtain the Eular ═ roll, pitch, yaw } of the current gesture of the user in real time.

Specifically, in this embodiment, the bending sensor is a current-type bidirectional bending sensor, the motion processing component is a 9-axis motion processing component of the type MPU9250, and in the execution process of the steps of the finger bending data calibration method and the visual angle and orientation calibration method, the data glove performs automatic real-time online data reading on the bending sensor and the motion processing component through the main control module, normalizes the reading values of the bending sensor, calibrates the visual angle and orientation of the motion processing component, and provides gesture action data with a higher standard for the whole sign language translation process.

Specifically, in this embodiment, the current-type bidirectional bending sensor is implemented by using the current-type bidirectional bending sensor disclosed in chinese patent publication No. CN110095086A, and has a wide application range, convenience in angle measurement, and high measurement accuracy. The two ends of the current type bidirectional bending sensor are respectively a current input end and a current output end, wherein the two ends of the current type bidirectional bending sensor are electrically connected with the main control module through the acquisition interface, and are not described herein.

The main control module realizes real-time calculation of quaternion representing the attitude information by driving a Digital Motion Processor (DMP) in the MPU 9250. Before the main control module converts the quaternion into the Euler angle, each element of the quaternion needs to be divided by 2 in advance³⁰Reducing the quaternion to a floating point number, converting the quaternion into Euler angle data through an asin () and atan2() function expression according to the conversion relation between the quaternion direction cosine array and the Euler angle, wherein the conversion expression is as follows:

in this embodiment, the data glove periodically sends gesture motion data through the information collecting and exchanging interface in a frame-by-frame manner. For example, at time t0, the data glove collects and sends out data frames in the format:

Frame_t0＝{F1_t0,F2_t0,F3_t0,F4_t0,F5_t0,pitch_t0,roll_t0,yaw_t08 channel data in total, wherein F1 to F5 are finger bending data of 5 fingers, F1 respectively_t0,F2_t0,F3_t0,F4_t0,F5_t0The finger bending data of 5 fingers at time t0, and pitch, roll, and yaw are gesture attitude angle data of different euler angles.

When detecting that a user starts to express sign language, the data glove starts to send finger bending data and/or gesture attitude angle data to the translation terminal; and when the data glove detects that the user finishes expressing the sign language, finishing sending the data finger bending data and/or the gesture angle data to the translation terminal.

In the process, in order to realize the function of detecting the beginning and ending of the expression of the sign language of the user, the data glove adopts a mode of triggering a specific gesture, and the hand gesture and the finger bending state of the user are detected in real time through the main control module. For example, assuming that the standard range of the reading value of the calibration bending sensor is [0,100], when finger bending data is detected as Frame { F1 e [90,100], F2 e [90,100], F3 e [90,100], F4 e [90,100], F5 e [90,100], P e [ -50, -90], R e [ -180,180 ], Y e [ -180,180 }, indicating that five fingers bend and make a fist in a state of natural hand drop, determining a gesture result as that the current user starts to express a sign language according to a nonlinear calibration value obtained from the finger bending data, and defining the gesture result as a starting gesture; when the finger bending data is detected to be Frame [ { F1 ∈ [90,100], F2 ∈ [0,10], F3 ∈ [0,10], F4 ∈ [0,10], F5 ∈ [0,10], P ∈ [ 50, -90], R ∈ [ 180,180 ], Y ∈ [ 180,180 ]) }, the thumb is tightly held in a naturally-drooping state, the rest four fingers are unfolded and straightened, a gesture result is judged to be that the current user finishes expressing the sign language according to a nonlinear calibration value obtained by the finger bending data, and the gesture result is defined as an end gesture; and the like. It should be understood that the gesture result may also be a gesture result corresponding to different meaning sign languages, and is not described herein in detail. Wherein the start gesture and the end gesture are set to be specific gestures.

Example 2:

the embodiment provides a sign language inter-translation system, which is used for implementing the sign language inter-translation method in the embodiment 1, and the sign language inter-translation system comprises a translation terminal; as shown in fig. 5, the translation terminal includes a sign language translation module, a speech recognition module, a chinese word segmentation module, a speech synthesis module, and a sign language animation synthesis module;

the sign language translation module is used for receiving the gesture action data, converting the gesture action data into first character information and then sending the first character information to the voice synthesis module;

the voice synthesis module is used for receiving the current first character information and carrying out voice synthesis processing on the current first character information to obtain sign language translation voice information;

the voice recognition module is used for receiving the voice data, converting the voice data into second character information and then sending the second character information to the Chinese word segmentation module;

the Chinese word segmentation module is used for receiving second character information or externally input third character information, analyzing the current second character information or the current third character information to obtain second character information after word segmentation or third character information after word segmentation, and then sending the second character information after word segmentation or the third character information after word segmentation to the sign language animation synthesis module; in the embodiment, when the Chinese word segmentation module analyzes the second character information or the third character information, the grammar analysis of the natural language and the Chinese word segmentation can be realized;

and the sign language animation synthesis module is used for receiving and processing the second character information after word segmentation or the third character information after word segmentation to obtain sign language animation information matched with the second character information after word segmentation or the third character information after word segmentation.

In this embodiment, the sign language intertranslation system further includes a data glove; the data glove comprises a bending sensor, a motion processing assembly and a main control module, wherein in the embodiment, the main control module is electrically connected with the bending sensor and the motion processing assembly;

the bending sensor is used for acquiring finger bending data and then sending the finger bending data to the main control module;

the motion processing assembly is used for acquiring gesture attitude angle data and then sending the gesture attitude angle data to the main control module;

the main control module is used for acquiring gesture action data and then sending the gesture action data to the sign language translation module; wherein the gesture motion data comprises finger bend data and gesture pose angle data.

In this embodiment, as shown in fig. 6, the sign language translation module includes a feature extraction unit, a daily sign language template library, a custom sign language template library, and a sign language recognition unit;

the feature extraction unit is used for receiving the gesture action data, extracting a key frame set from the gesture action data and then sending the key frame set to the sign language recognition unit;

the sign language recognition unit is used for receiving the key frame set sent by the feature extraction unit, performing feature matching on the key frame set and sign language template data in a daily sign language template library and/or a user-defined sign language template library to obtain first character information of a gesture template corresponding to the key frame set, and then sending the first character information to the voice synthesis module. It should be understood that the daily sign language template library and the custom sign language template library both contain a plurality of gesture templates corresponding to gesture action data and first text information matched with the gesture templates.

Specifically, in this embodiment, the gesture motion data is received from the main control module in the data glove via the feature extraction unit, and a key frame set with high information content is extracted. Then the sign language recognition unit respectively matches the key frame set with all sign language template data in the daily sign language template library and the user-defined sign language module library in a characteristic way; if a sign language template which has higher matching degree with the key frame set on the data characteristics and is optimal in matching confidence coefficient can be screened out from a daily sign language template library or a custom sign language module library, the sign language template is used as a sign language recognition result, then first character information corresponding to the sign language template is output, and the first character information is sent to a voice synthesis module to be converted into sign language translation voice information so as to be displayed and played through mobile equipment.

In this embodiment, the feature extraction unit receives gesture motion data sent by the main control module in the data glove through the information exchange interface, and specifically, the information exchange interface is used to cache the gesture motion data uploaded by the data glove. Because each frame of data periodically uploaded by the data glove is fixed to be 8 channels, the gesture action data is cached by adopting a data structure of a two-dimensional array A [ n ] [8], wherein,

wherein, the row vector sequence of A [ n ] [8] [ [ F1t0, …, Rt0, Yt0], [ F1t1, …, Rt1, Yt1], …, [ F1tn, …, Rtn, Ytn ] ] represents a sequence of n elements in total with one frame data as a unit for each element; the column vector sequence of a [ n ] [8] [ [ F1t0, F1t1, …, F1n-1, F1n ], [ F2t0, F2t1, …, F2n-1, F2n ], …, [ Yt0, Yt1, …, Yn-1, Yn ] ] represents a sequence of 8 elements in total for each element in a unit of single-channel time series data.

When the sign language is expressed, the hand motions of the user are coherent, and some key motions in the sign language often have important information. For example, when the user has a sign language consistent action in the expression "two, three, four, five, six, seven, ninety", the user tends to move from "one" to "two" in gesture after the particular gesture action expressing "one". The specific gesture actions such as 'one' and 'two' belong to key gestures, and the transition gesture belongs to the process of changing from the 'one' gesture action to the 'two' gesture action. In sign language expression, key gestures are often clearly expressed, while transition gestures are relatively blurred in the motion change trend. Therefore, when the user expresses the sign language, in the gesture motion data collected by the data glove, the data change trend of the key gesture part is relatively stable, and the data change trend of the transition gesture part is relatively fluctuated. Therefore, the feature extraction unit is used for extracting a key frame set with high information content from the gesture motion data, namely feature data of key motion in the gesture motion.

In this embodiment, gesture motion data in a period of continuous time is referred to as sign language time sequence data, and according to the fluctuation characteristic of the sign language time sequence data, the feature extraction unit calculates the change rate of the dimension time sequence data by performing first-order derivation on the time sequence data of each dimension channel. And according to the change rate of the time domain data on each dimension, a plurality of key frames with relatively small discrete degree are calculated through a standard deviation formula. The key frames refer to key gesture motion data extracted from sign language time sequence data, and have the characteristic that the change trend on time domain data is relatively small.

Specifically, the method for extracting the features of the time series data of the hand language specifically comprises the following steps:

E1. the received sign language time sequence two-dimensional array A [ n ] [8], extracting each element of a column vector sequence from the A [ n ] [8] through a feature extraction unit, respectively performing first-order derivation calculation on each element, and generating a two-dimensional array B [ n-1] [8] according to a time sequence order group by data calculated after derivation;

E2. and respectively carrying out standard deviation calculation on each element of the B row vector sequence of the two-dimensional array, wherein the calculation result reflects the dispersion degree of the element data set. Combining the calculation result values of each standard deviation according to the calculated order to generate a dimensional array a [ n-1 ];

E3. setting a threshold value K, traversing the array a [ n-1], comparing the elements with the K value one by one, and marking the index of the element value lower than the threshold value K;

E4. leading the marked index set into a two-dimensional array A [ n ] [8], and extracting key frame data of row vectors in the array A [ n ] [8] according to the index position; wherein the key frame data is not zero.

In this embodiment, the sign language database includes a daily sign language template library and a custom sign language template library. The daily sign language template library is a fixed sign language template library preset by a sign language translation module, integrates a plurality of sign language templates daily communicated by hearing-impaired people, each sign language template consists of time sequence data of 8 channels, such as a two-dimensional array A [ n ] [8], and can be accessed to a cloud server through a network communication unit for data updating. The user-defined sign language template library is an incremental library which faces the openness of users, each sign language template consists of time sequence data of 8 channels, such as a two-dimensional array A [ n ] [8], the user-defined sign language template library allows the user to freely input a user-defined sign language action, and the user further performs personalized supplement on the sign language database template, so that the data of the sign language template library can further meet the personalized requirements of the user, the sign language translation system can be compatible with the sign language expression habits of multiple users, the sign language translation process has higher expansibility and flexibility, and good experience effects are brought to the user.

The sign language recognition unit has the functions of analyzing the data characteristics of the gesture actions, recognizing the sign language actions, counting sign language recognition results and the like. Specifically, after the user gesture is extracted by the feature extraction unit, the sign language recognition unit can perform matching recognition on the key frame set by combining with the sign language template library, output the recognition result as first character information, and then send the first character information to the speech synthesis module. Along with the increase of the sign language translation times of the user, the sign language recognition unit gradually accumulates a large number of sign language translation results, counts the characteristic difference between each gesture action and the sign language recognition result, can analyze the action habit of the user when the user carries out sign language expression, and deeply excavates data characteristics to fit the action habit of the user. And the minimum value, the maximum value and the minimum value of the relevant time sequence signals can be extracted according to the last recognition result of the user, and the data characteristic identification weight of the sign language database is corrected, so that the database tends to the sign language action habit of the user continuously, and an experience value is provided for the next sign language recognition. When a user performs a new round of sign language action expression, the sign language translation module can strengthen the reliability of the sign language translation module for the sign language recognition of the user according to the learned experience value of the historical data.

In the process of sign language action recognition, the method comprises the following steps:

F1. inputting gesture motion data to be processed into a feature extraction unit, and extracting a key frame set A;

F2. and traversing all sign language models in the custom hand language database and the daily hand language database, and setting a tolerance interval for the data point of each time sequence data frame. Expanding the time sequence data frame of each sign language model into a surface from a line according to the upper limit and the lower limit of the tolerance interval;

F3. traversing the key frame set A, and taking out one key frame f;

F4. matching all sign language templates in the user-defined sign language library and the daily sign language library with the key frame f, and if the data frame tolerance interval of a certain sign language template covers the key frame f, performing confidence level accumulation on the sign language template;

F5. repeating the step F3 and the step F4 in a circulating manner until all the key frames in the key frame set A are matched with all the templates in the phrase library;

F6. and counting the matching results of all the sign language templates in the sign language library and the key frame set A, and taking the sign language template with the optimal confidence as the current sign language recognition result.

F7. And displaying the character information of the sign language recognition result, and synthesizing voice broadcast.

In this embodiment, the sign language animation synthesis module includes a sign language animation combination unit and a sign language animation database;

when the voice recognition module detects that voice is input, recognizing the voice information to be processed, and transmitting a recognition result to the Chinese word segmentation module in a character form; similarly, the system also supports external manual character input, and the input character information is directly transmitted to the Chinese word segmentation module; the Chinese word segmentation module carries out word segmentation operation on input word information (second word information or third word information) to form a plurality of independent words, namely the word segmented second word information or the word segmented third word information; calling a sign language animation synthesis module, searching the second character information after word segmentation or the third character information after word segmentation to obtain animation data of corresponding character labels in a sign language animation library, and matching sign language animation data of each key word according to the one-to-one correspondence relationship of the character labels; the sign language animation combination unit combines a plurality of independent sign language animation data into a group of coherent sign language animation results by mutually independent animation data in a transition motion frame insertion mode. The sign language animation result represents the thought content of the input voice and the characters.

In this embodiment, the voice input from the outside is recognized as the natural language L by the voice recognition module, or the natural language L is input through a medium such as a keyboard. The natural language L is participled through a Chinese Word segmentation module to generate Words { Word1, Word 2. If the natural language L is equal to 'today is Friday', the natural language L is input to a Chinese word segmentation module for word segmentation, and Words { 'today', 'Yes', 'week', 'five' }isgenerated. And then, the sign language animation combination unit in the sign language animation module can retrieve the local sign language animation database according to word information of the Words, and a group of coherent sign language animation results are generated by the retrieved animation data through the sign language animation combination unit, and the animation results are abstractly expressed as that the virtual animation small person plays coherent sign language actions on the display screen.

The sign language animation database covers a plurality of independent sign language animation sequences. Each animation sequence comprises a plurality of animation key frames, and the animation key frames represent data of the finger bending state and the arm joint rotation state of the virtual animation small person at periodic time points. Each sign language animation sequence is pre-entered by the system and stored on the ROM memory. Each animation sequence has a unique text label, and the thought expressed by the text content of the animation sequence is consistent with the sign language meaning of the virtual animation small person. Meanwhile, the text label is also suitable for being matched with the input words.

The sign language animation combination unit is responsible for carrying out frame insertion combination on a plurality of independent sign language animation data. Because each independent sign language animation data has the characteristics of boundary and high cohesion, when two independent sign language animations are directly subjected to incremental combination, the data boundaries of the two sign language animations are prone to have mutation, so that the sign language animations have gesture action jumping in the expression process, and the actions of the sign language animations do not have a coherent effect. Therefore, the sign language animation combination unit combines the independent animation data into a group of coherent sign language animation results by using a transition motion frame interpolation mode.

In this embodiment, the sign language animation combining unit performs frame-filling combination on the two sets of data as follows:

G1. inputting natural language L into Chinese Word segmentation module to generate Word ═ Word₁，Word₂,...,Word_nN is more than or equal to 1;

G2. inputting the vocabulary information of Words into a sign language animation database, and searching out a sign language animation Sequence set Sequence ═ S₁，S₂,...,S_nWherein n is more than or equal to 1. Wherein each animation sequence S has a plurality of key frames F, namely S_n＝{Fn₁,Fn₂,...,Fn_kWherein k is more than or equal to 1;

G3. reading a sign language animation Sequence set Sequence, and respectively inserting a transition frame placeholder T between every two adjacent animation sequences;

G4. traversing the sign language animation Sequence set Sequence, reading all the key frames of the animation Sequence according to the arrangement Sequence, and combining the key frames into a set K;

G5. assigning and frame inserting the placeholders of the transition frames by using all key frame data in the key frame set K through a third-order spline frame inserting method;

G6. combining all the key frame data and the transition frame data into a group of coherent sign language animation data according to the time sequence order;

G7. and rendering the sign language animation combination result to a virtual animation small person for playing.

The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: modifications of the technical solutions described in the embodiments or equivalent replacements of some technical features may still be made. And such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Finally, it should be noted that the present invention is not limited to the above alternative embodiments, and that various other forms of products can be obtained by anyone in light of the present invention. The above detailed description should not be taken as limiting the scope of the invention, which is defined in the claims, and which the description is intended to be interpreted accordingly.