1. A multi-modal interaction method for a vehicle based on Bayesian theorem is characterized by comprising the following steps:

step 1: comprehensively acquiring multi-mode behavior information by multiple sensors:

shooting the behavior of the driver through a special gesture detection camera to obtain gesture changes of the driver during different operations; shooting a picture of the face of a driver through a front camera to obtain expression information of the driver; capturing blinking actions through an eye tracker sensor, and judging the fatigue degree of a driver; obtaining voice information of a driver through a vehicle-mounted intelligent voice system, and obtaining a specific instruction of the driver;

step 2: feature extraction:

carrying out feature extraction on data of information collected by a sensor to form a bottom-layer expression of human body behaviors of a driver; starting from the characteristics, further performing digital representation on each information, determining the corresponding relation between the information and the Bayesian network nodes, and determining values;

and step 3: bayesian network building and training based on reality data:

acquiring multi-sensor acquired data of a vehicle running in a past practical scene, extracting features of the data, constructing a training data set by combining real behavior intentions of a driver, constructing and training a Bayesian network by using samples in the training data set, determining the network structure and probability distribution of the Bayesian network, and obtaining the trained Bayesian network, so that a mapping relation between the features and the behavior intentions of the user is established, and a one-to-one mapping relation between various features and single result judgment is realized;

and 4, step 4: inputting the obtained multi-modal characteristic information into a Bayesian network built based on real data to obtain the real intention of the user;

and 5: the feedback execution means performs appropriate feedback to the user in multiple modes such as visual, auditory, and tactile according to the inferred user intention.

2. The Bayesian theorem-based vehicle multi-modal interaction method according to claim 1, wherein: in the step 1, the behavior of the driver is shot by a special gesture detection camera to obtain gesture changes of the driver during different operations; shooting a picture of the face of a driver through a front camera to obtain expression information of the driver; capturing blinking actions through an eye tracker sensor, and judging the fatigue degree of a driver; and obtaining the voice information of the driver through a vehicle-mounted intelligent voice system, and obtaining a specific instruction of the driver.

3. The Bayesian theorem-based vehicle multi-modal interaction method according to claim 1, wherein: in the step 2, the special camera for gesture detection can process and obtain the skeleton model of the operator and the three-dimensional coordinate data of the key nodes in the skeleton model in real time through a human skeleton tracking technology, most people use the body language of the upper limbs when driving, the real intention of the user is conveyed through the upper half body movement, seven joint points of a left wrist joint point, a right wrist joint point, a left elbow joint point, a right elbow joint point, a left shoulder joint point, a right shoulder joint point and a head joint point are selected as key points to obtain three-dimensional coordinate data of the key points, then, coordinate data is detected, the moving range of each point is divided into 9 areas which are represented by 0 to 8, when the coordinates of a certain key point fall into a certain area, the coordinates are expressed by the number of the area, and the processed seven key point data are saved to the feature E.₁,E₂,E₃,E₄,E₅,E₆,E₇Wherein, the values of the two are all 0 to 8;

the face picture of the user shot by the front camera passes the trained depthConvolutional neural network recognition expression E₈With different expressions in E₈Are indicated by different numbers;

the eye tracker sensor can obtain the characteristic PERCLOS, namely the percentage of the eye closing time in unit time, the characteristic BF, namely the blinking frequency, the characteristic MECD, namely the duration of the longest eye closing in a certain time through data analysis, is used for detecting the concentration and the fatigue degree, the PERCLOS characteristic, the BF characteristic and the characteristic MECD are respectively subjected to discretization processing, are divided into five grades, are respectively represented by numbers 0-4 and are stored in the characteristic E₉、E₁₀And E₁₁Performing the following steps;

the vehicle-mounted intelligent voice system acquires voice information through a microphone, converts the voice information into corresponding text description information through natural language identification, and extracts preset keyword information to E₁₂In E, different voice information is₁₂Are indicated by different numbers.

4. The Bayesian theorem-based vehicle multi-modal interaction method according to claim 1, wherein: in the step 3, multi-sensor collected data of a vehicle running in a past practical situation are obtained, feature extraction is carried out on the data, a training data set is constructed by combining the real behavior intention of a driver, a Bayesian network is constructed and trained by using samples in the data set, the network structure and probability distribution of the Bayesian network are determined, and the trained Bayesian network is obtained, so that the mapping relation between the features and the behavior intention of the user is established, and the one-to-one mapping relation between various features and single result judgment is realized;

the bayesian network is a complex causal relationship network, and can be regarded as a binary group B ═ G, P >, where G is a network structure, G ═ X, a > is a Directed Acyclic Graph (DAG), X represents nodes, a represents arcs between nodes, each node represents a variable, i.e., an event, and the arcs between variables represent direct causal relationships of the occurrence of the event; p is probability distribution, elements in P represent the conditional probability density of the node X, and the Bayesian network is successfully constructed by solving the network structure G and the probability distribution P;

the method comprises the steps that a network structure G is determined by adopting a K2 algorithm, an arc is continuously added into a network by the K2 algorithm according to a greedy search method to obtain a new network structure and score the new network structure, the network structure with the highest score is finally obtained and determined as the network structure of the Bayesian network which is required by us, heuristic search is carried out on the Bayesian network structures on the assumption that the Bayesian network structures have equal prior probability, father node sets of nodes are searched according to the sequence of node variables, the score of a local Bayesian network structure is improved by increasing the father nodes, the score of the finally obtained Bayesian network structure is maximized, D is a known data set, P (G, D) is the score of a certain network structure under the given data set, and N is X₁,X₂,…,X_nFor all nodes in the Bayesian network structure, and X_i∈{x_i1,x_i2,...,x_iri},r_iMore than or equal to 2, i ═ 1, 2., n, where X is_iIn common r_iA possible value x_i1,x_i2,...,x_iri，N_ijkFor variable X in data set D_iTake the kth value, whose parent set takes the number of combinations of the jth value, anAssuming that the prior probability distribution of each variable node is subject to uniform distribution, and C is a constant, the K2 scoring method is expressed as:

the probability distribution P is determined by adopting an EM algorithm, because the acquired data is not complete, the EM algorithm of non-complete data is selected, the EM algorithm is divided into an E step and an M step, the E step utilizes a formula (2) to calculate expected sufficient statistical factors of default data in a sample through a network structure and parameters, the M step utilizes a formula (3) to complete a default data set by utilizing the expected sufficient statistical factors, the optimal parameters of the current model are re-estimated, and finally the probability distribution of the Bayesian network is obtained, wherein Z is the currently known data,z is missing data, θ is probability distribution, corner mark i is ith data, Q_i(Z⁽ⁱ⁾) The weight corresponding to the ith sample;

Q_i(Z⁽ⁱ⁾)＝P(z⁽ⁱ⁾|Z⁽ⁱ⁾,θ) (2)

5. the Bayesian theorem-based vehicle multi-modal interaction method according to claim 1, wherein: in the step 4, E in the step 2 is₁To E₁₂Inputting a Bayesian network constructed based on the real data in the step 3 to obtain the posterior probability of the driver behavior, wherein the concrete formula is as follows:

wherein E ═ E (E)₁,E₂,…,E₁₂) If the behavior intention of the driver is c, P (E | c) is the probability that the user behavior E occurs when the behavior intention of the user is c, and P (c | E) is the probability that the behavior intention of the user is c when the behavior intention of the user occurs;

according to the maximum posterior probability decision logic, c with the maximum posterior probability P (c | E) is selected as a diagnosis decision result, namely, the behavior intention of the user is considered to be c when the multi-modal information E is collected.

6. The Bayesian theorem-based vehicle multi-modal interaction method according to claim 1, wherein: in the step 5, the user behavior intention after the Bayesian analysis is sent to a corresponding content service provider or a corresponding execution module, namely a feedback execution device; the feedback execution device performs appropriate feedback on the user according to the requirement of the user through various modes such as auditory sense, visual sense, touch sense and the like, wherein the auditory feedback comprises voice feedback and music feedback; visual feedback includes image feedback, video feedback, and the like; the tactile feedback includes vibration feedback and ultrasonic feedback.

Background

With the rapid development of the information era, human-computer interaction plays an increasingly important role in the life and work of people, the information entertainment function in automobiles is also greatly improved from the previous single radio function to the navigation and internet function, and then to the current intelligent auxiliary function which is mainly realized by voice interaction. The rapid development of the new technology enables the traveling experience of a driver to be improved continuously, the requirements of the driver to be improved continuously, the driver needs to strengthen the cognition on the automobile continuously, the related technology invention can be accurately used, and the learning cost is increased continuously.

In the field of automotive HMIs currently available, drivers typically interact with the automobile through key touch and voice systems. The point-touch type automobile touch screen interaction method has the advantages that the point-touch type automobile touch screen interaction method is very complicated in the process of interacting with an automobile, a driver is usually difficult to accurately achieve the purpose through the method in the driving process, various safety problems are easily caused in the traveling process, and the frequency of interaction use of touch clicking is very low. For the voice interaction system, various noises, such as wind noise, engine noise and the like, generally exist in the vehicle, and the noises can interfere with the voice recognition system in the vehicle, so that the voice recognition system in the vehicle cannot accurately judge the user intention, and the user use experience is reduced. Besides noise, various factors such as dialect and accent of the driver also affect the accuracy of speech recognition.

Patent publication of "multi-mode depth fusion airborne cabin man-machine interaction method": CN 109933272 a, proposes to determine its intention to complete the corresponding operation through touch control command or voice input to the pilot, but only based on passive acceptance of pilot command, and does not actively detect information such as fatigue of pilot, inattention, etc.

Disclosure of Invention

The invention provides a vehicle multi-mode interaction method based on Bayesian theorem, which is used for improving the effectiveness and reliability of an automobile interaction system.

The technical scheme adopted by the invention is that the method comprises the following steps:

step 1: comprehensively acquiring multi-mode behavior information by multiple sensors:

shooting the behavior of the driver through a special gesture detection camera to obtain gesture changes of the driver during different operations; shooting a picture of the face of a driver through a front camera to obtain expression information of the driver; capturing blinking actions through an eye tracker sensor, and judging the fatigue degree of a driver; obtaining voice information of a driver through a vehicle-mounted intelligent voice system, and obtaining a specific instruction of the driver;

step 2: feature extraction:

carrying out feature extraction on data of information collected by a sensor to form a bottom-layer expression of human body behaviors of a driver; starting from the characteristics, further performing digital representation on each information, determining the corresponding relation between the information and the Bayesian network nodes, and determining values;

and step 3: bayesian network building and training based on reality data:

acquiring multi-sensor acquired data of a vehicle running in a past practical scene, extracting features of the data, constructing a training data set by combining real behavior intentions of a driver, constructing and training a Bayesian network by using samples in the training data set, determining the network structure and probability distribution of the Bayesian network, and obtaining the trained Bayesian network, so that a mapping relation between the features and the behavior intentions of the user is established, and a one-to-one mapping relation between various features and single result judgment is realized;

and 4, step 4: inputting the obtained multi-modal characteristic information into a Bayesian network built based on real data to obtain the real intention of the user;

and 5: the feedback execution means performs appropriate feedback to the user in multiple modes such as visual, auditory, and tactile according to the inferred user intention.

In the step 1, the behavior of the driver is shot by a special gesture detection camera to obtain gesture changes of the driver during different operations; shooting a picture of the face of a driver through a front camera to obtain expression information of the driver; capturing blinking actions through an eye tracker sensor, and judging the fatigue degree of a driver; obtaining voice information of a driver through a vehicle-mounted intelligent voice system, and obtaining a specific instruction of the driver;

in the step 2, the special camera for gesture detection can process and obtain the skeleton model of the operator and the three-dimensional coordinate data of the key nodes in the skeleton model in real time through a human skeleton tracking technology, most people use the body language of the upper limbs when driving, the real intention of the user is conveyed through the upper half body movement, seven joint points of a left wrist joint point, a right wrist joint point, a left elbow joint point, a right elbow joint point, a left shoulder joint point, a right shoulder joint point and a head joint point are selected as key points to obtain three-dimensional coordinate data of the key points, then, coordinate data is detected, the moving range of each point is divided into 9 areas which are represented by 0 to 8, when the coordinates of a certain key point fall into a certain area, the coordinates are expressed by the number of the area, and the processed seven key point data are saved to the feature E.₁,E₂,E₃,E₄,E₅,E₆,E₇Wherein, the values of the two are all 0 to 8;

expression E of user facial picture shot by front-facing camera is recognized through trained deep convolutional neural network₈With different expressions in E₈Are indicated by different numbers.

The eye tracker sensor can obtain the characteristic PERCLOS, namely the percentage of the eye closing time in unit time, the characteristic BF, namely the blinking frequency, the characteristic MECD, namely the duration of the longest eye closing in a certain time through data analysis, is used for detecting the concentration and the fatigue degree, the PERCLOS characteristic, the BF characteristic and the characteristic MECD are respectively subjected to discretization processing, are divided into five grades, are respectively represented by numbers 0-4 and are stored in the characteristic E₉、E₁₀And E₁₁Performing the following steps;

vehicle-mounted intelligent voice system pass microphoneObtaining voice information by wind, converting the voice information into corresponding text description information through natural language identification, and extracting preset keyword information to E₁₂In E, different voice information is₁₂Are indicated by different numbers.

In the step 3, multi-sensor collected data of a vehicle running in a past practical situation are obtained, feature extraction is carried out on the data, a training data set is constructed by combining the real behavior intention of a driver, a Bayesian network is constructed and trained by using samples in the data set, the network structure and probability distribution of the Bayesian network are determined, and the trained Bayesian network is obtained, so that the mapping relation between the features and the behavior intention of the user is established, and the one-to-one mapping relation between various features and single result judgment is realized;

the bayesian network is a complex causal relationship network, and can be regarded as a binary group B ═ G, P >, where G is a network structure, G ═ X, a > is a Directed Acyclic Graph (DAG), X represents nodes, a represents arcs between nodes, each node represents a variable, i.e., an event, and the arcs between variables represent direct causal relationships of the occurrence of the event; p is the probability distribution, and the elements in P represent the conditional probability density of node X. Obtaining the network structure G and the probability distribution P to successfully construct a Bayesian network;

the method comprises the steps that a network structure G is determined by adopting a K2 algorithm, an arc is continuously added into a network by the K2 algorithm according to a greedy search method to obtain a new network structure and score the new network structure, the network structure with the highest score is finally obtained and determined as the network structure of the Bayesian network which is required by us, heuristic search is carried out on the Bayesian network structures on the assumption that the Bayesian network structures have equal prior probability, father node sets of nodes are searched according to the sequence of node variables, the score of a local Bayesian network structure is improved by increasing the father nodes, the score of the finally obtained Bayesian network structure is maximized, D is a known data set, P (G, D) is the score of a certain network structure under the given data set, and N is X₁,X₂,…,X_nFor all nodes in the Bayesian network structure, and X_i∈{x_i1,x_i2,...,x_iri},r_iMore than or equal to 2, i ═ 1, 2., n, where X is_iIn common r_iA possible value x_i1,x_i2,...,x_iri，N_ijkFor variable X in data set D_iTake the kth value, whose parent set takes the number of combinations of the jth value, anAssuming that the prior probability distribution of each variable node is subject to uniform distribution, and C is a constant, the K2 scoring method is expressed as:

the probability distribution P is determined by adopting an EM algorithm, because the acquired data is not complete, the EM algorithm of non-complete data is selected, the EM algorithm is divided into an E step and an M step, the E step utilizes a formula (2), an expected sufficient statistical factor of default data in a sample is calculated through a network structure and parameters, the M step utilizes a formula (3) to complete a default data set by utilizing the expected sufficient statistical factor, the optimal parameters of the current model are re-estimated, and finally the probability distribution of the Bayesian network is obtained, wherein Z is currently known data, Z is missing data, theta is probability distribution, an angle index i is ith data, Q is Q_i(Z⁽ⁱ⁾) The weight corresponding to the ith sample;

Q_i(Z⁽ⁱ⁾)＝P(z⁽ⁱ⁾|Z⁽ⁱ⁾,θ) (2)

in the step 4, E in the step 2 is₁To E₁₂Inputting a Bayesian network constructed based on reality data in the step 3, and obtaining the posterior probability of the driver behavior as shown in the attached figure 3, wherein the concrete formula is as follows:

wherein E ═ E (E)₁,E₂,…,E₁₂) If the behavior intention of the driver is c, P (E | c) is the probability that the user behavior E occurs when the behavior intention of the user is c, and P (c | E) is the probability that the behavior intention of the user is c when the behavior intention of the user occurs;

according to the maximum posterior probability decision logic, c with the maximum posterior probability P (c | E) is selected as the diagnosis decision result. Namely, the behavior intention of the user is considered to be c when the multi-modal information E is collected;

in the step 5, the user behavior intention after the Bayesian analysis is sent to a corresponding content service provider or a corresponding execution module, namely a feedback execution device; the feedback execution device performs appropriate feedback on the user according to the requirement of the user through various modes such as auditory sense, visual sense, touch sense and the like, wherein the auditory feedback comprises voice feedback and music feedback; visual feedback includes image feedback, video feedback, and the like; the tactile feedback includes vibration feedback and ultrasonic feedback.

The invention has the beneficial effects that: the collection of signal adopts the multisensor to gather, draws the conclusion through two kinds of information synthesis of sound and driver's action information, judges driver's state or intention, compares in prior art, and this kind of data acquisition mode reliability is higher, and the result that obtains after calculating through the Bayesian formula is closer with actual conditions, to feedback execution device, through covering the comparatively comprehensive feedback execution mode of visual sense of hearing sense of touch such as voice prompt, text prompt, vibrations, provides more accurate feedback for the user.

Drawings

FIG. 1 is a schematic diagram of the overall system of the present invention;

FIG. 2 is a multi-modal interaction diagram of the present invention;

FIG. 3 is a schematic diagram of a Bayesian network of the present invention.

Detailed Description

Specific embodiments of the present invention will be described below with reference to the accompanying drawings, which show exemplary embodiments of the present invention, but it will be understood by those skilled in the art that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of the present invention.

As shown in fig. 1; comprises the following steps:

step 1: comprehensively acquiring multi-mode behavior information by multiple sensors:

shooting the behavior of the driver through a special gesture detection camera to obtain gesture changes of the driver during different operations; shooting a picture of the face of a driver through a front camera to obtain expression information of the driver; capturing blinking actions through an eye tracker sensor, and judging the fatigue degree of a driver; obtaining voice information of a driver through a vehicle-mounted intelligent voice system, and obtaining a specific instruction of the driver;

step 2: feature extraction:

carrying out feature extraction on data of information collected by a sensor to form a bottom-layer expression of human body behaviors of a driver; starting from the characteristics, further performing digital representation on each information, determining the corresponding relation between the information and the Bayesian network nodes, and determining values;

and step 3: bayesian network building and training based on reality data:

acquiring multi-sensor acquired data of a vehicle running in a past practical scene, extracting features of the data, constructing a training data set by combining real behavior intentions of a driver, constructing and training a Bayesian network by using samples in the training data set, determining the network structure and probability distribution of the Bayesian network, and obtaining the trained Bayesian network, so that a mapping relation between the features and the behavior intentions of the user is established, and a one-to-one mapping relation between various features and single result judgment is realized;

and 4, step 4: inputting the obtained multi-modal characteristic information into a Bayesian network built based on real data to obtain the real intention of the user;

and 5: the feedback execution means performs appropriate feedback to the user in multiple modes such as visual, auditory, and tactile according to the inferred user intention.

In the step 1, the behavior of the driver is shot by a special gesture detection camera to obtain gesture changes of the driver during different operations; shooting a picture of the face of a driver through a front camera to obtain expression information of the driver; capturing blinking actions through an eye tracker sensor, and judging the fatigue degree of a driver; obtaining voice information of a driver through a vehicle-mounted intelligent voice system, and obtaining a specific instruction of the driver;

in the step 2, the special camera for gesture detection can process and obtain the skeleton model of the operator and the three-dimensional coordinate data of the key nodes in the skeleton model in real time through a human skeleton tracking technology, most people use the body language of the upper limbs when driving, the real intention of the user is conveyed through the upper half body movement, seven joint points of a left wrist joint point, a right wrist joint point, a left elbow joint point, a right elbow joint point, a left shoulder joint point, a right shoulder joint point and a head joint point are selected as key points to obtain three-dimensional coordinate data of the key points, then, coordinate data is detected, the moving range of each point is divided into 9 areas which are represented by 0 to 8, when the coordinates of a certain key point fall into a certain area, the coordinates are expressed by the number of the area, and the processed seven key point data are saved to the feature E.₁,E₂,E₃,E₄,E₅,E₆,E₇Wherein, the values of the two are all 0 to 8;

expression E of user facial picture shot by front-facing camera is recognized through trained deep convolutional neural network₈With different expressions in E₈Are indicated by different numbers.

The eye tracker sensor can obtain the characteristic PERCLOS, namely the percentage of the eye closing time in unit time, the characteristic BF, namely the blinking frequency, the characteristic MECD, namely the duration of the longest eye closing in a certain time through data analysis, is used for detecting the concentration and the fatigue degree, the PERCLOS characteristic, the BF characteristic and the characteristic MECD are respectively subjected to discretization processing, are divided into five grades, are respectively represented by numbers 0-4 and are stored in the characteristic E₉、E₁₀And E₁₁Performing the following steps;

vehicle-mounted intelligent voice system acquires voice through microphoneThe information is converted into corresponding text description information through natural language identification, and preset keyword information is extracted to E₁₂In E, different voice information is₁₂Are indicated by different numbers.

In the step 3, multi-sensor collected data of a vehicle running in a past practical situation are obtained, feature extraction is carried out on the data, a training data set is constructed by combining the real behavior intention of a driver, a Bayesian network is constructed and trained by using samples in the data set, the network structure and probability distribution of the Bayesian network are determined, and the trained Bayesian network is obtained, so that the mapping relation between the features and the behavior intention of the user is established, and the one-to-one mapping relation between various features and single result judgment is realized;

the bayesian network is a complex causal relationship network, and can be regarded as a binary group B ═ G, P >, where G is a network structure, G ═ X, a > is a Directed Acyclic Graph (DAG), X represents nodes, a represents arcs between nodes, each node represents a variable, i.e., an event, and the arcs between variables represent direct causal relationships of the occurrence of the event; p is the probability distribution, and the elements in P represent the conditional probability density of node X. Obtaining the network structure G and the probability distribution P to successfully construct a Bayesian network;

the method comprises the steps that a network structure G is determined by adopting a K2 algorithm, an arc is continuously added into a network by the K2 algorithm according to a greedy search method to obtain a new network structure and score the new network structure, the network structure with the highest score is finally obtained and determined as the network structure of the Bayesian network which is required by us, heuristic search is carried out on the Bayesian network structures on the assumption that the Bayesian network structures have equal prior probability, father node sets of nodes are searched according to the sequence of node variables, the score of a local Bayesian network structure is improved by increasing the father nodes, the score of the finally obtained Bayesian network structure is maximized, D is a known data set, P (G, D) is the score of a certain network structure under the given data set, and N is X₁,X₂,…,X_nFor all nodes in the Bayesian network structure, and X_i∈{x_i1,x_i2,...,x_iri},r_iMore than or equal to 2, i ═ 1, 2., n, where X is_iIn common r_iA possible value x_i1,x_i2,...,x_iri，N_ijkFor variable X in data set D_iTake the kth value, whose parent set takes the number of combinations of the jth value, anAssuming that the prior probability distribution of each variable node is subject to uniform distribution, and C is a constant, the K2 scoring method is expressed as:

the probability distribution P is determined by adopting an EM algorithm, because the acquired data is not complete, the EM algorithm of non-complete data is selected, the EM algorithm is divided into an E step and an M step, the E step utilizes a formula (2), an expected sufficient statistical factor of default data in a sample is calculated through a network structure and parameters, the M step utilizes a formula (3) to complete a default data set by utilizing the expected sufficient statistical factor, the optimal parameters of the current model are re-estimated, and finally the probability distribution of the Bayesian network is obtained, wherein Z is currently known data, Z is missing data, theta is probability distribution, an angle index i is ith data, Q is Q_i(Z⁽ⁱ⁾) The weight corresponding to the ith sample;

Q_i(Z⁽ⁱ⁾)＝P(z⁽ⁱ⁾|Z⁽ⁱ⁾,θ) (2)

in the step 4, E in the step 2 is₁To E₁₂Inputting a Bayesian network constructed based on reality data in the step 3, and obtaining the posterior probability of the driver behavior as shown in the attached figure 3, wherein the concrete formula is as follows:

wherein E ═ E (E)₁,E₂,…,E₁₂) If the behavior intention of the driver is c, P (E | c) is the probability that the user behavior E occurs when the behavior intention of the user is c, and P (c | E) is the probability that the behavior intention of the user is c when the behavior intention of the user occurs;

according to the maximum posterior probability decision logic, c with the maximum posterior probability P (c | E) is selected as the diagnosis decision result. Namely, the behavior intention of the user is considered to be c when the multi-modal information E is collected;

in the step 5, the user behavior intention after the Bayesian analysis is sent to a corresponding content service provider or a corresponding execution module, namely a feedback execution device; the feedback execution device performs appropriate feedback on the user according to the requirement of the user through various modes such as auditory sense, visual sense, touch sense and the like, wherein the auditory feedback comprises voice feedback and music feedback; visual feedback includes image feedback, video feedback, and the like; the tactile feedback comprises vibration feedback and ultrasonic feedback, and is specifically divided into three categories, namely fuzzy decision, active feedback and danger prompt;

and when the fuzzy decision is made, namely the user puts forward a fuzzy demand, the system judges the demand of the user to make further decision and inquires the user. If the user says 'put a song of Zhou Jie Lun', the system obtains the instruction and considers the user to be in the drowsy state according to the collected data, and identifies E according to the expression in the Bayesian network behavior intention data set₈For the selected music at happy time, a selection is made and asked: "can come a cheerful cowboy? "

The active feedback is to actively make judgment and inquire the user according to the collected user information. When the special camera for gesture detection detects that the user acts as a hand quickly waving, the intention of the user is obtained according to Bayes inference: "the user is very hot, needs the cooling", initiatively sends the suggestion: "is the air conditioner required to be turned on? ", select whether to turn on the air conditioner according to the user's answer.

And when the dangerous prompt detects that the user is over-tired or has dangerous behaviors, the user is in a dangerous state according to Bayesian inference, and the user is reminded through multi-modal behaviors such as seat vibration, screen flicker, voice alarm and the like. For example, when the user is in the state of being tired excessively, the eye tracker detects that the user is in the state of closing eyes for a long time, deduces that the user is probably in the state of being tired excessively at this moment according to bayes, and sends out a prompt through vibration of a steering wheel and screen flickering at this moment: "you are in the drowsy state at this moment, need to stop the car to the safe area immediately and take a rest" to remind the user to drive safely.