1. A kitchen oil smoke detection method based on image processing is characterized by comprising the following steps;

step (ii) of: acquiring basic information, including personnel information perception and oil smoke concentration information perception, acquiring personnel information and oil smoke concentration information, and generating a reward matrix subsequently;

step (ii) of: generating a reward matrix based on the oil smoke concentration and the personnel action, and acquiring a reference personnel action sequence;

the specific steps of obtaining the reward matrix are as follows:

1) generating an initial reward matrix by taking the oil smoke concentration as row information and taking personnel actions as column information, wherein element values in the initial reward matrix are all 0;

2) setting a group of oil smoke concentration and personnel action can only correspond to a reward value, namely, under a certain oil smoke concentration, taking a certain action reward value, the firstOil smoke concentration of each intervalIs shown as followsIndividual category of person acts toShow, then awardThe excitation value can be expressed as，Is shown inAndas a mapping function of the parameters, the reward value and the variation value of the oil smoke concentrationIn the negative correlation, no matter the concentration of the oil smoke is any value, the reward value corresponding to the action is all the valueWherein, in the step (A),the oil smoke concentration is shown inThe reward value for taking a pouring action at each interval,the oil smoke concentration is shown inThe reward value of the stir-frying action is taken in each interval,the oil smoke concentration is shown inReward value for taking wait action at intervals;

3) constructing a reward matrix generation network, wherein the network input is under discrete time sequenceA sequence;the sequence is sent into a feature extraction encoder to obtain a feature tensor, the feature tensor is sent into a reward matrix generation decoder to output a reward matrix, namely the network output is the reward matrix, and the size of the reward matrix is consistent with that of the initial reward matrix;

step (ii) of: determining a frame difference interval, wherein the frame difference interval is used for self-adaptively acquiring the frame difference interval so as to improve the accuracy of oil smoke detection;

step (ii) of: and oil smoke detection for sensing the oil smoke concentration and further controlling an air port of the range hood.

2. The kitchen oil smoke detection method based on image processing according to claim 1, characterized by the steps ofThe information perception of the middle personnel is specifically as follows: acquiring human body key points, kitchen ware key points and container key points through a key point detection network, wherein the input of the key point network is a single-frame image, the output of the key point network is a key point thermodynamic diagram, and each frame image in a video sequence with a known timestamp is taken as the input to acquire a corresponding key point thermodynamic diagram; the oil smoke concentration information perception specifically comprises the following steps: acquiring the oil smoke concentration at each moment by an oil smoke concentration detection sensor, wherein the oil smoke concentration at each moment takes a timestamp for acquiring information asAnd time identification, wherein the personnel information and the oil smoke concentration information are in one-to-one correspondence through timestamps.

3. The method for detecting kitchen oil smoke based on image processing as claimed in claim 2, wherein said human body key points comprise 8 key point categories of head, hand, elbow, shoulder and root nodes, said kitchen ware key points comprise pan, shovel, spoon and chopsticks, and said container key points comprise bowl, bottle and can.

4. The kitchen oil smoke detection method based on image processing as claimed in claim 3, wherein the reward matrix generation network training process is specifically as follows: multiple sets of video sequences based on different cooking contents in different scenes are acquiredAs a training data set, the loss function of the network is(ii) a Wherein the content of the first and second substances,in order to account for motion losses, specifically,,the loss is used to ensure that the size of the reward value corresponding to the action is satisfiedWhen the objective of using the exponential function is to make the variable less than 0,smaller, a number less than 1, and when the variable is greater than 0,the items are large and are numbers larger than 1, and in order to enlarge the influence relation, a scaling coefficient is added，Is a large integer;in order to vary the amount of loss, specifically,i.e. when the value of the concentration becomes large, the current prize value should be greater than the prize value at a future moment, and vice versa, usually by using a scaling factor Q and an exponential function to ensure thatIs a number less than 0.

5. The kitchen oil smoke detection method based on image processing according to claim 4, characterized in that the stepsThe obtaining of the middle frame difference interval specifically comprises the following steps:

1) according to the acquired reward matrix, the optimal action combination is selected based on the initial state, the influence of the characteristics of the action of a person is considered at the moment, namely for the cooking process, when the cooking process is not finished, the cooking quality is influenced if the person is in a waiting state for a long time, and the cooking process is taken as an example, the cooking mode such as cooking is defaulted, and no large oil smoke is generated;

2) the optimal action combination selection strategy is as follows: obtaining duration of cooking processCurrent action categoryAnd duration of current actionObtaining the occurrence probability of each action under the current condition through statisticsRespectively multiplying the probability value with the elements of the reward matrix based on the action category to obtain a modified reward matrix;

3) based on the modified reward matrix, the current optimal reward selection is carried out based on the forgetting idea, namelyIn the formula (I), wherein,to select action categoriesIs evaluated in the final reward(s) of (c),to select action categoriesThe current time of day prize value of (c),to select action categoriesFuture reward value of;a forgetting coefficient for determining the degree of reward after consideration is set to 0.8 in the present invention; selectingThe corresponding action category is taken as the currently selected action category;

4) updating the duration of the cooking process according to the selected action categoryCurrent action categoryAnd duration of current actionRepeating the stepsUp to，An expected cooking time, an empirical value; thus, obtaining an optimal action sequence;

5) performing deviation analysis according to the actual action sequence and the reference action sequence, performing frame-by-frame comparison, mapping the difference by a modified Sigmoid function when the difference exists, and rounding the mapping value downwards to obtain an inter-frame interval correction value, wherein the modified Sigmoid function specifically comprises the following steps:。

Background

At present, in the prior art, oil smoke detection is usually directly detected by a frame difference method, for example, chinese patent No. CN108760590B discloses a kitchen oil smoke concentration detection and interference elimination method based on image processing, which directly performs frame difference operation processing with fixed frames to further obtain oil smoke motion information, and by adopting the method, if the selected frame number is small, oil smoke change may be not obvious, and oil smoke change condition is not easily distinguished; if the number of the selected frames is large, the oil smoke change degree is possibly too large, so that the detail information of the change process is lost, the reference oil smoke concentration change information is not convenient to obtain, and the accuracy of oil smoke detection is low.

Disclosure of Invention

Aiming at the problems, the invention provides a kitchen oil smoke detection method based on image processing, which comprises the following steps:

step (ii) of: acquiring basic information, including personnel information perception and oil smoke concentration information perception, acquiring personnel information and oil smoke concentration information, and generating a reward matrix subsequently;

step (ii) of: generating a reward matrix based on the oil smoke concentration and the personnel action, and acquiring a reference personnel action sequence;

step (ii) of: determining a frame difference interval, wherein the frame difference interval is used for self-adaptively acquiring the frame difference interval so as to improve the accuracy of oil smoke detection;

step (ii) of: and oil smoke detection for sensing the oil smoke concentration and further controlling an air port of the range hood.

Has the advantages that:

(1) compared with the prior art, the reward matrix obtaining step has the advantages that the reference action under the current concentration is determined through the reward value, and the change information of the reference oil smoke concentration is convenient to obtain;

(2) based on the frame difference interval determining step, compared with the prior art, the method has the advantages that deviation amount is obtained according to comparison of actual action sequence information and action sequence information, and further frame difference interval is obtained in a self-adaptive mode, so that accuracy of oil smoke detection based on a frame difference method is improved.

Detailed Description

In order to make the present invention more comprehensible to those skilled in the art, the present invention is described below with reference to examples and the accompanying drawings.

In order to realize the content, the invention designs a kitchen oil smoke detection method based on image processing, which comprises the following steps:

step (ii) ofAcquiring basic information, wherein the purpose of the step is as follows: the method has the advantages that personnel information and oil smoke concentration information are acquired, the method is used for generating a reward matrix in the follow-up process, and the prior knowledge is as follows: the key point detects the network.

The input is as follows: the video sequence and the sensor detection value sequence are used for sensing personnel information and oil smoke concentration information, and the output is as follows: personnel information and oil smoke concentration information.

Wherein the personnel information perception specifically comprises: acquiring human body key points, kitchen ware key points and container key points through a key point detection network, wherein the human body key points comprise 8 key point categories including head, hand, elbow, shoulder and root nodes; the network outputs a thermodynamic diagram with ten channels; the kitchen ware comprises a slice, a spoon, chopsticks and the like; the container comprises a bowl, a pot and the like; the method comprises the steps that a single frame image is input into a key point network, the single frame image is output into a key point thermodynamic diagram, and the corresponding key point thermodynamic diagram is obtained by taking each frame image in a video sequence with a known timestamp as input;

it should be noted that, in the invention, the default position of the cooking utensil is fixed, and the position of the center point of the cooking utensil is easy to be positioned in the image because the camera of the invention is fixedThe operation is marked by an implementer according to the kitchen environment information; setting a first radius with the center point of the cooking utensil as the center of a circleAnd a second radiusA first circular area is generated by using the circle center and the first radius as a first interested area, a second circular area is generated by using the circle center and the second radius, an area which does not belong to the first circular area in the second circular area is a second interested area, and it should be noted that the second radius isGreater than the first radius(ii) a When the hand key point in a certain frame of image is located in the second region of interest and the kitchen ware key point is located in the first region of interest, judging that the frame is a stir-frying action, wherein the action type identifier is 2; when the hand key point in a certain frame of image is located in the second region of interest and the container key point is located in the first region of interest, judging that the frame is a dumping action, and identifying the action type as 3; otherwise, judging that the waiting action type identifier is 1; each frame of action information is personnel information;

wherein, the oil smoke concentration information perception specifically is: acquiring the oil smoke concentration at each moment through an oil smoke concentration detection sensor, wherein the oil smoke concentration at each moment takes a timestamp of acquired information as a time identifier; the personnel information and the oil smoke concentration information are in one-to-one correspondence through timestamps;

step (ii) ofAnd obtaining a reward matrix: the purpose of this step is: the method has the advantages that the reward matrix with concentration and action as row information and column information can be obtained, the reference oil smoke concentration change information can be obtained based on the reward information, and the action and action of personnel are consideredThe influence of the occurrence time on the change of the oil smoke concentration is obtained, and more accurate reference oil smoke concentration change information is obtained;

the input is as follows: personnel information and oil smoke concentration information carry out reward matrix and acquire the processing, and the output is: a reward matrix.

The reward matrix acquisition specifically comprises:

1) the oil smoke concentration is used as row information, the personnel action is used as column information to generate an initial reward matrix, element values in the initial reward matrix are all 0, for example, the oil smoke concentration is divided into ten intervals, in order to refine the reward matrix and obtain a better analysis effect, an implementer can set the number of the intervals to be a larger value, the personnel action is obtained into three categories based on the personnel information, and the size of the matrix is 10 x 3;

2) under the condition of not considering the characteristic influence of the personnel action, one group of oil smoke concentration and the personnel action can only correspond to one reward value, namely, under a certain oil smoke concentration, the reward value of taking a certain action is the first reward valueOil smoke concentration of each intervalIs shown as followsIndividual category of person acts toIndicating that the prize value may be expressed as，Is shown inAndas a mapping function of the parameters, the reward value and the variation value of the oil smoke concentrationIn a negative correlation relationship, it is obvious that no matter the concentration of the oil smoke is any value, the reward value corresponding to the action should be the same as the valueWherein, in the step (A),the oil smoke concentration is shown inThe reward value for taking a pouring action at each interval,the oil smoke concentration is shown inThe reward value of the stir-frying action is taken in each interval,the oil smoke concentration is shown inReward value for taking wait action at intervals;

3) constructing a reward matrix generation network, wherein the network input is under discrete time sequenceA sequence;the sequence is sent to a feature extraction encoder to obtain a feature tensor, the feature tensor is sent to a reward matrix generation decoder to output a reward matrix, namely, the network output is the reward matrixThe incentive matrix is the same size as the initial incentive matrix.

The network training process for generating the reward matrix specifically comprises the following steps: multiple sets of video sequences based on different cooking contents in different scenes are acquiredAs a training data set, the training of the network does not need to be marked artificially, and the loss function of the network is(ii) a Wherein the content of the first and second substances,in order to account for motion losses, specifically,,the loss is used to ensure that the size of the reward value corresponding to the action is satisfiedWhen the objective of using the exponential function is to make the variable less than 0,the term is small, a number less than 1, and when the variable is greater than 0,the items are large and are numbers larger than 1, and in order to enlarge the influence relation, a scaling coefficient is added，Is a large integer;in order to vary the amount of loss, specifically,i.e. when the value of the concentration becomes large, the current prize value should be greater than the prize value at a future moment, and vice versa, usually by using a scaling factor Q and an exponential function to ensure thatIs a number less than 0.

Step (ii) ofFrame difference interval determination, the purpose of this step is: the method has the advantages that the frame difference interval of the frame difference method is determined in a self-adaptive mode, and the accuracy rate of oil smoke detection through the frame difference method is improved.

The input is as follows: and (3) rewarding the matrix, performing frame difference interval acquisition processing, and outputting: a frame difference interval.

The frame difference interval acquisition specifically comprises:

1) according to the acquired reward matrix, the optimal action combination is selected based on the initial state, the influence of the characteristics of the action of a person is considered at the moment, namely for the cooking process, when the cooking process is not finished, the long-time waiting state can influence the cooking quality, and the cooking method such as cooking and the like can not generate large oil smoke by default by taking the dish frying process as an example;

2) the optimal action combination selection strategy is as follows: obtaining duration of cooking processCurrent action categoryAnd duration of current actionObtaining the occurrence probability of each action under the current condition through statisticsRespectively multiplying the probability value with the elements of the reward matrix based on the action category to obtain a modified reward matrix;

3) based on the modified reward matrix, the current optimal reward selection is carried out based on the forgetting idea, namelyIn the formula (I), wherein,to select action categoriesIs evaluated in the final reward(s) of (c),to select action categoriesThe current time of day prize value of (c),to select action categoriesFuture reward value of;a forgetting coefficient for determining the degree of reward after consideration is set to 0.8 in the present invention; selectingThe corresponding action category is taken as the currently selected action category;

4) updating the duration of the cooking process according to the selected action categoryCurrent action categoryAnd duration of current actionRepeating the stepsUp to，An expected cooking time, an empirical value; thus, obtaining an optimal action sequence;

5) performing deviation analysis according to the actual action sequence and a reference action sequence, specifically, performing frame-by-frame comparison, and when there is a difference, mapping the difference (i.e. the action category identification difference value, the range is-2, -1, 1, 2) by using a modified Sigmoid function, wherein the mapping value is an inter-frame interval correction value after being rounded down, and the modified Sigmoid function specifically is as follows:the initial inter-frame interval is set by an implementer, and is set to 4 frames in the application;

6) therefore, self-adaptive inter-frame interval acquisition is realized, and the accuracy rate of detecting oil smoke by a subsequent frame difference method is improved.

Step (ii) ofOil smoke detection, the purpose of this step: the oil smoke concentration detection can bring the advantages of accurately sensing the oil smoke concentration and controlling the ventilation opening according to the oil smoke concentration;

the input is as follows: and frame difference interval, adopting a frame difference method to process according to the frame difference interval, and outputting: the concentration of oil smoke.

The frame difference method for detecting the oil smoke is the prior art, and is not described herein again, and an implementer can select the existing implementation method, and the scheme provided by the application aims to provide a self-adaptive frame difference interval, and then select video frames at the frame difference interval to detect the oil smoke concentration by the frame difference method, so as to improve the accuracy of the oil smoke concentration.