1. A holographic-based multidimensional tongue image analysis method is characterized by comprising the following steps:

acquiring a tongue image to be analyzed of a target user, wherein the tongue image to be analyzed comprises at least one of a tongue surface image and a tongue bottom image;

acquiring at least one tongue area image to be analyzed of the tongue of the target user according to the tongue image to be analyzed;

inputting the tongue area image to be analyzed into a pre-trained tongue characteristic neural network to obtain a tongue characteristic vector of the tongue area image to be analyzed;

acquiring an analyzed tongue region image in a preset analysis database, calculating Euclidean distance between the tongue region image to be analyzed and the analyzed tongue region image according to the tongue feature vector, and acquiring tongue features according to an analyzed result of the analyzed tongue region image when the Euclidean distance is smaller than a preset distance threshold;

and acquiring a tongue analysis result of the target user according to the tongue characteristic.

2. The holographic-based multidimensional tongue image analysis method of claim 1, wherein the tongue image to be analyzed further comprises at least one of a lingual image, an image of an upper jaw, an image of a lower jaw, an image of a gum, and an image of a tooth.

3. The holographic-based multidimensional tongue image analysis method of claim 2, wherein the step of obtaining the tongue analysis result of the target user according to the tongue characteristics is preceded by the steps of:

and sending a photo supplement prompt to the target user so that the target user supplements other tongue images to be analyzed according to the photo supplement prompt.

4. The holographic-based multidimensional tongue image analysis method of claim 1, wherein the step of obtaining the tongue analysis result of the target user according to the tongue characteristics comprises:

and acquiring the self-describing information of the target user, and acquiring the tongue analysis result by combining the self-describing information.

5. The holographic-based multidimensional tongue image analysis method of claim 1, wherein the step of obtaining at least one tongue area image to be analyzed of a tongue of a target user from the tongue image to be analyzed comprises:

and carrying out image segmentation on the tongue image to be analyzed according to the corresponding reflection relationship between each area of the tongue and the internal organs to obtain at least one tongue area image to be analyzed.

6. The holographic-based multi-dimensional tongue image analysis method of claim 1,

when the tongue image to be analyzed includes a tongue surface image, the tongue characteristics include: irregular texture, tongue surface change, tongue speckle and deformation;

when the tongue image to be analyzed includes a tongue fundus image, the tongue features include: vein and vein characteristics of the bottom of the tongue, color of the bottom of the tongue, morphology of the bottom of the tongue, unevenness, lumps, and polyps.

7. The holographic-based multidimensional tongue image analysis method of claim 1, wherein the step of inputting the tongue region image to be analyzed into a pre-trained tongue feature neural network is preceded by:

constructing a tongue characteristic neural network, wherein the tongue characteristic neural network is a deep convolutional neural network and comprises a global pooling layer, a discarding layer, an expanding layer, a smoothing layer, a Dense layer and a Softmax layer;

defining a tongue characteristic loss function to reduce Euclidean distances between similar images and increase Euclidean distances between non-similar images;

and inputting the tongue characteristic training sample image into the tongue characteristic neural network for training to obtain the pre-training tongue characteristic neural network.

8. The holographic-based multidimensional tongue image analysis method of claim 1, wherein the step of obtaining at least one tongue area image to be analyzed of a tongue of a target user from the tongue image to be analyzed comprises:

and inputting the tongue image to be analyzed into a pre-training tongue segmentation neural network to obtain the at least one tongue image to be analyzed.

9. The holographic-based multidimensional tongue image analysis method of claim 8, wherein said step of inputting said tongue image to be analyzed into a pre-trained tongue segmentation neural network is preceded by the steps of:

preparing a tongue segmentation training image, and segmenting and labeling the tongue segmentation training image according to a preset standard, wherein the preset standard comprises viscera organs corresponding to a tongue region;

generating a single-channel area gray image according to the marked tongue segmentation training image, and acquiring a tongue segmentation result image according to the single-channel area gray image;

and acquiring tongue segmentation training data according to each tongue segmentation training image and the corresponding tongue segmentation result image, inputting the tongue segmentation training data into a tongue segmentation neural network, and acquiring the pre-training tongue segmentation neural network.

10. The holographic-based multidimensional tongue image analysis method of claim 1, wherein the step of obtaining the tongue analysis result of the target user according to the tongue characteristics comprises: acquiring viscera organs corresponding to the tongue area of the tongue area image, acquiring the conditions of the viscera organs according to the tongue characteristics, and acquiring the tongue analysis result according to the conditions of the viscera organs.

11. The holographic-based multidimensional tongue image analysis method of claim 1, wherein the step of calculating the euclidean distance between the tongue region image to be analyzed and the analyzed tongue region image based on the tongue feature vector comprises:

and calculating Euclidean distances between the tongue characteristic vectors of the preset quantity dimension of the tongue region image to be analyzed and the analyzed vectors of the preset quantity dimension of the analyzed image.

12. The holographic-based multidimensional tongue image analysis method of claim 11, wherein said step of calculating euclidean distances between a predetermined number of said tongue region images to be analyzed and a predetermined number of analyzed vectors of said analyzed images comprises:

calculating the euclidean distance according to the following formula:

wherein x is_iIs the i-th dimension tongue feature vector, y_iAnalyzed vectors for the ith dimension.

13. The holographic-based multidimensional tongue image analysis method of claim 1, wherein the step of obtaining a tongue image to be analyzed of a target user comprises:

and acquiring a self-timer tongue image provided by a target user, and inputting the self-timer tongue image into a pre-trained tongue extraction neural network to acquire the tongue image to be analyzed.

14. The holographic-based multidimensional tongue image analysis method of claim 1, wherein the step of obtaining the tongue analysis result of the target user according to the tongue characteristics comprises:

and acquiring a preliminary analysis result according to the target tongue characteristic, acquiring an auxiliary analysis result according to the auxiliary tongue characteristic, and synthesizing the preliminary analysis result and the auxiliary analysis result to acquire the tongue analysis result.

15. The holographic-based multi-dimensional tongue image analysis method of claim 1, wherein the preset distance threshold is 0.5.

16. A holographic-based multidimensional tongue image analysis system, comprising:

the analysis device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a tongue image to be analyzed of a target user, and the tongue image to be analyzed comprises at least one of a tongue surface image and a tongue bottom image;

the image module is used for acquiring at least one tongue area image to be analyzed of the tongue of the target user according to the tongue image to be analyzed;

the network module is used for inputting the tongue area image to be analyzed into a pre-trained tongue characteristic neural network and acquiring a tongue characteristic vector of the tongue area image to be analyzed;

the characteristic module is used for acquiring an analyzed tongue region image in a preset analysis database, calculating Euclidean distance between the tongue region image to be analyzed and the analyzed tongue region image according to the tongue characteristic vector, and acquiring tongue characteristics according to an analyzed result of the analyzed tongue region image when the Euclidean distance is smaller than a preset distance threshold;

and the result module is used for acquiring a tongue analysis result of the target user according to the tongue characteristic.

17. A storage medium, characterized in that a computer program is stored which, when being executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 15.

18. A computer arrangement comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 15.

Background

At present, all tongue diagnosis technologies intelligently replace manual differentiation of symptoms by traditional tongue diagnosis, most of the tongue diagnosis technologies are limited to eight-principle differentiation of symptoms by looking at the tongue surface, and specific diseases cannot be directly screened out by determining symptoms such as yin and yang exterior cold, interior heat, deficiency and excess and the like.

In addition, there are some external factors when observing the tongue coating, tongue proper and tongue status on the tongue surface by traditional tongue diagnosis. If colored food and drink are eaten, the tongue coating is colored, and the colored tongue coating is analyzed, so that the analysis result can be influenced. If the tongue surface is injured, it is considered that the zang-fu organs are diseased, and there is a possibility that the analysis results will be affected. Every time the tongue is extended out of the tongue surface for shooting, the tongue has different shape results; for example, if the patient is stretched obliquely, the patient will be misdiagnosed as stroke if the patient is not suffering from stroke; the same tongue has different extending force and angle, so that a plurality of forms such as a fat tongue, a long and thick tongue or a curled tongue can appear, and the possibility of influencing the analysis result exists.

Disclosure of Invention

In view of the above, it is necessary to provide a holographic-based multidimensional tongue image analysis method, system, device and storage medium.

A holographic-based multidimensional tongue image analysis method comprises the following steps: acquiring a tongue image to be analyzed of a target user, wherein the tongue image to be analyzed comprises at least one of a tongue surface image and a tongue bottom image; acquiring at least one tongue area image to be analyzed of the tongue of the target user according to the tongue image to be analyzed; inputting the tongue area image to be analyzed into a pre-trained tongue characteristic neural network to obtain a tongue characteristic vector of the tongue area image to be analyzed; acquiring an analyzed tongue region image in a preset analysis database, calculating Euclidean distance between the tongue region image to be analyzed and the analyzed tongue region image according to the tongue feature vector, and acquiring tongue features according to an analyzed result of the analyzed tongue region image when the Euclidean distance is smaller than a preset distance threshold; and acquiring a tongue analysis result of the target user according to the tongue characteristic.

Wherein the tongue image to be analyzed further comprises at least one of a tongue both-side image, an upper jaw image, a lower jaw image, a gum image, and a tooth image.

Before the step of obtaining the tongue analysis result of the target user according to the tongue feature, the method further includes: and sending a photo supplement prompt to the target user so that the target user supplements other tongue images to be analyzed according to the photo supplement prompt.

Wherein, the step of obtaining the tongue analysis result of the target user according to the tongue characteristics comprises: and acquiring the self-describing information of the target user, and acquiring the tongue analysis result by combining the self-describing information.

Wherein, the step of obtaining at least one tongue area image to be analyzed of the tongue of the target user according to the tongue image to be analyzed includes: and carrying out image segmentation on the tongue image to be analyzed according to the corresponding reflection relationship between each area of the tongue and the internal organs to obtain at least one tongue area image to be analyzed.

Wherein, when the tongue image to be analyzed includes a tongue surface image, the tongue characteristics include: irregular texture, tongue surface change, tongue speckle and deformation; when the tongue image to be analyzed includes a tongue fundus image, the tongue features include: vein and vein characteristics of the bottom of the tongue, color of the bottom of the tongue, morphology of the bottom of the tongue, unevenness, lumps, and polyps.

Wherein, before the step of inputting the tongue region image to be analyzed into the pre-trained tongue feature neural network, the method comprises the following steps: constructing a tongue characteristic neural network, wherein the tongue characteristic neural network is a deep convolutional neural network and comprises a global pooling layer, a discarding layer, an expanding layer, a smoothing layer, a Dense layer and a Softmax layer; defining a tongue characteristic loss function to reduce Euclidean distances between similar images and increase Euclidean distances between non-similar images; and inputting the tongue characteristic training sample image into the tongue characteristic neural network for training to obtain the pre-training tongue characteristic neural network.

Wherein, the step of obtaining at least one tongue area image to be analyzed of the tongue of the target user according to the tongue image to be analyzed includes: and inputting the tongue image to be analyzed into a pre-training tongue segmentation neural network to obtain the at least one tongue image to be analyzed.

Wherein, before the step of inputting the tongue image to be analyzed into the pre-training tongue segmentation neural network, the method comprises the following steps: preparing a tongue segmentation training image, and segmenting and labeling the tongue segmentation training image according to a preset standard, wherein the preset standard comprises viscera organs corresponding to a tongue region; generating a single-channel area gray image according to the marked tongue segmentation training image, and acquiring a tongue segmentation result image according to the single-channel area gray image; and acquiring tongue segmentation training data according to each tongue segmentation training image and the corresponding tongue segmentation result image, inputting the tongue segmentation training data into a tongue segmentation neural network, and acquiring the pre-training tongue segmentation neural network.

Wherein, the step of obtaining the tongue analysis result of the target user according to the tongue characteristics comprises: acquiring viscera organs corresponding to the tongue area of the tongue area image, acquiring the conditions of the viscera organs according to the tongue characteristics, and acquiring the tongue analysis result according to the conditions of the viscera organs.

Wherein, the step of calculating the euclidean distance between the tongue region image to be analyzed and the analyzed tongue region image according to the tongue feature vector includes: and calculating Euclidean distances between the tongue characteristic vectors of the preset quantity dimension of the tongue region image to be analyzed and the analyzed vectors of the preset quantity dimension of the analyzed image.

Wherein, the step of calculating the euclidean distance between the tongue feature vector of the preset number dimension of the tongue region image to be analyzed and the analyzed vector of the preset number dimension of the analyzed image comprises: calculating the euclidean distance according to the following formula:

wherein x is_iIs the i-th dimension tongue feature vector, y_iAnalyzed vectors for the ith dimension.

Wherein, the step of obtaining the tongue image to be analyzed of the target user comprises: and acquiring a self-timer tongue image provided by a target user, and inputting the self-timer tongue image into a pre-trained tongue extraction neural network to acquire the tongue image to be analyzed.

Wherein, the step of obtaining the tongue analysis result of the target user according to the tongue characteristics comprises: and acquiring a preliminary analysis result according to the target tongue characteristic, acquiring an auxiliary analysis result according to the auxiliary tongue characteristic, and synthesizing the preliminary analysis result and the auxiliary analysis result to acquire the tongue analysis result.

Wherein the preset distance threshold is 0.5.

A holographic-based multi-dimensional tongue image analysis system, comprising: the analysis device comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a tongue image to be analyzed of a target user, and the tongue image to be analyzed comprises at least one of a tongue surface image and a tongue bottom image; the image module is used for acquiring at least one tongue area image to be analyzed of the tongue of the target user according to the tongue image to be analyzed; the network module is used for inputting the tongue area image to be analyzed into a pre-trained tongue characteristic neural network and acquiring a tongue characteristic vector of the tongue area image to be analyzed; the characteristic module is used for acquiring an analyzed tongue region image in a preset analysis database, calculating Euclidean distance between the tongue region image to be analyzed and the analyzed tongue region image according to the tongue characteristic vector, and acquiring tongue characteristics according to an analyzed result of the analyzed tongue region image when the Euclidean distance is smaller than a preset distance threshold; and the result module is used for acquiring a tongue analysis result of the target user according to the tongue characteristic.

A storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of the method as described above.

A computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of the method as described above.

The embodiment of the invention has the following beneficial effects:

acquiring a tongue image to be analyzed comprising at least one of a tongue surface image and a tongue bottom image, and acquiring at least one tongue area image to be analyzed of a target user tongue according to the tongue image to be analyzed; performing image analysis on each tongue area image to be analyzed to obtain tongue characteristics corresponding to each tongue area image to be analyzed; the tongue analysis result of the target user is obtained according to the tongue characteristics, the problem that the tongue analysis is easily affected by external adverse effects when the tongue bottom image is used for analysis only according to the tongue surface image can be effectively solved, the target user can analyze the tongue of the target user only by providing the image comprising the tongue, and the method is simple, convenient and reliable.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Wherein:

FIG. 1 is a schematic flow chart of a holographic-based multidimensional tongue image analysis method according to a first embodiment of the present invention;

FIG. 2 is a schematic diagram of a labeled tongue extraction training image provided by the present invention;

FIG. 3 is a schematic diagram of a single-channel tongue grayscale image provided by the present invention;

FIG. 4 is a schematic view of a first embodiment of a tongue image to be analyzed according to the present invention;

FIG. 5 is a schematic view of the correspondence between the tongue surface and the tongue base and the zang-fu organs according to the present invention;

FIG. 6 is a schematic diagram of the correspondence between the left and right sides of the tongue and the triple foci provided by the present invention;

FIG. 7 is a schematic diagram of the relationship between the upper and lower jaws and the internal organs according to the present invention;

FIG. 8 is a diagram of the relationship between the teeth and the organs of the human body according to the present invention;

FIG. 9 is a labeled tongue segmentation training image provided by the present invention;

FIG. 10 is a schematic diagram of a single channel region grayscale image provided by the present invention;

FIG. 11 is a schematic diagram of a self-timer tongue image according to a first embodiment of the present invention;

FIG. 12 is a schematic diagram of a self-timer tongue image according to a second embodiment of the present invention;

FIG. 13 is a schematic diagram of a self-timer tongue image according to a third embodiment of the present invention;

FIG. 14 is a schematic diagram of a fourth embodiment of a self-timer tongue image provided by the present invention;

FIG. 15 is a schematic diagram of a fifth embodiment of a self-timer tongue image provided by the present invention;

FIG. 16 is a diagram of a self-timer tongue image according to a sixth embodiment of the present invention;

FIG. 17 is a diagram of a seventh embodiment of a self-timer tongue image provided by the present invention;

FIG. 18 is a diagram of an eighth embodiment of a self-timer tongue image provided by the present invention;

FIG. 19 is a diagram of a seventh embodiment of a self-timer tongue image provided by the present invention;

FIG. 20 is a diagram of an eighth embodiment of a self-timer tongue image provided by the present invention;

FIG. 21 is a schematic diagram of one embodiment of a neural network portion architecture provided by the present invention;

FIG. 22 is a schematic structural diagram of a tongue feature neural network according to an embodiment of the present invention;

FIG. 23 is a diagram of a ninth embodiment of a self-timer tongue image provided by the present invention;

FIG. 24 is a schematic view of a second embodiment of a tongue image to be analyzed according to the present invention;

FIG. 25 is a schematic view of an embodiment of a tongue region image to be analyzed provided by the present invention;

FIG. 26 is a schematic diagram of an embodiment of the present invention for calculating Euclidean distance;

FIG. 27 is a schematic diagram of an embodiment of a holographic-based multidimensional tongue image analysis system according to the present invention;

FIG. 28 is a schematic block diagram of an embodiment of a computer apparatus provided by the present invention;

fig. 29 is a schematic structural diagram of an embodiment of a storage medium provided in the present application.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, fig. 1 is a schematic flow chart of a holographic-based multidimensional tongue image analysis method according to a first embodiment of the present invention. The invention provides a holographic-based multidimensional tongue image analysis method, which comprises the following steps:

s101: and acquiring a tongue image to be analyzed of the target user, wherein the tongue image to be analyzed comprises at least one of a tongue surface image and a tongue bottom image.

In a specific implementation scenario, the target user may use a smart device with a camera, such as a mobile phone and a tablet computer, or a camera or a video camera to shoot by himself or another person may use the device to help the target user shoot, and obtain a self-portrait tongue image. The self-portrait tongue image is uploaded to an analysis system for analysis. And carrying out image extraction on the self-timer tongue image to obtain a to-be-analyzed tongue image of the self-timer tongue image. For example, the self-portrait tongue image may include elements which are not relevant to analysis, such as a part of a face of a target user, a background during shooting, and the like, and these elements are deleted, and a tongue image to be analyzed of the tongue of the target user relevant to analysis is retained, so that the data volume required to be analyzed and processed during subsequent analysis can be effectively reduced, and the processing speed and efficiency can be effectively improved.

In the implementation scenario, the self-timer tongue image may be subjected to image recognition and image segmentation to obtain a tongue image to be analyzed, the self-timer tongue image may be input into a pre-trained tongue extraction neural network to obtain the tongue image to be analyzed, and the self-timer tongue image may be intercepted by a user or an analyst to obtain the tongue image to be analyzed.

The pre-trained tongue needs to be trained before it can be used to extract the neural network. A large number of tongue extraction training images can be obtained in advance, image annotation is performed on the tongue extraction training images by using an image annotation tool such as Labelme, and tongue regions in every other tongue extraction training image are annotated. Referring to fig. 2, fig. 2 is a schematic diagram of a labeled tongue extraction training image according to the present invention. And extracting a training image for each labeled tongue, setting the pixels of the labeled tongue area as 1, and setting the pixels of the rest areas as 0, and generating a corresponding single-channel tongue gray image. Referring to fig. 3, fig. 3 is a schematic diagram of a single-channel tongue grayscale image according to the present invention. Acquiring a tongue extraction result image according to the single-channel tongue gray image, taking each tongue extraction training image and the tongue extraction result image thereof as a group of tongue extraction training data, inputting a large amount of tongue extraction training data into a tongue extraction neural network, and acquiring a pre-trained tongue extraction neural network with a tongue extraction function. And inputting the self-timer tongue image into a pre-trained tongue extraction neural network to obtain a tongue image to be analyzed. Referring to fig. 4, fig. 4 is a schematic diagram of a tongue image to be analyzed according to the present invention. In this implementation scenario, the tongue extraction neural network is the deplab v3+ neural network model.

In this implementation scenario, the tongue image to be analyzed includes at least one of the tongue surface image and the tongue bottom image, because misdiagnosis is likely to occur when the tongue coating on the tongue surface is analyzed under the conditions of food staining, injury, and the like, which affects the reliability and accuracy of the analysis. The tongue bottom is not polluted by food, so that the analysis result cannot be influenced by the color change of the tongue coating, the tongue bottom is basically not injured, the analysis result cannot be influenced by the damage of the tongue texture, and the tongue bottom is in a tongue state and is upwards tilted, so that the analysis result cannot be influenced by the change of the tongue state.

In other implementations, the tongue image to be analyzed further includes at least one of a tongue side image, a maxilla image, a mandible image, a gum image, and a teeth image. The upper jaw, the lower jaw and the gum teeth can not be stained by food, the gum teeth can not be injured basically under the protection of muscles, and the tongue state is avoided, so that the reliability and the accuracy of analysis are higher.

S102: and acquiring at least one tongue area image to be analyzed of the tongue of the target user according to the tongue image to be analyzed.

In a specific implementation scenario, at least one tongue area image to be analyzed of the tongue of the target user is acquired from the tongue image to be analyzed. The tongue region image to be analyzed may be the entire tongue image to be analyzed as shown in fig. 4, or may be the tongue region image to be analyzed, which is obtained by segmenting the tongue image to be analyzed, and obtaining the tongue region image to be analyzed of each partition of the tongue, such as the tongue tip image to be analyzed, the tongue root image to be analyzed, and the like. Specifically, the analysis may be performed according to the analysis requirement, if the result can be analyzed only according to the image of the tongue partial area such as the tongue tip and the tongue root, the image segmentation may be performed to obtain the image of the tongue area to be analyzed corresponding to the partial area, and if the analysis result requires the image of the entire tongue area, the image of the tongue to be analyzed may be used as the image of the tongue area to be analyzed.

In other implementations, the image segmentation can be performed according to the corresponding reflection relationship between the tongue region and the viscera. Referring to fig. 5-8, fig. 5 is a schematic view of the relationship between the tongue surface and the tongue base and the viscera according to the present invention. FIG. 6 is a schematic diagram of the correspondence between the left and right sides of the tongue and the triple foci. FIG. 7 is a schematic diagram of the correspondence between the upper and lower jaws and the zang-fu organs according to the present invention. FIG. 8 is a diagram of the relationship between the teeth and the organs of the zang-fu organs according to the present invention.

If the tongue image to be analyzed needs to be segmented, the tongue image to be analyzed can be input into a pre-training tongue segmentation neural network to obtain at least one tongue image to be analyzed. Or the tongue image to be analyzed is divided into at least one tongue image to be analyzed according to preset division positions (for example, equally divided into three in the vertical direction or divided into three in the horizontal direction in a ratio of 1:2: 1). The tongue image to be analyzed may also be segmented by a user or an analyst to obtain at least one tongue image to be analyzed.

The tongue segmentation neural network needs to be trained when using a pre-trained tongue segmentation neural network. Specifically, a large number of tongue segmentation training images may be prepared in advance. The tongue segmentation training image is segmented and labeled according to a preset standard (for example, according to corresponding areas of different internal organs). Referring to fig. 9, fig. 9 is a labeled tongue segmentation training image according to the present invention. And generating a single-channel area gray image according to the marked tongue segmentation training image. Referring to fig. 10, fig. 10 is a schematic diagram of a single-channel area grayscale image provided by the present invention. Acquiring a tongue segmentation result image according to a single-channel area gray image, acquiring tongue segmentation training data according to each tongue segmentation training image and the corresponding tongue segmentation result image, inputting a large amount of tongue segmentation training data into a tongue segmentation neural network, and acquiring a pre-trained tongue segmentation neural network with a tongue extraction function. And inputting the tongue image to be analyzed into a pre-trained tongue segmentation neural network to obtain a tongue area image to be analyzed. In this implementation scenario, the tongue segmentation neural network is the deedlabv 3+ neural network model.

S103: and inputting the tongue area image to be analyzed into a pre-trained tongue characteristic neural network to obtain a tongue characteristic vector of the tongue area image to be analyzed.

In a specific implementation scenario, the tongue region image to be analyzed is input into a pre-trained tongue feature neural network, and a tongue feature vector of the tongue region image to be analyzed is obtained. The tongue feature vector is used for analyzing the corresponding situation of the tongue area image to be analyzed.

In this implementation scenario, the tongue characteristic neural network needs to be trained to obtain a pre-trained tongue characteristic neural network. Specifically, a tongue eigen neural network is constructed, the tongue eigen neural network uses the inclusion resnetv2 as a backbone network, the last density layer of the inclusion resnetv2 network is removed, and the tongue eigen neural network is replaced by the neural network structure shown in fig. 20. Referring to fig. 21 and 22 in combination, fig. 21 is a schematic diagram of an embodiment of a partial structure of a neural network provided in the present invention. Fig. 22 is a schematic structural diagram of a tongue feature neural network according to an embodiment of the present invention. In the implementation scenario shown in fig. 22, when training the tongue eigen-Neural network, the anchor sample (a), the positive sample (p), and the negative sample (n) are input into a CNN (Convolutional Neural network) network (e.g., an implicit rennet v2 network), and after passing through an L2 regularization layer, an Embedding layer (converting discrete variables into continuous vectors), and a loss function Triplet loss, three corresponding 128-dimensional vectors are obtained.

In the structure shown in fig. 21, globalaveragepoiling 2D is a global pooling layer for reducing feature dimensions and reducing network parameters. Dropout is a discard layer used to appropriately discard some neurons to avoid an overfitting state when training the network. Dense is an unfolding layer for unfolding the tongue region image into a preset number of dimensions, for example 128 dimensions. Lambda is a smoothing layer for performing the L2 normalization operation, i.e., vector element squaring and re-squaring, to smooth features so that different types of feature vectors can be of the same order of magnitude. The Dense layer and the following Softmax layer are classifiers: for assisting the convergence of the loss function. Since the use of only the Loss function Triplet Loss makes it difficult for the entire network to converge combining the two Loss functions of Cross-entry Loss and Triplet Loss as the total Loss, a classifier is constructed to assist the convergence of the Loss function Triplet Loss.

In this implementation scenario, the loss function Triplet loss is defined as:

L＝max(d(a，p)-d(a，n)+margin，0)

where d (a, p) represents the distance of the anchor sample from the positive sample and d (a, n) represents the distance of the anchor sample from the negative sample. In the implementation scenario, the input is a triplet including an anchor sample, a positive sample, and a negative sample, and similarity calculation between the samples is realized by optimizing that the distance between the anchor sample and the positive sample is smaller than the distance between the anchor sample and the negative sample. The margin is constant 1, which forces the neural network to learn more, so that the Euclidean distance value between the positive sample and the negative sample is larger.

Defining the triple Loss function can enable the Euclidean distance between vectors of a normal tongue region image and an abnormal tongue region image after feature extraction to be larger, so that the accuracy and reliability of a judgment result of whether the tongue region image to be analyzed and the analyzed tongue region image correspond to the same symptom or not are improved.

S104: and acquiring an analyzed tongue region image in a preset analysis database, calculating Euclidean distance between the tongue region image to be analyzed and the analyzed tongue region image according to the tongue feature vector, and acquiring a tongue feature according to an analyzed result of the analyzed tongue region image when the Euclidean distance is smaller than a preset distance threshold value.

In a specific implementation scenario, the tongue region image to be analyzed is input into a pre-trained tongue feature neural network, after a 128-dimensional tongue feature vector is obtained, an analyzed tongue region image in the same region in the database corresponding to the tongue region image to be analyzed is obtained, the 128-dimensional tongue feature vector of the tongue region image to be analyzed and the 128-dimensional analyzed feature vector of the analyzed tongue region image are calculated, and the euclidean distance is obtained. The database includes a plurality of analyzed tongue region images that have been acquired after manual or neural network analysis of their corresponding tongue regions and corresponding analysis scenarios. The analyzed tongue image is input into the neural network model shown in fig. 20 or fig. 21, and the analyzed feature vector of 128 thereof is obtained.

Calculating the Euclidean distance between the tongue region image to be analyzed and the analyzed tongue region image according to the following formula:

wherein x is_iIs the i-th dimension tongue feature vector, y_iAnalyzed vectors for the ith dimension.

If there is an analyzed tongue region image whose euclidean distance from the tongue region image to be analyzed is smaller than a preset distance threshold (e.g., 0.1, 0.3, 0.5) in the database, an analyzed result of the analyzed tongue region image is obtained, and a tongue feature is obtained according to an analyzed feature of the analyzed tongue region image, for example, the analyzed result is used as the tongue feature of the tongue region image to be analyzed.

In other implementation scenarios, one of the to-be-analyzed tongue region images corresponding to the plurality of tongue regions may be selected as a target tongue region image according to the determination requirement, a target tongue feature of the target tongue region image is obtained according to the above steps, a preliminary analysis result is obtained according to the target tongue feature, in order to further enhance the accuracy and reliability of the analysis, at least one of the to-be-analyzed tongue region images is selected as an auxiliary tongue region image, an auxiliary tongue feature of the auxiliary tongue region image is obtained, and the tongue analysis result is obtained by integrating the preliminary analysis result and the auxiliary analysis result.

In an embodiment, please refer to fig. 23-26 in combination, fig. 23 is a schematic diagram of a ninth embodiment of a self-timer tongue image according to the present invention. FIG. 24 is a schematic view of a second embodiment of a tongue image to be analyzed according to the present invention. FIG. 25 is a schematic view of an embodiment of a tongue region image to be analyzed according to the present invention. Fig. 26 is a schematic diagram of an embodiment of calculating the euclidean distance provided by the present invention.

The target user provides the self-timer tongue image shown in fig. 21, performs image extraction on the self-timer tongue image according to the method in the foregoing, obtains the to-be-analyzed tongue image shown in fig. 22, and performs image segmentation on the to-be-analyzed tongue image according to the method in the foregoing, so as to obtain a plurality of to-be-analyzed tongue region images shown in fig. 23. And extracting a tongue region image to be analyzed in the tongue tip region to obtain a tongue characteristic vector of the tongue region image. The euclidean distance between the tongue feature vector and the tongue feature vector of the analyzed tongue region image of the corresponding tongue tip region in the data is calculated. As shown in fig. 24, the euclidean distance is 0.156, which is less than the preset threshold value of 0.5. It is determined that the tongue tip of the target user corresponds to the analysis result of the analyzed tongue region image (e.g., depression).

Further, in order to ensure the reliability of the analysis result, the image of the tongue region to be analyzed in the tongue edge (corresponding to the hepatobiliary region) region is extracted, the Euclidean distance between the image of the tongue region to be analyzed and the image of the analyzed tongue region in the corresponding tongue edge region in the database is calculated, and whether the characteristic of liver-qi stagnation is met or not is judged according to the Euclidean distance. And extracting a to-be-analyzed tongue region image of a tongue root (corresponding to the kidney) region, calculating the Euclidean distance between the to-be-analyzed tongue region image and the analyzed tongue region image of the corresponding tongue root region in the database, and judging whether the to-be-analyzed tongue region image accords with the characteristics of insomnia and dreaminess according to the Euclidean distance. If the characteristics of liver-qi stagnation and insomnia and dreaminess are met, the target user can be more accurately judged to have the depression.

And acquiring a preliminary analysis result according to the target tongue characteristic, acquiring an auxiliary analysis result according to the auxiliary tongue characteristic, and synthesizing the preliminary analysis result and the auxiliary analysis result to acquire a tongue analysis result, so that the accuracy and reliability of the tongue analysis result can be further improved.

In a specific implementation scenario, image analysis is performed on a tongue image to be analyzed, and acquisition of tongue features corresponding to each tongue area image to be analyzed can also be achieved through technologies such as image recognition.

In this implementation scenario, the detected angles are different for tongue region images corresponding to different tongue regions, and the acquired tongue features are different. For example, the tongue region image to be analyzed on the tongue surface can detect whether abnormal textures such as fissures, transverse lines, longitudinal lines, lines and the like exist, and can also detect surface changes of the tongue, such as whether high bulges, depressions, lumps, hyperplasia, ulceration, desquamation and the like exist. It is also possible to detect the presence or absence of tongue spots, such as red spots, black spots, bluish purple, and sarcoma abnormalities. Whether the change of high convexity, concavity, texture and the like and the deformation of both sides exist is detected for the tongue region image of the tongue root part. And detecting whether the conditions of petechiae, venation, tumor and the like exist or not aiming at the tongue region images of the images at the two sides of the tongue. The trend, shape, thickness, length, and nodes of the vein of the tongue bottom are detected for the tongue region image of the tongue bottom image, the shape, unevenness, mass, polyp, and the like of the tongue bottom can be detected, and the color of the tongue bottom, such as red, black, cyan, purple, white, yellow, and the like, can be detected. And detecting aspects such as morphology, color quality and the like of the tongue region image to be analyzed of the upper jaw image and the lower jaw image. The morphological change thereof is detected with respect to the tongue region image to be analyzed of the tooth image of the gum image.

S105: and acquiring a tongue analysis result of the target user according to the tongue characteristics.

In a specific implementation scenario, the tongue characteristics of the tongue region image are used to obtain the conditions of the organs corresponding to the tongue region of the tongue region image, and the tongue analysis result of the target user is obtained according to the conditions of the organs. For example, the visceral organs corresponding to the tongue region where the tongue feature is located can be searched through fig. 5-8, and the condition of the visceral organs can be obtained according to the tongue feature. And (5) integrating the conditions of all the internal organs to obtain the tongue analysis result of the target user.

In other implementation scenarios, analysis may be performed only according to the tongue image to be analyzed provided by the target user, the obtained analysis result is not high in reliability and accuracy, and a photo supplement prompt may be issued to the target user, so that the target user supplements other tongue images to be analyzed according to the photo supplement prompt, for example, a tongue surface image provided by the user, and since a tongue coating may be stained, the user may be prompted to supplement a tongue bottom image and tongue side images, and the analysis result obtained by synthesizing a plurality of tongue images to be analyzed is more reliable.

In another implementation scenario, when obtaining the tongue analysis result, the self-describing information of the target user is obtained, and the tongue analysis result is obtained by combining the self-describing information. Various conditions corresponding to the tongue characteristics can be deleted or selected by combining the self-describing information, and parts which do not accord with the self-describing information or are irrelevant to the self-describing information are removed, so that the accuracy and the reliability of an analysis result are further improved.

Referring to fig. 11 and 12 in combination, fig. 11 is a schematic diagram of a self-timer tongue image according to a first embodiment of the present invention, and fig. 12 is a schematic diagram of a self-timer tongue image according to a second embodiment of the present invention. In one implementation scenario, the target user uploads the self-tongue image shown in fig. 11 and 12, and as can be seen from the analysis of fig. 11 and 12:

tongue tip (cardiopulmonary area): tongue shape is high convex: deficiency of heart-lung and heart-yang, chest distress and shortness of breath. Red, reddish and greasy coating: headache, dizziness, vexation, insomnia, dry eyes, red eyes. Narrowing of tongue tip: discomfort of neck and shoulder

In the tongue (liver, gallbladder, spleen and stomach region): the two sides of the tongue are raised: liver qi stagnation, listlessness, general debilitation, pain and discomfort. Red tongue edge, petechia: hyperactivity of liver-fire and gallbladder-fire, irritability, female mammary swelling, and menoxenia. Tongue concavity and convexity: stomach fullness, inappetence.

Tongue root (renal region): sunken tongue root: kidney deficiency, soreness and weakness of the waist and knees, and poor function.

According to the analysis, the target user is easy to have headache, dizziness, dry eyes, red eyes, stuffiness, short breath, dysphoria, irritability, insomnia, palpitation, inappetence, gastrectasia, neck and shoulder discomfort, soreness and weakness of waist and knees and lumbago. Female breast distention and irregular menstruation. As can be seen from the above symptom analysis, the target user may have depression.

Referring to fig. 13 and 14 in combination, fig. 13 is a schematic diagram of a third embodiment of a self-timer tongue image provided by the present invention, and fig. 14 is a schematic diagram of a fourth embodiment of a self-timer tongue image provided by the present invention. In one implementation scenario, the target user uploads the self-timer tongue image shown in fig. 13 and 14, and the analysis of fig. 13 and 14 shows that:

lingual surface analysis (heart-lung area): collapse of tongue tip: heart yang deficiency, chest distress and short breath; anterior tongue shape: restlessness, insomnia, cervical discomfort.

Tongue root (renal region): turbid phlegm with greasy coating: kidney deficiency, hypertension, and tumor.

Tongue bottom: petechia and red dots: hypertension, blood stasis and accumulation of qi and blood.

Tongue base (cardio-cerebral region): texture and venation: cerebral stroke or premonitory brain stroke has occurred.

Based on the above analysis, the target user has cerebral apoplexy or aura, hypertension, chest distress, short breath, vexation, insomnia, and cervical vertebra discomfort. As can be seen from the above symptom analysis, the target user may have a cerebral stroke or a precursor.

Referring to fig. 15 and 16 in combination, fig. 15 is a schematic diagram of a fifth embodiment of a self-timer tongue image provided by the present invention, and fig. 16 is a schematic diagram of a sixth embodiment of a self-timer tongue image provided by the present invention. In one implementation scenario, the target user uploads the self-timer tongue image shown in fig. 15 and 16, and as a result of analyzing fig. 15 and 16, it can be known that:

tongue tip (cardiopulmonary area): dizziness, amnesia, unresponsiveness, slow response, and shoulder discomfort.

Tongue tip (mammary region): high convex: breast lumps, swelling pain of axillary lymph nodes, chest distress and short breath.

On both sides of the tongue (hepatobiliary region): swelling and defects exist: there are liver and gallbladder diseases.

Tongue root (renal region): high convex and concave: kidney deficiency and tumor.

Tongue bottom surface: petechia: stagnant blood and blood stagnation.

From the above analysis, the target user may have breast disease (lumps or early stage tumors), dizziness, amnesia, depression, reaction retardation, shoulder discomfort, breast lumps, distending pain, chest distress, short breath, liver and gall. As can be seen from the above symptom analysis, the target user may have breast disease (lump or early tumor).

Referring to fig. 17 and 18 in combination, fig. 17 is a schematic diagram of a seventh embodiment of a self-timer tongue image provided by the present invention, and fig. 18 is a schematic diagram of an eighth embodiment of a self-timer tongue image provided by the present invention. In one implementation scenario, the target user uploads the self-timer tongue image shown in fig. 17 and 18, and the analysis of fig. 17 and 18 shows that:

gum and tooth: hemorrhoid and toxoplasma exist.

Tongue surface and tongue bottom: with hemorrhoids

According to the analysis, the target user may suffer from hemorrhoids and toxoplasma.

Referring to fig. 19 and 20 in combination, fig. 19 is a schematic diagram of a seventh embodiment of a self-timer tongue image provided by the present invention, and fig. 20 is a schematic diagram of an eighth embodiment of a self-timer tongue image provided by the present invention. In one implementation scenario, the target user uploads the self-timer tongue image shown in fig. 19 and 20, and the analysis of fig. 19 and 20 shows that:

the upper jaw and the bottom of the tongue contain the lower jaw: bending the center pillar: lateral curvature of cervical vertebra, lateral curvature of thoracic vertebra, and lumbar pain.

From the above analysis, the target user may have cervical lateral curvature, thoracic lateral curvature, and lumbar pain.

As can be seen from the above description, in this embodiment, a to-be-analyzed tongue image including at least one of a tongue surface image and a tongue bottom image is acquired, and at least one to-be-analyzed tongue area image of a tongue of a target user is acquired according to the to-be-analyzed tongue image; performing image analysis on each tongue area image to be analyzed to obtain tongue characteristics corresponding to each tongue area image to be analyzed; the tongue analysis result of the target user is obtained according to the tongue characteristics, the problem that the tongue analysis is easily affected by external adverse effects when the tongue bottom image is used for analysis only according to the tongue surface image can be effectively solved, the target user can analyze the tongue of the target user only by providing the image comprising the tongue, and the method is simple, convenient and reliable.

Referring to fig. 27, fig. 27 is a schematic structural diagram of an embodiment of a holographic-based multidimensional tongue image analysis system according to the present invention. The holographic-based multi-dimensional tongue image analysis system 10 includes: an acquisition module 11, an image module 12, a network module 13, a feature module 14, and a results module 15.

The obtaining module 11 is configured to obtain a tongue image to be analyzed of a target user, where the tongue image to be analyzed includes at least one of a tongue surface image and a tongue bottom image. The image module 12 is configured to obtain at least one tongue area image to be analyzed of the tongue of the target user according to the tongue image to be analyzed. The network module 13 is configured to input the tongue region image to be analyzed into the pre-trained tongue feature neural network, and obtain a tongue feature vector of the tongue region image to be analyzed. The feature module 14 is configured to obtain an analyzed tongue region image in a preset analysis database, calculate a euclidean distance between the tongue region image to be analyzed and the analyzed tongue region image according to the tongue feature vector, and obtain a tongue feature according to an analyzed result of the analyzed tongue region image when the euclidean distance is smaller than a preset distance threshold. The result module 15 is configured to obtain a tongue analysis result of the target user according to the tongue feature.

Wherein the tongue image to be analyzed further comprises at least one of a tongue both-side image, an upper jaw image, a lower jaw image, a gum image, and a tooth image.

The obtaining module 11 is further configured to issue a photo supplement prompt to the target user, so that the target user supplements other tongue images to be analyzed according to the photo supplement prompt.

The result module 15 is further configured to obtain the self-describing information of the target user, and obtain a tongue analysis result by combining the self-describing information.

The image module 12 is further configured to perform image segmentation on the tongue image to be analyzed according to the corresponding reflection relationship between each area of the tongue and the internal organs to obtain at least one tongue area image to be analyzed.

Wherein, when waiting to analyze tongue image and including tongue face image, the tongue characteristic includes: irregular texture, tongue surface change, tongue speckle and deformation; when the tongue image to be analyzed includes a tongue fundus image, the tongue characteristics include: vein and vein characteristics of the bottom of the tongue, color of the bottom of the tongue, morphology of the bottom of the tongue, unevenness, lumps, and polyps.

The tongue analysis system 10 further comprises a training module 15, wherein the training module 15 is used for constructing a tongue characteristic neural network, and the tongue characteristic neural network is a deep convolutional neural network and comprises a global pooling layer, a discarding layer, an expanding layer, a smoothing layer, a Dense layer and a Softmax layer; defining a tongue characteristic loss function to reduce Euclidean distances between similar images and increase Euclidean distances between non-similar images; and inputting the tongue characteristic training sample image into a tongue characteristic neural network for training to obtain a pre-training tongue characteristic neural network.

The image module 12 is further configured to input the tongue image to be analyzed into the pre-training tongue segmentation neural network, and obtain at least one tongue image to be analyzed.

The image module 12 is further configured to prepare a tongue segmentation training image, and segment and label the tongue segmentation training image according to a preset standard, where the preset standard includes viscera organs corresponding to the tongue region; generating a single-channel area gray image according to the marked tongue segmentation training image, and acquiring a tongue segmentation result image according to the single-channel area gray image; and acquiring tongue segmentation training data according to each tongue segmentation training image and the corresponding tongue segmentation result image thereof, inputting the tongue segmentation training data into a tongue segmentation neural network, and acquiring a pre-training tongue segmentation neural network.

The result module 15 is further configured to obtain the viscera organs corresponding to the tongue region of the tongue region image, obtain the conditions of the viscera organs according to the tongue characteristics, and obtain the tongue analysis result according to the conditions of the viscera organs.

The feature module 14 is further configured to calculate euclidean distances between the predetermined number of dimensional tongue feature vectors of the tongue region image to be analyzed and the predetermined number of dimensional analyzed vectors of the analyzed image.

The feature module 14 is also configured to calculate the euclidean distance according to the following formula:

wherein x is_iIs the i-th dimension tongue feature vector, y_iAnalyzed vectors for the ith dimension.

The obtaining module 11 is further configured to obtain a self-tongue image provided by the target user, and input the self-tongue image into a pre-trained tongue extraction neural network to obtain a tongue image to be analyzed.

The result module 15 is further configured to obtain a preliminary analysis result according to the target tongue characteristic, obtain an auxiliary analysis result according to the auxiliary tongue characteristic, and obtain a tongue analysis result by integrating the preliminary analysis result and the auxiliary analysis result.

Wherein the preset distance threshold is 0.5.

As can be seen from the above description, in this embodiment, a tongue image to be analyzed including at least one of a tongue surface image and a tongue bottom image is acquired based on a holographic multidimensional tongue image analysis system, and at least one tongue area image to be analyzed of a tongue of a target user is acquired according to the tongue image to be analyzed; performing image analysis on each tongue area image to be analyzed to obtain tongue characteristics corresponding to each tongue area image to be analyzed; the tongue analysis result of the target user is obtained according to the tongue characteristics, the problem that the tongue analysis is easily affected by external adverse effects when the tongue bottom image is used for analysis only according to the tongue surface image can be effectively solved, the target user can analyze the tongue of the target user only by providing the image comprising the tongue, and the method is simple, convenient and reliable.

Referring to fig. 28, fig. 28 is a schematic structural diagram of a computer device according to an embodiment of the present invention. The content detection device 20 comprises a processor 21, a memory 22. The processor 21 is coupled to a memory 22. The memory 22 has stored therein a computer program which is executed by the processor 21 in operation to implement the method as shown in fig. 1 and 21. The detailed methods can be referred to above and are not described herein.

Referring to fig. 29, fig. 29 is a schematic structural diagram of a storage medium according to an embodiment of the present disclosure. The storage medium 30 stores at least one computer program 31, and the computer program 31 is used for being executed by a processor to implement the method shown in fig. 1 and fig. 21, and the detailed method can be referred to above and is not described herein again. In one embodiment, the computer readable storage medium 30 may be a memory chip in a terminal, a hard disk, or other readable and writable storage tool such as a removable hard disk, a flash disk, an optical disk, or the like, and may also be a server or the like.

Compared with the prior art, the method has the advantages that the problem that the tongue bottom image is easily influenced by external adverse effects when the tongue bottom image is used for analyzing according to the tongue surface image can be effectively solved, the target user can analyze the tongue of the target user only by providing the image comprising the tongue, and the method is simple, convenient and reliable.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, and the program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims. Please enter the implementation content part.