1. A customer service robot knowledge base ambiguity detection method is characterized by comprising the following steps:

constructing a knowledge base, wherein the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question sentence, and each FAQ is of one category;

dividing the knowledge base into a test set and a training set of a deep learning model;

training a deep learning model on a training set, and performing ambiguity detection by using the deep learning model which is learned; the ambiguity detection includes: the category ambiguity detection, the labeling error detection and the labeling ambiguity detection, wherein the ambiguity detection by using the learned deep learning model comprises the following steps: detecting ambiguity by using a shallow classifier in a deep learning model, comprising:

counting the deep learning model classification results and forming a confusion matrix, wherein the confusion matrix comprises a plurality of classesEach row i of the confusion matrix corresponds to a labeled category, each column j corresponds to a category predicted by the deep learning model, and an element x_ijIs the number of question sentences labeled as category i, and the model predicts as category j, element x_jiThe number of question sentences marked as category j and predicted as category i by the model;

calculating the number of samples marked as a category i in the data set, wherein the number of the samples of the category i isWherein k is of any class;

calculating the number of samples marked as a category j in the data set, wherein the number of the samples of the category j isWherein k is of any class;

calculating a proportion P of samples in the data set labeled as class i to be predicted as class j by the deep learning model_ijRatio P to class i of samples predicted as class j_jiSaid P is_ijAnd P_jiThe calculation formulas are respectively as follows:the category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set;

calculating the confusion degree of the class pair (class i, class j), wherein the confusion degree is P_ijAnd P_jiHarmonic mean value of S_ijSaid

Judging whether ambiguity exists between the category i and the category j according to the confusion degree;

updating the knowledge base according to an ambiguity detection result;

and repeating the steps until the learning effect is not improved any more, and obtaining the knowledge base for eliminating the ambiguity.

2. The method of claim 1, wherein the partitioning the knowledge base into a test set and a training set of deep learning models comprises: randomly extracting a preset number of similar question sentences corresponding to each FAQ as test data of FAQ corresponding categories, and taking the rest similar question sentences as training data of the FAQ corresponding categories; the test data of all classes constitutes a test set and the training data of all classes constitutes a training set.

3. The method of claim 1, wherein the deep learning model comprises: the deep learning model is trained on a training set, and the deep learning model comprises the following steps:

inputting the question sentences in the training set into the deep learning model as an input part;

converting the question sentences in the input part into feature vectors by using a feature extractor in the deep learning model;

calculating a prediction result by utilizing a shallow classifier in the deep learning model according to the feature vector, wherein the prediction result is a category corresponding to a question in an input part;

optimizing a training model by using an optimizer, and minimizing the average difference between the actual category marked by the question in the training set and the prediction result of the deep learning model;

and (4) evaluating the trained model by using the test set, and calculating the consistency rate of the model prediction result and the actual category marked by the question in the test set to be used as the evaluation of the model learning effect.

4. The method according to claim 1, wherein the determining whether the category i and the category j are ambiguous according to the confusion comprises:

sorting the calculated confusion degrees;

and extracting the categories with the third preset number of confusion degrees ranked at the top, and manually detecting whether category ambiguity exists.

5. The method of claim 1, wherein detecting ambiguity using a deep learning model shallow classifier further comprises: and finding out data with inconsistent actual categories labeled in a data set and categories predicted by the deep learning model, and manually checking whether labeling errors exist, wherein the data set comprises a training set or/and a testing set.

6. The method of claim 1, wherein updating the knowledge base based on ambiguity detection comprises:

manually rewriting and manually re-labeling the detected ambiguous question, and deleting the original label;

and recombining and distributing the similar question sentences for the detected ambiguous categories, and deleting the original ambiguous categories.

7. A customer service robot knowledge base ambiguity detection device is characterized by comprising:

the system comprises a construction module, a query module and a query module, wherein the construction module is used for constructing a knowledge base, the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question sentence, and each FAQ is of one category;

the dividing module is used for dividing the knowledge base into a test set and a training set of a deep learning model;

the training module is used for training a deep learning model on a training set and carrying out ambiguity detection by utilizing the deep learning model which is learned; the ambiguity detection includes: the category ambiguity detection, the labeling error detection and the labeling ambiguity detection, wherein the ambiguity detection by using the learned deep learning model comprises the following steps: detecting ambiguity by using a shallow classifier in a deep learning model, comprising: counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to a labeled category, each column j corresponds to a category predicted by the deep learning model, and an element x_ijIs the number of question sentences labeled as category i, and the model predicts as category j, element x_jiThe number of question sentences marked as category j and predicted as category i by the model; calculating the number of samples marked as a category i in the data set, wherein the number of the samples of the category i isWherein k is of any class; calculating the number of samples marked as a category j in the data set, wherein the number of the samples of the category j isWherein k is of any class; calculating a proportion P of samples in the data set labeled as class i to be predicted as class j by the deep learning model_ijRatio P to class i of samples predicted as class j_jiSaid P is_ijAnd P_jiThe calculation formulas are respectively as follows:the category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set; calculating the confusion degree of the class pair (class i, class j), wherein the confusion degree is P_ijAnd P_jiHarmonic mean value of S_ijSaidJudging whether ambiguity exists between the category i and the category j according to the confusion degree;

the updating module is used for updating the knowledge base according to an ambiguity detection result;

and the repeating module is used for repeating the steps until the learning effect is not improved any more, and obtaining the knowledge base for eliminating the ambiguity.

8. An electronic device, comprising:

at least one memory for storing a program;

at least one processor configured to load the program to perform the customer service robot knowledge base ambiguity detection method of any one of claims 1-6.

9. A computer-readable storage medium in which a program executable by a processor is stored, comprising:

the processor executable program when executed by a processor is for performing the customer service robot knowledge base ambiguity detection method of any one of claims 1-6.

Background

With the increase of internet users, the service pressure of the customer service department of the enterprise is continuously increased. Since most of the questions encountered by users are repeated, these repeated questions can be answered with a fixed template. In order to reduce the labor cost of a customer service center, a robot customer service can be introduced, the type of a question of a user is judged by a program, if the question belongs to FAQ (Frequently Asked Questions), a standard answer is directly given, otherwise, manual service is switched to carry out special intervention.

In the related art, a customer service robot recognizes user intentions by using a machine learning technology, and converts the intention recognition into question classification problems. Each FAQ corresponds to a category, and each category has more than one similar question sentence. All FAQs and corresponding similar questions constitute the knowledge base of the robot.

The effect of the machine learning model usually depends on training data selected from a knowledge base, and especially the labeling accuracy of the training data has a great influence on the model effect. However, due to the limitations of time and manual effort, a large amount of ambiguity often exists in the knowledge base, for example, a question corresponds to an incorrect category, and the category and category semantics coincide, and the ambiguity may cause the model to learn the incorrect knowledge, thereby causing a negative effect on the accuracy of the model.

Disclosure of Invention

In order to overcome the problems in the related art at least to a certain extent, the application provides a method, a device and related equipment for detecting ambiguity of a knowledge base of a customer service robot.

In a first aspect, the present application provides a method for detecting ambiguity in a knowledge base of a customer service robot, including:

constructing a knowledge base, wherein the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question sentence, and each FAQ is of one category;

dividing the knowledge base into a test set and a training set of a deep learning model;

training a deep learning model on a training set, and performing ambiguity detection by using the deep learning model which is learned;

updating the knowledge base according to an ambiguity detection result;

and repeating the steps until the learning effect is not improved any more, and obtaining the knowledge base for eliminating the ambiguity.

Further, the dividing the knowledge base into a test set and a training set of the deep learning model includes: and randomly extracting a preset number of similar question sentences corresponding to each FAQ as test data of the FAQ corresponding type, and taking the rest similar question sentences as training data of the FAQ corresponding type. The test data of all classes constitutes a test set and the training data of all classes constitutes a training set.

Further, the deep learning model comprises: the deep learning model is trained on a training set, and the deep learning model comprises the following steps:

inputting the question sentences in the FAQ in the training set into the deep learning model as an input part;

converting the question sentences in the input part into feature vectors by using a feature extractor in the deep learning model;

calculating a prediction result by utilizing a shallow classifier in the deep learning model according to the feature vector, wherein the prediction result is a category corresponding to a question in an input part;

optimizing a training model by using an optimizer, and minimizing the average difference between the actual category marked by the question in the training set and the prediction result of the deep learning model;

and (4) evaluating the trained model by using the test set, and calculating the consistency rate of the model prediction result and the actual category marked by the question in the test set to be used as the evaluation of the model learning effect.

Further, the ambiguity detection includes: the category ambiguity detection, the labeling error detection and the labeling ambiguity detection, wherein the ambiguity detection by using the learned deep learning model comprises the following steps:

detecting ambiguity by using a feature extractor in a deep learning model;

and detecting ambiguity by using a shallow classifier in the deep learning model.

Further, the detecting ambiguity by using the feature extractor in the deep learning model includes:

converting similar question sentences in a data set into feature vectors by using a feature extractor in the deep learning model, wherein the data set comprises a training set or/and a test set;

combining the feature vectors corresponding to the question into a question feature vector pair (x, y), wherein the question corresponding to the feature vector x and the question corresponding to the feature vector y are from different categories respectively;

calculating the vector similarity cos (x, y) of each question feature vector pair, the

And sorting all question feature vector pairs from high to low according to the vector similarity, selecting the question feature vector pair with the vector similarity ranked at the top, and judging whether ambiguity exists according to the question feature vector pair with the vector similarity ranked at the top.

Further, the determining whether there is ambiguity according to the question feature vector pair ranked at the top in the vector similarity includes:

judging whether the labeling ambiguity or the labeling error exists: extracting a first preset number of question sentence feature vector pairs with the similarity ranking at the top, and manually checking whether the corresponding question sentence pairs have labeling ambiguity and labeling errors;

judging whether category ambiguity exists: counting the repeated occurrence times of the corresponding category pairs for the question feature vector pairs of the first preset number, sorting the category pairs of the second preset number from high to low according to the occurrence times, and manually checking whether category ambiguity exists.

Further, the detecting ambiguity by using the deep learning model shallow classifier includes:

counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to a labeled category, each column j corresponds to a category predicted by the deep learning model, and an element x_ijIs the number of question sentences labeled as category i, and the model predicts as category j, element x_jiThe number of question sentences marked as category j and predicted as category i by the model;

calculating the number of samples marked as a category i in the data set, wherein the number of the samples of the category i isWherein k is of any class;

calculating the number of samples marked as a category j in the data set, wherein the number of the samples of the category j isWherein k is of any class;

calculating a proportion P of samples in the data set labeled as class i to be predicted as class j by the deep learning model_ijRatio P to class i of samples predicted as class j_jiSaid P is_ijAnd P_jiThe calculation formulas are respectively as follows:the category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set;

calculating the confusion degree of the class pair (class i, class j), wherein the confusion degree is P_ijAnd P_jiHarmonic mean value of S_ijSaid

And judging whether ambiguity exists between the category i and the category j according to the confusion degree.

Further, the determining whether the category i and the category j have ambiguity according to the confusion degree includes:

sorting the calculated confusion degrees;

and extracting the categories with the third preset number of confusion degrees ranked at the top, and manually detecting whether category ambiguity exists.

Further, the detecting ambiguity by using the deep learning model shallow classifier further comprises: and finding out data with inconsistent actual categories labeled in a data set and categories predicted by the deep learning model, and manually checking whether labeling errors exist, wherein the data set comprises a training set or/and a testing set.

Further, the updating the knowledge base according to the ambiguity detection result includes:

manually rewriting and manually re-labeling the detected ambiguous question, and deleting the original label;

and recombining and distributing the similar question sentences for the detected ambiguous categories, and deleting the original ambiguous categories.

In a second aspect, the present application provides a customer service robot knowledge base ambiguity detection apparatus, including:

the system comprises a construction module, a query module and a query module, wherein the construction module is used for constructing a knowledge base, the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question sentence, and each FAQ is of one category;

the dividing module is used for dividing the knowledge base into a test set and a training set of a deep learning model;

the training module is used for training a deep learning model on a training set and carrying out ambiguity detection by utilizing the deep learning model which is learned; the ambiguity detection includes: the category ambiguity detection, the labeling error detection and the labeling ambiguity detection, wherein the ambiguity detection by using the learned deep learning model comprises the following steps: detecting ambiguity by using a shallow classifier in a deep learning model, comprising: counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to a labeled category, each column j corresponds to a category predicted by the deep learning model, and an element x_ijIs the number of question sentences labeled as category i, and the model predicts as category j, element x_jiThe number of question sentences marked as category j and predicted as category i by the model; calculating the number of samples marked as a category i in the data set, wherein the number of the samples of the category i isWherein k is of any class; calculating the number of samples marked as a category j in the data set, wherein the number of the samples of the category j isWherein k is of any class; calculating a proportion P of samples in the data set labeled as class i to be predicted as class j by the deep learning model_ijRatio P to class i of samples predicted as class j_jiSaid P is_ijAnd P_jiThe calculation formulas are respectively as follows:the category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set; calculating the confusion degree of the class pair (class i, class j), wherein the confusion degree is P_ijAnd P_jiHarmonic mean value of S_ijSaidJudging whether ambiguity exists between the category i and the category j according to the confusion degree;

the updating module is used for updating the knowledge base according to an ambiguity detection result;

and the repeating module is used for repeating the steps until the learning effect is not improved any more, and obtaining the knowledge base for eliminating the ambiguity.

In a third aspect, the present application provides an electronic device, comprising:

at least one memory for storing a program;

at least one processor configured to load the program to perform the customer service robot knowledge base ambiguity detection method of any one of the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium having stored therein a program executable by a processor, comprising:

the processor executable program when executed by the processor is for performing the customer service robot knowledge base ambiguity detection method of any one of the first aspect.

The technical scheme provided by the embodiment of the application can have the following beneficial effects:

the knowledge base is updated according to the ambiguity detection result, the training step is repeated until the learning effect reaches the expected standard, the ambiguity of the knowledge base can be found and corrected in an auxiliary mode, the ambiguity eliminated knowledge base is obtained, data are extracted from the ambiguity eliminated knowledge base and serve as a training set and a testing set of the deep learning model, and the learning effect of the deep learning model is further improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

Fig. 1 is a schematic flowchart of a customer service robot knowledge base ambiguity detection method according to an embodiment of the present application.

Fig. 2 is a flowchart illustrating a method for ambiguity detection in a knowledge base of a customer service robot according to another embodiment of the present application.

Detailed Description

The invention is described in detail below with reference to the figures and examples.

Fig. 1 is a schematic flowchart of a customer service robot knowledge base ambiguity detection method according to an embodiment of the present application.

As shown in fig. 1, the method of the present embodiment includes:

s11: and constructing a knowledge base, wherein the knowledge base is divided according to FAQs, each FAQ is provided with a variable number of similar question sentences, and each FAQ is a category.

The knowledge base is developed on the basis of large-scale knowledge processing, is applied to the industry, is suitable for the technical industries of large-scale knowledge processing, natural language understanding, knowledge management, an automatic question and answer system, reasoning and the like, and intelligent customer service not only provides a fine-grained knowledge management technology for enterprises, but also establishes a quick and effective technical means based on natural language for communication between the enterprises and mass users. Taking a customer service robot knowledge base of an e-commerce enterprise as an example, the knowledge base comprises a plurality of FAQs, such as a 'return flow' and a 'refund flow'. Taking the "return flow" as an example, the FAQ may include the following similar question: "how do I buy yesterday to return goods? "," how should I want to return goods? ".

S12: the knowledge base is divided into a test set and a training set of deep learning models.

And selecting N FAQs needing to detect ambiguity from the knowledge base as N categories. For each FAQ, a preset number of similar question sentences are randomly extracted as the test data of the category, and the rest similar question sentences are used as the training data of the category. The test data of all classes constitutes a test set and the training data of all classes constitutes a training set.

For example, 10 FAQs are contained in the knowledge base, each FAQ contains 20 similar questions, a preset amount, for example, 3 similar questions are randomly extracted from each category of the knowledge base as a test set of the deep learning model, then 30 similar questions are contained in the test set, and the other 170 similar questions are included in the training set of the deep learning model.

It should be noted that the number of categories included in the knowledge base according to the present invention and the number of similar question sentences included in each category are not limited to the examples in the embodiments, and are not described herein again.

S13: and training a deep learning model on a training set, and performing ambiguity detection by using the deep learning model.

The deep learning model comprises: a feature extractor and a shallow classifier.

The ambiguity detection includes: category ambiguity detection, annotation error detection and annotation ambiguity detection;

the ambiguity includes:

category ambiguity: that is, the meaning of the two categories is very similar, for example, category 1 is "order problem", category 2 is "change cancel problem of product", and the semantics of category 1 and the semantics of category 2 overlap, because category 1 can basically cover category 2;

and (3) annotation ambiguity: that is, the question may be timescaled to multiple classes, for example: category 1 is "return question of product", category 2 is "price question of product", if the question is "this thing is too expensive, i want to return goods", there is a label ambiguity in this sentence, because the question contains the meanings of the two categories at the same time;

marking errors: the question corresponds to the wrong category, for example, category 1 is "return question of product", category 2 is "price question of product", and if the question is "i do not want" but is labeled as category 2, a labeling error is generated.

The ambiguity detection is for a test set or/and a training set.

The using the learned deep learning model for ambiguity detection includes:

detecting ambiguity by using a feature extractor in a deep learning model;

detecting ambiguity by using a shallow classifier in a deep learning model;

the detection of the ambiguity by using the feature extractor in the deep learning model comprises the following steps:

converting similar question sentences in a data set into feature vectors by using a feature extractor in the deep learning model, wherein the data set comprises a training set or/and a test set;

combining the feature vectors corresponding to the question into a question feature vector pair (x, y), wherein the question corresponding to the feature vector x and the question corresponding to the feature vector y are from different categories respectively;

calculating the vector similarity cos (x, y) of each question feature vector pair, the

And sorting all question feature vector pairs from high to low according to the vector similarity, selecting the question feature vector pair with the vector similarity ranked at the top, and judging whether ambiguity exists according to the question feature vector pair with the vector similarity ranked at the top.

The step of judging whether ambiguity exists according to the question feature vector pair ranked at the top according to the vector similarity comprises the following steps:

judging whether the labeling ambiguity or the labeling error exists: extracting a first preset number of question sentence feature vector pairs with the similarity ranking at the top, for example, 30, and manually checking whether the corresponding question sentence pairs have labeling ambiguity and labeling error;

judging whether category ambiguity exists: counting the repeated occurrence times of the corresponding category pairs for the question feature vector pairs with the first preset number, sorting the category pairs from high to low according to the occurrence times, taking a second preset number, such as 20 category pairs, and manually checking whether category ambiguity exists.

The method for detecting the ambiguity by using the deep learning model shallow classifier comprises the following steps:

counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to a labeled category, each column j corresponds to a category predicted by the deep learning model, and an element x_ijIs the number of question sentences labeled as category i, and the model predicts as category j, element x_jiThe number of question sentences marked as category j and predicted as category i by the model;

calculating the number of samples marked as a category i in the data set, wherein the number of the samples of the category i isWherein k is of any class;

calculating the number of samples marked as a category j in the data set, wherein the number of the samples of the category j isWherein k is of any class;

calculating a proportion P of samples in the data set labeled as class i to be predicted as class j by the deep learning model_ijRatio P to class i of samples predicted as class j_jiSaid P is_ijAnd P_jiThe calculation formulas are respectively as follows:the category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set;

calculating the confusion degree of the class pair (class i, class j), wherein the confusion degree is P_ijAnd P_jiHarmonic mean value of S_ijSaid

And judging whether ambiguity exists between the category i and the category j according to the confusion degree.

Judging whether ambiguity exists in the category i and the category j according to the confusion degree comprises the following steps:

sorting the calculated confusion degrees;

and extracting a third preset number of category pairs with the confusion degrees of 5, for example, ranking at the top, and manually detecting whether category ambiguity exists.

The detecting ambiguity by using the deep learning model shallow classifier further comprises: and finding out data with inconsistent actual categories labeled in a data set and categories predicted by the deep learning model, and manually checking whether labeling errors exist, wherein the data set comprises a training set or/and a testing set.

S14: updating the knowledge base according to the ambiguity detection result, comprising:

manually rewriting and manually re-labeling the detected ambiguous question, and deleting the original label;

and recombining and distributing the similar question sentences for the detected ambiguous categories, and deleting the original ambiguous categories.

S15: and repeating the steps until the learning effect is not improved any more, and obtaining the knowledge base for eliminating the ambiguity.

The learning effect is the consistency ratio of the model prediction result and the actual category labeled by the question in the test set, and the consistency ratio is, for example, the prediction accuracy ratio, that is, the number of the question with the consistent prediction result is divided by the total number of the question. The learning effect is no longer improved, for example, the prediction accuracy is improved by less than 0.5%.

When the learning effect of the model is not improved any more, the performance reduction of the model caused by ambiguity of the knowledge base is eliminated, and the model can be trained by using the knowledge base and deployed in a production environment for use.

In this embodiment, the knowledge base is updated according to the ambiguity detection result, the training step is repeated until the learning effect reaches the expected standard, the ambiguity of the knowledge base can be found and corrected manually in an auxiliary manner, the ambiguity-resolved knowledge base is obtained, data is extracted from the ambiguity-resolved knowledge base and is used as a training set and a test set of the deep learning model, and the learning effect of the deep learning model is further improved.

Fig. 2 is a flowchart illustrating a method for ambiguity detection in a knowledge base of a customer service robot according to another embodiment of the present application.

As shown in fig. 2, the training of the deep learning model on the training set includes:

the concept of deep learning is derived from the research of an artificial neural network and comprises a multi-layer perceptron with multiple hidden layers. Deep learning forms a more abstract class or feature of high-level representation properties by combining low-level features to discover a distributed feature representation of the data. The deep learning model includes a feature extractor and a shallow classifier.

S21: inputting the question sentences in the training set into the deep learning model as an input part;

s22: converting the question sentences in the input part into feature vectors by using a feature extractor in the deep learning model;

the feature extractor is, for example, a recurrent neural network. The model reads each word in the question sentence in sequence and outputs a feature vector with fixed dimensionality. It should be noted that the feature extractor is not limited to the circular neural network illustrated, and any method that can convert a question into a feature vector of a fixed dimension may be used as the feature extractor.

S23: calculating a prediction result by utilizing a shallow classifier in the deep learning model according to the feature vector, wherein the prediction result is a category corresponding to a question in an input part;

the shallow classifier is, for example, a linear classifier. The classifier reads in a feature vector with a fixed dimension, calculates the linear combination of vector elements to obtain the score of each category, and takes the category with the highest score as a prediction result. It should be noted that the shallow classifier is not limited to the illustrated linear classifier, and any method that can convert a feature vector of a fixed dimension into scores of various categories can be used as the shallow classifier.

S24: optimizing a training model by using an optimizer, and minimizing the average difference between the actual category marked by the question in the training set and the prediction result of the deep learning model;

the average difference is for example a loss function. The loss function is for example cross entropy.

The optimizer is for example a gradient descent method. The Gradient Descent is one of iterative methods, and when solving model parameters of a machine learning algorithm, namely an unconstrained optimization problem, the Gradient Descent (Gradient) is one of the most commonly adopted methods. When the minimum value of the loss function is solved, iterative solution can be carried out step by step through a gradient descent method, and the minimized loss function and the corresponding model parameter value are obtained.

S25: and (3) evaluating the trained model by using the test set, and calculating the consistency rate of the model prediction result and the actual category marked by the question in the test set as the evaluation of the model learning effect, wherein the consistency rate is the prediction accuracy rate, namely the number of the question with the consistent prediction result is divided by the total number of the question.

In this embodiment, the deep learning model is used to train the concentrated FAQ, the optimizer is used to continuously optimize the model in the training process, the learning effect of the deep learning model is continuously improved by iteration, and the ambiguity detection accuracy is continuously improved.

An embodiment of the present application provides a customer service robot knowledge base ambiguity detection apparatus, including:

the system comprises a construction module, a query module and a query module, wherein the construction module is used for constructing a knowledge base, the knowledge base is divided according to FAQs, each FAQ is provided with at least one similar question sentence, and each FAQ is of one category;

the dividing module is used for dividing the knowledge base into a test set and a training set of a deep learning model;

the training module is used for training a deep learning model on a training set and carrying out ambiguity detection by utilizing the deep learning model which is learned; the ambiguity detection includes: the category ambiguity detection, the labeling error detection and the labeling ambiguity detection, wherein the ambiguity detection by using the learned deep learning model comprises the following steps: detecting ambiguity by using a shallow classifier in a deep learning model, comprising: counting the classification results of the deep learning model and forming a confusion matrix, wherein each row i of the confusion matrix corresponds to a labeled category, and each column j corresponds to the depth studyClass of learning model prediction, element x_ijIs the number of question sentences labeled as category i, and the model predicts as category j, element x_jiThe number of question sentences marked as category j and predicted as category i by the model; calculating the number of samples marked as a category i in the data set, wherein the number of the samples of the category i isWherein k is of any class; calculating the number of samples marked as a category j in the data set, wherein the number of the samples of the category j isWherein k is of any class; calculating a proportion P of samples in the data set labeled as class i to be predicted as class j by the deep learning model_ijRatio P to class i of samples predicted as class j_jiSaid P is_ijAnd P_jiThe calculation formulas are respectively as follows:the category i and the category j belong to different categories, and the data set comprises a training set or/and a testing set; calculating the confusion degree of the class pair (class i, class j), wherein the confusion degree is P_ijAnd P_jiHarmonic mean value of S_ijSaidJudging whether ambiguity exists between the category i and the category j according to the confusion degree;

the updating module is used for updating the knowledge base according to an ambiguity detection result;

and the repeating module is used for repeating the steps until the learning effect is not improved any more, and obtaining the knowledge base for eliminating the ambiguity.

In some embodiments, further comprising:

the random extraction module is used for dividing the knowledge base into a test set and a training set of the deep learning model, and comprises the following steps: randomly extracting a preset number of similar question sentences corresponding to each FAQ as test data of FAQ corresponding categories, and taking the rest similar question sentences as training data of the FAQ corresponding categories; the test data of all classes constitutes a test set and the training data of all classes constitutes a training set.

And the sorting module is used for sorting the calculated confusion degrees, extracting the categories with the third preset number of confusion degrees ranked at the top, and manually detecting whether category ambiguity exists.

The labeling module is used for detecting ambiguity by utilizing a shallow classifier of the deep learning model, and further comprises: and finding out data with inconsistent actual categories labeled in a data set and categories predicted by the deep learning model, and manually checking whether labeling errors exist, wherein the data set comprises a training set or/and a testing set.

One embodiment of the present application provides an electronic device, including:

at least one memory for storing a program;

at least one processor, configured to load the program to perform the customer service robot knowledge base ambiguity detection method according to the above embodiments.

One embodiment of the present application provides a computer-readable storage medium, in which a program executable by a processor is stored, comprising:

the processor executable program when executed by the processor is configured to perform the customer service robot knowledge base ambiguity detection method according to the above embodiments.

It is understood that the same or similar parts in the above embodiments may be mutually referred to, and the same or similar parts in other embodiments may be referred to for the content which is not described in detail in some embodiments.

It should be noted that, in the description of the present application, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Further, in the description of the present application, the meaning of "a plurality" means at least two unless otherwise specified.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.

It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.

It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.

In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc.

In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.

It should be noted that the present invention is not limited to the above-mentioned preferred embodiments, and those skilled in the art can obtain other products in various forms without departing from the spirit of the present invention, but any changes in shape or structure can be made within the scope of the present invention with the same or similar technical solutions as those of the present invention.