1. A Chinese entity identification method facing the mechanical chemical field comprises the following steps:

(1) extracting effective content by adopting short text preprocessing;

(2) adopting a Chinese word segmentation device optimized by a dictionary to carry out Chinese word segmentation and part-of-speech tagging to screen out nouns;

(3) a weighting function formed by the word frequency and the class priority function is used for weight calculation, and the short text highest weight key words are extracted based on rule type optimization weighting;

(4) and searching the context of the keyword with the highest weight, and simultaneously performing context expansion on the keyword based on the constructed directed probability state transition graph so as to form the target entity.

2. The method for recognizing Chinese entities in the field of mechanical and chemical engineering as claimed in claim 1, wherein: the short text preprocessing in the step (1) specifically comprises the following steps:

(1-1) text regularization; in order to process dirty data, the text regularization includes the extraction of pure Chinese and disregards the content in all brackets of short text, wherein the bracket content is a special annotation, which has no obvious effect on entity identification and is discarded;

(1-2) processing special words; the short text of the mechanical and chemical industry type contains unique characteristics including product names and product models, words of 'models', 'specifications', 'specification models' can help to quickly and directly locate the position of a target product entity, only the product name needs to be searched in the context after the position of the model is located, namely the context is directly used as a candidate keyword, consumption in the keyword extraction step can be reduced, all nouns in the short text do not need to be used as the candidate keyword to use a weight formula, or directly used as a rule formula in the step (3), and recognition accuracy can be improved.

3. The method for recognizing Chinese entities in the field of mechanical and chemical engineering as claimed in claim 1, wherein: and (3) the dictionary optimization in the step (2) is the dictionary optimization of the Chinese word segmentation device, and comprises the steps of adding stop words and self-defined dictionaries and updating a corpus according to recognition result statistics.

4. The method for recognizing Chinese entities in the field of mechanical and chemical engineering as claimed in claim 1, wherein: the weight function used for extracting the keywords in the step (3), the keyword extraction strategy is an enhanced improved version based on a TF-IDF keyword extraction strategy, the TF-IDF strategy is a common text classification statistical method, and the word frequency and reverse file frequency are used as weights, namely TF_i,j*idf_i(ii) a Wherein:

because of the inverse file frequency idf_iThe extraction and recognition efficiency in short text is extremely low, so the class priority function is used

Alternatively, the weighting function is optimized to tf_i,jF (t), wherein

t is the number of final keywords of the entity/the number of candidate keywords of the entity in the whole short text, i.e. the entity is the number of the final keywords

t represents the strength of the candidate keyword becoming the final word, the ideal range is [0,1], when t → 0, the word cannot become the final word, when t → 1, the word appears and is inevitably the final word, therefore, the range is enlarged by using function change through f (t) under the condition of not influencing the concave-convex property of the actual function of the function, the weight difference is enlarged, the effect of t is more favorably embodied, the core aim is to improve the hit probability of the final keyword, and the constant 1.01 is to prevent the situation that the divisor is 0 in the actual operation;

when t occurs in actual operation>1, range correction is performed to divide all t by t_maxThereby ensuring a range t<＝1。

5. The method for recognizing Chinese entities in the field of mechanical and chemical engineering as claimed in claim 1, wherein: the rule-based optimization weighting in the step (3) specifically comprises the following steps:

because f (t) is the fully supervised learning, when the training set is too disordered and f (t) has obvious errors, a short text may contain a plurality of final keywords in the actual situation, or t does not exist, which means that the word does not appear, and the class priority function is invalid; at the moment, rule-based optimization weighting is adopted, wherein the rule includes but is not limited to a certain characteristic generated by clustering by using a result set, including the relative position of a short text occupied by a product entity and the character size range of the product entity;

n rule formulas are set, and the occupation percentage of the result set conforming to the rule formulas is x₁,x₂,x₃,……x₁When x is_n>When 0.5, the rule formula is considered to have greater practicability, and the weight is added on the basis of the weight function

Wherein C is a constant, actually represents the proportion of the rule formula in the total weight, the default is 1/n, when the resource is enough, in order to achieve a better constant C, a better solution is obtained by using partial operation results based on gradient descent, and the characteristic value is that two parts are respectively a weight function and a weighting function; the weighting function is essentially the inverse of the sigmoid activation function, so in practice x is_n<0.5 may also participate in optimizing the weighting but the effect is not good, and it can be found that x is not good_n<0.5 is also negative, which is left out for reduced operating cost considerations.

6. The method for recognizing Chinese entities in the field of mechanical and chemical engineering as claimed in claim 1, wherein: the constructing of the directed probability state transition diagram in the step (4) specifically includes:

because the screened entity keywords are nouns, the initial states of the probability state conversion graph are all nouns n, all state statistics is carried out according to the context searched by the training set, and the probability converted into the states is calculated to form a directed graph, and the method comprises the following steps:

(4-1) combining the part of speech of the statistical result set; that is, the part-of-speech combination of the statistical target result itself, the product name may not be formed by only a single noun, but may be formed by a plurality of words of different parts-of-speech (such as nn, nnn, an, etc., where n represents a noun and a represents an adjective), so that all part-of-speech combinations of the statistical result set serve as the final state of the directed state probability transformation graph and serve as one of the rings of the probability calculation;

(4-2) counting a context combination having only one noun; a part of result set is composed of single nouns, original short text context part of speech combinations of the part are counted, the counted number of the part of speech combinations represents the number of the part of speech combinations which are not converted by the single nouns, and the part of speech combinations are used as another ring of probability calculation;

(4-3) carrying out probability transformation on the two statistical results to form a state transformation probability graph; the statistical number of words that can be converted into a certain part-of-speech combination is obtained from the step specification (4-1)Obtaining the statistical number of the part of speech combinations which can not be converted from the unit nouns from the step specification (4-2)According to a formula of probability calculation

The state transition probability can be calculated;

when the directed probability state transition graph is used, only traversal needs to be searched for comparing probability to judge whether context expansion is carried out, and theoretically, the state transition probability P is obtained_kIf the conversion rate is more than 0.5, the conversion is needed, but the conversion can be adjusted according to the actual situation, the accuracy of the conversion can be improved by increasing the parameter by 0.5, but the conversion quantity is also reduced, and the conversion success quantity is equal to the conversion quantityAnd (4) fitting a quadratic curve by using a small amount of operation results and combining a least square method under the condition that resources allow, and then taking out a probability parameter when the conversion success quantity is the maximum as a conversion standard.

7. The system for implementing the Chinese entity recognition method oriented to the mechanical chemical field in claim 1 is characterized in that: the system comprises a short text preprocessing module, a Chinese word segmentation and part-of-speech tagging module, a weight calculation and rule type optimization weighting module and a keyword search and expansion module which are sequentially connected, wherein:

the short text preprocessing module extracts effective contents by adopting short text preprocessing, and the short text preprocessing specifically comprises the following steps:

(1-1) text regularization; in order to process dirty data, the text regularization includes the extraction of pure Chinese and disregards the content in all brackets of short text, wherein the bracket content is a special annotation, which has no obvious effect on entity identification and is discarded;

(1-2) processing special words; the short text of the mechanical and chemical engineering type contains unique characteristics including product names and product models, the words of 'model', 'specification model' can help to quickly and directly locate the position of a target product entity, only the product name needs to be searched in the context after the position of the model is located, namely the context is directly used as a candidate keyword, the consumption in the keyword extraction step can be reduced, all nouns in the short text do not need to be used as the candidate keyword to use a weight formula, or directly used as a rule formula in the step (3), and the identification accuracy can be improved;

the Chinese word segmentation and part-of-speech tagging module adopts a Chinese word segmenter subjected to dictionary optimization to perform Chinese word segmentation and part-of-speech tagging to screen out nouns, wherein the dictionary optimization is dictionary optimization of the Chinese word segmenter and comprises adding stop words and a user-defined dictionary and counting and updating a corpus according to recognition results;

the weight calculation and rule-based optimization weighting module utilizes a weight function consisting of word frequency and class priority functions as weight calculation and extracts the highest weight keyword of the short text based on rule-based optimization weighting;

the keyword searching and expanding module searches the context of the keyword with the highest weight and simultaneously expands the context of the keyword based on the constructed directed probability state transition graph so as to form a target entity, wherein the constructed directed probability state transition graph specifically comprises the following steps:

because the screened entity keywords are nouns, the initial states of the probability state conversion graph are all nouns n, all state statistics is carried out according to the context searched by the training set, and the probability converted into the states is calculated to form a directed graph, and the method comprises the following steps:

(4-1) combining the part of speech of the statistical result set; that is, the part-of-speech combination of the statistical target result itself, the product name may not be formed by only a single noun, but may be formed by a plurality of words of different parts-of-speech (such as nn, nnn, an, etc., where n represents a noun and a represents an adjective), so that all part-of-speech combinations of the statistical result set serve as the final state of the directed state probability transformation graph and serve as one of the rings of the probability calculation;

(4-2) counting a context combination having only one noun; a part of result set is composed of single nouns, original short text context part of speech combinations of the part are counted, the counted number of the part of speech combinations represents the number of the part of speech combinations which are not converted by the single nouns, and the part of speech combinations are used as another ring of probability calculation;

(4-3) carrying out probability transformation on the two statistical results to form a state transformation probability graph; the statistical number of words that can be converted into a certain part-of-speech combination is obtained from the step specification (4-1)Obtaining the statistical number of the part of speech combinations which can not be converted from the unit nouns from the step specification (4-2)According to a formula of probability calculation

The state transition probability can be calculated;

when the directed probability state transition graph is used, only traversal needs to be searched for comparing probability to judge whether context expansion is carried out, and theoretically, the state transition probability P is obtained_kIf the conversion is more than 0.5, the conversion is needed, but the appropriate amount of adjustment can be performed according to the actual situation, the accuracy of the conversion can be improved by increasing the parameter 0.5, but the number of conversions is also reduced at this time, the number of successful conversions is equal to the number of conversions.

8. The system of claim 7, wherein: the weight calculation and rule type optimization weighting module extracts the weight function used by the key words, the key word extracting strategy is an enhanced improved version based on a TF-IDF key word extracting strategy, the TF-IDF strategy is a common text classification statistical method, and the word frequency and reverse file frequency is used as weighting, namely TF_i,j*idf_i(ii) a Wherein:

because of the inverse file frequency idf_iThe extraction and recognition efficiency in short text is extremely low, so the class priority function is used

Alternatively, the weighting function is optimized to tf_i,jF (t), wherein

t is the number of final keywords of the entity/the number of candidate keywords of the entity in the whole short text, i.e. the entity is the number of the final keywords

t represents the strength of the candidate keyword becoming the final word, the ideal range is [0,1], when t → 0, the word cannot become the final word, when t → 1, the word appears and is inevitably the final word, therefore, the range is enlarged by using function change through f (t) under the condition of not influencing the concave-convex property of the actual function of the function, the weight difference is enlarged, the effect of t is more favorably embodied, the core aim is to improve the hit probability of the final keyword, and the constant 1.01 is to prevent the situation that the divisor is 0 in the actual operation;

when t occurs in actual operation>1, range correction is performed to divide all t by t_maxThereby ensuring a range t<＝1；

The rule-based optimization weighting specifically includes:

because f (t) is the fully supervised learning, when the training set is too disordered and f (t) has obvious errors, a short text may contain a plurality of final keywords in the actual situation, or t does not exist, which means that the word does not appear, and the class priority function is invalid; at the moment, rule-based optimization weighting is adopted, wherein the rule includes but is not limited to a certain characteristic generated by clustering by using a result set, including the relative position of a short text occupied by a product entity and the character size range of the product entity;

n rule formulas are set, and the occupation percentage of the result set conforming to the rule formulas is x₁,x₂,x₃,……x₁When x is_n>When 0.5, the rule formula is considered to have greater practicability, and the weight is added on the basis of the weight function

Wherein C is a constant, actually a generationThe proportion of a rule formula in the total weight is shown, the default is 1/n, when the resources are enough, in order to achieve a better constant C, a part of operation results are used for obtaining a better solution based on gradient descent, and the characteristic value is that two parts are respectively a weight function and a weighting function; the weighting function is essentially the inverse of the sigmoid activation function, so in practice x is_n<0.5 may also participate in optimizing the weighting but the effect is not good, and it can be found that x is not good_n<0.5 is also negative, which is left out for reduced operating cost considerations.

Background

With the popularization of computers and the wide introduction of various electronic texts, a great deal of information brings a serious challenge to the information acquisition and processing of people, people urgently need some automatic tools to help the information data processing, information extraction, information inspection, machine translation and other technologies, and the most fundamental and important problem is named entity identification, and the quality of named entity identification directly influences a series of subsequent data operations.

[1] In the development process of named entity identification, the method based on the rules has strong field, and lacks robustness and portability; although the statistical method has certain objectivity, human language use is not a simple random process, and the application range of the statistical model is limited due to serious data sparsity and system processing capacity limitation. The combined use of statistical models and rule knowledge will have better trainability and adaptability, and the cost for maintaining performance is much lower, which is a future development trend of NER, see "named entity recognition research [ J ]" in japan, computer science, 2005(04) ": 44-48.

In recent years, the NER technology of the Chinese text is more and more mature, and a large number of Chinese word segmentation tools are mature, for example, open source technologies such as SnowNLP, Thulac, HanLP, LTP, CoreNLP and the like in python can process the Chinese text to a certain extent, but most of Chinese word segmenters have low recognition efficiency on named entities in a specific field, for example, the recognition accuracy can only reach about 50% in the field of mechanization.

Disclosure of Invention

The invention aims to overcome the defects in the prior art, provides a Chinese entity identification method and system for the field of mechanical and chemical engineering, and realizes product entity identification with high accuracy.

The invention relates to a Chinese entity identification method facing the mechanical and chemical engineering field, which comprises the following steps:

(1) extracting effective content by adopting short text preprocessing;

(2) adopting a Chinese word segmentation device optimized by a dictionary to carry out Chinese word segmentation and part-of-speech tagging to screen out nouns;

(3) a weighting function formed by the word frequency and the class priority function is used for weight calculation, and the short text highest weight key words are extracted based on rule type optimization weighting;

(4) searching the context of the keyword with the highest weight, and performing context expansion of the keyword based on the constructed directed probability state transformation graph so as to form a target entity; the whole flow chart is as shown in the attached figure 1.

Further, the short text preprocessing described in step (1) specifically includes:

(1-1) text regularization. The short text of the mechanization industry type can contain a large amount of disordered dirty data, the dirty data mainly comprises some irregular texts or non-Chinese data which can greatly affect the normal operation of the data, the text regularization is a common and effective method for processing the dirty data, particularly for the short text of Chinese, the text regularization mainly comprises the extraction of pure Chinese and ignores the content in all brackets of the short text, wherein the bracket content is basically a special annotation, and the method has no obvious effect on entity identification and is discarded.

And (1-2) processing the special words. The short text of the mechanical and chemical engineering type always contains some unique characteristics, and mainly comprises a product name, a product model and the like, so that the words such as the model, the specification model and the like can help to quickly and directly position the position of a target product entity, for example, the product name is always around the product model, and after the position of the model is positioned, the product name is only needed to be searched in the context, namely, the context is directly used as a candidate keyword, the consumption in the keyword extraction step can be reduced, a weight formula is not needed to be used by taking all nouns in the short text as candidate keywords, the rule that the product name is always around the product model can be subjected to regularization so as to be directly used as the rule formula in the step (3), and the identification accuracy can be improved.

The dictionary optimization in the step (2) specifically comprises the following steps:

and (2-1) corpus updating. Most Chinese word segmenters are provided with corpora, but most of the corpora provided by the Chinese word segmenters can only be used for analyzing common daily sentences, the analysis capability of special words in a certain field is weak, the corpora is the basis of Chinese word segmentation and part of speech tagging, the basic accuracy rate of Chinese named entity recognition is directly determined, aiming at the field of mechanical and chemical engineering, the basic recognition accuracy rate can be effectively improved by using the adaptive corpora, and certain promotion effect on accurate recognition can be achieved by adding stop words, and the above methods are common methods for Chinese named entity recognition.

And (2-2) statistically updating the corpus according to the recognition result. In the process of program operation, high-frequency product nouns found by statistics can be used for expanding a corpus and effectively improving the accuracy of product entity identification.

The weighting function in the step (3) is:

F(t)＝tf_i，j*f(t) (1)

wherein:

the function composition analysis is as follows:

the keyword extraction strategy is an enhanced improved version based on a TF-IDF keyword extraction strategy, the TF-IDF strategy is a common text classification statistical method, and the word frequency and reverse file frequency are used as weights, namely TF_i，j*idf_i. Wherein:

because of the inverse file frequency idf_iExtraction in short textClassification efficiency is very low, so class priority function is used

Alternatively, the weighting function is optimized to tf_i，jF (t), wherein

t is the number of final keywords of the entity/the number of candidate keywords of the entity in the whole short text, i.e. the entity is the number of the final keywords

t represents the strength of the candidate keyword becoming the final word, the ideal range is [0,1], when t → 0, the word cannot become the final word, when t → 1, the word appears and is inevitably the final keyword, therefore, the range is enlarged by using function change through f (t) under the condition of not influencing the actual function and the concave-convex property of the function, the weight difference can be enlarged, the effect of t is more favorably embodied, the core goal is to improve the hit probability of the final keyword, and the constant 1.01 is to prevent the situation that the divisor is 0 in the actual operation and can be properly adjusted.

In actual operation, t > 1 may occur, for example, a plurality of identical candidate keywords appear in a short text, and range correction may be performed at this time, where range correction is performed by dividing all t by t_maxThus ensuring a range t < 1. The weight can greatly improve the final keyword hit probability of the product entity.

The rule-based optimization weighting in the step (3) is as follows: because f (t) is the fully supervised learning, when the training set is too disordered, f (t) has obvious errors, and as a practical situation, a short text may include a plurality of final keywords and the like; or t is absent, which means that the word has not occurred, when the class priority function is invalid; at this time, a rule-based optimization weighting may be adopted, where the rule-based optimization weighting includes, but is not limited to, a certain feature generated by clustering the result set, such as a relative position of the product entity occupying the short text, a character size range of the product entity, and the like.

N rule formulas are set, and the occupation percentage of the result set conforming to the rule formulas is x₁，x₂，x₃，...x₁When x is_nIf the weight is more than 0.5, the rule formula is considered to have greater practicability, and the weight is added on the basis of the weight function

Wherein C is a constant and actually represents the proportion of the rule formula in the total weight, the default is 1/n, when the resource is enough, in order to achieve a better constant C, a better solution is obtained by using partial operation results based on gradient descent, and the characteristic value is that the two parts are respectively a weight function and a weighting function. The weighting function is essentially the inverse of the sigmoid activation function, so in practice x is_nLess than 0.5 may also participate in optimizing the weighting but the effect is not good, and it can be found that x is not good_nBelow 0.5 the weighting function is negative and is left out for reduced operating cost considerations.

The rule-based optimization weighting has an unobvious effect when the training set is better or the class priority function has no obvious error, but has an excellent effect when the class priority function has a larger error or cannot function due to disorder of the training set, and is taken as a complementary means at the moment.

The step (4) of constructing the directed probability state transition graph comprises the following steps:

the entity identification of the mechanical and chemical products often comprises combined words, and a single noun keyword often cannot become a final target, so that when the number of words of the keyword with the highest weight is less, the context of the keyword needs to be searched, and whether the context is expanded or not is judged according to the part of speech and the directed probability state conversion diagram.

Constructing a directed probability state conversion graph: because the screened entity keywords are nouns, the initial states of the probability state conversion graph are all nouns n, all state statistics is carried out according to the context searched by the training set, the probability converted into the states is calculated to form a directed graph, the detailed steps are constructed as shown in fig. 2, and part of the steps are explained as follows:

and (4-1) combining the part of speech of the statistical result set. That is, the part-of-speech combination of the statistical target result itself, the product name may not be formed by only a single noun, but may be formed by a plurality of words of different parts-of-speech (e.g., nn, nnn, an, etc., where n represents a noun and a represents an adjective), and thus all part-of-speech combinations of the statistical result set serve as the final state of the directed state probability transformation graph and serve as one of the rings of the probability calculation.

(4-2) statistics of context combinations with only one noun. A part of result set is composed of single nouns, the original short text context part of speech combination of the part is counted, the counted number of the part of speech combination represents the number of the part of speech combination which is not converted by the single nouns, and the number is used as another ring of probability calculation.

And (4-3) carrying out probability transformation on the two statistical results to form a state transformation probability chart. The statistical number of words that can be converted into a certain part-of-speech combination is obtained from the step specification (4-1)Obtaining the statistical number of the part of speech combinations which can not be converted from the unit nouns from the step specification (4-2)According to a formula of probability calculation

The state transition probability can be calculated.

When the directed probability state transition graph is used, only traversal needs to be searched for comparing probability to judge whether context expansion is carried out, and theoretically, the state transition probability P is obtained_kIf the conversion rate is more than 0.5, the conversion is needed, but the appropriate adjustment can be carried out according to the actual situation, and the accuracy of the conversion can be improved by increasing the parameter by 0.5, but the conversion rate can also be improved at the momentAnd reducing the number of conversions, wherein the number of successful conversions is the number of conversions and the accuracy of the conversions, fitting a quadratic curve by using a small number of operation results and combining a least square method under the condition of resource permission, and taking a probability parameter when the number of successful conversions is the maximum as a conversion standard.

The invention discloses a system for implementing a Chinese entity identification method oriented to the field of mechanical and chemical engineering, which is characterized in that: the system comprises a short text preprocessing module, a Chinese word segmentation and part-of-speech tagging module, a weight calculation and rule type optimization weighting module and a keyword search and expansion module which are sequentially connected.

The invention has the advantages that: the method realizes high-accuracy identification of the Chinese short text entity in the mechanical and chemical engineering field under the condition of permission of both supervised and unsupervised learning, and has higher robustness and expansibility.

Drawings

FIG. 1 is a general flow diagram of the process of the present invention.

FIG. 2 is a flow chart of the construction of the directed probability state transition graph of the present invention.

FIG. 3 is a detailed flow chart of an embodiment of the method of the present invention.

Detailed Description

The following will further explain the overall process steps of the scheme by taking a part of short texts of the mechanical and chemical engineering class as an example with reference to the accompanying drawings, and a detailed implementation flow chart is shown in fig. 3.

Step 1: traversing a large amount of short texts of the mechanical and chemical engineering class for the first time, and preprocessing each short text to obtain a preprocessed text; and (4) stopping importing a word stock and a user-defined word stock to the Chinese word segmentation device, wherein the two word stocks need to be correspondingly established aiming at the field of mechanical and chemical engineering.

Step 2: traversing the preprocessed short text, performing word segmentation and part-of-speech recognition tagging by using a Chinese word segmentation device, and extracting nouns; the keyword extraction strategy is used to calculate the weight corresponding to each noun, and the rule type optimization weighting can be used in the period. The data required by the word frequency and the class priority function of the keyword extraction strategy are extracted according to a dictionary, the data in the dictionary generally consists of a large number of similar tuples (words, parts of speech and word frequency), the extraction mode is adjusted according to a Chinese word segmentation device, and different Chinese word segmentation devices can be different. The rule formula is obtained by taking the features extracted by the clustering algorithm as a rule formula, taking the character size range as a rule formula, manually observing a certain rule as a rule formula and the like, and the purpose is to use the rule formula with the accuracy rate of more than 50% as rule formula weighting as far as possible, and the rule formula with the higher accuracy rate is used as weighting effect better. The keyword extraction strategy uses the extracted keyword with the highest weight as the input of the next step.

And step 3: and judging whether the final keyword needs to be subjected to context expansion according to the directed state probability transformation graph. Firstly, establishing a directed state probability transformation graph according to training set statistical data, then searching the part of speech of a context to form a combined part of speech aiming at a keyword with a small character size, and finally traversing the directed state probability transformation graph to judge whether context combination is carried out to form a final word.

And 4, step 4: the frequency of counting and finding the appearing combined words is high, namely the combined words are added into a dictionary to update the dictionary, and the combined words are not required to be recombined when the same combined words are encountered in the next operation, so that the recognition efficiency is improved.

The invention relates to a system for implementing a Chinese entity identification method facing the field of mechanical and chemical engineering, which comprises a short text preprocessing module, a Chinese word segmentation and part-of-speech tagging module, a weight calculation and rule type optimization weighting module and a keyword search and expansion module which are sequentially connected, wherein:

the short text preprocessing module extracts effective content by adopting short text preprocessing, wherein the short text preprocessing specifically comprises the following steps:

(1-1) text regularization; in order to process dirty data, the text regularization includes the extraction of pure Chinese and disregards the content in all brackets of short text, wherein the bracket content is a special annotation, which has no obvious effect on entity identification and is discarded;

(1-2) processing special words; the short text of the mechanical and chemical engineering type contains unique characteristics including product names and product models, the words of 'model', 'specification model' can help to quickly and directly locate the position of a target product entity, only the product name needs to be searched in the context after the position of the model is located, namely the context is directly used as a candidate keyword, the consumption in the keyword extraction step can be reduced, all nouns in the short text do not need to be used as the candidate keyword to use a weight formula, or directly used as a rule formula in the step (3), and the identification accuracy can be improved;

the Chinese word segmentation and part-of-speech tagging module adopts a Chinese word segmenter subjected to dictionary optimization to perform Chinese word segmentation and part-of-speech tagging to screen out nouns, wherein the dictionary optimization is dictionary optimization of the Chinese word segmenter and comprises adding stop words and a user-defined dictionary and counting and updating a corpus according to recognition results;

the weight calculation and rule-based optimization weighting module utilizes a weight function consisting of word frequency and class priority functions as weight calculation and extracts the highest weight keyword of the short text based on rule-based optimization weighting;

the weight calculation and rule type optimization weighting module extracts the weight function used by the key words, the key word extracting strategy is an enhanced improved version based on a TF-IDF key word extracting strategy, the TF-IDF strategy is a common text classification statistical method, and the word frequency and reverse file frequency is used as weighting, namely TF_i，j*idf_i(ii) a Wherein:

because of the inverse file frequency idf_iThe extraction and recognition efficiency in short text is extremely low, so the class priority function is used

Alternatively, the weighting function is optimized to tf_i，jF (t), itIn

t is the number of final keywords of the entity/the number of candidate keywords of the entity in the whole short text, i.e. the entity is the number of the final keywords

t represents the strength of the candidate keyword becoming the final word, the ideal range is [0,1], when t → 0, the word cannot become the final word, when t → 1, the word appears and is inevitably the final word, therefore, the range is enlarged by using function change through f (t) under the condition of not influencing the concave-convex property of the actual function of the function, the weight difference is enlarged, the effect of t is more favorably embodied, the core aim is to improve the hit probability of the final keyword, and the constant 1.01 is to prevent the situation that the divisor is 0 in the actual operation;

when t is larger than 1 in actual operation, range correction is carried out, and the range correction is that all t is divided by t_maxSo as to ensure that the range t is less than 1;

the rule-based optimization weighting specifically includes:

because f (t) is the fully supervised learning, when the training set is too disordered and f (t) has obvious errors, a short text may contain a plurality of final keywords in the actual situation, or t does not exist, which means that the word does not appear, and the class priority function is invalid; at the moment, rule-based optimization weighting is adopted, wherein the rule includes but is not limited to a certain characteristic generated by clustering by using a result set, including the relative position of a short text occupied by a product entity and the character size range of the product entity;

n rule formulas are set, and the occupation percentage of the result set conforming to the rule formulas is x₁，x₂，x₃，...x₁When x is_nIf the weight is more than 0.5, the rule formula is considered to have greater practicability, and the weight is added on the basis of the weight function

Wherein C is a constant, actually represents the proportion of the rule formula in the total weight, the default is 1/n, when the resource is enough, in order to achieve a better constant C, a better solution is obtained by using partial operation results based on gradient descent, and the characteristic value is that two parts are respectively a weight function and a weighting function; the weighting function is essentially the inverse of the sigmoid activation function, so in practice x is_nLess than 0.5 may also participate in optimizing the weighting but the effect is not good, and it can be found that x is not good_nBelow 0.5 the weighting function is negative and is left out for reduced operating cost considerations.

The keyword searching and expanding module searches the context of the keyword with the highest weight and simultaneously expands the context of the keyword based on the constructed directed probability state transition graph so as to form a target entity, wherein the constructed directed probability state transition graph specifically comprises the following steps:

because the screened entity keywords are nouns, the initial states of the probability state conversion graph are all nouns n, all state statistics is carried out according to the context searched by the training set, and the probability converted into the states is calculated to form a directed graph, and the method comprises the following steps:

(4-1) combining the part of speech of the statistical result set; that is, the part-of-speech combination of the statistical target result itself, the product name may not be formed by only a single noun, but may be formed by a plurality of words of different parts-of-speech (such as nn, nnn, an, etc., where n represents a noun and a represents an adjective), so that all part-of-speech combinations of the statistical result set serve as the final state of the directed state probability transformation graph and serve as one of the rings of the probability calculation;

(4-2) counting a context combination having only one noun; a part of result set is composed of single nouns, original short text context part of speech combinations of the part are counted, the counted number of the part of speech combinations represents the number of the part of speech combinations which are not converted by the single nouns, and the part of speech combinations are used as another ring of probability calculation;

(4-3) carrying out probability transformation on the two statistical results to form a state transformation probability graph; the statistical number of words that can be converted into a certain part-of-speech combination is obtained from the step specification (4-1)Obtaining the statistical number of the part of speech combinations which can not be converted from the unit nouns from the step specification (4-2)According to a formula of probability calculation

The state transition probability can be calculated;

when the directed probability state transition graph is used, only traversal needs to be searched for comparing probability to judge whether context expansion is carried out, and theoretically, the state transition probability P is obtained_kIf the conversion is more than 0.5, the conversion is needed, but the appropriate amount of adjustment can be performed according to the actual situation, the accuracy of the conversion can be improved by increasing the parameter 0.5, but the number of conversions is also reduced at this time, the number of successful conversions is equal to the number of conversions.

The embodiments described in this specification are merely illustrative of implementations of the inventive concept and the scope of the present invention should not be considered limited to the specific forms set forth in the embodiments but rather by the equivalents thereof as may occur to those skilled in the art upon consideration of the present inventive concept.