1. A truth value discovery method for annotation tasks under a multi-agent annotation scene is characterized by comprising the following steps:

step 1, collecting confidence: collecting the confidence degree of the label provided by the intelligent agent of the labeling service purchased by the buyer to the labeling task;

step 2, pretreatment: comparing the confidence of each label collected in the step 1 with a threshold value of the confidence given by a buyer, marking the label with the confidence higher than the threshold value of the confidence given by the buyer as true, and marking the label with the confidence lower than or equal to the threshold value of the confidence given by the buyer as false;

step 3, determining reliability parameters of the agent and confidence parameters of the labeling task:

determining the reliability parameters of the intelligent agent according to the following formula by using the label after the truth value is determined in the step 2:

in the above formula (1), ρ_wkThe reliability parameter of the agent is the ratio of the number of the marking tasks which are correctly answered by the agent to the total number of the answered marking tasks; x is the number of^tRepresenting the T-th annotation task in the whole annotation tasks T;providing the confidence level of the kth label of the t-th labeling task for the agent w;is the true value of the kth label; y is the confidence value range of the label: 0 or 1; the T represents that the current intelligent body does not provide confidence for the kth label of the tth labeling task;

taking the reliability parameter of the agent determined in the step 3 as a weight, and processing the collected tags according to the following formulas (2) and (3) to obtain a confidence coefficient parameter of the labeling task of the agent:

in the above-mentioned formulas (2) and (3),for the confidence coefficient distribution of the kth label, the initial value of the parameter is given by averaging random distribution;

updating the reliability parameter of the agent and the confidence coefficient parameter of the labeling task through iterative processing until the confidence coefficient parameter of the labeling task is determined to converge when the absolute value of the variation difference value of the confidence coefficient parameter of the labeling task obtained in the adjacent iteration turns is less than 0.0001, wherein the obtained confidence coefficient parameter of the labeling task is the confidence coefficient parameter of the final labeling task;

and 4, comparing the confidence coefficient parameter of the final labeling task obtained in the step 3 with a buyer-given threshold, and if the confidence coefficient parameter is higher than the buyer-given threshold, determining that the label of the labeling task is a correct label.

2. The method as claimed in claim 1, wherein the difference between the confidence coefficient parameters of the labeling tasks in adjacent iteration turns is a direct subtraction of the confidence coefficient parameters of the labeling tasks in adjacent iteration turns.

3. The method for finding the truth value of the labeling task in the multi-agent labeling scene as claimed in claim 1 or 2, wherein in the step 3 of the method, if the plurality of different labels of the labeling task are independently distributed labels, the confidence coefficient is separately obtained for the plurality of different labels of each labeling task, and the confidence coefficient of each label is as follows:

in the above formula (4), P (y)^t|A^tΨ) represents the confidence of each tag; k represents the total number of labels of each labeling task; k represents the serial number of the processed tag; k takes a value from 1 to K;represents the true value of the kth tag; w^tRepresenting a set of annotation tasks that have provided answers for the tth annotation task; a. the^tRepresenting the confidence set collected by the t-th labeling task; psi denotes the iterative parameter set, which is the set of reliability parameters of the agent and confidence parameters of the annotation task, and t denotes the sequence number of the annotation task.

4. The method for finding the truth of the labeling task in the multi-agent labeling scenario as claimed in claim 1 or 2, wherein in the step 3 of the method, if the multiple different labels of the labeling task are associated labels, the multiple different labels of each labeling task are jointly modeled and inferred by a Hidden Markov Model (HMM), and the confidence of each label is obtained as follows:

in the above formula (5), λ^tDenotes pi^t、A set of parameters of (a);a probability function representing whether two tags are included or not included at the same time;representing views in Hidden Markov Models (HMMs)Measuring the state to generate a probability function; pi^tRepresents a mean distribution of the K dimensions; k represents the total number of labels of each labeling task; k represents the serial number of the processed tag; k takes a value from 1 to K;represents the true value of the kth tag; t denotes the sequence number of the annotation task.

Background

Today's intelligent systems (i.e., AI systems) urgently need a large number of labeled data sets to build their own intelligent models. The unlabeled data set is usually completed by manual labeling, and due to the diversified quality and time-consuming characteristic, the requirement of rapid development of the current intelligent system cannot be met.

There are many API agent tagging services based on machine learning models and deep learning models, such as: MLaaS and BaidusI can provide richer and faster labeling functions. If an intelligent image retrieval AI system is considered to be constructed, a buyer of the annotation service needs to add a set of semantic tags to each data (e.g., image, voice, video) in an unmarked data set. Buyers can purchase a series of API agent tagging services on these agent tagging platforms and use them to generate as many tags as possible to describe each image, thereby improving the quality of search results.

Although various annotation agents may be used, buyers still have some difficulty in successfully completing data annotations. Such as: (1) the ability of a single agent to label is limited: a single annotating agent can typically only output partial tags based on certain aspects of the data. Thus, in many cases, such as image retrieval and text sentiment analysis, a series of tags are often required to obtain a broad understanding of the data. Also, for these annotation tasks, the more diverse the labels that are typically collected, the better the service provided. (2) The diversity and reliability of each tagged agent is unknown: while prior efforts have been made to provide increasingly powerful annotating agents to perform various tasks, the output of the annotating agents has not yet been necessarily correct. Each annotating agent outputs its own classification result and its confidence. It is even possible that different labeled agents output completely different results for the same task. (3) A large number of annotation tasks have high requirements on data processing capacity: the buyer generally needs to complete a large number of annotation tasks, which means that the buyer obtains even more annotation results of the annotation tasks, and after the collection of the label results is completed, the buyer will have a huge data processing challenge.

Disclosure of Invention

Aiming at the problems in the prior art, the invention aims to provide a truth value discovery method for labeling tasks under a multi-agent labeling scene, which can solve the problems that the labeling capacity of a single agent is limited, the diversity and the reliability of each labeled agent are unknown, the requirement of a large number of labeling tasks on data processing capacity is high and the like in the conventional agent labeling.

The purpose of the invention is realized by the following technical scheme:

the embodiment of the invention provides a truth value discovery method for a labeling task under a multi-agent labeling scene, which comprises the following steps:

step 1, collecting confidence: collecting the confidence degree of the label provided by the intelligent agent of the labeling service purchased by the buyer to the labeling task;

step 2, pretreatment: comparing the confidence of each label collected in the step 1 with a threshold value of the confidence given by a buyer, marking the label with the confidence higher than the threshold value of the confidence given by the buyer as true, and marking the label with the confidence lower than or equal to the threshold value of the confidence given by the buyer as false;

step 3, determining reliability parameters of the agent and confidence parameters of the labeling task:

determining the reliability parameters of the intelligent agent according to the following formula by using the label after the truth value is determined in the step 2:

in the above formula (1), ρ_wkThe reliability parameter of the agent is the ratio of the number of the marking tasks which are correctly answered by the agent to the total number of the answered marking tasks; x is the number of^tw represents the T-th labeling task in the whole labeling tasks T;providing the confidence level of the kth label of the t-th labeling task for the agent w;is the true value of the kth label; y is the confidence value range of the label: 0 or 1; the T represents that the current intelligent body does not provide confidence for the kth label of the tth labeling task;

taking the reliability parameter of the agent determined in the step 3 as a weight, and processing the collected tags according to the following formulas (2) and (3) to obtain a confidence coefficient parameter of the labeling task of the agent:

in the above-mentioned formulas (2) and (3),for the confidence coefficient distribution of the kth label, the initial value of the parameter is given by averaging random distribution;

updating the reliability parameter of the agent and the confidence coefficient parameter of the labeling task through iterative processing until the confidence coefficient parameter of the labeling task is determined to converge when the absolute value of the variation difference value of the confidence coefficient parameter of the labeling task obtained in the adjacent iteration turns is less than 0.0001, wherein the obtained confidence coefficient parameter of the labeling task is the confidence coefficient parameter of the final labeling task;

step 4, judging the labeling task: and (3) comparing the confidence coefficient parameter of the final labeling task obtained in the step (3) with a buyer-given threshold, and if the confidence coefficient parameter is higher than the buyer-given threshold, determining that the label of the labeling task is a correct label.

According to the technical scheme provided by the invention, the truth value discovery method for the labeling task in the multi-agent labeling scene provided by the embodiment of the invention has the following beneficial effects:

by jointly estimating the reliability of the agents and the truth value of the label of the labeling task, the reliability degree of each labeled agent can be estimated in the labeling scheme of more than two agents in the intelligent API labeling market, and the inference result and the confidence coefficient of a plurality of labels of the labeling task are output.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.

Fig. 1 is a flowchart of a reliability estimation and truth value discovery method in a multi-agent labeling scenario according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a first probabilistic graphical model and parameters of a method according to an embodiment of the present invention;

FIG. 3 is a second probabilistic graphical model and parameter diagram of the method according to the embodiment of the present invention;

fig. 4 is a schematic diagram of an output result of the agent annotation model of the method according to the embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the specific contents of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention. Details which are not described in detail in the embodiments of the invention belong to the prior art which is known to the person skilled in the art.

Referring to fig. 1, an embodiment of the present invention provides a truth value discovery method for a annotation task in a multi-agent annotation scenario, including:

step 1, collecting confidence: collecting the confidence degree of the label provided by the intelligent agent of the labeling service purchased by the buyer to the labeling task;

step 2, pretreatment: comparing the confidence of each label collected in the step 1 with a threshold value of the confidence given by a buyer, marking the label with the confidence higher than the threshold value of the confidence given by the buyer as true, and marking the label with the confidence lower than or equal to the threshold value of the confidence given by the buyer as false;

step 3, determining reliability parameters of the agent and confidence parameters of the labeling task:

determining the reliability parameters of the intelligent agent according to the following formula by using the label after the truth value is determined in the step 2:

in the above formula (1), ρ_wkThe reliability parameter of the agent is the ratio of the number of the marking tasks which are correctly answered by the agent to the total number of the answered marking tasks; x is the number of^tRepresenting the T-th annotation task in the whole annotation tasks T;providing the confidence level of the kth label of the t-th labeling task for the agent w;is the true value of the kth label; y is the confidence value range of the label: 0 or 1; the T represents that the current intelligent body does not provide confidence for the kth label of the tth labeling task;

taking the reliability parameter of the agent determined in the step 3 as a weight, and processing the collected tags according to the following formulas (2) and (3) to obtain a confidence coefficient parameter of the labeling task of the agent:

in the above-mentioned formulas (2) and (3),for the confidence coefficient distribution of the kth label, the initial value of the parameter is given by averaging random distribution;

updating the reliability parameter of the agent and the confidence coefficient parameter of the labeling task through iterative processing until the confidence coefficient parameter of the labeling task is determined to converge when the absolute value of the variation difference value of the confidence coefficient parameter of the labeling task obtained in the adjacent iteration turns is less than 0.0001, wherein the obtained confidence coefficient parameter of the labeling task is the confidence coefficient parameter of the final labeling task;

and 4, comparing the confidence coefficient parameter of the final labeling task obtained in the step 3 with a buyer-given threshold, and if the confidence coefficient parameter is higher than the buyer-given threshold, determining that the label of the labeling task is a correct label.

In the above method, the change difference of the confidence coefficient parameters of the labeling tasks in the adjacent iteration turns is a direct subtraction value of the confidence coefficient parameters of the labeling tasks obtained in the adjacent iteration turns.

In step 3 of the method, if the multiple different labels of the labeling task are independently distributed labels, confidence levels are separately obtained for the multiple different labels of each labeling task, and the confidence level of each label is as follows:

in the above formula (4), P (y)^t|A^tΨ) represents the confidence of each tag; k represents the total number of labels of each labeling task; k represents the serial number of the processed tag; k takes a value from 1 to K;represents the true value of the kth tag; w^tRepresenting a set of annotation tasks that have provided answers for the tth annotation task; a. the^tIs shown asA confidence set that t labeling tasks have been collected; Ψ represents an iterative parameter set, which is a set of reliability parameters of the agent and confidence parameters of the labeling task, and t represents a sequence number of the labeling task.

In step 3 of the above method, if the multiple different labels of the labeling task are associated labels, jointly modeling and inferring the multiple different labels of each labeling task by using a hidden markov model HMM to obtain a confidence level of each label as:

in the above formula (5), λ^tDenotes pi^t、A set of parameters of (a);a probability function representing whether two tags are included or not included at the same time;representing the probability function generated by the observation state in the hidden Markov model HMM, wherein the function can be obtained by the derivation of the formulas (2) and (3); pi^tRepresents a mean distribution of the K dimensions; k represents the total number of labels of each labeling task; k represents the serial number of the processed tag; k takes a value from 1 to K;represents the true value of the kth tag; t denotes the sequence number of the annotation task.

The confidence of each label, whether the labels are independently distributed or the labels are associated, is used in the formula (1) of the step 3 for replacementTo solve for the reliability parameters of the agent.

The method can estimate the reliability degree of each labeled intelligent agent and output the inference results and confidence degrees of a plurality of labels of the labeling tasks in the labeling scheme of more than two intelligent agents in the intelligent API labeling market by jointly estimating the reliability of the intelligent agents and the truth values of the labels of the labeling tasks.

The embodiments of the present invention are described in further detail below.

Examples

Referring to fig. 1, the present embodiment provides a method for discovering a truth value of a annotation task in a multi-agent annotation scenario, which is a method for providing a truth value of an annotation task for a service buyer, and includes the following steps:

step 1, collecting annotation data: for the annotation service of a plurality of agents purchased by a buyer, collecting and recording the confidence of the labels provided by the agents to the annotation task (see fig. 4);

step 2, pretreatment: preprocessing the confidence degree of the collected labels, setting the confidence degree of the intelligent agent answers higher than the confidence degree of a threshold value of the confidence degree given by the buyer to be 1, indicating that the labels labeled by the confidence degree are correct labels, and otherwise, setting the confidence degree to be 0, indicating that the labels labeled by the confidence degree are wrong labels; because the confidence degree of the label provided by the agent for the labeling task is the probability of whether the label is correct or not, and is usually less than 1 (see fig. 4), the confidence degree of the label can be simply divided into 1 or 0 through the processing of the step, which indicates that the label is correct or wrong, and is convenient for subsequent processing;

step 3, determining reliability parameters of the agent and confidence parameters of the labeling task:

determining the reliability parameters of the intelligent agent according to the following formula by using the label after the truth value is determined in the step 2:

in the above formula (1), ρ_wkThe reliability parameter of the agent is the ratio of the number of the marking tasks which are correctly answered by the agent to the total number of the answered marking tasks; x is the number of^tRepresenting the T-th annotation task in the whole annotation tasks T;providing the confidence level of the kth label of the t-th labeling task for the agent w;is the true value of the kth label; y is the confidence value range of the label: 0 or 1; t indicates that the current agent does not provide confidence for the kth tag of the tth labeling task (see fig. 2);

taking the reliability parameter of the agent determined in the step 3 as a weight, and processing the collected tags according to the following formulas (2) and (3) to obtain a confidence coefficient parameter of the labeling task of the agent:

in the above-mentioned formulas (2) and (3),for the confidence distribution of the kth label, the initial value of the parameter is given by averaging the random distribution (see fig. 3);

updating the reliability parameter of the agent and the confidence coefficient parameter of the labeling task through iterative processing until the confidence coefficient parameter of the labeling task is determined to converge when the absolute value of the variation difference value of the confidence coefficient parameter of the labeling task obtained in the adjacent iteration turns is less than 0.0001, wherein the obtained confidence coefficient parameter of the labeling task is the confidence coefficient parameter of the final labeling task;

step 4, judging the labeling task: and (3) comparing the confidence coefficient parameter of the final labeling task obtained in the step (3) with a buyer-given threshold, and if the confidence coefficient parameter is higher than the buyer-given threshold, determining that the label of the labeling task is a correct label.

In step 3 of the above method, if the multiple different labels of the labeling task are independently distributed labels, obtaining confidence levels separately for the multiple different labels of each labeling task, where the confidence level of each label is:

in the above formula (4), P (y)^t|A^tΨ) represents the confidence of each tag; k represents the total number of labels of each labeling task; k represents the serial number of the processed tag; k takes a value from 1 to K;represents the true value of the kth tag; w^tRepresenting a set of annotation tasks that have provided answers for the tth annotation task; a. the^tRepresenting the confidence set collected by the t-th labeling task; Ψ represents an iterative parameter set, which is a set of reliability parameters of the agent and confidence parameters of the labeling task, and t represents a sequence number of the labeling task. Derived confidence P (y) for each tag^t|λ^t，A^t) Substitution of in equation (1)And solving the reliability parameters of the intelligent agent.

In step 3 of the above method, if the multiple different labels of the labeling task are associated labels, jointly modeling and inferring the multiple different labels of each labeling task by using a Hidden Markov Model (HMM), and obtaining a confidence level of each label as:

in the above formula (5), λ^tDenotes pi^t、A set of parameters of (a);a probability function representing whether two tags are included or not included at the same time;representing an observation state generation probability function in a Hidden Markov Model (HMM); pi^tRepresents a mean distribution of the K dimensions; k represents the total number of labels of each labeling task; k represents the serial number of the processed tag; k takes a value from 1 to K;represents the true value of the kth tag; t denotes the sequence number of the annotation task. Derived confidence P (y) for each tag^t|λ^t，A^t) Substitution of in equation (1)And solving the reliability parameters of the intelligent agent.

Those of ordinary skill in the art will understand that: all or part of the processes of the methods for implementing the embodiments may be implemented by a program, which may be stored in a computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods as described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.