1. An error correction method in automatic marking of individual cells in a marrow image, comprising:

marking the marrow pathological image by using a cell marking model to obtain an initial marking result; the initial marking result comprises initial cell types and initial marking frame position information;

performing secondary confirmation on the initial marking result by using a class error corrector to obtain a classification score of the cell and the cell type after error correction, and correcting the mark with classification error to obtain a marking result after primary error correction;

comparing the constraint relation between each cell marker frame and the corresponding cell size in the marker result after the first correction by using a positioning error corrector, judging whether a positioning error exists or not, and correcting the marker of the positioning error to obtain a marker result after the second correction;

and respectively calculating the overlapping rate of the mark frame areas of any two cells in the mark result after the second error correction by using a fuzzy confirmer, judging whether a fuzzy mark problem exists according to the ratio of the overlapping rate to the overlapping threshold value and the class of the mark frame, and correcting the fuzzy mark problem to obtain the final error corrected mark result.

2. The method of claim 1, wherein the initial labeling result is obtained by labeling a pathological image of bone marrow with a trained cell labeling model.

3. The method of claim 1, wherein the constraints on the sizes of the individual cells are obtained by clustering the data of the individual bone marrow cells.

4. The method according to claim 2, wherein the constraints of cell size include aspect ratio constraints and area constraints of the category.

5. The method according to claim 1, wherein if the marker frame for correcting the cell type does not satisfy the constraint relation of the corresponding cell size, the marker frame is deleted from the detection result.

6. The method according to claim 1, wherein if the overlapping rate is greater than the overlap threshold, further determining whether the mark frames have the same category, and if so, directly deleting the mark frame with the lower classification score; if the two mark frames are different, adding the category and the position information of the two mark frames into the cell list to be confirmed, and deleting the mark frame with lower score.

7. The method of claim 1, wherein the overlap threshold is dynamically adjusted during the error correction process until an optimal threshold is obtained.

8. An error correction system in automatic labeling of individual cells in a bone marrow image, comprising:

the automatic image marking module is used for marking the pathological image of the bone marrow by using the cell marking model to obtain an initial marking result; the initial marking result comprises initial cell types and initial marking frame position information;

the classification error correction confirmation module is used for performing secondary confirmation on the initial marking result by using the classification error corrector to obtain the classification score of the cell and the cell type after error correction, and correcting the mark with classification error to obtain the marking result after primary error correction;

the positioning error correction confirmation module is used for comparing the constraint relation between each cell marker frame in the marking result after the first correction and the corresponding cell size by using the positioning error corrector, judging whether a positioning error exists or not, and correcting the marker of the positioning error to obtain a marking result after the second correction;

and the fuzzy error correction confirmation module is used for respectively calculating the overlapping rate of the mark frame areas of any two cells in the mark result after the second error correction by using the fuzzy confirmer, judging whether a fuzzy mark problem exists according to the ratio of the overlapping rate to the overlapping threshold value and the class of the mark frame, and correcting the fuzzy mark problem to obtain the final error corrected mark result.

9. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the steps of the method for error correction in the automatic labeling of individual cells of an image of bone marrow according to any one of claims 1 to 7.

10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the method for error correction in automatic labeling of individual cells of a bone marrow image according to any one of claims 1 to 7.

Background

The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.

The morphological analysis and classification statistics of various cells in the bone marrow pathological images are helpful for assisting pathologists to accurately diagnose acute leukemia (ALL or AML), myelodysplastic syndrome (MDS), multiple myeloma and other blood system diseases. The bone marrow cell automatic marking model based on deep learning is established, so that the initial marking of the bone marrow cells can be realized, but the error is still larger. The method for correcting errors in the automatic marking of the marrow cells has important significance for optimizing pathological data sets, establishing accurate marrow cell detection models and further realizing an auxiliary pathological analysis system.

The existing bone marrow cell public data set is not high in quality. For example, the ALL-IDB1 database, which consists of 109 bone marrow pathology images of 2592 × 1944 and 260 bone marrow pathology images of 257 × 257, can only be used for identifying primitive cells in bone marrow cells, and cannot be directly applied to the differential counting and quantification percentages of the bone marrow cells; the bone marrow pathology image data set provided by the image bank online website of the american society for hematology can only be used to identify normal and abnormal pathology images. The inventor finds that the reason for this phenomenon is mainly that the building of the bone marrow cell data set is highly dependent on the manual marking of the pathologist, and consumes a great deal of time and energy of the pathologist, and the marking quality is easily influenced by the subjective effect of the pathologist. In this case, it is difficult to improve the accuracy of the bone marrow cell automatic labeling model.

Disclosure of Invention

In order to solve the technical problem of large error of an automatic marking model of marrow cells in the background technology, the invention provides an error correction method and an error correction system in automatic marking of individual cells of a marrow image, which can improve the quality of a marrow pathological data set and the accuracy of an automatic marking system, further replace manual marking and realize standardized management.

In order to achieve the purpose, the invention adopts the following technical scheme:

the first aspect of the invention provides an error correction method in automatic marking of individual cells of a bone marrow image.

An error correction method in automatic labeling of individual cells in a bone marrow image, comprising:

marking the marrow pathological image by using a cell marking model to obtain an initial marking result; the initial marking result comprises initial cell types and initial marking frame position information;

performing secondary confirmation on the initial marking result by using a class error corrector to obtain a classification score of the cell and the cell type after error correction, and correcting the mark with classification error to obtain a marking result after primary error correction;

comparing the constraint relation between each cell marker frame and the corresponding cell size in the marker result after the first correction by using a positioning error corrector, judging whether a positioning error exists or not, and correcting the marker of the positioning error to obtain a marker result after the second correction;

and respectively calculating the overlapping rate of the mark frame areas of any two cells in the mark result after the second error correction by using a fuzzy confirmer, judging whether a fuzzy mark problem exists according to the ratio of the overlapping rate to the overlapping threshold value and the class of the mark frame, and correcting the fuzzy mark problem to obtain the final error corrected mark result.

A second aspect of the invention provides an error correction system in automatic labeling of individual cells in a bone marrow image.

An error correction system in automatic labeling of individual cells in a bone marrow image, comprising:

the automatic image marking module is used for marking the pathological image of the bone marrow by using the cell marking model to obtain an initial marking result; the initial marking result comprises initial cell types and initial marking frame position information;

the classification error correction confirmation module is used for performing secondary confirmation on the initial marking result by using the classification error corrector to obtain the classification score of the cell and the cell type after error correction, and correcting the mark with classification error to obtain the marking result after primary error correction;

the positioning error correction confirmation module is used for comparing the constraint relation between each cell marker frame in the marking result after the first correction and the corresponding cell size by using the positioning error corrector, judging whether a positioning error exists or not, and correcting the marker of the positioning error to obtain a marking result after the second correction;

and the fuzzy error correction confirmation module is used for respectively calculating the overlapping rate of the mark frame areas of any two cells in the mark result after the second error correction by using the fuzzy confirmer, judging whether a fuzzy mark problem exists according to the ratio of the overlapping rate to the overlapping threshold value and the class of the mark frame, and correcting the fuzzy mark problem to obtain the final error corrected mark result.

A third aspect of the invention provides a computer-readable storage medium.

A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method for error correction in automatic labeling of individual cells of an image of bone marrow as described above.

A fourth aspect of the invention provides a computer apparatus.

A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method for error correction in automatic marking of individual cells of an image of bone marrow as described above when executing the program.

Compared with the prior art, the invention has the beneficial effects that:

the invention provides an error correction method and an error correction system in automatic marking of individual cells of a marrow image, which solve the problems of class errors, positioning errors and fuzzy marks possibly existing in primary marking, adopt classification error correction confirmation, positioning error correction confirmation and fuzzy error correction confirmation to correct the problems of classification errors, positioning errors and fuzzy marks in sequence, obtain a more accurate result by automatic error correction on the basis of automatic marking, enable the marking process of marrow cells to be more automatic and accurate, and greatly improve the marking efficiency of marrow cells and the accuracy of an automatic marking model.

Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the invention and together with the description serve to explain the invention and not to limit the invention.

FIG. 1 is a flowchart of an error correction method in automatic labeling of individual cells in a bone marrow pathology image according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of the application of error correction in the automatic labeling of individual cells in a bone marrow pathology image according to the present invention.

Fig. 3 is a schematic diagram of a classification error correction validation process according to an embodiment of the present invention.

Fig. 4 is a schematic diagram of a positioning error correction confirmation process according to an embodiment of the present invention.

FIG. 5 is a flow chart illustrating a fuzzy error correction validation process according to an embodiment of the present invention.

FIG. 6 is a schematic structural diagram of an error correction system for automatic labeling of individual cells in a bone marrow pathology image according to an embodiment of the present invention.

Detailed Description

The invention is further described with reference to the following figures and examples.

It is to be understood that the following detailed description is exemplary and is intended to provide further explanation of the invention as claimed. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.

The invention provides an error correction method and system in marrow image individual cell automatic marking by establishing a small artificial marrow cell marking data set and training a detection model taking marrow cells as objects to realize automatic preliminary marking based on a detection result, wherein the problems of category errors, positioning errors and fuzzy marking may exist in the preliminary marking. The specific technical scheme of the invention is explained in detail by adopting the following embodiments:

example one

As shown in fig. 1, this embodiment provides an error correction method in automatic labeling of individual cells in a bone marrow image, which specifically includes the following steps:

step S101: marking the marrow pathological image by using a cell marking model to obtain an initial marking result; the initial labeling result includes initial cell type and initial labeling frame position information.

In this embodiment, the initial labeling result is obtained by labeling the bone marrow pathology image using a trained cell labeling model.

It can be understood that the cell marker model is an existing network model, the structure of which can be specifically selected according to actual situations, and parameters of the cell marker model can be obtained by automatically marking and training positions of frames and types of the frames, namely cell types.

The process of initial labeling can refer to steps 1 and 2 in fig. 2:

in step 1, a bone marrow pathology image is imported. Bone marrow pathology image formats include, but are not limited to, PNG format, JPEG format, TIFF format, and the like. The person skilled in the art can import a single picture or a plurality of pictures according to his own needs without limiting the size of the picture.

In step 2, the bone marrow pathological image is marked by using the trained cell marking model, and the cell type and position information is recorded as an initial marking result.

Step S102: the class error corrector is used to perform secondary confirmation on the initial labeling result to obtain the classification score of the cell and the cell type after error correction, and the label with classification error is corrected to obtain the labeling result after first error correction, as shown in fig. 3.

This step of this embodiment corresponds to step 3 in fig. 2, and the result obtained in step S2 is automatically corrected. And (5) performing secondary confirmation on the marking information of the S2, including the cell type and the coordinates of the cell marking frame, and automatically judging whether a classification error exists. And (3) taking the trained type error corrector such as VGGnet, inclusion _ v3, Overfeat, Alexnet and the like as a referee, and counting the classification result according to the grading condition of each type error corrector.

And when the statistical classification result is different from the detection result obtained by the initial marking, indicating that a classification error exists, and modifying the original marking. The evaluation statistical method involved in the step is as follows:

wherein N represents the number of class error corrector, M represents the number of class,a weighting factor representing the jth class of the ith class error corrector,represents the score, N, of the jth class of the ith class corrector^jThe number of error correctors with a non-zero jth class Score is shown, and Score indicates the final class Score.

Step S103: and comparing the constraint relation between each cell marker frame in the first corrected marker result and the corresponding cell size by using a positioning error corrector, judging whether a positioning error exists, and correcting the marker of the positioning error to obtain a second corrected marker result, which is shown in fig. 4.

This step of the present embodiment corresponds to step 4 in fig. 2. Specifically, constraints on the sizes of various types of cells are obtained by clustering individual bone marrow cell data. For example: and (5) carrying out clustering analysis on the individual bone marrow cells by using K-means to obtain various bone marrow cell size constraints.

Wherein the constraints on cell size include an aspect ratio constraint and an area constraint of the class.

And if each cell marker frame in the marker result after the first correction does not meet the constraint relation of the corresponding cell size, deleting the marker frame from the detection result.

Specifically, obtaining information of each cell marker frame in the marker result after first correction, sequentially extracting single marker information, calculating whether the marker frame meets aspect ratio constraint of the category to which the marker frame belongs, and deleting the marker frame from the detection result if the marker frame does not meet the constraint; if the constraint is satisfied, further judging whether the area of the mark frame satisfies the area constraint of the category to which the mark frame belongs, and deleting the mark frame which does not satisfy the constraint. The formula involved in this step is as follows:

wherein, F_LHFor the width and height constraint in cells, F_OffFor the constraint of the intracellular area, N is the number of labeled boxes, M is the number of classes, w_i,h_iRespectively indicate the width and height of the frame marked as the j-type cell, respectively represents the minimum and maximum labeled frame width-to-height ratio of the j-th cell, mu represents the width-to-height ratio threshold value, (W)^j×H^j)_Min、(W^j×H^j)_MaxRespectively representing the minimum and maximum labeled box area of the jth cell, wherein eta represents an area threshold, and mu and eta are dynamically adjusted along with the error correction process until an optimal threshold is obtained. When F is present_LHWhen the mark frame width-to-height ratio is 1, the mark frame width-to-height ratio is in the category of the mark frame width-to-height ratio, and when F is_LHWhen the mark frame width/height ratio is 0, it indicates that the mark frame width/height ratio does not belong to the category. When F is present_OffWhen the mark frame area is 1, the mark frame area is in the category of the mark frame area, and F_OffWhen the mark frame area is 0, it indicates that the mark frame area does not belong to the category.

Step S104: and respectively calculating the overlapping rate of the mark frame areas of any two cells in the mark result after the second error correction by using a fuzzy confirmer, judging whether a fuzzy mark problem exists according to the ratio of the overlapping rate to the overlapping threshold value and the class of the mark frame, and correcting the fuzzy mark problem to obtain the final error corrected mark result.

This step of the present embodiment corresponds to step 5 in fig. 2. If the overlapping rate is greater than the overlapping threshold value, further judging whether the types of the mark frames are the same, and if so, directly deleting the mark frame with the lower classification score in the two mark frames; if the two mark frames are different, adding the category and the position information of the two mark frames into the cell list to be confirmed, and deleting the mark frame with lower score.

As shown in fig. 5, a single mark frame is sequentially extracted as a decision frame, the overlapping rate of the area of the decision frame and the area of another mark frame is respectively calculated, and whether the overlapping rate is greater than the overlapping threshold α is determined. If the overlapping rate of the mark frame and a certain mark frame is greater than the overlapping threshold value alpha, further judging whether the mark frame types are the same, and if the mark frames belong to the same type, directly deleting the frames with lower classification scores in the mark frame type and the mark frame type; if the types of the cells are different, adding the types and the position information of the two marking frames into a cell list to be confirmed, and deleting the frame with lower score. The formula involved in this step is as follows:

wherein N is the number of the mark frames,respectively the coordinate position of the upper left corner and the coordinate position of the lower right corner of the ith mark frame, S_i∪S_jRepresenting the union area of two marker boxes, S_i∩S_jIndicates the intersection area of two marker boxes, R_ijIndicates the overlapping ratio of two mark frames, c_ij0 indicates that the two marker boxes are not of the same class, ci_jNot equal to 0 indicates that the label box category is the same. Alpha represents an overlap threshold value, which is dynamically adjusted in the error correction process until an optimal threshold value is obtained. F_ij1 is the basis of class ambiguity determination, F_ij0 is the basis of positioning fuzzy decision among the same classes, F_ijNo. 1 is not a fuzzy label basis.

And after actual error correction is finished, storing the whole bone marrow picture and the error-corrected marking result into a database. For example: the image of bone marrow cells is stored in BLOB format, and the list of marked and confirmed cells is stored in TXT format.

The embodiment adopts classification error correction confirmation, positioning error correction confirmation and fuzzy error correction confirmation, corrects the problems of classification errors, positioning errors and fuzzy marking in sequence, obtains a more accurate result through automatic error correction on the basis of marking, enables the marking process of the bone marrow cells to be more automatic and accurate, and greatly improves the marking efficiency of the bone marrow cells and the accuracy of an automatic marking model.

Example two

As shown in fig. 6, the present embodiment provides an error correction system for automatic labeling of individual cells in a bone marrow image, which specifically includes the following modules:

the automatic image marking module 11 is used for marking the marrow pathology image by using the cell marking model to obtain an initial marking result; the initial marking result comprises initial cell types and initial marking frame position information;

a classification error correction confirmation module 12, configured to perform secondary confirmation on the initial labeling result by using a classification error corrector to obtain a classification score of the cell and a cell type after error correction, and correct a label with a classification error to obtain a labeling result after primary error correction;

a positioning error correction confirmation module 13, configured to compare the constraint relationship between each cell marker frame in the first corrected marker result and the corresponding cell size by using a positioning error corrector, determine whether a positioning error exists, and correct the marker of the positioning error to obtain a second corrected marker result;

and the fuzzy error correction confirming module 14 is configured to calculate, by using a fuzzy confirmer, overlapping rates of the mark frame areas of any two cells in the mark result after the second error correction, determine whether a fuzzy mark problem exists according to a ratio of the overlapping rates to an overlapping threshold and a category to which the mark frame belongs, and correct the fuzzy mark problem to obtain a final error-corrected mark result.

It should be noted that, each module in the error correction system for automatic marking of individual cells of bone marrow images in this embodiment corresponds to each step in the error correction method for automatic marking of individual cells of bone marrow images in the first embodiment one by one, and the specific implementation process is the same, which will not be described herein again.

The embodiment adopts classification error correction confirmation, positioning error correction confirmation and fuzzy error correction confirmation, corrects the problems of classification errors, positioning errors and fuzzy marking in sequence, obtains a more accurate result through automatic error correction on the basis of marking, enables the marking process of the bone marrow cells to be more automatic and accurate, and greatly improves the marking efficiency of the bone marrow cells and the accuracy of an automatic marking model.

EXAMPLE III

The present embodiment provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps in the method for error correction in automatic labeling of individual cells in a bone marrow image as described in the first embodiment above.

The embodiment adopts classification error correction confirmation, positioning error correction confirmation and fuzzy error correction confirmation, corrects the problems of classification errors, positioning errors and fuzzy marking in sequence, obtains a more accurate result through automatic error correction on the basis of marking, enables the marking process of the bone marrow cells to be more automatic and accurate, and greatly improves the marking efficiency of the bone marrow cells and the accuracy of an automatic marking model.

Example four

The embodiment provides a computer device, which comprises a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor executes the program to realize the steps of the method for correcting errors in the automatic marking of individual cells in the bone marrow image according to the first embodiment.

It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general purpose processors, digital signal processors DSP, application specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and so on. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory may include both read-only memory and random access memory, and may provide instructions and data to the processor, and a portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

The embodiment adopts classification error correction confirmation, positioning error correction confirmation and fuzzy error correction confirmation, corrects the problems of classification errors, positioning errors and fuzzy marking in sequence, obtains a more accurate result through automatic error correction on the basis of marking, enables the marking process of the bone marrow cells to be more automatic and accurate, and greatly improves the marking efficiency of the bone marrow cells and the accuracy of an automatic marking model.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.