1. A method of pathological image processing, comprising:

carrying out target detection on a pathological image to be processed to obtain a target detection result, wherein the target detection result comprises thyroid nodule information;

carrying out thyroid nodule component identification processing on the target detection result through a fine-grained classification model to obtain first probability information of a category to which a nodule component corresponding to the thyroid nodule information belongs;

determining second probability information of a category to which a nodule component corresponding to the thyroid nodule information belongs, based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, and pixel information corresponding to each pixel included in the thyroid nodule information;

and determining the category of the corresponding thyroid nodule component in the thyroid nodule information according to the second probability information of the category of the nodule component corresponding to the thyroid nodule information.

2. The method according to claim 1, wherein the performing target detection on the pathological image to be processed to obtain a target detection result comprises:

and carrying out target detection on the pathological image to be processed through a Mask recursive convolutional neural network Mask-RCNN model to obtain the target detection result.

3. The method according to claim 1, wherein the obtaining of the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs by performing thyroid nodule component identification processing on the target detection result through a fine-grained classification model includes:

and carrying out thyroid nodule component identification processing on the target detection result through a recursive attention convolutional neural network (RA-CNN) model to obtain first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

4. The method according to claim 1, wherein the determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and pixel information corresponding to each pixel included in the thyroid nodule information includes:

determining pixel values corresponding to all pixels contained in the thyroid nodule information;

determining third probability information of a category to which a nodule component corresponding to the thyroid nodule information belongs based on a relationship between a pixel value corresponding to each pixel contained in the thyroid nodule information and a component classification threshold;

and determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

5. The method according to claim 4, wherein the determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs comprises:

determining weight information corresponding to the first probability information and the third probability information respectively;

and determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, and weight information corresponding to each of the first probability information and the third probability information.

6. The method of claim 1, further comprising:

obtaining training samples, the training samples comprising: a plurality of thyroid nodule images;

performing data enhancement processing on at least one thyroid nodule image in the training sample in a padding mode;

and training the fine-grained classification model based on the training sample after data enhancement processing.

7. The method according to claim 6, wherein the determining the category to which the thyroid nodule component corresponding to the thyroid nodule belongs according to the second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs further comprises:

and training the fine-grained classification model based on the target detection result and the category of the thyroid nodule component corresponding to the thyroid nodule.

8. An apparatus for pathological image processing, comprising:

the target detection module is used for carrying out target detection on the pathological image to be processed to obtain a target detection result, and the target detection result comprises thyroid nodule information;

the component identification processing module is used for carrying out thyroid nodule component identification processing on the target detection result through a fine-grained classification model to obtain first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs;

a first determining module, configured to determine, based on first probability information of a category to which a nodule component corresponding to the thyroid nodule information belongs and pixel information corresponding to each pixel included in the thyroid nodule information, second probability information of a category to which the nodule component corresponding to the thyroid nodule information belongs;

and the second determining module is used for determining the category of the corresponding thyroid nodule component in the thyroid nodule information according to the second probability information of the category of the nodule component corresponding to the thyroid nodule information.

9. An electronic device, comprising:

one or more processors;

a memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: method of performing pathological image processing according to any one of claims 1 to 7.

10. A computer readable storage medium, characterized in that it stores at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by said processor to implement the method of pathological image processing according to any one of claims 1 to 7.

Background

The deep learning technology has been developed rapidly in recent years, and also shows great application potential in detection and identification in the field of medical imaging, for example, benign and malignant differentiation can be performed on pathological images containing Thyroid nodules, or Thyroid imaging reporting and data (Tirad) level differentiation can be performed on pathological images containing Thyroid nodules.

In the related art, there are some ways to distinguish benign and malignant pathological images containing thyroid nodules, or to distinguish Tirad from pathological images containing thyroid nodules, for example, to distinguish benign and malignant pathological images containing thyroid nodules by means of deep learning, or to distinguish Tirad levels from pathological images containing thyroid nodules by means of deep learning.

However, in the related art, the differentiation between benign and malignant images of pathological images including thyroid nodules is performed only by means of deep learning, or the differentiation at the Tirad level is performed, which depends on the accuracy of the extracted features and the accuracy of model training, and in the related art, the differentiation of pathological images including thyroid nodules is performed only by means of deep learning, which is a relatively coarse-grained differentiation, so that it is an important issue to obtain finer features based on pathological images including thyroid nodules to assist doctors in accurate diagnosis.

Disclosure of Invention

The present application aims to provide a method, an apparatus, an electronic device and a readable storage medium for pathological image processing, which are used to solve at least one of the above technical problems.

The above object of the present application is achieved by the following technical solutions:

in a first aspect, a method for pathological image processing is provided, which includes:

carrying out target detection on the pathological image to be processed to obtain a target detection result, wherein the target detection result comprises thyroid nodule information;

carrying out thyroid nodule component identification processing on the target detection result through a fine-grained classification model to obtain first probability information of a category to which a nodule component corresponding to thyroid nodule information belongs;

determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and pixel information corresponding to each pixel contained in the thyroid nodule information;

and determining the category of the corresponding thyroid nodule component in the thyroid nodule information according to the second probability information of the category of the nodule component corresponding to the thyroid nodule information.

In one possible implementation manner, performing target detection on a pathological image to be processed to obtain a target detection result includes:

and carrying out target detection on the pathological image to be processed through a Mask recursive convolutional neural network Mask-RCNN model to obtain a target detection result.

In another possible implementation manner, performing thyroid nodule component identification processing on the target detection result through a fine-grained classification model to obtain first probability information of a category to which a nodule component corresponding to thyroid nodule information belongs, including:

and carrying out thyroid nodule component identification processing on the target detection result through a recursive attention convolutional neural network RA-CNN model to obtain first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

In another possible implementation manner, determining, based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the pixel information corresponding to each pixel included in the thyroid nodule information, the second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs includes:

determining pixel values corresponding to all pixels contained in thyroid nodule information;

determining third probability information of a category to which a nodule component corresponding to thyroid nodule information belongs based on a relationship between a pixel value corresponding to each pixel contained in the thyroid nodule information and a component classification threshold;

and determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

In another possible implementation manner, determining, based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, the second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs includes:

determining weight information corresponding to the first probability information and the third probability information respectively;

and determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, and weight information corresponding to the first probability information and the third probability information respectively.

In another possible implementation manner, the method further includes:

obtaining training samples, the training samples comprising: a plurality of thyroid nodule images;

performing data enhancement processing on at least one thyroid nodule image in the training sample in a padding mode;

and training the fine-grained classification model based on the training sample after the data enhancement processing.

In another possible implementation manner, determining a category to which a thyroid nodule component corresponding to a thyroid nodule belongs according to second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, and then:

and training a fine-grained classification model based on the target detection result and the category of the thyroid nodule component corresponding to the thyroid nodule.

In a second aspect, there is provided an apparatus for pathological image processing, comprising:

the target detection module is used for carrying out target detection on the pathological image to be processed to obtain a target detection result, and the target detection result comprises thyroid nodule information;

the component identification processing module is used for carrying out thyroid nodule component identification processing on the target detection result through a fine-grained classification model to obtain first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs;

the first determining module is used for determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and pixel information corresponding to each pixel contained in the thyroid nodule information;

and the second determining module is used for determining the category of the corresponding thyroid nodule component in the thyroid nodule information according to the second probability information of the category of the nodule component corresponding to the thyroid nodule information.

In a possible implementation manner, the target detection module, when performing target detection on a pathological image to be processed to obtain a target detection result, is specifically configured to:

and carrying out target detection on the pathological image to be processed through a Mask recursive convolutional neural network Mask-RCNN model to obtain a target detection result.

In another possible implementation manner, the component identification processing module is specifically configured to, when performing thyroid nodule component identification processing on the target detection result through a fine-grained classification model to obtain first probability information of a category to which a nodule component corresponding to thyroid nodule information belongs:

and carrying out thyroid nodule component identification processing on the target detection result through a recursive attention convolutional neural network RA-CNN model to obtain first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

In another possible implementation manner, the first determining module is specifically configured to, when determining, based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the pixel information corresponding to each pixel included in the thyroid nodule information, the second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, specifically:

determining pixel values corresponding to all pixels contained in thyroid nodule information;

determining third probability information of a category to which a nodule component corresponding to thyroid nodule information belongs based on a relationship between a pixel value corresponding to each pixel contained in the thyroid nodule information and a component classification threshold;

and determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

In another possible implementation manner, the first determining module is specifically configured to, when determining, based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, the second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs:

determining weight information corresponding to the first probability information and the third probability information respectively;

and determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, and weight information corresponding to the first probability information and the third probability information respectively.

In another possible implementation manner, the apparatus further includes: an acquisition module, a data enhancement processing module, and a first training module, wherein,

an acquisition module for acquiring a training sample, the training sample comprising: a plurality of thyroid nodule images;

the data enhancement processing module is used for performing data enhancement processing on at least one thyroid nodule image in the training sample in a padding mode;

and the first training module is used for training the fine-grained classification model based on the training samples after the data enhancement processing.

In another possible implementation manner, the apparatus further includes: a second training module, wherein,

and the second training module is used for training the fine-grained classification model based on the target detection result and the category of the thyroid nodule component corresponding to the thyroid nodule.

In a third aspect, an electronic device is provided, which includes:

one or more processors;

a memory;

one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to: the corresponding operations of the method of pathological image processing according to any one of the possible implementations of the first aspect are performed.

In a fourth aspect, there is provided a computer readable storage medium storing at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to implement the method of pathological image processing as shown in any one of the possible implementations of the first aspect.

Compared with the prior art that benign and malignant differentiation is carried out on pathological images containing thyroid nodules only in a deep learning mode or Tirad level differentiation is carried out, in the method, target detection is carried out on the pathological images to be processed to obtain target detection results, the target detection results contain thyroid nodule information, then thyroid nodule component identification processing is carried out on the target detection results through a fine-grained classification model to obtain first probability information of classes to which nodule components corresponding to the thyroid nodule information belong, then second probability information of the classes to which the nodule components corresponding to the thyroid nodule information belong is determined based on the first probability information of the classes to which the nodule components corresponding to the thyroid nodule information belong and pixel information corresponding to each pixel contained in the thyroid nodule information respectively, and then determining the category of the corresponding thyroid nodule component in the thyroid nodule information according to the second probability information of the category of the nodule component corresponding to the thyroid nodule information. Namely, the pathological images can obtain the category of the components in the thyroid nodule in the above mode, so that more precise characteristics can be obtained to assist doctors in accurate diagnosis.

Drawings

Fig. 1 is a flowchart illustrating a method of pathological image processing according to an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a pathological image processing apparatus according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of an electronic device for pathological image processing according to an embodiment of the present application.

Detailed Description

The present application is described in further detail below with reference to the attached drawings.

The present embodiment is only for explaining the present application, and it is not limited to the present application, and those skilled in the art can make modifications of the present embodiment without inventive contribution as needed after reading the present specification, but all of them are protected by patent law within the scope of the claims of the present application.

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship, unless otherwise specified.

Thyroid nodules are a very common finding in thyroid tissue and may be indicative of thyroid cancer. In the united states, europe and japan, approximately 2% to 7% of the population is diagnosed. Thyroid cancer is a curable disease when it is found early, and therefore, accurate differentiation between malignant and benign thyroid nodules is essential to ensure proper management of malignant nodules. The advent of high resolution ultrasound technology as a pre-operative diagnostic tool has made it possible to obtain detailed information about thyroid structures. Today, in contemporary ultrasound systems, thyroid nodules may be located, measured and examined to determine whether surgical intervention is required. In addition to echogenicity and microcalcification within the nodule, other ultrasound features (e.g., the shape and contour of the nodule) may also be used as risk factor criteria for malignancy. Unclear borders or irregular shapes may indicate malignant tumors, while circular areas indicate benign lesions. Despite its non-invasive, low cost and easy-to-use real-time application, ultrasound imaging suffers from granular patterns called speckles. This is a result of various constructive and destructive interference phenomena that occur when the distance between tissue scatterers is less than the axial resolution limit of the system. It can lead to anatomical deformations and random fluctuations in the image intensity distribution. If the image is damaged by speckle, there will be no area where the intensity distribution is approximately constant even if the reflecting tissue is completely uniform. In addition, certain ultrasound properties may cause misleading effects in ultrasound images. Reverberation, shadowing, refraction, side lobes and grating lobes can reduce the resolution of the ultrasound image, thereby reducing its overall quality. The above-mentioned problems arising from the complexity of ultrasound imaging constitute an accurate boundary detection, which is a difficult task even for physicians with a rich expertise. Proper boundary estimation of thyroid nodules may play a key role in ultrasound thyroid imaging applications, such as classification based on the shape, size, and location of the nodule. In addition, it may help with accurate needle placement during fine needle biopsy. A variety of computer segmentation methods have been employed in ultrasound imaging of prostate, kidney, cardiac anatomy, ovary, fetal head, and breast lesions. All these algorithms can be classified into five types, depending on the strategy chosen for subdividing the regions of interest. Segmentation methods based on edge detection, texture or feature analysis, deformable models and activity models, methods based on the above algorithms to combine the optimization results and methods based on multi-scale algorithms, which can visualize ultrasound images of different resolutions. The edge-based segmentation algorithm can detect any abrupt changes in the gray values within the ultrasound image. For final contour extraction, an additional process is performed to select and link edge pixels. In texture analysis, rather than trying to locate edges in an ultrasound image, several texture features are used for region characterization. These features are typically used as input to a classification or clustering algorithm to distinguish a group of pixels as correct or incorrect regions. Model-based segmentation methods either use a priori knowledge with named active contours and deformation models or use statistical models that do not use any a priori information about the region of interest. The multiscale approach decomposes the input ultrasound image into several different levels of resolution in order to obtain all the subdivisions available for efficient use. Generally, the poor quality of ultrasound images, combined with the disadvantages associated with the ultrasound properties, have limited the performance of the various segmentation methods proposed in the past decade. Until the rise of deep learning.

The automatic thyroid cancer identification by the traditional machine learning method is usually divided into two steps, wherein the characteristics are firstly extracted, and then a classifier is used for classification. The quality of the classifier depends heavily on the quality of the extracted features. The deep learning replaces the process of manually extracting features, and then the deep learning has a dominance in the field of image recognition, and is gradually applied to the field of ultrasound automation due to the excellent performance of the deep learning on various visual tasks such as image recognition, positioning and segmentation and the increase of the amount of ultrasound data.

The inventor finds in research that deep learning methods in related technologies mainly focus on rough classification automatic identification of thyroid nodules, whether thyroid nodules are benign or malignant, and whether Tirad and the like, and studies are rarely made on the fine granularity of ultrasonic image features such as nodule components, however, the fine granularity features can better determine nodule information in an ultrasonic image, and the finer features can better provide information to help doctors to diagnose.

The embodiment of the application provides a pathological image processing method, which is used for carrying out component identification processing on an image containing thyroid nodules on the basis of machine learning/deep learning in artificial intelligence so as to determine component information of the thyroid nodules contained in the pathological image.

Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Machine Learning (ML) is a multi-domain cross discipline, and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal education learning.

The embodiments of the present application will be described in further detail with reference to the drawings attached hereto.

As shown in fig. 1, the pathological image processing method provided in the embodiment of the present application may be executed by an electronic device, where the electronic device may be a server or a terminal device, where the server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services. The terminal device may be a smart phone, a tablet computer, a notebook computer, a desktop computer, and the like, but is not limited thereto, and the terminal device and the server may be directly or indirectly connected through wired or wireless communication, and the embodiment of the present application is not limited thereto. The method comprises the following steps:

and S101, carrying out target detection on the pathological image to be processed to obtain a target detection result.

Wherein, the target detection result comprises thyroid nodule information.

For the embodiment of the application, thyroid nodule information is contained in the pathological image to be processed. In the embodiment of the present application, the pathological image to be processed may include an ultrasound effect, a Computed Tomography (CT) image, and the like, and may further include other medical images, and the type of the pathological image is not limited in the embodiment of the present application.

For the embodiment of the present application, since the pathological image to be processed containing the thyroid nodule information further contains other tissue regions, such as skin tissue regions, carotid arteries, blood vessels, and the like, in order to more accurately classify components of the thyroid nodule and further reduce the complexity of a fine-grained classification model of the thyroid nodule, the thyroid nodule region needs to be located in the pathological image to be processed. In the embodiment of the present application, target detection is performed on a pathological image to be processed in a target detection manner to detect an image region (may be referred to as a thyroid nodule image) including a thyroid nodule in the pathological image to be processed, and a specific detection manner is described in detail in the following embodiments.

And step S102, carrying out thyroid nodule component identification processing on the target detection result through a fine-grained classification model to obtain first probability information of the category of the nodule component corresponding to the thyroid nodule information.

For the embodiment of the present application, the category to which the nodule component corresponding to the thyroid nodule information belongs may include: cystic and solid component categories. That is to say, in the embodiment of the present application, the target detection result is subjected to thyroid nodule component identification processing by using a fine-grained classification model, and first probability information that a component of a thyroid nodule in the target detection result region belongs to a cystic class or first probability information that a component of a thyroid nodule in the target detection result region belongs to a real class is obtained.

Step S103 determines second probability information of a category to which a nodule component corresponding to thyroid nodule information belongs, based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and pixel information corresponding to each pixel included in the thyroid nodule information.

For the embodiment of the present application, the pixel information corresponding to each pixel included in the thyroid nodule information may be pixel value information or may be other information related to the pixel. This is not limited in the embodiments of the present application.

Further, in the embodiment of the present application, before step S103, the method may further include: and determining pixel information corresponding to each pixel in the thyroid nodule region contained in the target detection result. The step of determining the pixel information corresponding to each pixel in the thyroid nodule region included in the target detection result may be performed before step S102, may also be performed after step S102, and may also be performed simultaneously with step S102, where a specific execution step is not limited in this embodiment of the application.

And step S104, determining the category of the corresponding thyroid nodule component in the thyroid nodule information according to the second probability information of the category of the nodule component corresponding to the thyroid nodule information.

For the embodiment of the present application, after the second probability information of the category to which the nodule component corresponding to the thyroid nodule information included in the pathological image to be processed belongs is determined by the above embodiment, the category to which the corresponding thyroid nodule component in the thyroid nodule information belongs is determined based on the obtained second probability information.

Compared with the prior art that benign and malignant differentiation is performed on pathological images containing thyroid nodules only in a deep learning mode or Tirad-level differentiation is performed, in the pathological images to be processed, target detection results are obtained and contain thyroid nodule information, then thyroid nodule component identification processing is performed on the target detection results through a fine-grained classification model, first probability information of the categories to which the nodule components corresponding to the thyroid nodule information belong is obtained, second probability information of the categories to which the nodule components corresponding to the thyroid nodule information belong is determined based on the first probability information of the categories to which the nodule components corresponding to the thyroid nodule information belong and pixel information corresponding to each pixel contained in the thyroid nodule information, and then determining the category of the corresponding thyroid nodule component in the thyroid nodule information according to the second probability information of the category of the nodule component corresponding to the thyroid nodule information. Namely, the pathological images can obtain the category of the components in the thyroid nodule in the above mode, so that more precise characteristics can be obtained to assist doctors in accurate diagnosis.

In a possible implementation manner of the embodiment of the present application, step S101 may specifically include: and performing target detection on the pathological image to be processed through a Mask-recursive Convolutional Neural Network (Mask-RCNN) model to obtain a target detection result.

For the embodiment of the application, the Mask-RCNN model is an extension of a Faster RCNN model, FCN is used for semantic segmentation of each extraction prior frame of the fast RCNN, and segmentation is completed while detection is completed; in addition, because Region Of Interest Pooling (Region Of Interest Pooling, RoI Pooling) is not aligned according to pixels one by one, RoI Align has higher precision than RoI Pooling, and RoI Align is also introduced into the Mask-RCNN model to replace RoI Pooling in the fast RCNN model; in addition, the Mask-RCNN adds an FPN network structure, the FPN adopts a pyramid feature map form, and the FPN not only uses a deep feature map in a Visual Geometry Group (VGG) model, but also extracts a shallow feature map. The characteristic diagrams are efficiently integrated through bottom-up, top-down and transverse connection, multi-scale characteristic diagram information is obtained, and detection accuracy is greatly improved. In the embodiment of the present application, the ROI Align operation includes: the method comprises the following steps of canceling quantization operation, obtaining an image numerical value on a pixel point with a floating point number as a coordinate by using a bilinear interpolation method, and converting the whole feature aggregation process into a continuous operation, wherein in the specific algorithm operation, ROI Align does not simply supplement coordinate points on the boundary of a candidate region, and then pooling the coordinate points, but traverses each candidate region and keeps the boundary of the floating point number not quantized; dividing the candidate area into k × k units, and not quantizing the boundary of each unit; fixed four coordinate positions are calculated in each cell, the values of the four positions are calculated by a bilinear interpolation method, and then the maximum pooling operation is performed.

Specifically, in the embodiment of the application, the Mask-RCNN divides the target pixel while realizing target detection by adding a branch network on the basis of the fast-RCNN. In an embodiment of the present application, the Mask-RCNN model may include: feature Pyramid (FPN) networks, Region pro-active Network (RPN) networks, ROI Align layers, full Convolution layers (full Convolution networks).

Further, in step S101, the method of performing target detection on the pathological image to be processed to obtain the target detection result may be the method described in the embodiment of the present application, and may also be the method described in the related art to obtain the target detection result. The embodiments of the present application are not limited.

In another possible implementation manner of the embodiment of the present application, step S102 may specifically include: and carrying out thyroid nodule component identification processing on the target detection result through a Recurrent Attention Convolutional Neural Network (RA-CNN) model to obtain first probability information of the type of the nodule component corresponding to the thyroid nodule information.

For the present embodiment, the RA-CNN is an overlay network whose inputs are fine-grained local regions from full image to multi-scale. The network structure design mainly comprises 3 sub-networks with different scales, the network structure of each sub-network with different scales is the same, only the network parameters are different, and each sub-network with different scales comprises two types of networks: a classification network and an Attention suggestion sub-network (APN) network. The input image is subjected to feature extraction and classification through a classification network, then the APN network is trained based on the extracted features to obtain attention area information, the area is cut out and amplified and then used as the input of a second scale network, so that the output results of 3 scale networks can be obtained by repeating the operation for 3 times, and the final output result is obtained by fusing the results of different scale networks. In the embodiment of the application, a target recognition result (thyroid nodule area image) is extracted and classified through a classification network in a first scale network, an APN network obtains attention area information based on the extracted features, the attention area is cut out and enlarged and then is used as input of a second scale network, the features are extracted and classified through the classification network of the second scale network, the APN network obtains the attention area information based on the extracted features, the area is cut out and enlarged and is used as input of a third scale network, the features are extracted and classified through the classification network of the third scale network, the APN network obtains the attention area information based on the extracted features, the attention area is cut out and enlarged, and then the last full-connection layer of each classification sub-network is stacked, then, they are connected to a full connection layer, and then classified through the softmax layer, so as to obtain a classification result, that is, first probability information (first probability information that thyroid nodule is cystic and/or first probability information that thyroid nodule is substantial) of a category to which the nodule component corresponding to the thyroid nodule information belongs.

For the embodiment of the application, the target detection result (thyroid nodule region image) is subjected to thyroid nodule component identification processing through the RA-CNN to obtain the first probability information of the category to which the thyroid nodule component belongs, and the RA-CNN can be fused with the output results of networks with different scales, so that the accuracy of the obtained category to which the thyroid nodule component belongs is higher, namely the classification accuracy is higher.

Further, in the embodiment of the present application, the manner of obtaining the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on the target detection result (including the image of the thyroid nodule region) may be the manner described in the embodiment of the present application, may also be the manner described in the related art, and is not limited in the embodiment of the present application.

In another possible implementation manner of the embodiment of the present application, step S103 may specifically include: step S1031 (not shown in the figure), step S1032 (not shown in the figure), and step S1033 (not shown in the figure), wherein,

step S1031 determines pixel values corresponding to the respective pixels included in the thyroid nodule information.

For the present embodiment, the thyroid nodule information in step S1031 includes: thyroid nodule area images. That is, the pixel values corresponding to the respective pixels in the thyroid nodule region image are determined.

Step S1032 determines third probability information of a category to which a nodule component corresponding to thyroid nodule information belongs, based on a relationship between a pixel value corresponding to each pixel included in the thyroid nodule information and a component classification threshold.

For the embodiment of the present application, the component classification threshold may be preset or input by a user, and further, the component classification threshold may be a fixed value or a range. The embodiments of the present application are not limited thereto.

Specifically, in this embodiment of the present application, the category to which the nodule component corresponding to the thyroid nodule information belongs may include: the cystic classification and the real classification determine the number of pixels belonging to the cystic classification in the thyroid nodule information and/or the number of pixels belonging to the real classification in the thyroid nodule information based on the relationship between the pixel value corresponding to each pixel in the thyroid nodule information and the component classification threshold, and further determine the proportion (probability) of the nodule component corresponding to the thyroid nodule information belonging to the cystic classification based on the number of pixels belonging to the cystic classification in the thyroid nodule information and the number of pixels contained in the thyroid nodule information, and/or determine the proportion (probability) of the nodule component corresponding to the thyroid nodule belonging to the real classification based on the number of pixels belonging to the real classification in the thyroid nodule information and the number of pixels contained in the thyroid nodule information.

For example, if the thyroid nodule information corresponding to a pixel having a pixel value of less than 30 is described as cystic property, and if the thyroid nodule information corresponding to a pixel having a pixel value of not less than 30 is described as real property, and the number of pixels included in the thyroid nodule information (thyroid nodule region image) is 100, and if the number of pixels having a pixel value of less than 30 in the image is 60 and the number of pixels having a pixel value of not less than 30 is 40, it is determined that the ratio of the thyroid nodule information corresponding to the thyroid nodule information to cystic property is 0.6 and the ratio of the thyroid nodule information to real property is 0.4, that is, the third probability information that the thyroid nodule information corresponding to the thyroid nodule information is cystic property is 0.6, and the third probability information that the thyroid nodule information corresponding to the thyroid nodule information is real property is 0.4.

Step S1033 determines second probability information of a category to which a nodule component corresponding to thyroid nodule information belongs, based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

Specifically, step S1033 may specifically include: determining weight information corresponding to the first probability information and the third probability information respectively; and determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, and weight information corresponding to the first probability information and the third probability information respectively.

Specifically, in the embodiment of the application, second probability information of a category to which a nodule component corresponding to thyroid nodule information belongs is determined through formula 1;

score = a × RA-CNN _ Score + b × color _ Score formula 1;

wherein, Score is second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs; RA-CNN _ Score is first probability information of a category to which a nodule component corresponding to thyroid nodule information belongs; the color _ Score is third probability information of a category to which a nodule component corresponding to thyroid nodule information belongs; a. b is a parameter indicating weight information corresponding to the first probability information and the second probability information, respectively.

Wherein a + b = 1, and 1 > a > 0, 1 > b > 0. In the embodiment of the application, the values of a and b can be determined by traversing all the parameter spaces in 0-1 at intervals of 0.01 and taking score as the maximum value. Further, in the embodiments of the present application, the determination of the values a and b is not limited to the manner in the embodiments of the present application, and may be, for example, a preset fixed value, which is not limited in the embodiments of the present application.

Further, in order to enable the fine-grained classification model to perform thyroid nodule component identification processing, or further improve the accuracy of the fine-grained classification model for performing thyroid nodule component identification processing, the fine-grained classification model may be trained, which is specifically described in the following embodiments.

In another possible implementation manner of the embodiment of the present application, the method may further include: step Sa (not shown), step Sb (not shown), and step Sc (not shown), wherein,

and step Sa, obtaining a training sample.

Wherein, training the sample includes: a plurality of thyroid nodule images.

For the embodiment of the application, the training sample may further include component classification labeling information corresponding to each thyroid nodule image, or may also include component classification labeling information corresponding to part of thyroid nodule images.

And Sb, performing data enhancement processing on at least one thyroid nodule image in the training sample in a padding mode.

For the embodiment of the application, data enhancement is the most common means for improving the robustness of the model. In the deep learning era, the larger the scale and the higher the quality of data are, the better generalization ability of the model can be possessed, and the data directly determines the upper limit of model learning. However, in actual engineering, collected data is difficult to cover all scenes, such as the illumination conditions of images, images shot by the same scene may have great differences due to different light, and thus illumination data enhancement needs to be added during model training. Such as other influencing factors like angle, direction, shading, etc., data enhancement can be used to improve the robustness of the model.

Further, considering that the size of the intercepted thyroid nodule area is often much smaller than the size of 224 × 224 input by the classification model network, and many original detail information of the image is lost by directly scaling, in order to retain the original information, the intercepted image is filled in by padding, the size after filling is 256 × 256, and data enhancement is completed by randomly cutting, flipping and rotating. In the embodiment of the present application, padding filling may include: zero padding, constant padding, mirror padding, and repeat padding, etc., as well as other padding approaches. In the embodiment of the present application, when performing data enhancement processing by a padding method, the method is not limited to the padding filling method.

And step Sc, training a fine-grained classification model based on the training sample after data enhancement processing.

In the embodiment of the present application, step Sa, step Sb, and step Sc may be performed before step S102, may be performed after step S102, may be performed simultaneously with step S102, and are not limited in the embodiment of the present application.

In order to further improve the thyroid nodule component identification processing performed by the fine-grained classification model, and obtain more accurate probability information of a category to which a nodule component corresponding to thyroid nodule information belongs, in an embodiment of the present application, the determining, according to the second probability information of the category to which the nodule component corresponding to thyroid nodule information belongs, a category to which a thyroid nodule component corresponding to a thyroid nodule belongs may further include: and training a fine-grained classification model based on the target detection result and the category of the thyroid nodule component corresponding to the thyroid nodule.

Specifically, after the category to which the thyroid nodule component corresponding to the thyroid nodule belongs is determined according to the second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, the target detection result and the category to which the thyroid nodule component corresponding to the thyroid nodule belongs can be used as training samples, and then the fine-grained classification model is trained. In the embodiment of the application, after the category to which the thyroid nodule component corresponding to the thyroid nodule information belongs is determined according to the second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, the fine-grained classification model is trained based on the determined category, a certain number of target recognition results and the categories to which the thyroid nodule components corresponding to the target recognition results belong may be collected, and the collected categories are used as training samples to train the fine-grained classification model.

The above embodiments describe a method for pathological image processing from the perspective of method flow, and the following embodiments describe an apparatus for pathological image processing from the perspective of modules or units, which are described in detail in the following embodiments.

The embodiment of the present application provides a pathological image processing apparatus, and as shown in fig. 2, the pathological image processing apparatus 20 may specifically include: an object detection module 21, a component recognition processing module 22, a first determination module 23, and a second determination module 24, wherein,

the target detection module 21 is configured to perform target detection on the pathological image to be processed to obtain a target detection result, where the target detection result includes thyroid nodule information;

the component identification processing module 22 is configured to perform thyroid nodule component identification processing on the target detection result through a fine-grained classification model to obtain first probability information of a category to which a nodule component corresponding to thyroid nodule information belongs;

a first determining module 23, configured to determine, based on first probability information of a category to which a nodule component corresponding to thyroid nodule information belongs and pixel information corresponding to each pixel included in the thyroid nodule information, second probability information of a category to which the nodule component corresponding to the thyroid nodule information belongs;

and the second determining module 24 is configured to determine, according to the second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, the category to which the thyroid nodule component corresponding to the thyroid nodule information belongs.

For the embodiment of the present application, the first determining module 23 and the second determining module 24 may be the same determining module or different determining modules, and are not limited in the embodiment of the present application.

In another possible implementation manner of the embodiment of the present application, when the target detection module 21 performs target detection on a pathological image to be processed to obtain a target detection result, the target detection module is specifically configured to:

and carrying out target detection on the pathological image to be processed through a Mask recursive convolutional neural network Mask-RCNN model to obtain a target detection result.

In another possible implementation manner of the embodiment of the present application, when performing thyroid nodule component identification processing on a target detection result through a fine-grained classification model to obtain first probability information of a category to which a nodule component corresponding to thyroid nodule information belongs, the component identification processing module 22 is specifically configured to:

and carrying out thyroid nodule component identification processing on the target detection result through a recursive attention convolutional neural network RA-CNN model to obtain first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

In another possible implementation manner of the embodiment of the present application, when performing thyroid nodule component identification processing on a target detection result through a fine-grained classification model to obtain first probability information of a category to which a nodule component corresponding to thyroid nodule information belongs, the component identification processing module 22 is specifically configured to:

and carrying out thyroid nodule component identification processing on the target detection result through a recursive attention convolutional neural network RA-CNN model to obtain first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

In another possible implementation manner of the embodiment of the present application, when determining, based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the pixel information corresponding to each pixel included in the thyroid nodule information, the first determining module 23 is specifically configured to:

determining pixel values corresponding to all pixels contained in thyroid nodule information;

determining third probability information of a category to which a nodule component corresponding to thyroid nodule information belongs based on a relationship between a pixel value corresponding to each pixel contained in the thyroid nodule information and a component classification threshold;

and determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs.

In another possible implementation manner of the embodiment of the present application, when determining, based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and the third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, the first determining module 23 is specifically configured to:

determining weight information corresponding to the first probability information and the third probability information respectively;

and determining second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs based on first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, third probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, and weight information corresponding to the first probability information and the third probability information respectively.

In another possible implementation manner of the embodiment of the present application, the apparatus 20 further includes: an acquisition module, a data enhancement processing module, and a first training module, wherein,

an acquisition module for acquiring a training sample, the training sample comprising: a plurality of thyroid nodule images;

the data enhancement processing module is used for performing data enhancement processing on at least one thyroid nodule image in the training sample in a padding mode;

and the first training module is used for training the fine-grained classification model based on the training samples after the data enhancement processing.

In another possible implementation manner of the embodiment of the present application, the apparatus 20 further includes: a second training module, wherein,

and the second training module is used for training the fine-grained classification model based on the target detection result and the category of the thyroid nodule component corresponding to the thyroid nodule.

Further, in this embodiment of the application, the first training module and the second training module may be the same training module or different training modules, and are not limited in this embodiment of the application.

Compared with the prior art that benign and malignant differentiation is performed on pathological images containing thyroid nodules only in a deep learning mode or Tirad-level differentiation is performed, in the pathological image processing device, target detection is performed on the pathological images to be processed to obtain target detection results, the target detection results contain thyroid nodule information, thyroid nodule component identification processing is performed on the target detection results through a fine-grained classification model to obtain first probability information of categories to which the nodule components corresponding to the thyroid nodule information belong, second probability information of the categories to which the nodule components corresponding to the thyroid nodule information belong is determined based on the first probability information of the categories to which the nodule components corresponding to the thyroid nodule information belong and pixel information corresponding to each pixel contained in the thyroid nodule information, and then determining the category of the corresponding thyroid nodule component in the thyroid nodule information according to the second probability information of the category of the nodule component corresponding to the thyroid nodule information. Namely, the pathological images can obtain the category of the components in the thyroid nodule in the above mode, so that more precise characteristics can be obtained to assist doctors in accurate diagnosis.

Further, the apparatus for processing a pathological image provided in the embodiment of the present application is applicable to the method embodiment described above, and is not described herein again.

In an embodiment of the present application, an electronic device is provided, as shown in fig. 3, where the electronic device 300 shown in fig. 3 includes: a processor 301 and a memory 303. Wherein processor 301 is coupled to memory 303, such as via bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that the transceiver 304 is not limited to one in practical applications, and the structure of the electronic device 300 is not limited to the embodiment of the present application.

The Processor 301 may be a CPU (Central Processing Unit), a general-purpose Processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other Programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 301 may also be a combination of computing functions, e.g., comprising one or more microprocessors, a combination of a DSP and a microprocessor, or the like.

Bus 302 may include a path that transfers information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 302 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 3, but this does not mean only one bus or one type of bus.

The Memory 303 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical Disc storage, optical Disc storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), a magnetic Disc storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to these.

The memory 303 is used for storing application program codes for executing the scheme of the application, and the processor 301 controls the execution. The processor 301 is configured to execute application program code stored in the memory 303 to implement the aspects illustrated in the foregoing method embodiments.

Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

The present application provides a computer-readable storage medium, on which a computer program is stored, which, when running on a computer, enables the computer to execute the corresponding content in the foregoing method embodiments. Compared with the prior art that benign and malignant differentiation is performed on pathological images containing thyroid nodules only in a deep learning mode or in a Tirad level differentiation mode, in the embodiment of the application, target detection is performed on pathological images to be processed to obtain a target detection result, thyroid nodule information is contained in the target detection result, thyroid nodule component identification processing is performed on the target detection result through a fine-grained classification model to obtain first probability information of a category to which a nodule component corresponding to the thyroid nodule information belongs, second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs is determined based on the first probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs and pixel information respectively corresponding to each pixel contained in the thyroid nodule information, and then according to the second probability information of the category to which the nodule component corresponding to the thyroid nodule information belongs, and determining the category of the corresponding thyroid nodule component in the thyroid nodule information. Namely, the pathological images can obtain the category of the components in the thyroid nodule in the above mode, so that more precise characteristics can be obtained to assist doctors in accurate diagnosis.

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

The foregoing is only a partial embodiment of the present application, and it should be noted that, for those skilled in the art, several modifications and decorations can be made without departing from the principle of the present application, and these modifications and decorations should also be regarded as the protection scope of the present application.