1. A breast tissue pathological image classification method is characterized in that a migration learning method is used on two network models of VGG16 and ResNet50 to freeze a shallow convolution, and the shallow general features are used to improve the network accuracy and generalization capability; secondly, adding an attention module to extract breast lesion region information so as to enhance feature description; and finally, fusing class probability results of the two individual classifiers in a soft voting manner to obtain a final eight-classification result.

2. The breast pathology image classification method according to claim 1, comprising the following specific steps:

step S1: image preprocessing is performed on the breast histopathology image dataset;

step S2: a neural network constructed by taking a VGG16 network as a basic framework is used as a first individual classifier, the VGG16 network comprises 13 convolutional layers, and a migration learning fine tuning experiment is performed by taking an ImageNet data set as a source domain and a mammary gland histopathology image data set as a target domain in a layer-by-layer parameter freezing mode;

step S3: a neural network constructed by taking a ResNet50 network as a basic framework is used as a second individual classifier, the ResNet50 network comprises 4 types of residual blocks, and a migration learning fine tuning experiment is carried out by taking an ImageNet data set as a source domain and a breast histopathology image data set as a target domain in a mode of gradually freezing parameters of the residual blocks;

step S4: adding channel attention and spatial attention in a VGG16 network and a ResNet50 network respectively, wherein the adding position is after the last convolutional layer and before the full connection layer, the sequence is that the channel attention is in front, and the spatial attention is in back;

step S5: constructing a mammary gland image classification network combining attention and dual-model migration fusion;

step S6: and training a mammary gland image classification network combined with attention and dual-model migration for classification detection of mammary gland histopathology images.

Background

Breast cancer is one of the most common cancers in women worldwide and is also one of the most mortality cancers. With the continuous progress of computer science and technology, computer-aided diagnosis has become a popular research field in the medical field.

The traditional breast cancer tissue pathology image classification method needs to manually extract image features, but due to the fact that the number of professional pathology experts is limited, manual selection is subjective, the selected features are not representative, and the precision of the traditional breast cancer image classification method cannot meet clinical requirements. Compared with the traditional artificial diagnosis, the computer-aided diagnosis technology can overcome the subjectivity of the diagnosis of doctors, reduce the misdiagnosis and missed diagnosis phenomenon of the artificial diagnosis, and can process a large amount of image data in a short time by benefiting from the strong computing power of a computer, thereby remarkably improving the diagnosis speed and assisting the doctors to make reasonable decisions more quickly. The image classification method based on deep learning is widely applied to various classification tasks, and due to the particularity and complexity of medical images, the data set is very difficult to obtain, so that the data volume of the existing public data set is small, and the phenomenon of overfitting of the network is easily caused when a neural network is used for classifying breast cancer histopathology images.

In the image preprocessing stage, the most common method at present is to perform block processing on the image, which not only can reduce the training time of the network, but also can multiply the data volume. However, the blocking process is only suitable for the data set with pixel level labels, the labeling process cost is high due to the complexity of the breast pathological image, the Breakhis breast pathological image data set has only image level labels, the labels of the original images are assigned to the corresponding generated blocks by using the blocking process, however, benign regions exist in the breast malignant images, and the benign regions can cause wrong labeling of the blocks, thereby causing the performance of network classification to be reduced.

Disclosure of Invention

The invention relates to a classification method of a breast tissue pathological image, which solves the problem of over-fitting of wrongly marked blocks and small samples when a computer assists classification of the breast tissue pathological image in the prior art.

In order to achieve the purpose, the technical scheme of the invention is as follows:

a breast tissue pathological image classification method comprises the steps of firstly, freezing shallow convolution on two network models of VGG16 and ResNet50 by using a transfer learning method, and improving the network accuracy and generalization capability by using shallow universal characteristics; secondly, adding an attention module to extract breast lesion region information so as to enhance feature description; and finally, fusing class probability results of the two individual classifiers in a soft voting manner to obtain a final eight-classification result.

Further, the method comprises the following specific steps:

step S1: image preprocessing is performed on the breast histopathology image dataset;

step S2: a neural network constructed by taking a VGG16 network as a basic framework is used as a first individual classifier, the VGG16 network comprises 13 convolutional layers, and a migration learning fine tuning experiment is performed by taking an ImageNet data set as a source domain and a mammary gland histopathology image data set as a target domain in a layer-by-layer parameter freezing mode;

step S3: a neural network constructed by taking a ResNet50 network as a basic framework is used as a second individual classifier, the ResNet50 network comprises 4 types of residual blocks, and a migration learning fine tuning experiment is carried out by taking an ImageNet data set as a source domain and a breast histopathology image data set as a target domain in a mode of gradually freezing parameters of the residual blocks;

step S4: adding channel attention and spatial attention in a VGG16 network and a ResNet50 network respectively, wherein the adding position is after the last convolutional layer and before the full connection layer, the sequence is that the channel attention is in front, and the spatial attention is in back;

step S5: constructing a mammary gland image classification network combining attention and dual-model migration fusion;

step S6: and training a mammary gland image classification network combined with attention and dual-model migration for classification detection of mammary gland histopathology images.

Compared with the prior art, the invention has the beneficial effects that:

the early diagnosis of the breast cancer is crucial to the later treatment, and good and malignant classification is carried out through a computer-aided breast cancer pathological image, so that good prevention and detection effects can be achieved. According to the invention, a transfer learning mechanism is applied to a breast image classification task, a large public data set is used in advance to obtain certain priori knowledge, the network learning efficiency is improved, and overfitting of model training is avoided. The attention mechanism integrates feature information of space attention and channel attention, the salient features of the space and the channel in the image are learned, and attention salient features are selected so as to better capture key information, inhibit influence of interference areas such as noise and background on decision making and improve performance of a classification network. The soft voting algorithm is adopted to vote the class probability results of the two classifiers, and the class labels are directly voted by comparing with the hard voting, so that the classification capability of the confusable classes can be improved. The invention can realize high-precision classification of the breast histopathology images, including benign and malignant two-classification and eight-subtype multi-classification of the breast.

Drawings

FIG. 1 is a schematic diagram of the structure of a classification network for breast histopathology images according to the present invention;

FIG. 2 is a view of a channel attention map;

FIG. 3 is a spatial attention block diagram;

FIG. 4 is a layer-by-layer freezing effect diagram of VGG 16;

fig. 5 is a diagram of the effect of resenet 50 freezing layer by layer (residual block);

FIG. 6 is an eight class confusion matrix;

FIG. 7 is a two-class confusion matrix.

Detailed Description

The present invention will be described in further detail with reference to the following detailed description and accompanying drawings. In the following description, numerous details are set forth in order to provide a better understanding of the present application. However, those skilled in the art will readily recognize that some of the features may be omitted or replaced with other elements, materials, methods in different instances. According to the invention, a classification network of the breast histopathology image combining attention and dual-model migration fusion is constructed, the characteristic information of space attention and channel attention is fused, and the significance characteristics of the space and the channel in the image are learned, so that the classification effect of the network is greatly improved; the soft voting algorithm is adopted to vote the class probability results of the two classifiers, and the class labels are directly voted by comparing with the hard voting, so that the classification capability of the confusable classes can be improved. The invention can realize high-precision classification of the breast histopathology images, including benign and malignant two-classification and eight-subtype multi-classification of the breast.

Example (b):

referring to fig. 1, the method of the present embodiment includes the following steps:

step S1: image preprocessing is performed on the breast histopathology image dataset;

the training effect of the neural network can be determined by the amount of data, 7909 pictures are contained in the breast histopathology Breakhis data set, but the learning of the neural network is far from enough, so that overfitting of the network is easily caused, and therefore, data enhancement is necessary. Common data enhancement methods are: cutting, rotating, turning, translating, adjusting contrast, and the like. The breast histopathology image has the characteristic of rotation invariance, the diagnosis of a pathologist does not change due to the change of the image angle, but the image is completely different after rotation transformation for a machine.

Step S1 includes the following steps:

step S101: performing image preprocessing with the Breakwheis image size of the breast histopathology image dataset [700,460], redefining the total image [224,224] as input to a neural network;

step S102: dividing a data set, taking 65% of pictures as a training set and 35% of pictures as a testing set;

step S103: and performing data enhancement, including random clipping, random rotation, random horizontal turnover and random vertical turnover.

Step S2: a neural network constructed by taking a VGG16 network as a basic framework is used as a first individual classifier, the VGG16 network comprises 13 convolutional layers, and a migration learning fine-tuning experiment is performed by taking an ImageNet data set as a source domain and a mammary gland histopathology image data set as a target domain in a layer-by-layer parameter freezing mode.

The deepening of the neural network causes the parameter quantity to increase, if the training data set samples are insufficient, the neural network cannot fully learn the characteristic rule of the target task, and the random initialization also prolongs the network fitting time. When the data volume of the target domain is insufficient, with the help of the migration learning, the experimental result of the target task can be obviously improved by means of the related task with the sufficient data volume existing in the source domain. The network parameters are trained in advance in the large-scale public database by the transfer learning instead of random initialization and complete de novo training, so that the training time can be shortened, the overfitting phenomenon can be effectively inhibited, and the network generalization capability is improved.

The ImageNet data set has more than 1400 million marked pictures, and is trained on a classical network model, so that the obtained training parameters can be migrated to a classification task of the breast histopathology image. However, the pictures in the ImageNet data set are mainly real pictures in daily life and have low similarity with breast tissue pathological images, so that a model fine-tuning method is used, the method belongs to a parameter (model) -based transfer learning method, the difference between samples can be overcome, an optimization model suitable for a target task is obtained in a mode of transferring model parameters, the specific method is to transfer knowledge to a breast image classification task in a mode of freezing the parameters layer by layer, and the capability and the efficiency of extracting network features are improved by utilizing shallow features such as edges, contours and textures.

Step S2 includes the following steps:

step S201: changing the output dimension of the classifier from 1000 to 8 according to a classification task by taking a VGG16 network as a basic framework, and adding a Dropout layer between the last three fully-connected layers;

step S202: selecting an Adam optimizer to minimize a cross entropy loss function as a training strategy, wherein the initial value of the learning rate is 0.0001;

step S203: pre-training the ImageNet data set by the network, and selecting and migrating a first layer of convolution, wherein the specific method is that the first layer of convolution parameters are frozen, and then the convolution layer parameters are continuously updated along with network learning;

step S204: inputting preprocessed breast histopathology images, disordering images in a data set, inputting 16 images in each batch into a neural network, learning for 50 rounds, and storing results and models;

step S205: repeating the steps S23 and S24, wherein the step S23 shifts convolution to be increased step by step, namely freezing the former two layers of convolution parameters, the former three layers of convolution parameters … … and the like;

step S206: and comparing the migration learning fine adjustment results based on the VGG16 network structure, and selecting the most appropriate migration strategy to perform a fusion experiment.

Step S3: a neural network constructed by taking a ResNet50 network as a basic framework is used as a second individual classifier, the ResNet50 network comprises 4 types of residual blocks, and a migration learning fine tuning experiment is carried out by taking an ImageNet data set as a source domain and a breast histopathology image data set as a target domain in a mode of gradually freezing parameters of the residual blocks;

step S3 includes the following steps:

step S301: changing the output dimension of the classifier from 1000 to 8 according to a classification task by taking a ResNet50 network as a basic framework, and adding a Dropout layer between the last three fully-connected layers;

step S302: selecting an Adam optimizer to minimize a cross entropy loss function as a training strategy, wherein the initial value of the learning rate is 0.0001;

step S303: pre-training the ImageNet data set by the network, and selecting and migrating a first residual block, wherein the specific method is freezing the parameter of the first residual block, and then continuously updating the parameter of the convolutional layer along with network learning;

step S304: inputting preprocessed breast histopathology images, disordering images in a data set, inputting 16 images in each batch into a neural network, learning for 50 rounds, and storing results and models;

step S305: repeating the steps S33 and S34, wherein the step S33 shifts the residual block to increase step by step, i.e., freezes the first two residual block parameters, the first three residual block parameters … …, and so on;

step S306: and comparing the migration learning fine adjustment results based on the ResNet50 network structure, and selecting the most appropriate migration strategy to perform a fusion experiment.

Step S4: adding channel attention and space attention in a VGG16 migration network and a ResNet50 migration network respectively;

the human visual attention mechanism can quickly lock a target area in a complex scene, simulate the visual perception of human eyes, select attention key characteristics and omit unimportant redundant information. The attention mechanism firstly learns the weight of the corresponding dimension to express the attention degree of the information, the interference of the non-important characteristic on the classification result can be restrained when the weight is small, the contribution of the useful information to the network decision can be enhanced when the weight is large, and finally the final attention diagram is obtained by multiplying the input characteristic diagram. The attention mechanism used by the invention combines two dimensions of channel attention and space attention to carry out adaptive feature optimization.

Step S4 includes the following steps:

step S401: and (3) respectively adding a channel attention module after the last convolutional layer of each network by using the VGG16 framework network model with the best migration effect in the step 2 and the ResNet50 framework network model with the best migration effect in the step 3.

The characteristics extracted by different channels in the convolutional neural network are different, such as texture, color, contour and other information, and the contribution degree of different characteristics to the target task is different, so that the problem of 'what' of the target can be solved by performing weighting processing on each channel through the channel attention module.

Referring to fig. 2 and 3, step S401 specifically includes:

step S411: compressing the space dimension of the given feature map F, specifically performing global average pooling and global maximum pooling to obtain the feature of the channel dimension;

step S412: respectively carrying out feature learning through two layers of convolution networks;

step S413: adding the two feature maps;

step S414: obtaining a channel attention weight through the activation of a Sigmoid function;

step S415: the channel attention weight is multiplied by the input feature map F.

The channel attention module enables the network to focus on important dimensional features by assigning weights to different channel features, i.e., giving different attention to different features.

The spatial attention module simulates human eye perception mechanism and solves the problem of 'where' the target is. And weight distribution is carried out on the space dimension by compressing the dimension of the channel, so that the focus of the space position of the focus area in the pathological image is realized.

Step S402: a spatial attention module is added after the channel attention.

Referring to fig. 2, step S402 specifically includes:

step S421: compressing the channel dimension for the given feature map F, specifically performing global average pooling and global maximum pooling to obtain the features of the space dimension;

step S422: connecting the two characteristic graphs in channel dimension;

step S423: performing feature learning through a layer of convolutional neural network;

step S424: obtaining a space attention weight through the activation of a Sigmoid function;

step S425: the spatial attention weight is multiplied with the input feature map F.

The space attention module compresses the feature map through the channel dimension and performs feature calibration in the space dimension, and weights are distributed to feature values of each position in space, so that the significance information of the space dimension is extracted, and irrelevant background areas are suppressed.

Step S5: constructing a mammary gland image classification network combining attention and dual-model migration fusion;

the classifier fusion is to combine decision results of a plurality of individual classifiers in some way to judge classification, and common fusion ways include a majority voting method, a weighted voting method and the like. The research proves that the classification by adopting the multi-classifier fusion is far superior to an individual classifier. The majority voting method and the weighted voting method adopt class label voting, and the soft voting algorithm adopts a class probability voting mode.

Step S5 specifically includes:

step S501: changing the output of the VGG16 framework migration network combined with attention and the ResNet50 framework migration network combined with attention, and directly outputting the octasubtype class probability;

step S502: and fusing the class probability results of the VGG16 framework migration network combined with attention and the ResNet50 framework migration network combined with attention by a soft voting algorithm.

TABLE 1

Table 1 shows comparison of evaluation results of different models in an eight-classification task, and experimental results show that the method of the present invention learns salient features of a space and a channel in an image due to fusion of feature information of space attention and channel attention, which greatly improves a classification effect of a network.

Step S6: and training a mammary gland image classification network combined with attention and dual-model migration for classification detection of mammary gland histopathology images.

Step S6 specifically includes:

step S601: the two models select an Adam optimizer to minimize a cross entropy loss function as a training strategy, and the initial value of the learning rate is 0.0001;

step S602: inputting preprocessed breast histopathology images, disordering images in a data set, inputting 2 images in each batch into a neural network, learning 100 rounds and storing results and models;

in order to fully evaluate the network model, the model of the present invention is compared with the model with better performance by using various indexes, including accuracy, precision, recall rate and F1 value, and the specific comparison result is shown in table 2. Aiming at the two classification tasks of the breast tissue pathological images, compared with other classification models, the method has higher accuracy and excellent performance of various indexes, and the model has good performance.

TABLE 2

The experimental results show that aiming at the two-classification task and the eight-classification task of the breast histopathology image, the multi-model fusion algorithm for fusing the attention mechanism and the deep migration learning provided by the invention is superior to other algorithms such as BHCNet, and the training time is not excessively increased by adding the attention module and fusing the two models, so that the model of the invention achieves better effects on the classification accuracy and the training efficiency.

The depth migration layer-by-layer freezing effect of the VGG16 framework of the individual classifier is shown in FIG. 4, and it can be known that the convolution effect of the two layers before the VGG16 framework network is frozen is best. The effect of deep migration per-residual block freezing of the individual classifier ResNet50 framework is shown in FIG. 5, and it can be seen from the figure that the ResNet50 framework network has the highest precision when freezing the first 5 residual blocks. Fig. 6 is a confusion matrix test performed on a test set picture according to the present invention, where the confusion matrix presents classification results in the form of a matrix, and can represent the confusion degree of a neural network model on a classification task, and rows of the confusion matrix in the figure represent real categories and columns represent prediction categories of the model. FIG. 7 is a benign and malignant binary confusion matrix.

The above are specific embodiments of the present invention, but the structural features of the present invention are not limited thereto, and the present invention can be applied to similar products, and any changes or modifications within the scope of the present invention by those skilled in the art are covered by the claims of the present invention.