1. A zebra fish morphological classification method based on Mask R-CNN is characterized by comprising the following steps:

s1, acquiring the appearance phenotype data of the zebra fish juvenile fish, and establishing an image data set of the appearance phenotype of the zebra fish juvenile fish;

s2, dividing the image data set into a training data set and a testing data set, and preprocessing and data enhancing the zebra fish shape image in the training data set by using imgauge;

s3, building a Mask-RCNN target instance segmentation network model, and inputting the image data set image and the labeling information into a network for training;

s4, carrying out overall index evaluation on the model;

s5, calculating the appearance image of the zebra fish juvenile fish by using the trained model to obtain the classification result, the class confidence, the outer surrounding frame and the segmentation outline of all zebra fish in the image.

2. The zebra fish morphological classification method based on Mask R-CNN as claimed in claim 1, wherein step S1 includes: the method comprises the steps of placing the zebra fish juvenile fish under a microscope, shooting and collecting images of the zebra fish juvenile fish with eight common morphological types including normal hatching, spinal curvature, pericardial edema, necrosis after hatching, tail downward bending, tail upwarping, yolk elongation deformity and yolk sac edema, and establishing an image data set of the morphological phenotype of the zebra fish juvenile fish.

3. The zebra fish morphological classification method based on Mask R-CNN as claimed in claim 1, wherein step S2 includes: and (3) preprocessing the image data set, labeling the amplified image data set by using a Labelme open source labeling tool, acquiring the label type and contour of the individual zebra fish juvenile fish, and calculating the data obtained by data amplification according to the label type and contour of the labeled data.

4. The zebra fish morphological classification method based on Mask R-CNN as claimed in claim 1, wherein step S2 includes: and performing data amplification on the morphology categories with less data quantity by using an image warping algorithm through an open source software library imgauge in the field of computer vision, and performing data amplification on all training data in a brightness conversion, contrast conversion and noise adding mode.

5. The zebra fish morphological classification method based on Mask R-CNN as claimed in claim 1, wherein step S2 includes: creation of TFRecord file data, the image data set is divided into a training data set and a testing data set according to a 7:3 ratio, and packed into TFRecord format.

6. The zebra fish morphological classification method based on Mask R-CNN as claimed in claim 1, wherein the step S3 is based on a network model of an encoder and a decoder, the encoder part utilizes down-sampling to perform feature extraction, the structure of the encoder part comprises a backbone network ResNet-50, RPN and FPN, the backbone network is used for extracting image information to generate semantic feature maps of different scales, the RPN detects the object on the semantic feature maps of different scales according to the size proportion of the object in the original image, and the FPN performs dimension reduction on the object potential region on the semantic feature maps of different scales through convolution of 1 x 1 to be fused to one semantic feature map.

7. The zebra fish morphological classification method based on Mask R-CNN as claimed in claim 6, wherein the decoder is divided into ROIAlign, Mask head, class head and bbox head, ROI Align improves the matching precision of the compressed feature map and the original map according to bilinear interpolation algorithm; the Mask head predicts an image Mask of the target according to the target region feature map passing through ROIAlign; meanwhile, the target area feature map passing through ROIAlign is sent to a full connection layer for decoding, a class head and a bbox head are accessed behind the full connection layer, and the classification result and the coordinates of the outer surrounding frame of the target area are obtained.

8. The zebra fish morphological classification method based on Mask R-CNN as claimed in claim 1, wherein step S3 includes: training is based on a Tensorflow deep learning framework, a graphic processor GeForce RTX 2080Ti of NVIDIA is used for calculation, and a CUDA10.1 library is used for acceleration.

9. The zebra fish morphological classification method based on Mask R-CNN as claimed in claim 1, wherein step S4 includes: and evaluating the trained model to obtain the accuracy index of the model, including classification accuracy mAP, intersection ratio of the surrounding frames and intersection ratio of the outlines.

10. The zebra fish morphological classification method based on Mask R-CNN as claimed in claim 1, wherein step S5 includes: and inputting the test set picture into a trained model for prediction, and drawing the zebra fish pathological classification result, the classification confidence coefficient, the outer surrounding frame and the outline segmentation obtained through prediction on the original picture.

Background

Zebra fish is an ideal model organism for screening environmental toxic substances, artificial chemical substances and medicines, and embryos of the zebra fish are mostly used in acute toxicity screening to evaluate the biological toxicity of the chemical substances. In the traditional method, researchers observe the development condition of zebra fish embryos through a microscope, and judge the influence degree of chemical substances on the embryos through the toxicity end points (shapes such as death, hatching, development delay, deformity and the like). However, in the existing zebra fish morphological evaluation system, because a plurality of moving fishes usually exist in one visual field at the same time, a lot of time and labor cost are needed for rapidly and accurately observing and recording the morphological characteristics of the fishes and making a judgment. And when data are compared in parallel in mutually independent researches, the form evaluation of the zebra fish is based on subjective judgment of different researchers, and the possibility of data deep mining is limited due to the influence of artificial priori knowledge. With accumulation of chemical substances requiring toxicity test and explosive increase of image data, an image detection method capable of rapidly and efficiently automatically analyzing the morphology and the type of zebra fish embryos is needed.

Patent CN111583207A discloses a zebra fish juvenile fish heart contour determination method, which collects a plurality of test set zebra fish juvenile fish microscopic images and training set zebra fish juvenile fish microscopic images; cutting the microscopic image of each training set zebra fish juvenile fish to obtain a juvenile fish area after cutting; carrying out ellipse fitting on each juvenile fish area to obtain the characteristic point coordinates of the heart contour of the zebra fish juvenile fish; training a convolutional neural network according to each juvenile fish area and the corresponding characteristic point coordinates; optimizing the trained network model according to the test set zebra fish juvenile microscopic image; inputting the zebra fish juvenile fish microscopic image to be predicted as input into the optimized network model to obtain the characteristic point coordinates of the heart contour of the juvenile fish to be predicted; the heart contour of the zebra fish juvenile fish is determined according to the characteristic point coordinates of the heart contour of the juvenile fish to be predicted, however, the method is only used for predicting the heart position of the zebra fish juvenile fish under the condition of single prediction, and the function is limited; according to the method, the head part of the zebra fish juvenile fish needs to be manually intercepted before the heart position of the zebra fish is predicted, and manpower is consumed; the patent generates heart areas of the zebra fish juvenile fish by predicting eight key points on an ellipse, and cannot finely fit the heart positions of the zebra fish juvenile fish in various forms. The method can predict the types and contour masks of all zebra fish juvenile fishes contained in the picture at the same time; the image shot by the experiment can be directly processed, and the middle process does not need manual participation; the whole position of zebra fish juvenile fish is predicated through the outline mask of pixel level to this patent, and meticulous degree is high, need not be restricted to specific form.

Disclosure of Invention

The invention aims to provide a zebra fish morphology classification method based on Mask R-CNN aiming at the problems that the zebra fish morphology evaluation process is time-consuming and the evaluation standard is ambiguous, and based on the existing deep learning technology, batch processing and automatic morphology classification of zebra fish morphology images are realized.

The purpose of the invention is realized by the following technical scheme:

a zebra fish morphological classification method based on Mask R-CNN comprises the following steps:

s1, acquiring morphology phenotype data of the zebra fish juvenile fish, wherein the morphology phenotype data comprises eight common morphology types of zebra fish juvenile fish images including normal hatching, spinal curvature, pericardial edema, necrosis after hatching, tail downward bending, tail upwarping, yolk extension deformity and yolk sac edema, and establishing an image data set of the morphology phenotype of the zebra fish juvenile fish;

s2, dividing the zebra fish shape data set into a training data set and a testing data set according to the proportion of 7:3, and preprocessing and enhancing data of the zebra fish shape images in the training set by using imgauge to improve the robustness of the model;

s3, building a Mask-RCNN target instance segmentation network model, and inputting the eight images of normal incubation, spinal curvature, pericardial edema, necrosis after incubation, tail downward bending, tail upwarping, yolk extension deformity, yolk sac edema and labeling information into the network for training. The model uses ResNet-50 as a backbone network, RPN and FPN are used for multi-scale target detection and feature fusion, the example segmentation precision is improved by using a bilinear difference method through an ROI Align structure, and target detection, example segmentation and classification prediction are divided into three heads to be trained simultaneously, so that the target example segmentation of zebra fish in the image is realized;

s4, carrying out overall index evaluation on the model;

s5, calculating the appearance image of the zebra fish juvenile fish by using the trained model to obtain the classification result, the class confidence, the outer surrounding frame and the segmentation outline of all zebra fish in the image.

Further, step S1 includes: placing the zebra fish juvenile fish (72-120 hpf) under a microscope, and shooting and collecting the zebra fish juvenile fish images with eight common morphology types including normal hatching, spinal curvature, pericardial edema, necrosis after hatching, tail downward bending, tail upward tilting, yolk elongation deformity and yolk sac edema. And establishing an image data set of the appearance phenotype of the zebra fish juvenile fish.

Further, step S2 includes: pre-processing of the image dataset. And labeling the amplified image data set by using a Labelme open source labeling tool to obtain the label type and contour of the individual zebra fish juvenile fish. And calculating the data obtained by data amplification according to the label type and the outline of the labeled data.

Further, step S2 includes: augmentation of the image dataset. In order to improve the robustness of the model for identifying the appearance of the zebra fish juvenile fish and avoid the influence of unbalanced quantity of training data of all appearance categories on the accuracy of the model, an open source software library imgauge in the field of computer vision is used for carrying out data amplification on the appearance categories with less data quantity through an image distortion algorithm, and all training data are subjected to data amplification through brightness conversion, contrast conversion, noise addition and other modes. And carrying out data enhancement on the whole data set according to 13 image enhancement algorithms of vertical flip, horizontal flip, rotation, scaling, pixel multiplication, adjacent pixel multiplication, contrast adjustment, Gaussian noise, Gaussian blur, sharpening, super-pixel enhancement, single-channel enhancement and color space enhancement in a random threshold and random combination mode.

Further, step S2 includes: and (5) creation of TFRecord file data. The image data sets were classified, 70% as training data set and 30% as test data set, and packed into TFRecord format for reading and processing in TensorFlow.

Further, the network model in step S3 is based on the encoder and decoder structures, the encoder part performs feature extraction by using downsampling, the structure of the encoder part is composed of a backbone network ResNet-50, an RPN and an FPN, the backbone network is used for extracting image information to generate semantic feature maps of different scales, the RPN detects the target on the semantic feature maps of different scales according to the size ratio of the target in the original image, and the FPN performs dimension reduction on the target potential area on the semantic feature maps of different scales by convolution of 1 × 1 to be fused to one semantic feature map.

The decoder part is divided into ROI Align, mask head, class head and bbox head, and the ROI Align improves the matching precision of the compressed feature map and the original image according to a bilinear interpolation algorithm, so that a more accurate image mask is generated; the Mask head predicts an image Mask of the target according to the characteristic graph of the target region passing through the ROI Align; meanwhile, the characteristic diagram of the target region passing through the ROI Align is sent to the full-connection layer for decoding, and a class head and a bbox head are accessed behind the full-connection layer to obtain a classification result and an outer surrounding frame coordinate of the target region.

Further, step S3 includes: training is based on a Tensorflow deep learning framework, a graphic processor GeForce RTX 2080Ti of NVIDIA is used for calculation, and a CUDA10.1 library is used for acceleration.

Further, step S4 includes: and evaluating the trained model to obtain the accuracy indexes of the model, including classification accuracy mAP, intersection ratio of the surrounding frames and intersection ratio of the outlines.

Further, step S5 includes: and inputting the test set picture into a trained model for prediction, and reading a prediction picture. Defining an accumulative scoring rule, wherein the test set picture comprises eight common morphology types including normal hatching, spinal curvature, pericardial edema, necrosis after hatching, tail downward bending, tail upward warping, yolk elongation deformity and yolk sac edema, the accumulative scoring rule is that the total score is 5, one score is deducted for each appearance type, the morphology types displayed by the prediction graph are obtained for accumulative scoring, and the results are output to a text file one by one.

Compared with the prior art, the invention has the following beneficial effects:

the zebra fish image and video analysis method is based on the existing classical network design mode of a Mask R-CNN network, integrates a new model application concept, objectively classifies the morphology types by identifying and detecting the morphology characteristics, establishes an objective zebra fish morphology classification system, realizes rapid and accurate batch processing of experimental pictures, reduces the labor cost, improves the efficiency and ensures the accuracy and the reproducibility. The method can simultaneously predict the types and contour masks of all the zebra fish juvenile fishes contained in the picture, namely, multiple targets can be simultaneously identified in a single visual field, the types can be accurately analyzed, and different types of deformities (deformity superposition) can be accurately identified; the image shot in the experiment can be directly processed, the intermediate process does not need manual participation, the image can be accurately identified without being limited by the image background, and the accurate identification is not influenced because the longitudinal background has noise or the background value is not clean; the method predicts the whole position of the zebra fish juvenile fish through the pixel-level contour mask, has high fineness degree, and is not limited to a specific form, namely the recognition accuracy is not limited by the direction, the angle and the view of the juvenile fish; the accurate identification of the patent is not limited by microscopic magnification and data of different test batches, and the generalization capability of the model is strong.

Drawings

FIG. 1 is a flow chart of an embodiment of the present invention;

FIG. 2 is an original image containing different topographical types of zebra fish;

FIG. 3 is an original image with tail sag type after data enhancement according to the present invention;

FIG. 4 is a schematic structural diagram of a Mask R-CNN model used in the present invention;

fig. 5 is a diagram of a backbone network architecture used in the present invention;

FIG. 6 illustrates training loss during model training according to the present invention;

FIG. 7 is a display of the results of the prediction of images after model training in accordance with the present invention.

Detailed Description

The invention is described in detail below with reference to the figures and specific embodiments.

A zebra fish morphological classification method based on Mask R-CNN (fig. 5), fig. 1 is an implementation flow chart of the present invention, comprising the following steps:

s1, obtaining the appearance phenotype data of the zebra fish juvenile fish:

and (3) placing the zebra fish juvenile fish (72-120 hpf) under an integrated digital liquid crystal microscope for shooting.

As shown in fig. 2, images of zebra fish juvenile fish including eight common morphological types including normal hatching, spinal curvature, pericardial edema, post-hatching necrosis, caudal recurvation, caudal upwarping, yolk elongation deformity and yolk sac edema are collected, and an image data set of the morphological phenotype of the zebra fish juvenile fish is established.

S2, preprocessing of the image data set and data enhancement:

pre-processing of the image dataset. And labeling the amplified image data set by using a Labelme open source labeling tool to obtain the zebra fish juvenile fish individual and tissue outlines of the heart, yolk sac, back, tail and swim bladder of the zebra fish juvenile fish.

And (5) creation of TFRecord file data. The image data sets were classified, 70% as training data set and 30% as test data set, and packed into TFRecord format, facilitating their reading and processing in TensorFlow.

The TFRecord data file is a binary file which uniformly stores the characteristic data and the labels corresponding to the characteristic data, and can enable the data to be quickly copied, moved, read and stored in Tensflow.

The TFRecord file uses a Protocol Buffer (Protocol Buffer) structure data serialization tool, and realizes the data storage and exchange functions between the memory and the hard disk by performing serialization definition on the structured data. When image data is read for network training, only one binary file needs to be loaded at one time.

The invention improves the calculation processing efficiency by dividing a large amount of training data into a plurality of TFRecord files.

Augmentation of the image dataset. According to the method, the problem of unbalanced quantity of types of all appearance types of the image data set is solved in a data enhancement mode, and the robustness of the model for zebra fish instance segmentation and classification is improved.

Fig. 3 shows the data enhancement effect of the present invention on the image data set through the imgauge development library. A total of 14 image processing algorithms including vertical flipping, horizontal flipping, rotation, scaling, pixel multiplication, adjacent pixel multiplication, contrast adjustment, gaussian noise, gaussian blur, warping, sharpening, superpixel enhancement, single channel enhancement, and color space enhancement are applied.

Since zebra fish larvae usually appear in the image in a variety of positional modalities, the present invention first processes the data set with vertical flipping, horizontal flipping, rotation, and scaling of the image.

Calculation formula of vertical flipping: [ x, y,1 ]]＝[x₀,H-y₀,1]

Wherein x and y are inverted coordinates, x₀，y₀H is the image height.

Calculation formula of horizontal turning: [ x, y,1 ]]＝[W-x₀,y₀,1]

Wherein x and y are inverted coordinates, x₀，y₀W is the coordinate before flipping and W is the image width.

Scaling calculation formula:

wherein x and y are inverted coordinates, x₀，y₀As coordinates before flipping, S_xTo a horizontal scaling of S_yIs vertical scaling.

Calculation formula of rotation:

wherein x and y are inverted coordinates, x₀，y₀Is the coordinate before turning over, alpha is the rotation angle, x_cIs the abscissa of the center of rotation, y_cIs the ordinate of the centre of rotation.

Calculation formula of image blur:

calculation formula of gaussian blur:

the calculation formula of the contrast is as follows: c ═ Σ_δδ(i,j)²P_δ(i, j), where δ (i, j) ═ i-j |, is the difference in gray levels between adjacent pixels; p_δ(i, j) is a pixel distribution probability that the gray scale difference between adjacent pixels is δ.

The pixel multiplication implements the change of the image characteristics by multiplying all pixels in the image by a random coefficient.

The adjacent pixel multiplication realizes the change of image characteristics by multiplying different random coefficients by different pixels in the image, and the algorithm ensures that the adjacent pixels cannot use the same coefficients.

The image sharpening algorithm performs convolution operation on the whole image through a Laplacian operator, so that the change of image characteristics is realized.

The super-pixel enhancement algorithm generates a certain number of super-pixels for each image according to the pixel average value of the original image, and replaces a single pixel in the original image with the super-pixels at random.

The single-channel enhancement algorithm multiplies a certain channel in RGB by a random coefficient and adds a random offset value, thereby changing the color characteristics of the original image.

The color space enhancement algorithm is to convert the original image into HSV color space, add a random value to the whole, and finally display the whole image matrix in RGB color space.

As in fig. 6, the model loss functions fall into two categories: 1) loss of RPN 2) loss of Mask RCNN headers.

The RPN loss is divided into foreground and background classification loss RPN class loss and target frame regression loss RPN bbox loss, and is responsible for improving the capability of the model for providing a potential target area from an original image.

RPN loss function:

representing the classification loss, P, of all anchors_iThe category true value of each anchor and the predicted value of each anchor. RPN only concerns the foreground and background partition of anchor, so for two classes, cross entropy loss is used.

Representing the regression loss of the outer bounding box, the smoothL1 function was used.

The Mask-RCNN headers loss is divided into classification loss class, regression loss bbox loss and outline Mask loss, and is responsible for improving the capability of the model in positioning, classifying and generating the outline Mask for the targets contained in the potential target area.

Mask-RCNN loss function is: l ═ L_cls+L_box+L_mask

WhereinThe penalty function for this portion is the same as the RPN penalty.

Where k is a certain category, m is mask side length of mask head, y_ijIs the label truth value of the (i, j) point in the mask,is the predicted value of the k category at the (i, j) point in the mask.

S3, building a Mask-RCNN target instance segmentation network model (as shown in figure 5), and inputting the image data set image and the labeling information into a network for training;

the model is based on the structures of an encoder and a decoder, the encoder part utilizes down sampling to extract features, the structure of the encoder part comprises a backbone network ResNet-50, an RPN and an FPN, the backbone network is used for extracting image information to generate semantic feature maps with different scales, the RPN detects targets on the semantic feature maps with different scales according to the size proportion of the targets in an original image, and the FPN reduces the dimensions of target potential areas on the semantic feature maps with different scales through convolution of 1 multiplied by 1 to be fused on one semantic feature map.

The decoder part is divided into ROI Align, mask head, class head and bbox head, and the ROI Align improves the matching precision of the compressed feature map and the original image according to a bilinear interpolation algorithm; the Mask head predicts an image Mask of the target according to the characteristic graph of the target region passing through the ROI Align; meanwhile, the characteristic diagram of the target region passing through the ROI Align is sent to a full-connection layer for decoding, a class head and a bbox head are accessed behind the full-connection layer, and a classification result and an outer surrounding frame coordinate of the target region are obtained.

Training is based on a Tensorflow deep learning framework, a graphic processor GeForce RTX 2080Ti of NVIDIA is used for calculation, and a CUDA10.1 library is used for acceleration.

S4, carrying out overall index evaluation on the model;

and evaluating the trained model to obtain the accuracy indexes of the model, including classification accuracy mAP, intersection ratio of the surrounding frames and intersection ratio of the outlines.

S5, calculating the appearance image of the zebra fish juvenile fish by using the trained model to obtain the classification result, the class confidence, the outer surrounding frame and the segmentation outline of all zebra fish in the image.

And inputting the test set picture into a trained model for prediction, and reading a prediction picture. Defining an accumulative scoring rule, wherein a test set picture comprises eight common morphology types including normal hatching, spinal curvature, pericardial edema, necrosis after hatching, tail downward bending, tail upward warping, yolk elongation deformity and yolk sac edema, the accumulative scoring rule is that the total score is 5, one score is deducted for each appearance type, the morphology types displayed by a prediction graph are obtained for accumulative scoring, results are output to a text file one by one, and fig. 7 shows the results of the prediction of the image after model training.

Table 1 shows the classification accuracy after model training, and the method can realize rapid and accurate batch processing of experimental pictures, reduce labor cost, improve efficiency, and ensure accuracy and reproducibility.

TABLE 1 Classification accuracy after model training

Category (Class)	Accuracy (Accuracy)
		Curvature of spine	87.14％
Necrosis after incubation	88.70％
		Downward bending of tail	74.20％
Yolk elongation deformity	83.94％
		Normal hatching	83.37％
Pericardial edema	88.38％
		The tail part upwarps	91.38％
Yolk sac edema	83.00％

The embodiments described above are described to facilitate an understanding and use of the invention by those skilled in the art. It will be readily apparent to those skilled in the art that various modifications to these embodiments may be made, and the generic principles described herein may be applied to other embodiments without the use of the inventive faculty. Therefore, the present invention is not limited to the above embodiments, and those skilled in the art should make improvements and modifications within the scope of the present invention based on the disclosure of the present invention.