1. A lung 4D-CT medical image registration method is characterized in that: the method comprises the following steps:

acquiring CT image data in a respiratory cycle;

at least two times of down sampling are carried out on the obtained CT image data in a respiratory cycle, and the obtained result is used as the input of a first full-connection network to obtain a first deformation field;

performing primary up-sampling operation on the first deformation field, performing primary down-sampling operation on the acquired 3D lung CT image in one period, and using the first deformation field and the acquired 3D lung CT image as the input of a second fully-connected network to obtain a second deformation field;

and performing one-time up-sampling operation on the second deformation field, and using the second deformation field and the obtained CT image data in one respiratory cycle as the input of a third fully-connected network together to obtain a third deformation field.

2. The pulmonary 4D-CT medical image registration method of claim 1, wherein:

the degree of alignment between the two images is calculated using the normalized cross-correlation as a measure of similarity for each fully connected network.

3. The pulmonary 4D-CT medical image registration method of claim 1, wherein:

the first, second and third fully connected networks each include a smoothness constraint in the loss function.

4. The pulmonary 4D-CT medical image registration method of claim 1, wherein:

and applying local folding constraint to the deformation field by adopting a Jacobian as a space folding penalty loss function and utilizing a Jacobian regularization method.

5. The pulmonary 4D-CT medical image registration method of claim 1, wherein:

the loss functions of the first fully connected network, the second fully connected network and the third fully connected network are: a weighted sum of a similarity measure loss function, a smooth regularization loss function, a spatial folding penalty loss function, and a period constraint loss function.

6. The pulmonary 4D-CT medical image registration method of claim 1, wherein:

the first, second and third fully connected networks each comprise four layers of encoders and decoders with a hopping connection layer.

7. The pulmonary 4D-CT medical image registration method of claim 1, wherein:

the first, second and third fully connected networks are structurally identical and share initial parameters.

8. A pulmonary 4D-CT medical image registration system, characterized by: the method comprises the following steps:

a data acquisition module configured to: acquiring CT image data in a respiratory cycle;

a first deformation field acquisition module configured to: at least two times of down sampling are carried out on the obtained CT image data in a respiratory cycle, and the obtained result is used as the input of a first full-connection network to obtain a first deformation field;

a second deformation field acquisition module configured to: performing primary up-sampling operation on the first deformation field, performing primary down-sampling operation on the acquired 3D lung CT image in one period, and using the first deformation field and the acquired 3D lung CT image as the input of a second fully-connected network to obtain a second deformation field;

a third deformation field acquisition module configured to: and performing one-time up-sampling operation on the second deformation field, and using the second deformation field and the obtained CT image data in one respiratory cycle as the input of a third fully-connected network together to obtain a third deformation field.

9. A computer readable storage medium having a program stored thereon, which program, when being executed by a processor, is adapted to carry out the steps of a method for registration of a 4D-CT medical image of a lung as claimed in any one of the claims 1 to 7.

10. An electronic device comprising a memory, a processor and a program stored on the memory and executable on the processor, wherein the processor when executing the program performs the steps in a method of registration of a 4D-CT medical image of a lung as claimed in any one of claims 1 to 7.

Background

The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.

Pulmonary 4D-CT refers to a collection of CT images of the same patient during a respiratory cycle, which can reflect the motion of a tumor in the lung. By registering the tumors in the 4D-CT images of the lung, the tumors at different moments are reflected in the same coordinate scale, and the lung tumors can be subjected to radiotherapy. Registration of 4D-CT images of the lungs is therefore a critical step in the planning and delivery of lung cancer radiotherapy.

Deformation image registration has a history of decades as a fundamental task in medical image research and remains a popular research topic today. The purpose of the deformation image registration is to establish a dense nonlinear spatial correspondence between two images and adopt appropriate nonlinear transformation to register the two images, so that the similarity between the two images is maximized. Conventional deformation registration methods typically model the problem as an optimization problem and strive to minimize the energy function in an iterative manner. However, these methods typically represent the registration problem as an independent iterative optimization problem. Therefore, the registration time is significantly increased.

With the rapid development of deep learning in recent years, many deep learning-based medical image registration methods are proposed in succession, and these methods generally define the registration problem as the learning problem of the convolutional neural network. Compared with the traditional method, the method based on deep learning can effectively predict the conversion of the test image pair through a pre-trained model, and realizes the registration rapidly. Methods based on deep learning may be classified into a supervised learning method and an unsupervised learning method according to whether or not the supervised information is required. Supervised learning methods typically require image-specific "gold standards" (manual labeling by physicians, etc.) to guide the learning of the network.

The inventor finds that: although the method of supervised learning speeds up the process of image registration, supervised methods are often of limited effectiveness in practice due to the difficulty of collecting "gold standard" information; furthermore, the registration performance of supervised learning approaches depends strongly on the quality of the predefined "gold standard". Unsupervised methods learn dense spatial mappings between input image pairs in an unsupervised manner by using convolutional neural networks, spatial transformations, and differentiable similarity functions. The methods do not depend on any predefined supervision information when network training is carried out, and many researches show that the methods achieve the registration accuracy equivalent to that of a classical registration method while realizing quick registration. However, most of the above methods adopt a pair-wise mode to input images in a data set into a network for learning, so that although a good result is obtained, the obtained network training parameters are only related to a selected fixed image, and a part of image features are probably ignored. In addition, the method based on paired input also needs a large amount of data sets to train network parameters, while the number of medical image data sets is constrained by multiple parties, and there is not enough data to train, so in order to alleviate the lack of data volume, the existing method of paired training mixes CT images of different patients to train the network, which inevitably causes some registration errors. Finally, most of the existing deep learning-based methods generally only carry out regularization constraint on the deformation field, although the weight of the regularization constraint can control the flatness of the deformation field, the registration accuracy of the model can be reduced when the weight of the regularization constraint is large; folding phenomena in the transformation (usually implying registration errors) occur when the weights of the regularization constraints are small.

Disclosure of Invention

In order to solve the defects of the prior art, the disclosure provides a lung 4D-CT medical image registration method (Multiple-net) and a system, and the specific content is that an original image is a high-resolution image, the original image is respectively subjected to down-sampling once and down-sampling twice to obtain middle-resolution and low-resolution images, the images with different resolutions are input into a network to learn different scale characteristics, the obtained low-resolution deformation field is subjected to up-sampling once and is used as the input of a corresponding high-resolution network, and finally, a total registration deformation field is obtained, so that the image registration problem when a medical image data set is less is solved, and the registration accuracy is greatly improved.

In order to achieve the purpose, the following technical scheme is adopted in the disclosure:

the first aspect of the disclosure provides a lung 4D-CT medical image registration method.

A lung 4D-CT medical image registration method comprises the following processes:

acquiring CT image data in a respiratory cycle;

at least two down-sampling is carried out on the acquired CT image data in a respiratory cycle, and the obtained result is used as the input of a first full-connection network (namely, a low-resolution network) to obtain a first deformation field

The first deformation fieldPerforming an upsampling operation on the acquired 3D pulmonary CT image of one period, performing a downsampling operation on the acquired 3D pulmonary CT image of one period, and using the upsampling operation and the downsampling operation as the input of a second fully-connected network (namely, a medium-resolution network) to obtain a second deformation field

Subjecting the second shape to deformationPerforming an upsampling operation, and using the upsampling operation and the obtained CT image data in one respiratory cycle as the input of a third fully-connected network (i.e. high-resolution network) to obtain a third deformation field

Further, the degree of similarity between the two images is calculated using the normalized cross-correlation as a similarity measure for each fully connected network.

Further, the loss functions of the first fully-connected network, the second fully-connected network and the third fully-connected network all include smooth constraints, so that an unrealistic or discontinuous deformation field is avoided.

Furthermore, a Jacobian is adopted as a space folding penalty loss function, local folding constraint is applied to the deformation field by utilizing a Jacobian regularization method, the occurrence of deformation field folding is reduced, and high registration accuracy is kept.

Further, the loss functions of the first fully connected network, the second fully connected network and the third fully connected network are: a weighted sum of a similarity measure loss function, a smooth regularization loss function, a spatial folding penalty loss function, and a period constraint loss function.

Further, the first fully connected network, the second fully connected network and the third fully connected network each comprise four layers of encoders and decoders with a hopping connection layer.

Further, the first fully connected network, the second fully connected network and the third fully connected network have the same structure and share the initial parameters.

A second aspect of the present disclosure provides a pulmonary 4D-CT medical image registration system.

A pulmonary 4D-CT medical image registration system, comprising:

a data acquisition module configured to: acquiring CT image data in a respiratory cycle;

a first deformation field acquisition module configured to: at least two times of down-sampling are carried out on the obtained CT image data in a respiratory cycle, and the obtained result is used as the input of a first full-connection network to obtain a first deformation field

A second deformation field acquisition module configured to: the first deformation fieldPerforming one-time up-sampling operation on the acquired 3D of one periodThe lung CT image is subjected to one-time down-sampling operation, and the lung CT image are jointly used as the input of a second fully-connected network to obtain a second deformation field

A third deformation field acquisition module configured to: subjecting the second shape to deformationPerforming one-time up-sampling operation, and using the obtained CT image data in one respiratory cycle as the input of a third fully-connected network to obtain a third deformation field

A third aspect of the present disclosure provides a computer readable storage medium having stored thereon a program which, when executed by a processor, performs the steps in a method of registration of 4D-CT medical images of a lung as described in the first aspect of the present disclosure.

A fourth aspect of the present disclosure provides an electronic device comprising a memory, a processor and a program stored on the memory and executable on the processor, the processor implementing the steps in a method of registration of a 4D-CT medical image of a lung as described in the first aspect of the present disclosure when executing the program.

Compared with the prior art, the beneficial effect of this disclosure is:

1. according to the method, the system, the medium or the electronic device, the Multiple-net adopts the mode that the 3D lung CT image in one period is used as the input of the network to replace the pair input learning mode, the Multiple-net network does not need to input all data sets into the network for training, the network parameters are obtained by training the input 3D lung CT image in one period, and potential errors are effectively avoided.

2. The method, the system, the medium or the electronic equipment adopt a multi-resolution mechanism aiming at the condition of insufficient data quantity, an original image is high-resolution, the original image is subjected to down-sampling once and twice respectively to obtain a medium-resolution image and a low-resolution image, the images with different resolutions are input into a network to learn different scale characteristics, the obtained low-resolution deformation field is used as the input of a corresponding high-resolution network, and finally, a total registration deformation field is obtained, so that the registration precision is greatly improved.

3. The methods, systems, media, or electronic devices described in this disclosure use jacobian on spatial regularization constraints to reduce folding in the transformation, maintaining high registration accuracy.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure and are not to limit the disclosure.

Fig. 1 is a schematic network structure diagram of a lung 4D-CT medical image registration method provided in embodiment 1 of the present disclosure.

Fig. 2 is a schematic structural diagram of a fully connected network provided in embodiment 1 of the present disclosure.

Detailed Description

The present disclosure is further described with reference to the following drawings and examples.

It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.

The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.

Example 1:

the embodiment 1 of the present disclosure provides a lung 4D-CT medical image registration method, which includes the following processes:

as shown in fig. 1, the present embodiment preferably learns image features at three different resolutions;

the lowest layer of the overall architecture is a low-resolution image, and the specific operation is as follows:

making two down-sampling on a periodic 3D lung CT image, using the obtained result as the input of the network, and making the network learn the large-scale characteristics of the image to obtain the corresponding deformation field (i.e. the first deformation field)) And the obtained deformation fieldPerforming one-time up-sampling operation, and taking the up-sampling operation as the input of the middle layer of the overall architecture;

the middle layer of the overall architecture is a medium-resolution image, and the specific operations are as follows:

make a down-sampling of the 3D lung CT image of one period, the obtained result and the deformation field obtained at the bottom layer (i.e. the first deformation field)) The two are used as the input of the network, so that the network learns the medium-scale features of the image and the low-scale features at the same time to obtain the corresponding deformation field (i.e. the second deformation field)) And the obtained deformation fieldPerforming one-time up-sampling operation, and taking the up-sampling operation as the input of the uppermost layer of the overall architecture;

the top layer of the overall structure is a high-resolution image, namely an original image, and a deformation field obtained by the image and the middle layerInputting the three parameters into the network together for learning, so that the network learns the characteristics of three scales of low, medium and high to obtain the corresponding deformation field (i.e. the third deformation field)) This may improve network registration accuracy.

It is understood that in other embodiments, the image features may be learned at four or five or more different resolutions, as long as the progressive sampling and connection are performed according to the method disclosed in the present disclosure, and those skilled in the art may select the image features according to specific conditions, which is not described herein again.

As shown in fig. 2, the network framework designed by the present embodiment is a full-connection network similar to U-net, and the whole network is composed of four layers of coders and decoders with hopping connection layers. The fully connected networks in the different resolution layers in fig. 1 are the same, and only the lowest fully connected network will be described in this embodiment.

First, in the encoding stage, a set of 3D lung CT images of one period is input into the network (e.g., input into the leftmost network in fig. 1), a convolution kernel with a size of 3 × 3 × 3 and a step size of 1 is applied to obtain the leftmost convolutional layer in fig. 2, then a downsampling operation is performed on the convolutional layer with a step size of 2 and a size of 3 × 3 × 3, and further high-level features between the input image pairs are calculated and the image size is halved and sampled until the lowest size is reached (e.g., the convolutional layer size in fig. 2 gradually decreases from left to right).

The decoding stage first performs a deconvolution operation by applying a convolution kernel of size 2 × 2 × 2 with a step size of 2, so that the convolution layer obtained by the encoding stage is upsampled to twice its size. And (4) connecting the convolution layers with the same size obtained in the encoding stage through residual error network hopping so as to make up the characteristic information lost by the network. A convolution kernel of size 3 x 3 with step size 1 is applied to reduce the number of channels in the network after the connection. After 4 operations, the dimension of the convolutional layer is the same as that of the leftmost convolutional layer in FIG. 1. Finally, a convolution kernel with the size of 1 multiplied by 1 and the step size of 1 is applied to obtain the deformation field. In addition, the remaining convolutions, except for the last output convolution layer in the network, are each followed by a Relu activation function, and a padding operation is performed after each convolution operation with a step size of 1, in order to make the output and input convolution layers of consistent size.

The loss function for a fully connected network includes:

(1) similarity measure loss function

There are many functions for calculating the loss of similarity between two images, such as Mean Squared Error (MSE), Sum of Squared Distances (SSD), Mutual Information (MI), cross-correlation (CC), normalized cross-correlation (NCC), etc. For simplicity, the present embodiment uses normalized cross-correlation as a similarity measure for the network to calculate the degree of alignment between the two images. Let I_iAnd I_jTwo images (i ≠ j) of a set of CT images in a respiratory cycle,andeach being ω centered on x³The expected value of the image block, where ω is 9 in the experiment of the present embodiment. The normalized cross-correlation is thus defined as the following equation:

wherein Ω represents the image field, x_iIs represented by³The voxels in the image block that are adjacent to the center x. In addition, the value range of the normalized cross-correlation NCC is [ -1,1]The number-1 indicates that the two images are completely uncorrelated, and the number 1 indicates that the two images are extremely highly correlated. Therefore, the similarity measure loss function is designed to:

wherein, I^NRepresented by a set of CT images, T, over a respiratory cycle^NShows the set of all CT image transformation vector fields within one breathing cycle,refers to the set of ith and jth image transition vector fields.

(2) Smoothing regularized loss function

In previous studies, local smoothing constraints are usually applied to the spatial gradient to smooth the estimated deformation field, in order to avoid obtaining an unrealistic or discontinuous deformation field. Therefore, the present embodiment also uses a smoothing constraint in the loss function:

wherein, mu_i(p) represents the deformation vector field at position p for the ith image,representing the gradient of the deformation vector field, | · | | non-woven phosphor₂Is represented by₂Norm, in this embodiment, the spatial gradient is approximated by the difference between neighboring voxels.

(3) Space folding penalty loss function

Although the regularizer can control the flatness of the deformation field, the registration accuracy of the model may be reduced when the weight of the regularizer is large; folding in the transform occurs when the weight of the regularizer is small. The folding points are mathematically negative in the value of their jacobian, so in order to maximize the image registration accuracy when selecting smaller regularization weights, the present embodiment applies a jacobian regularization method that imposes local folding constraints on the estimated deformation field. The spatial folding penalty loss function is defined as follows:

where σ (·) represents a linear constraint, expressed as:

(4) in the formula (I), the compound is shown in the specification,representing a deformation vector field mu_nJacobian at point p.

The jacobian matrix is defined as follows:

the Jacobian matrix of the displacement field is a second-order tensor field formed by directional derivatives of points in various directions in the image domain, and the local behavior of the deformation field can be analyzed by using a Jacobian matrix, for exampleIt means that the deformation field at the point p maintains a direction in the neighborhood of p, whereas the deformation field at the point p is opposite in the neighborhood of p, and a fold is likely to be generated. So the jacobian is adopted as a space folding penalty loss function to avoid folding.

(4) Periodic constraint penalty function

The period constraint defines the sum of the deformation vectors corresponding to all voxels in a period.

The period constraint penalty function is defined as follows:

wherein the content of the first and second substances,the deformation vector of the corresponding point p from the first image to the nth image is shown. (6) A sum of zero indicates a complete periodic motion.

Thus, the integrity penalty function of this embodiment can be written as:

L_total(I^N,T^N)＝L_sim+λ₁L_reg+λ₂L_Jdet+λ₃L_pre (7)

wherein λ is₁,λ₂,λ₃And weights respectively representing a smooth regular loss function, a spatial folding penalty loss function and a periodic constraint loss function.

Example 2:

the embodiment 2 of the present disclosure provides a lung 4D-CT medical image registration system, including:

a data acquisition module configured to: acquiring CT image data in a respiratory cycle;

a first deformation field acquisition module configured to: at least two times of down-sampling are carried out on the obtained CT image data in a respiratory cycle, and the obtained result is used as the input of a first full-connection network to obtain a first deformation field

A second deformation field acquisition module configured to: the first deformation fieldPerforming one-time up-sampling operation on the acquired 3D lung CT image in one period, and taking the up-sampling operation and the down-sampling operation as the input of a second fully-connected network to obtain a second deformation field

A third deformation field acquisition module configured to: subjecting the second shape to deformationPerforming one-time up-sampling operation, and using the obtained CT image data in one respiratory cycle as the input of a third fully-connected network to obtain a third deformation field

Example 3:

embodiment 3 of the present disclosure provides a computer-readable storage medium, on which a program is stored, which when executed by a processor implements the steps in a lung 4D-CT medical image registration method according to embodiment 1 of the present disclosure.

Example 4:

embodiment 4 of the present disclosure provides an electronic device, which includes a memory, a processor, and a program stored in the memory and executable on the processor, and the processor executes the program to implement the steps in the lung 4D-CT medical image registration method according to embodiment 1 of the present disclosure.

As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.