1. A medical three-dimensional model display and cutting system based on a mobile terminal, wherein the system is located in the mobile terminal, and the mobile terminal comprises: rendering a bottom layer and a buffer zone of the mobile terminal, wherein the system comprises a Model-View-ViewModel architecture;

the Model-View Model architecture comprises a Model layer, a View layer and a ViewModel layer;

the View layer is used for receiving data operated by a user and transmitting the data interactively operated by the user to the ViewModel layer;

the ViewModel layer is used for processing the pre-stored Model file according to the user operation data and the pre-stored operation data after receiving the user operation data transmitted by the View layer, acquiring a processing result and storing the processing result to the Model layer;

the prestored model file is a model list stored by adopting a LiveData component;

the Model layer is connected with a database and permanently stores the data in the Model file;

the View layer comprises: the model loading module, the model display module comprising the renderer and the model management module.

2. The system of claim 1,

the model file comprises three-dimensional models: vertex data, normal vector data;

the data of the user operation comprises: cutting mode button information triggered by a user on a display screen of the mobile terminal and cutting range information triggered by the user;

the operational data includes: and creating a rendering program and loading a shader function, an internal cutting function, an external cutting function and a model rendering function.

3. The system of claim 2,

the model loading module is used for reading a preset initial model file in the mobile terminal to obtain a model file corresponding to the preset initial model file, and storing the model file into a preset model list of the ViewModel layer;

the initial model file comprises all vertex data and face index values in the three-dimensional model;

each face index value comprises position coordinates of three vertexes of a triangular face in the three-dimensional model in the model file;

the method includes that the model loading module reads a preset initial model file in the mobile terminal to obtain a model file corresponding to the preset model file, and stores the model file into a preset model list of the ViewModel layer, and specifically includes:

determining the coordinates of the vertexes forming the triangular surface in the vertex data according to the preset surface index value of the initial model file in the mobile terminal and sequentially storing the coordinates in the point data of the model file;

judging whether the preset initial model file has point normal vector data or not;

if yes, finding out the normal vector coordinates of the point from the point data in the initial model file according to the preset index value of the point vector, and storing the normal vector coordinates in the normal vector data of the model file;

if not, acquiring a corresponding normal vector of the triangular surface according to the three vertex coordinates forming the triangular surface in the vertex data, and taking the normal vector of the triangular surface as point normal vector data of the three vertex coordinates;

and assigning a color value to each model file, and adding the model files into a preset model list.

4. The system of claim 3,

and the model management module is used for storing the preset model list in the ViewModel in a LiveData mode.

5. The system of claim 4,

the model display module comprises a renderer and is used for reading model files in a model list in the ViewModel layer, preprocessing the model files and further displaying the model files; the method specifically comprises the following steps:

adding the point data and the normal vector data in each model file into a buffer area of a rendering bottom layer of the mobile terminal;

a renderer in the model display module reads a preset shader program, adopts the created rendering program for each model file, loads a shader function to create the rendering program and binds the shader;

transmitting a total transformation matrix and a color value of a model file constructed by a renderer in a model display module, and point data and normal vector data in the model file into a shader for calculating the final display position of a vertex and the display effect of illumination, and acquiring the coordinate displayed on a two-dimensional screen and the display effect of illumination corresponding to each vertex coordinate in the model file;

the total transformation matrix is formed by multiplying a scaling matrix, a translation matrix and a rotation matrix of a model in a renderer in the model display module;

and calling a preset drawing function to render the three-dimensional model in a rendering bottom layer of the mobile terminal, and displaying the model file in a display screen of the mobile terminal based on the coordinates displayed on the two-dimensional screen corresponding to each vertex coordinate in the model file and the illumination display effect.

6. The system of claim 5, further comprising:

the graphical operation module is used for acquiring an automatic rotation button of an X axis of the model selected by a user, and automatically rotating the model according to the X axis; or the like, or, alternatively,

an automatic rotation button for obtaining the Y axis of the model selected by the user, and automatically rotating the model according to the Y axis; or the like, or, alternatively,

and an automatic rotation button for acquiring the Z axis of the model selected by the user, and automatically rotating the model according to the Z axis.

7. The system of claim 6, further comprising:

the model cutting module is used for acquiring cutting mode button information triggered by a user on a display screen of the mobile terminal and cutting range information triggered by the user, cutting the model file according to the internal cutting function or the external cutting function and acquiring a cutting result;

the cutting mode button information comprises internal cutting mode button information and external cutting mode button information;

the user-triggered cutting range information includes: coordinate information each time a user's finger moves on a display screen of the mobile terminal.

8. The system of claim 7,

the method for acquiring the cutting mode button information triggered by the user and the cutting range information triggered by the user, cutting the model file and acquiring the cutting result specifically comprises the following steps:

acquiring a cutting range according to the cutting range information triggered by the user;

the cutting range includes: the coordinate information in the coordinate information set is connected end to form a closed polygonal area;

the coordinate information set comprises: coordinate information of each time a finger of a user moves on a display screen of the mobile terminal;

judging whether the vertex is in a cutting range, acquiring a judgment result, and determining whether to delete the vertex according to the judgment result and the cutting mode button information triggered by the user; the method specifically comprises the following steps:

if the judgment result is that the vertex is in the cutting range and the user triggers the information of the external cutting mode button, deleting the vertex;

if the judgment result is that the vertex is in the cutting range and the user triggers the button information of the internal cutting mode, the vertex is reserved;

if the judgment result is that the vertex is not in the cutting range and the user triggers the information of the external cutting mode button, the vertex is reserved;

if the judgment result is that the vertex is not in the cutting range and the user triggers the button information of the internal cutting mode, deleting the vertex;

the cutting result comprises a retained vertex;

the judging whether the vertex is in the cutting range specifically includes:

judging whether the first ray corresponding to the vertex has an odd number of intersection points with the cutting range, and if so, determining that the vertex of the model is in the cutting range;

the first ray corresponding to the vertex is a ray emitted to the right from the vertex.

9. The system of claim 8,

the model cutting module is further used for acquiring the starting positions of the vertexes and the number of the vertexes reserved after each cutting in the process of cutting the model file;

acquiring the interval of the reserved vertexes after each cutting according to the starting positions of the reserved vertexes after each cutting and the number of the vertexes;

acquiring a vertex which is overlapped in the vertex-reserved interval after each cutting in the process of cutting the model file according to the vertex-reserved interval after each cutting;

and rendering the coincident vertexes in the vertexes reserved after each cutting by adopting the preset model rendering function.

10. The system of claim 1, further comprising:

and the PC-end medical image three-dimensional reconstruction module is connected with the mobile terminal and used for reading medical image data acquired in advance to carry out three-dimensional reconstruction, reconstructing two-dimensional medical image data into a three-dimensional model and storing the three-dimensional model into a preset initial model file in the mobile terminal.

Background

In recent years, the rapid rise of the 5G technology and the rapid development of the mobile terminal software and hardware technology, the combination of medical treatment and the mobile terminal is a hot and important trend, and the better mobile terminal software and hardware technology makes it possible for the traditional user to view and analyze medical data only at the PC terminal, and to analyze and research the medical data at any time and any place at the mobile terminal. In addition, the three-dimensional reconstruction technology for medical image data accuracy is more mature, the medical three-dimensional model generated by the technology is displayed, so that the body structure and the diseased condition of a patient can be observed more intuitively by a user, and the method plays a great role in more convenient analysis and diagnosis of the patient for the user. Like DICOM medical images, most of medical three-dimensional model display after reconstruction on a PC terminal is concentrated on the PC terminal, so that a user can conveniently analyze a medical three-dimensional model generated by a real medical image of a patient on a mobile terminal, and the mobile terminal simultaneously displays a plurality of medical three-dimensional models and can perform operations such as cutting, rotating, freely displaying the models and the like to enable the user to intensively observe a certain affected part and an integral structure, thereby having practical application significance. The system can greatly help the user to analyze and diagnose the medical image data of the patient.

The chinese patent CN103136785A three-dimensional visualization system of human body medical model for mobile terminal and its implementation method provide a method for generating medical images into three-dimensional model at server end and then displaying the human body medical three-dimensional model on mobile terminal. According to the patent, a stl model is generated after three-dimensional reconstruction is carried out at a server side, and the stl model is converted into a 3ds model for simplification and then is transmitted into a mobile phone for display. The mobile terminal reads the model file into the memory by using the input and output stream of the operating system, then transmits the model data into the OpenGL ES for three-dimensional rendering and display, and finally realizes the rotation, scaling and translation operations of the three-dimensional medical model through gesture recognition.

The Chinese patent CN103136785A human body medical model three-dimensional visualization system for mobile terminal and the realization method thereof are as follows: firstly, generating a three-dimensional model by using medical image data at a server side, then simply displaying the human medical model on a mobile terminal, and adding the most basic graphical operations: rotation, zoom, and translation. Therefore, the mobile terminal is only a displayed tool, a user can only simply observe the model through the mobile terminal, more operations cannot be performed on the mobile terminal, and only one model can be loaded at a time for viewing, so that the advantage of high-speed development of the hardware of the mobile terminal at present is not fully utilized.

However, the conventional segmentation method directly acts on the point data, directly deletes the model point data and then renders the model point data again, and due to the limitation of the memory size on the mobile terminal, if the segmentation method is adopted, too much memory overhead and time overhead are generated for a displayed slightly large medical model, so that the cutting efficiency is low, even the cutting cannot be performed, and the requirement of smoothly using an auxiliary user cannot be met.

Disclosure of Invention

Technical problem to be solved

In view of the above disadvantages and shortcomings of the prior art, the present invention provides a medical three-dimensional model display and cutting system based on a mobile terminal.

(II) technical scheme

In order to achieve the purpose, the invention adopts the main technical scheme that:

in a first aspect, the embodiment of the invention provides a medical three-dimensional model display and cutting system based on a mobile terminal.

The embodiment of the invention provides a medical three-dimensional model display and cutting system based on a mobile terminal, which is characterized in that the system is positioned in the mobile terminal, and the mobile terminal comprises: rendering a bottom layer (rendering pipeline of Opengles) and a buffer area of the mobile terminal, wherein the system comprises a Model-View-ViewModel architecture;

the Model-View Model architecture comprises a Model layer, a View layer and a ViewModel layer;

the View layer is used for receiving data operated by a user and transmitting the data interactively operated by the user to the ViewModel layer;

the ViewModel layer is used for processing the pre-stored Model file according to the user operation data and the pre-stored operation data after receiving the user operation data transmitted by the View layer, acquiring a processing result and storing the processing result to the Model layer;

the prestored model file is a model list stored by adopting a LiveData component;

the Model layer is connected with a database and permanently stores the data in the Model file;

the View layer comprises: the model loading module, the model display module comprising the renderer and the model management module.

Preferably, the first and second liquid crystal materials are,

the model file comprises three-dimensional models: vertex data, normal vector data;

the data of the user operation comprises: cutting mode button information triggered by a user on a display screen of the mobile terminal and cutting range information triggered by the user;

the operational data includes: and creating a rendering program and loading a shader function, an internal cutting function, an external cutting function and a model rendering function.

Preferably, the first and second liquid crystal materials are,

the model loading module is used for reading a preset initial model file in the mobile terminal to obtain a model file corresponding to the preset initial model file, and storing the model file into a preset model list of the ViewModel layer;

the initial model file comprises all vertex data and face index values in the three-dimensional model;

each face index value comprises position coordinates of three vertexes of a triangular face in the three-dimensional model in the model file;

the method includes that the model loading module reads a preset initial model file in the mobile terminal to obtain a model file corresponding to the preset model file, and stores the model file into a preset model list of the ViewModel layer, and specifically includes:

determining the coordinates of the vertexes forming the triangular surface in the vertex data according to the preset surface index value of the initial model file in the mobile terminal and sequentially storing the coordinates in the point data of the model file;

judging whether the preset initial model file has point normal vector data or not;

if yes, finding out the normal vector coordinates of the point from the point data in the initial model file according to the preset index value of the point vector, and storing the normal vector coordinates in the normal vector data of the model file;

if not, acquiring a corresponding normal vector of the triangular surface according to the three vertex coordinates forming the triangular surface in the vertex data, and taking the normal vector of the triangular surface as point normal vector data of the three vertex coordinates;

and assigning a color value to each model file, and adding the model files into a preset model list.

Preferably, the first and second liquid crystal materials are,

and the model management module is used for storing the preset model list in the ViewModel in a LiveData mode.

Preferably, the first and second liquid crystal materials are,

the model display module comprises a renderer and is used for reading model files in a model list in the ViewModel layer, preprocessing the model files and further displaying the model files; the method specifically comprises the following steps:

adding the point data and the normal vector data in each model file into a buffer area of a rendering bottom layer of the mobile terminal;

a renderer in the model display module reads a preset shader program, adopts the created rendering program for each model file, loads a shader function to create the rendering program and binds the shader;

transmitting a total transformation matrix and a color value of a model file constructed by a renderer in a model display module, and point data and normal vector data in the model file into a shader for calculating the final display position of a vertex and the display effect of illumination, and acquiring the coordinate displayed on a two-dimensional screen and the display effect of illumination corresponding to each vertex coordinate in the model file;

the total transformation matrix is formed by multiplying a scaling matrix, a translation matrix and a rotation matrix of a model in a renderer in the model display module;

and calling a preset drawing function to render the three-dimensional model in a rendering bottom layer of the mobile terminal, and displaying the model file in a display screen of the mobile terminal based on the coordinates displayed on the two-dimensional screen corresponding to each vertex coordinate in the model file and the illumination display effect.

Preferably, the system further comprises:

the graphical operation module is used for acquiring an automatic rotation button of an X axis of the model selected by a user, and automatically rotating the model according to the X axis; or the like, or, alternatively,

an automatic rotation button for obtaining the Y axis of the model selected by the user, and automatically rotating the model according to the Y axis; or the like, or, alternatively,

and an automatic rotation button for acquiring the Z axis of the model selected by the user, and automatically rotating the model according to the Z axis.

Preferably, the system further comprises:

the model cutting module is used for acquiring cutting mode button information triggered by a user on a display screen of the mobile terminal and cutting range information triggered by the user, cutting the model file according to the internal cutting function or the external cutting function and acquiring a cutting result;

the cutting mode button information comprises internal cutting mode button information and external cutting mode button information;

the user-triggered cutting range information includes: coordinate information each time a user's finger moves on a display screen of the mobile terminal.

Preferably, the first and second liquid crystal materials are,

the method for acquiring the cutting mode button information triggered by the user and the cutting range information triggered by the user, cutting the model file and acquiring the cutting result specifically comprises the following steps:

acquiring a cutting range according to the cutting range information triggered by the user;

the cutting range includes: the coordinate information in the coordinate information set is connected end to form a closed polygonal area;

the coordinate information set comprises: coordinate information of each time a finger of a user moves on a display screen of the mobile terminal;

judging whether the vertex is in a cutting range, acquiring a judgment result, and determining whether to delete the vertex according to the judgment result and the cutting mode button information triggered by the user; the method specifically comprises the following steps:

if the judgment result is that the vertex is in the cutting range and the user triggers the information of the external cutting mode button, deleting the vertex;

if the judgment result is that the vertex is in the cutting range and the user triggers the button information of the internal cutting mode, the vertex is reserved;

if the judgment result is that the vertex is not in the cutting range and the user triggers the information of the external cutting mode button, the vertex is reserved;

if the judgment result is that the vertex is not in the cutting range and the user triggers the button information of the internal cutting mode, deleting the vertex;

the cutting result comprises a retained vertex;

the judging whether the vertex is in the cutting range specifically includes:

judging whether the first ray corresponding to the vertex has an odd number of intersection points with the cutting range, and if so, determining that the vertex of the model is in the cutting range;

the first ray corresponding to the vertex is a ray emitted to the right from the vertex.

Preferably, the model cutting module is further configured to obtain a starting position of each vertex and a number of vertices that are retained after each cutting in the process of cutting the model file;

acquiring the interval of the reserved vertexes after each cutting according to the starting positions of the reserved vertexes after each cutting and the number of the vertexes;

acquiring a vertex which is overlapped in the vertex-reserved interval after each cutting in the process of cutting the model file according to the vertex-reserved interval after each cutting;

and rendering the coincident vertexes in the vertexes reserved after each cutting by adopting the preset model rendering function.

Preferably, the system further comprises:

and the PC-end medical image three-dimensional reconstruction module is connected with the mobile terminal and used for reading medical image data acquired in advance to carry out three-dimensional reconstruction, reconstructing two-dimensional medical image data into a three-dimensional model and storing the three-dimensional model into a preset initial model file in the mobile terminal.

(III) advantageous effects

The invention has the beneficial effects that: according to the medical three-dimensional Model display and cutting system based on the mobile terminal, due to the adoption of the Model-View Model architecture, compared with the prior art, the Model-View Model architecture can enable a plurality of Model files to be loaded on the mobile terminal at the same time and various operations to be carried out, and can efficiently cut the Model files.

According to the medical three-dimensional Model display and cutting system based on the mobile terminal, due to the fact that the Model-View Model architecture is adopted, all modules are decoupled greatly, data in a Model file are shared globally, memory overhead is saved, real-time performance and correctness of the data are guaranteed, and reusability is improved.

In the medical three-dimensional model display and cutting system based on the mobile terminal, the model cutting module is used for storing the vertex data of the model which is not deleted or moved at the starting position of the point to be rendered and the number of the points during each cutting in the process of cutting the model file, so that under the condition that a plurality of models are displayed or the models are overlarge, the memory consumption and the time consumption are greatly saved, and the efficiency and the fluency of the system are improved.

Drawings

FIG. 1 is a schematic diagram of a medical three-dimensional model display and cutting system architecture based on a mobile terminal according to the present invention;

FIG. 2 is a general functional architecture diagram of a medical three-dimensional model display and cutting system based on a mobile terminal according to the present invention;

FIG. 3 is a flow chart of parsing and loading of a model file in a medical three-dimensional model display and cutting system based on a mobile terminal according to the present invention;

FIG. 4 is an overall flow chart of the transformation and display of model point coordinates in a medical three-dimensional model display and cutting system based on a mobile terminal according to the present invention;

FIG. 5 is a general flow chart of the medical three-dimensional model display and cutting range line display in the cutting system based on the mobile terminal according to the present invention;

FIG. 6 is a schematic diagram of a method for displaying a medical three-dimensional model based on a mobile terminal and determining a point within a cutting range in a cutting system according to the present invention;

FIG. 7 is a schematic diagram of a cutting method in a medical three-dimensional model display and cutting system based on a mobile terminal according to the invention;

FIG. 8 is a schematic diagram of a multi-cutting combination method in a medical three-dimensional model display and cutting system based on a mobile terminal according to the present invention;

fig. 9 is a flow chart of three-dimensional reconstruction of medical images in a medical three-dimensional model display and cutting system based on a mobile terminal according to the present invention.

Detailed Description

For the purpose of better explaining the present invention and to facilitate understanding, the present invention will be described in detail by way of specific embodiments with reference to the accompanying drawings.

In order to better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

Referring to fig. 1, the present embodiment provides a medical three-dimensional model display and cutting system based on a mobile terminal, where the system is located in the mobile terminal, and the mobile terminal includes: rendering a bottom layer (rendering pipeline of Opengles), and a buffer area of the mobile terminal (namely a part of a memory of the mobile terminal), wherein the system comprises a Model-View-ViewModel architecture; the Model-View-ViewModel architecture comprises a Model layer, a View layer and a ViewModel layer.

The View layer is used for receiving data of user interaction operation and transmitting the data of the user interaction operation to the ViewModel layer.

And the ViewModel layer is used for processing the pre-stored Model file according to the user operation data and the pre-stored operation data after receiving the user operation data transmitted by the View layer, acquiring a processing result and storing the processing result to the Model layer.

And the pre-stored model file is used for storing the model list by adopting a LiveData component.

The Model layer is connected with a database and permanently stores the data in the Model file.

The View layer comprises: the model loading module, the model display module comprising the renderer and the model management module.

In practical application of this embodiment, the model file includes the following three-dimensional models: vertex data, normal vector data.

The data of the user operation comprises: the method comprises the following steps of cutting mode button information triggered by a user on a display screen of the mobile terminal and cutting range information triggered by the user.

The operational data includes: and creating a rendering program and loading a shader function, an internal cutting function, an external cutting function and a model rendering function.

In practical application of this embodiment, the model loading module is configured to read a preset initial model file in the mobile terminal to obtain a model file corresponding to the preset initial model file, and store the model file in a preset model list of the ViewModel layer.

The initial model file includes all three-dimensional vertex data and face index values.

Each of the face index values includes position coordinates of three vertices constituting a triangular face in the model file.

The method includes that the model loading module reads a preset initial model file in the mobile terminal to obtain a model file corresponding to the preset model file, and stores the model file into a preset model list of the ViewModel layer, and specifically includes:

and determining the coordinates of the vertexes forming the triangular surface in the three-dimensional vertex data according to the preset surface index value of the initial model file in the mobile terminal, and sequentially storing the coordinates in the point data of the model file.

And judging whether the preset initial model file has point normal vector data or not.

If yes, finding out the normal vector coordinates of the point from the point data in the initial model file according to the preset index value of the point vector, and storing the normal vector coordinates in the normal vector data of the model file.

If not, calculating a normal vector of the triangular surface according to the three vertex coordinates forming the triangular surface in the three-dimensional vertex data, and then using the calculated normal vector of the triangular surface as point normal vector data of the three vertex coordinates.

And assigning a color value to each model file, and adding the model files into a preset model list.

In this embodiment, the mobile terminal provides an interface for the user to select the model file, and the user clicks the display interface of the mobile terminal to select the model file, so that the user can freely select a single model file or press the model file for multiple times. After the selection is completed, a display interface of the mobile terminal displays all model file paths selected by the user for the user to really perform, and after the user confirms that the model files are correct, the mobile terminal performs multithreading analysis and loading on the asynchronous background model files after clicking a display button.

Referring to fig. 3, the step of loading the file by the system in this embodiment is:

reading the face index value of the model file, finding out the point coordinates forming a triangular face from all the stored vertex data, and storing the point coordinates in the model file in the point data in sequence.

And judging whether the model file has point normal vector data of the model, if not, assigning coordinates of three points forming the model file to be used as point normal vectors of the three points according to an average surface normal vector strategy, otherwise, directly finding out the normal vector coordinates of the points from the vertex data according to the index value of the model file by the system and storing the normal vector coordinates into the normal vector data of the model.

Releasing the intermediate storage data, initializing the display state of each model into display and randomly giving a color value, adding the model into a model list by the system, and analyzing and loading the point data and the point normal vector data of the model.

In practical application of this embodiment, the model management module is configured to store the preset model list in the ViewModel by using LiveData.

In this embodiment, the model list is maintained and operated in the ViewModel, and the model list is stored in the ViewModel by using LiveData. The mode of ViewModel + LiveData has a very long life cycle, can run through the life cycle of the whole system, guarantee the system probably because the model data is preserved when the system is rebuilt between the horizontal and vertical screens, also guarantee our model data can share, can obtain the latest same data through this mode in the whole system, while saving the memory overhead, the life cycle of processing the memory and business is simpler, more benefit to the maintenance and the multiplexing of the system.

The main steps of hiding or showing the model by the user are as follows:

the user lets the system open the model display list by clicking on the model display list button.

And displaying the serial number and the name of the currently loaded model in the model list and a switch of whether the model file is currently displayed on the interface or not by the system.

And the user freely selects whether the currently loaded model file is displayed on the interface, the display is performed when the switch of the model file is opened, the display is not performed when the switch of the model file is closed, and the system records the display state of each loaded model file.

In practical application of this embodiment, the model display module includes a renderer, and is configured to read a model file in a model list in the ViewModel layer, perform preprocessing, and further display the model file; the method specifically comprises the following steps:

and adding the point data and the normal vector data in each model file into a buffer area of a rendering bottom layer of the mobile terminal.

And a renderer in the model display module reads a preset shader program, adopts the created rendering program for each model file, and loads a shader function to create the rendering program and bind with a shader.

And transmitting the total transformation matrix and the color value of each model file constructed by a renderer in a model display module, the point data in the model file and the normal vector data into a shader to calculate the final vertex display position and the illumination display effect, and acquiring the coordinates displayed on a two-dimensional screen and the illumination display effect corresponding to the coordinates of each vertex in the model file.

The total transformation matrix is formed by multiplying a scaling matrix, a translation matrix and a rotation matrix of the model in the renderer in the model display module.

And calling a preset drawing function to render the three-dimensional model in a rendering bottom layer of the mobile terminal, and displaying the model file in a display screen of the mobile terminal based on the coordinates displayed on the two-dimensional screen corresponding to each vertex coordinate in the model file and the illumination display effect.

In this embodiment, after the system performs file analysis in the background and finishes loading, a model list is automatically generated, and point data, normal vector data, display state, and color values of each model file to be displayed are stored in the model list. The system jumps to a display and operation interface of a model file, the model file is displayed by using an OpenGL ES2.0 bottom layer three-dimensional engine, the system is mainly provided with a model display window (GlSurfaceView) for displaying the model file and responding to graphical operation and a Renderer (Renderer) for rendering the model, the position of a vertex finally displayed on a two-dimensional interface of the mobile terminal is calculated by a model total transformation matrix and point data transmitted to a vertex shader, and the shader can calculate the illumination and color effect finally displayed on the two-dimensional interface of the mobile terminal by a model through normal vector data and transmitted color values of the point.

In this embodiment, the system performs preprocessing on the model file, specifically:

and reading the point data and the normal vector data in each model file and adding the point data and the normal vector data into a buffer area at the bottom rendering layer, so that the rendering time is saved.

And reading a preset shader program in the system, and creating a rendering program for each model file to be connected with the shader.

The transformation matrix of each model is built in the renderer to be transmitted to the shader for calculation of the final display position of the model point.

Referring to fig. 4, the main steps of constructing and displaying the transformation matrix of the model in this embodiment are as follows:

in order to enable the three-dimensional model to present a three-dimensional effect on a two-dimensional plane, a perspective projection matrix and a camera setting matrix are set. The camera setting matrix simulates the position of eyes, the perspective projection matrix simulates the visual angle size and distance of eyes to form a cone, and the object projection in the cone is displayed on the near plane of the cone.

Constructing a model matrix of a model file, which mainly comprises the following steps: and the scaling matrix, the translation matrix and the rotation matrix are used for changing the coordinate display position of the model point to realize the graphical operation of the user.

And transmitting the model total transformation matrix, the color value, the model point data and the model point normal vector data formed by multiplying all the transformation matrices into a shader for calculating the display position and the effect.

And calling a preset drawing function to render the three-dimensional model in the rendering pipeline and displaying the model in the view window.

In practical application of this embodiment, the system further includes:

the graphical operation module is used for acquiring an automatic rotation button of an X axis of the model selected by a user, and automatically rotating the model according to the X axis; or, an automatic rotation button for obtaining the Y axis of the model selected by the user, automatically rotating the model according to the Y axis; or, an automatic rotation button for acquiring the Z-axis of the model selected by the user automatically rotates the model according to the Z-axis.

In this embodiment, all the graphical operations are performed based on the transformation matrix of the model, and after the system has loaded the three-dimensional model data, the final model is displayed in the view window, and the user may control the scaling and rotation of the model in the view window, or may click a button in the model operation interface to perform operations such as automatic rotation, home position restoration, translation, and loading of more models on the displayed model.

In this embodiment, the main functions and implementations of the graphical operation module are as follows:

rotating: and the user slides a single finger on the view window, calculates the angle required to be rotated according to the finger moving distance, and then changes the rotation transformation matrix of the model so as to rotate the model.

Zooming: the user moves on the view window by using two fingers, the system firstly records the distance between the two fingers, and finally the scaling transformation matrix of the model is changed by comparing the moving distance between the two fingers of the user and the initial distance to realize the scaling of the model.

Automatically rotating the button: the user can freely select one of automatic rotation around the X axis, the Y axis and the Z axis, the system uses the system time as a seed point to calculate the angle required to rotate when the model is rendered each time, meanwhile, the system adjusts the rendering three-dimensional model to continuous rendering, the model is rendered again every 16ms, and the model in the view window can show the effect of automatic rotation along with the increase of time.

A home position restoring button: after the user clicks the button, the system initializes three model transformation matrices, namely a scaling matrix, a translation matrix and a rotation matrix of the model into an identity matrix, and the model is restored to the size and the position when the model is loaded.

Translating the model button: after the user clicks the button, the system locks the single-finger sliding 360-degree rotation function, so that the user can drag the model to move to the position where the user wants to put on the interface by the single finger, and meanwhile, the user can still perform the double-finger zooming function. The system calls data in the scaling matrix, calculates the position distance of the model needing to be moved according to the translation distance by the called scaling matrix data in a proportional mode, and changes the translation matrix of the model to enable the model to have the effect of changing the position by translation.

Load more model buttons: after the user clicks the button, the system calls the file manager, the user can perform single-selection model files or long-press multi-selection model files, the system multithreads asynchronously analyze and load the files in the background, after the loading is completed, the system initializes the display state of a newly loaded model and endows color values, and finally adds the model display state and the color values into a model display list and updates a view window.

In practical application of this embodiment, the system further includes: and the model cutting module is used for acquiring cutting mode button information triggered by a user on a display screen of the mobile terminal and cutting range information triggered by the user, cutting the model file according to the internal cutting function or the external cutting function and acquiring a cutting result.

The cutting mode button information includes internal cutting mode button information and external cutting mode button information.

Two model cutting modes are introduced in this example: the method comprises an external cutting mode and an internal cutting mode, wherein the external cutting mode is that a user draws a range to be cut and cuts off model points in the range, and the internal cutting mode is that the model points in the range drawn by the user are reserved, and all points outside the range are discarded.

The user-triggered cutting range information includes: coordinate information each time a user's finger moves on a display screen of the mobile terminal.

In the practical application of this embodiment, the obtaining of the cutting mode button information triggered by the user and the cutting range information triggered by the user cuts the model file to obtain the cutting result, and specifically includes:

and acquiring a cutting range according to the cutting range information triggered by the user.

The cutting range includes: the coordinate information in the coordinate information set is connected end to form a closed polygonal area.

The coordinate information set comprises: coordinate information each time a user's finger moves on a display screen of the mobile terminal.

Judging whether the vertex is in a cutting range, acquiring a judgment result, and determining whether to delete the vertex according to the judgment result and the cutting mode button information triggered by the user; the method specifically comprises the following steps:

and if the judgment result is that the vertex is in the cutting range and the user triggers the information of the external cutting mode button, deleting the vertex.

And if the judgment result is that the vertex is in the cutting range and the user triggers the button information of the internal cutting mode, reserving the vertex.

And if the judgment result is that the vertex is not in the cutting range and the user triggers the information of the external cutting mode button, reserving the vertex.

And if the judgment result is that the vertex is not in the cutting range and the user triggers the button information of the internal cutting mode, deleting the vertex.

The cut results include the retained vertices.

The first step in the present embodiment to perform the model cut first requires drawing an enclosed area on a two-dimensional plane following the user's finger. Since only two-dimensional screen coordinates of a user on a two-dimensional screen can be acquired, and the OpenGL ES display range line requires world three-dimensional coordinates, a series of coordinate conversion work needs to be performed on the range line first and then the range line is displayed.

Referring to fig. 5, the main steps of displaying the cutting range in this embodiment are as follows:

recording the coordinate information of each finger movement of the user, and adding the start coordinate to the coordinate information of the guarantee line at the tail of the coordinate information set every time to form a closed irregular polygon area which is the cutting range.

The system converts the two-dimensional screen coordinates of the line into two-dimensional normalized device coordinates and stores the two-dimensional normalized device coordinates for subsequent range comparison with points of the three-dimensional model point projected on the two-dimensional view.

Since the three-dimensional coordinate point projection of the model uses the perspective projection matrix and the camera matrix on the two-dimensional plane, the system finds the inverse matrix of the two matrices to convert the two-dimensional normalized device coordinates of the line into world three-dimensional coordinates and stores them.

And storing the world three-dimensional coordinate information of the line into a buffer area.

A line's rendering program is created and shaders are connected.

And transmitting the total transformation matrix to a shader of the range line, calling a rendering function, rendering the range line by the rendering pipeline and displaying the range line on a screen.

After the range drawn by the user is processed and displayed, the system takes out the two-dimensional normalization equipment coordinates of the line, takes out the point data of the model and the coordinates of the model point transmitted on the two-dimensional screen, which are obtained by multiplying the final transformation matrix of the model, and compares the point data with the coordinates, if the point data are within the cutting range, the point data are deleted, otherwise, the point data are kept unchanged.

The judging whether the vertex is in the cutting range specifically includes:

and judging whether the first ray corresponding to the vertex has an odd number of intersection points with the cutting range, and if so, determining that the vertex of the model is in the cutting range.

The first ray corresponding to the vertex is a ray emitted to the right from the vertex.

Referring to fig. 6, the method for determining whether the vertex is within the cutting range in this embodiment is as follows: and (3) emitting a ray to the right from the vertex of the model, if the ray has even number of intersection points with the cutting range, the vertex is out of the cutting range, otherwise, the ray has odd number of intersection points, the vertex is in the cutting range.

Meanwhile, the vertex data required to be transmitted by OpenGl ES is ordered, three vertexes are triangular surfaces, and if only one vertex is deleted, the relation of the whole vertex data surface is disordered to cause model display errors.

In practical application of this embodiment, the model cutting module is further configured to obtain a starting position and a number of vertices that are retained after each cutting in the process of cutting the model file.

And acquiring the interval of the reserved vertexes after each cutting according to the starting positions of the reserved vertexes after each cutting and the number of the vertexes.

And acquiring the vertex which is coincided in the vertex reserved interval after each cutting in the process of cutting the model file according to the vertex reserved interval after each cutting.

And rendering the coincident vertexes in the vertexes reserved after each cutting by adopting the preset model rendering function.

Referring to fig. 7, SP denotes the starting position of a rendering point (reserved vertex) stored by the system, and VC denotes the number of points (reserved vertices) that the system should render from this SP. The entire light color is all the vertices of the model, and the system initially stores the first vertex of the model and the number of all vertices so that the model is initially displayed in its entirety. After the cutting operation, the dark color part is the part of the vertex needing to be reserved, the system firstly adds the model state before cutting into the model state list, then stores the starting position and the number of the vertex of each segment of the dark color part, and the light color part is regarded as the cutting deletion.

Referring to fig. 8, the method adopted by the system for realizing the multiple cutting operations of the doctor is point-interval combination, the system finds the interval needing rendering points during the last cutting and finds the interval needing rendering points during the current cutting to perform interval combination, and the vertex in the intersection of the reserved intervals is the vertex needing rendering by the system finally, so that the multiple cutting effect of the model is achieved.

As shown in fig. 8, it is assumed that two times of cutting are required in the cutting process for the model file, a set a represents a vertex (a vertex to be retained) interval that needs to be rendered after the first cutting, a set B represents a vertex (a vertex to be retained) interval that needs to be rendered after the second cutting, and a set ans represents a vertex interval that needs to be rendered after intersection of a and B, that is, a final result that needs to be rendered finally.

The internal cutting and the external cutting of the system are realized by adopting the method, so that the internal and external mixed cutting of the model is also applicable. Each time the surgeon performs a cutting operation, the system records the state information of the previous model into a state list.

When the doctor chooses to undo the cut, the system retrieves the last state from the model state list to restore the model. Because the system stores the starting position of the point to be rendered and the vertex data of the model which is not deleted or moved in quantity when the system cuts each time, under the condition that a plurality of models are displayed or the models are overlarge, the memory consumption and the time consumption are greatly saved, and the efficiency and the fluency of the system are improved.

In practical application of this embodiment, the system further includes:

and the PC-end medical image three-dimensional reconstruction module is connected with the mobile terminal and used for reading medical image data acquired in advance to carry out three-dimensional reconstruction, reconstructing two-dimensional medical image data into a three-dimensional model and storing the three-dimensional model into a preset initial model file in the mobile terminal.

Referring to fig. 9, in the present embodiment, the main method for generating the medical three-dimensional model from the medical image data is as follows:

and reading the Dicom medical image data, and storing the read data as three-dimensional discrete point data, wherein each point data stores the pixel value of the Dicom image.

Segmenting a human tissue region in a medical image: selecting corresponding seed points according to different tissues, setting a gray threshold difference as a growth criterion, expanding outwards from the seed points until no more pixel points meet the condition, marking the points meeting the condition by segmentation as 1, and marking the points not meeting the condition as 0.

In the segmented three-dimensional discrete data field with the human tissue marks, every eight vertexes form a cube, an edge with two end vertexes marked differently is marked as a marked edge, an isosurface is on the marked edge, and finally the specific position of the marked edge is determined through linear interpolation.

And solving a centroid of an equivalent point in a marked edge contained in a cube, wherein the centroid point is the vertex of the model.

Because one edge is shared by four cubes, one marked edge corresponds to four centroid points, the four centroid points are connected to form two triangular surfaces, all marked edges in the three-dimensional data field are connected with the corresponding centroid points, and finally, a complete medical three-dimensional model formed by the triangular surface patches is obtained.

In the medical three-dimensional Model display and cutting system based on the mobile terminal in the embodiment, because the Model-View Model architecture is adopted, compared with the prior art, the Model-View Model architecture can enable the mobile terminal to load a plurality of Model files and perform various operations at the same time, and can efficiently cut the Model files.

In the medical three-dimensional Model display and cutting system based on the mobile terminal in the embodiment, as the Model-View-View Model architecture is adopted, the modules are decoupled greatly, and the data in the Model file is shared globally, the memory overhead is saved, the real-time performance and the correctness of the data are ensured, and the reusability is improved.

In the medical three-dimensional model display and cutting system based on the mobile terminal in the embodiment, the model cutting module is used for cutting the model file, the system stores the starting position of a point to be rendered and the number of points which are not deleted or the vertex data of the model is not moved during each cutting, under the condition that a plurality of models are displayed or the models are too large, the memory consumption and the time consumption are greatly saved, and the efficiency and the fluency of the system are improved.

Since the system described in the above embodiment of the present invention is a system used for implementing the method of the above embodiment of the present invention, a person skilled in the art can understand the specific structure and the modification of the system based on the method described in the above embodiment of the present invention, and thus the detailed description is omitted here. All systems adopted by the method of the above embodiments of the present invention are within the intended scope of the present invention.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions.

It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the terms first, second, third and the like are for convenience only and do not denote any order. These words are to be understood as part of the name of the component.

Furthermore, it should be noted that in the description of the present specification, the description of the term "one embodiment", "some embodiments", "examples", "specific examples" or "some examples", etc., means that a specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, the claims should be construed to include preferred embodiments and all changes and modifications that fall within the scope of the invention.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include such modifications and variations.