1. A method for automatically recommending AI programs based on performance analysis of structured reports, comprising:

recording modification logs of AI data output by different AI models with the same inspection purpose;

when the completion number of the image structured reports of the inspection purpose reaches a preset threshold value, outputting the coincidence rate of the AI data output by each AI model based on the modification log;

and determining the AI model with the highest coincidence rate as a default AI scheme under the examination purpose, and automatically recommending the default AI scheme to the doctor when the doctor prepares to write the image structured report.

2. The method for automatically recommending AI programs based on structured report performance analysis of claim 1, further comprising: the modification log includes: the exam objective, the list of lesions output by the AI model, the physician's confirmation of the list of lesions, measurements, modified measurements, key images, adjusted key images, and diagnostic conclusions sent into the visual structured report.

3. The method for automatically recommending AI programs based on the structured report performance analysis of claim 2, wherein said outputting the rate of compliance of the AI data output by each of said AI models comprises:

determining the probability of false positive lesions based on the lesion list output by the AI model and the lesion list confirmed by the doctor and sent to the image structured report;

determining a modification rate of the measurement value and the key image based on the measurement value, the modified measurement value, the key image, the adjusted key image, and the diagnostic conclusion;

and outputting the coincidence rate of the AI data output by each AI model according to the probability of the false positive focus and/or the modification rate.

4. The method for automatically recommending AI plans based on structured report performance analysis according to claim 1, characterized in that, while recommending said default AI plan to the doctor, the method further comprises: and prompting the AI data output by all other AI models under the examination purpose and the corresponding coincidence rate to a doctor.

5. The method for automatically recommending AI programs for performance analysis based on structured reports of claim 4, wherein when said doctor uses said default AI program to draw said diagnostic conclusion, the method further comprises:

calculating the coincidence rate of AI data output by the AI model corresponding to the default AI scheme, and defining the coincidence rate as the default AI scheme coincidence rate;

and when the default AI scheme coincidence rate is lower than the coincidence rate of the AI data output by any AI model under the examination purpose, automatically prompting a doctor.

6. The method for automatically recommending AI programs based on structured report performance analysis of claim 3, further comprising: counting the number of times of using each AI model in the preset threshold, and outputting the coincidence rate of the AI data output by each AI model according to a preset rule according to the probability of the false positive lesion and/or the modification rate and the number of times of using the AI model.

7. A system for automatically recommending AI programs based on performance analysis of structured reports, the system comprising: a recording module, a processing module and a recommending module, wherein,

the recording module is connected with the processing module and is used for recording modification logs of AI data output by different AI models with the same inspection purpose;

the processing module is respectively connected with the recording module and the recommending module and is used for outputting the coincidence rate of the AI data output by each AI model based on the modification log when the completion number of the image structured reports of the inspection purpose reaches a preset threshold value;

and the recommending module is connected with the processing module and is used for determining the AI model with the highest coincidence rate as a default AI scheme under the examination purpose, and automatically recommending the default AI scheme to a doctor when the doctor prepares to write the image structured report.

8. The system for automatically recommending AI programs for performance analysis based on structured reports of claim 7, wherein said modification log comprises: the exam objective, the list of lesions output by the AI model, the physician's confirmation of the list of lesions, measurements, modified measurements, key images, adjusted key images, and diagnostic conclusions sent into the visual structured report.

9. The system for automatically recommending AI programs based on structured report performance analysis of claim 8, wherein said processing module further comprises: a determination unit, a calculation unit and an output unit, wherein,

the determining unit is connected with the output unit and used for determining the probability of false positive focus based on the focus list output by the AI model and the focus list sent to the image structured report and confirmed by the doctor;

the computing unit is connected with the output unit and is used for determining the modification rate of the measured value and the key image based on the measured value, the modified measured value, the key image, the adjusted key image and the diagnosis conclusion;

and the output unit is respectively connected with the determining unit and the calculating unit and is used for outputting the coincidence rate of the AI data output by each AI model according to the probability of the false positive focus and/or the modification rate.

10. The system for automatically recommending AI schemes based on performance analysis of structured reports as claimed in claim 7, wherein said recommending module further comprises a prompting unit for prompting the doctor about the AI data outputted by all other AI models for the purpose of examination and the corresponding said coincidence rates, while recommending said default AI scheme to the doctor.

11. The system for automatically recommending AI plans based on structured report performance analysis of claim 10, wherein after the doctor uses said default AI plan to draw said diagnosis, the system further comprises: a calculation module and a comparison module, wherein,

the calculation module is connected with the recording module and used for calculating the coincidence rate of the AI data output by the AI model corresponding to the default AI scheme and defining the coincidence rate as the default AI scheme;

and the comparison module is respectively connected with the calculation module and the processing module and is used for automatically prompting a doctor when the compliance rate of the default AI scheme is lower than that of the AI data output by any AI model under the examination purpose.

12. The system for automatically recommending AI schemes based on structured report performance analysis according to claim 9, wherein said processing module further comprises a statistic unit connected to said output unit for counting the number of times each of said AI models is used in said preset threshold, and outputting the compliance rate of the AI data output by each of said AI models according to a preset rule based on the probability of said false positive lesion and/or said modification rate in combination with the number of times said AI models are used.

Background

The use of image AI diagnostic models by current image departments is increasing, and a plurality of image AI diagnostic models look more homogeneous, that is, in a business scene, AI diagnostic models of a plurality of manufacturers can be selected. When writing the image structured report, the diagnostician selects different AI diagnostic models, completely through personal preference, and manually clicks the menu for selection. The AI diagnosis model outputs auxiliary diagnosis data which is correct, good in quality, good in use and not objectively judged, and only one-line diagnosis doctor has a certain public praise, so that the AI diagnosis model has high subjectivity. The image department is difficult to evaluate which AI diagnostic model should be selected through objective performance; or to determine what AI diagnostic model to use in what circumstances.

Disclosure of Invention

In view of the above, the main objective of the present invention is to provide a method and a system for automatically recommending an AI scheme based on performance analysis of a structured report, which can solve the problem that in the prior art, an appropriate AI scheme cannot be automatically recommended for video diagnosis due to the fact that the quality of auxiliary diagnostic data output by an AI diagnostic model cannot be objectively evaluated.

In order to achieve the purpose, the technical scheme of the invention is realized as follows:

in one aspect, the present invention provides a method for automatically recommending an AI project based on performance analysis of a structured report, comprising: recording modification logs of AI data output by different AI models with the same inspection purpose; when the completion number of the image structured reports of the inspection purpose reaches a preset threshold value, outputting the coincidence rate of the AI data output by each AI model based on the modification log; the AI model with the highest rate of compliance is determined as the default AI protocol for the purpose of the examination, which is automatically recommended to the physician when the physician is ready to compose an image structured report.

Preferably, the method further comprises: the modification log includes: exam objectives, a list of lesions output by the AI model, physician confirmation of the list of lesions sent to the visual structured report, measurements, modified measurements, key images, adjusted key images, and diagnostic conclusions.

Preferably, outputting the coincidence rate of the AI data output by each AI model includes: confirming a focus list sent to an image structured report by a doctor based on the focus list output by the AI model, and determining the probability of false positive focus; determining a modification rate of the measurement values and the key images based on the measurement values, the modified measurement values, the key images, the adjusted key images, and the diagnostic conclusion; and outputting the coincidence rate of the AI data output by each AI model according to the probability and/or the modification rate of the false positive focus.

Preferably, while recommending the default AI protocol to the doctor, the method further comprises: the AI data output by all other AI models under the examination purpose and the corresponding coincidence rate are prompted to the doctor.

Preferably, when the doctor has concluded a diagnosis using the default AI protocol, the method further comprises: calculating the coincidence rate of AI data output by the AI model corresponding to the default AI scheme, and defining the coincidence rate as the default AI scheme coincidence rate; when the default AI scheme coincidence rate is lower than the coincidence rate of the AI data output by any AI model under the examination purpose, automatically prompting the doctor.

Preferably, the method further comprises: and counting the use times of each AI model in a preset threshold, and outputting the coincidence rate of the AI data output by each AI model according to a preset rule according to the probability and/or the modification rate of false positive lesions and the use times of the AI models.

In another aspect, the present invention provides a system for automatically recommending an AI scheme based on a performance analysis of a structured report, the system comprising: the device comprises a recording module, a processing module and a recommending module, wherein the recording module is connected with the processing module and used for recording modification logs of AI data output by different AI models with the same inspection purpose; the processing module is respectively connected with the recording module and the recommending module and is used for outputting the coincidence rate of the AI data output by each AI model based on the modification log when the completion number of the image structured reports of the inspection purpose reaches a preset threshold value; and the recommending module is connected with the processing module and is used for determining the AI model with the highest coincidence rate as a default AI scheme under the examination purpose, and automatically recommending the default AI scheme to the doctor when the doctor prepares to write an image structured report.

Preferably, the modification log comprises: exam objectives, a list of lesions output by the AI model, physician confirmation of the list of lesions sent to the visual structured report, measurements, modified measurements, key images, adjusted key images, and diagnostic conclusions.

Preferably, the processing module further comprises: the system comprises a determining unit, a calculating unit and an output unit, wherein the determining unit is connected with the output unit and used for determining the probability of false positive focuses on the basis of a focus list output by an AI model and a focus list sent to an image structured report and confirmed by a doctor; the computing unit is connected with the output unit and used for determining the modification rate of the measured value and the key image based on the measured value, the modified measured value, the key image, the adjusted key image and the diagnosis conclusion; and the output unit is respectively connected with the determining unit and the calculating unit and is used for outputting the coincidence rate of the AI data output by each AI model according to the probability and/or the modification rate of the false positive lesions.

Preferably, the recommending module further comprises a prompting unit for prompting the doctor with the AI data output by all other AI models under the examination purpose and the corresponding coincidence rate while recommending the default AI scheme to the doctor.

Preferably, when the doctor uses the default AI protocol to draw a diagnosis, the system further comprises: the calculation module is connected with the recording module and used for calculating the coincidence rate of AI data output by the AI model corresponding to the default AI scheme and defining the coincidence rate as the default AI scheme coincidence rate; and the comparison module is respectively connected with the calculation module and the processing module and is used for automatically prompting a doctor when the default AI scheme coincidence rate is lower than the coincidence rate of the AI data output by any AI model under the examination purpose.

Preferably, the processing module further includes a counting unit connected to the output unit, and configured to count the number of times each AI model is used in a preset threshold, and output a compliance rate of the AI data output by each AI model according to a preset rule according to the probability and/or modification rate of the false positive lesion in combination with the number of times the AI model is used.

The invention has the technical effects that:

1. the method can dynamically record the modification log of the AI data output by the doctor to the AI model, and output the coincidence rate of the AI data output by each AI model based on the modification log; determining the AI model with the highest coincidence rate as a default AI scheme under the examination purpose, and automatically recommending the default AI scheme to the doctor when the doctor prepares to write an image structured report; according to the modification of the filling content of the structured report by the imaging doctor and the analysis of the purpose of patient examination, whether the AI model is suitable for the diagnosis of the case or not can be objectively judged, and the AI model with the highest default quality can be automatically pushed for use;

2. the invention can also recommend the default AI scheme to the doctor, and simultaneously prompt the AI data output by all other AI models under the examination purpose and the corresponding coincidence rate to the doctor, so that the doctor can know the coincidence rate of the AI data output by each AI model under the same examination purpose, and the comparison between each AI model is convenient;

3. the method can also calculate the coincidence rate of the AI data output by the AI model corresponding to the default AI scheme after the doctor uses the default AI scheme to obtain a diagnosis conclusion, when the coincidence rate of the default AI scheme is lower than the coincidence rate of the AI data output by any AI model under the examination purpose, the method automatically prompts the doctor so as to facilitate the doctor to examine again, even can give up the report, directly calls another more reasonable AI data output by the AI model to rewrite the report, can eliminate the improper AI model, emerges the AI model with high quality, reduces the selection judgment cost of departments, and improves the efficiency and quality of the doctor in writing the report;

4. the method can also count the using times of each AI model in the preset threshold, output the coincidence rate of the AI data output by each AI model according to the preset rule based on the modification log and the using times of the AI models, and improve the accuracy of the coincidence rate, so that the recommended AI scheme is more reasonable and has stronger applicability.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:

fig. 1 is a flow chart illustrating a method for automatically recommending an AI scheme based on a structured report performance analysis according to an embodiment of the present invention;

fig. 2 is a schematic diagram of 3 AI diagnosis model interfaces displayed by the structured reporting system when a diagnostician opens a patient image and a report in the method for automatically recommending an AI scheme based on the performance analysis of the structured report according to the embodiment of the present invention;

fig. 3 is a schematic interface diagram illustrating that AI data output by an AI model is filled in each control element of a structured report when a diagnostician selects an AI model for report composition in a method for automatically recommending an AI scheme based on performance analysis of a structured report according to an embodiment of the present invention;

fig. 4 is a schematic diagram illustrating an interface for modifying AI data by a doctor in a method for automatically recommending an AI scheme based on performance analysis of a structured report according to an embodiment of the present invention;

fig. 5 is a schematic diagram illustrating an AI data interface returned by different AI models for the same inspection purpose in a method for automatically recommending an AI scheme based on performance analysis of a structured report according to an embodiment of the present invention;

fig. 6 is an interface diagram illustrating automatic recommendation of a default AI scenario when a doctor opens a report for a patient in a method for automatically recommending an AI scenario based on a structured report performance analysis according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of a system for automatically recommending an AI scheme based on the performance analysis of the structured report according to the second embodiment of the present invention;

fig. 8 is a schematic diagram of an interface of 3 AI diagnosis models displayed by the structured reporting system when a diagnostician opens a patient image and report in the system for automatically recommending an AI scheme based on the performance analysis of the structured report according to the second embodiment of the present invention;

fig. 9 is a schematic interface diagram illustrating that AI data output by an AI model is filled in each control element of a structural report when a diagnostician selects one AI model for report composition in the system for automatically recommending an AI scheme based on performance analysis of a structural report according to the second embodiment of the present invention;

fig. 10 is a schematic diagram illustrating an interface for modifying AI data by a doctor in the system for automatically recommending an AI scheme based on the performance analysis of the structured report according to the second embodiment of the present invention;

fig. 11 is a schematic diagram illustrating an AI data interface returned by different AI models for the same inspection purpose in the system for automatically recommending an AI scheme based on the performance analysis of the structured report according to the second embodiment of the present invention;

fig. 12 is an interface diagram illustrating automatic recommendation of a default AI scenario when a doctor opens a report of a certain patient in the system for automatically recommending an AI scenario based on performance analysis of a structured report according to the second embodiment of the present invention;

fig. 13 is a schematic diagram of a system for automatically recommending an AI scheme based on the performance analysis of the structured report according to a third embodiment of the present invention;

fig. 14 is a schematic diagram of a system for automatically recommending an AI scheme based on the performance analysis of the structured report according to a fourth embodiment of the present invention;

fig. 15 is a schematic diagram of a system for automatically recommending an AI scheme based on the performance analysis of the structured report according to the fifth embodiment of the present invention;

fig. 16 is a schematic structural diagram of a system for automatically recommending an AI scheme based on the performance analysis of the structured report according to a sixth embodiment of the present invention.

Detailed Description

The present invention will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Example one

Fig. 1 is a flowchart illustrating a method for automatically recommending an AI scheme based on a performance analysis of a structured report according to an embodiment of the present invention, and the method includes the following steps as shown in fig. 1:

step S101, recording modification logs of AI data output by different AI models with the same inspection purpose;

based on the integration of the image AI model and the structured report of the relevant disease category, the output of the image AI model can be automatically filled in the relevant control in the image structured report, and the embodiment of the invention can judge whether the AI model is suitable for the diagnosis of the case according to the modification of the AI data by the diagnostician and the analysis of the examination purpose of the patient. Based on this analysis, embodiments of the invention may automatically select a default higher quality AI model in the background for recommendation, which the diagnostician may use.

The way in which different video AI models are evaluated by structured reporting is personalized. However, in any image AI model, the performance evaluation is related to the type/severity of the image findings for the purpose of inspection, thereby determining the performance difference between different image AI models in the two categories under different scenes for dynamic, automatic and reasonable selection.

An AI diagnostic model of lung nodules is exemplified. Some lung nodules have particularly high AI sensitivity and poor specificity, but the discovered nodules are better arranged according to the risk (the risk is evaluated according to analysis of size, shape and content texture); some lung nodules have less high AI sensitivity but better specificity. The reason for some patients to do breast CT is examination screening; some reasons are to judge whether there is disease in chest; some patients are followed up with past nodules, and so on.

Typically, if there are multiple AIs for the same purpose, the PACS system will push images of the patient to the AIs at the same time. The AIs each generate measurements and key images for the patient and send to a receive cache of the structured report server.

Fig. 2 is a schematic diagram of 3 AI diagnosis model interfaces displayed by the structured reporting system when a diagnostician opens a patient image and a report in the method for automatically recommending an AI scheme based on the performance analysis of the structured report according to the embodiment of the present invention; as shown in FIG. 2, when the diagnostician double-clicks the patient list to open the patient image and report the diagnosis, the structured reporting system informs the diagnostician that the results of the 3 AI models are available for selection, and that the physician can select one of the output data to compose the report. In the interface schematic diagram, data returned by the AI models of the three lung nodules are displayed in the "select external data", and the doctor manually selects the data according to the DICOM image of the patient.

Fig. 3 is a schematic interface diagram illustrating that AI data output by an AI model is filled in each control element of a structured report when a diagnostician selects an AI model for report composition in a method for automatically recommending an AI scheme based on performance analysis of a structured report according to an embodiment of the present invention; as shown in fig. 3, when a diagnostician selects one of the AI diagnostic models, the AI data (measured values of lesions, key images, etc.) output by the diagnostician is automatically added to the corresponding controls of the structured report interface. For example, the volume of the nodes in the upper left Lung lobe is 111 cubic millimeters, and the Lung RADS given by the AI model is classified as class 2.

Step S102, when the completion number of the image structured reports of the inspection purpose reaches a preset threshold, outputting the coincidence rate of the AI data output by each AI model based on the modification log;

wherein modifying the log comprises: exam objectives, a list of lesions output by the AI model, physician confirmation of the list of lesions sent to the visual structured report, measurements, modified measurements, key images, adjusted key images, and diagnostic conclusions.

The preset threshold may be set at will according to requirements, for example, the number of completed image structured reports of lung nodule examination is set to 100 cases;

wherein, outputting the coincidence rate of the AI data output by each AI model comprises:

confirming a focus list sent to an image structured report by a doctor based on the focus list output by the AI model, and determining the probability of false positive focus;

determining a modification rate of the measurement values and the key images based on the measurement values, the modified measurement values, the key images, the adjusted key images, and the diagnostic conclusion;

and outputting the coincidence rate of the AI data output by each AI model according to the probability and/or the modification rate of the false positive focus.

For example, the number of completed image structured reports for a pulmonary nodule examination is set to 100, the examination purpose is a pulmonary nodule examination, 3 AI models corresponding to the examination purpose are used, 50 of the AI models are the a1 model, 30 of the AI models are the a2 model, 20 of the AI models are the A3 model, the probability of the false positive lesions of the a1 model is the average probability of the 50 cases, and the modification rate is the average modification rate of the 50 cases, and finally, the processing manner of the coincidence rate of the AI model, the coincidence rate of the a2 model, and the coincidence rate of the A3 model with the coincidence rate of the a1 model is output according to the average probability and the average modification rate.

In the invention, the calculation of the coincidence rate is automatically processed once when the preset threshold is met in the iterative processing of the coincidence rate, for example, the preset threshold is set as 100, that is, after 100 reports are reached, the system automatically recalculates the coincidence rate of the AI data output by each AI model under the checking purpose once, thereby reconfirming the default AI scheme again. The method can improve the accuracy of the default AI scheme, so that the system is more humanized.

Fig. 4 is a schematic diagram illustrating an interface for modifying AI data by a doctor in a method for automatically recommending an AI scheme based on performance analysis of a structured report according to an embodiment of the present invention; as shown in FIG. 4, the diagnostician modifies the AI data based on the DICOM image analysis of the patient.

The diagnostician reads the images, first checks through the list of lesions, selects certain lesions deemed reasonable, sends them to the structured report, and discards other lesion items found by the AI.

Fig. 5 is a schematic diagram illustrating an AI data interface returned by different AI models for the same inspection purpose in a method for automatically recommending an AI scheme based on performance analysis of a structured report according to an embodiment of the present invention; as shown in fig. 5, the physician clicks "send to report" in the interface after selecting a reasonable lesion.

Second, the doctor may correct the measurements output by the AI model, delete and supplement the key images, which is recorded by the structured report log system.

Third, the diagnostician completes the diagnostic analysis using the structured report, and composes a diagnostic decision. This diagnostic conclusion was also structured and labeled using RADLEX coding. For example, a decision "Lung-RADS is ranked 3" is given in the business of LUNG nodules.

In the case of lung nodules, after the report is complete, the system obtains the following records: examination goals, a list of lesions returned by the AI, a list of lesions that the physician confirms sent to the visual structured report, measurements, diagnostician-adjusted measurements, key images, diagnostician-adjusted key images, and a final diagnosis. Other AIs may be integrated with the structured report in a similar order, but the subsequent classification analysis logic may be changed, again without limitation.

The diagnostician may also review the returned results for different AI models, again taking the AI of the lung nodule as an example. If the patient is a health screener, then the assessment method for AI is the sequential rationality of the lesion list. That is, whether the order of the suspected lesions listed in the front is correct. Thus, the diagnostician can ensure that the lesions are analyzed for diagnostic value according to the AI submission sequence and that subsequent low-risk suspected lesion queues are discarded in time. If the patient is a febrile, coughing patient, the AI concern for the lung nodule is how much of the list of lesions it finds is selected by the physician to send into the report. Too many false positives are not a good output.

By analyzing the logic, when the AI model is loaded, the result loading of a certain AI model can be selected by default, and a plurality of AI results can be informed, and the performance differences of the AI results under different inspection purposes can be obtained.

Step S103, the AI model with the highest coincidence rate is determined as the default AI scheme under the examination purpose, and when the doctor prepares to write the image structured report, the default AI scheme is automatically recommended to the doctor.

Fig. 6 is an interface diagram illustrating automatic recommendation of a default AI scenario when a doctor opens a report for a patient in a method for automatically recommending an AI scenario based on a structured report performance analysis according to an embodiment of the present invention; as shown in fig. 6, if the doctor has a high match rate using XX lung nodule during the lung cancer screening examination, the XX lung nodule AI model is loaded by default.

Wherein, while recommending the default AI scenario to the physician, the method further comprises: the AI data output by all other AI models under the examination purpose and the corresponding coincidence rate are prompted to the doctor, so that the doctor can conveniently diagnose and compare the AI data.

Wherein, when the doctor uses the default AI protocol to draw a diagnosis, the method further comprises:

calculating the coincidence rate of AI data output by the AI model corresponding to the default AI scheme, and defining the coincidence rate as the default AI scheme coincidence rate;

when the default AI scheme coincidence rate is lower than the coincidence rate of the AI data output by any AI model under the examination purpose, automatically prompting the doctor. If the coincidence rate of the AI data output by one AI model is higher than the coincidence rate of the default AI scheme, the doctor is prompted, and the diagnostician can judge the accuracy of the measured value/key image when the default AI scheme is applied in the scene, so that the doctor can check the result or introduce a more reasonable AI model to rewrite the report.

Preferably, the method further comprises: and counting the use times of each AI model in a preset threshold, and outputting the coincidence rate of the AI data output by each AI model according to a preset rule according to the probability and/or the modification rate of false positive lesions and the use times of the AI models.

For example, the number of completed image structured reports for a pulmonary nodule examination is set to 100, the examination purpose is a pulmonary nodule examination, 3 AI models corresponding to the examination purpose are used, 50 of the AI models are an a1 model, 30 of the AI models are an a2 model, 20 of the AI models are an A3 model, the probability of false positive lesions of the a1 model is the average probability of the 50, and the modification rate is also the average modification rate of the 50, and finally, according to the average probability, the average modification rate and the number of uses, the processing mode of the coincidence rate of the AI model, the coincidence rate of the a2 model and the coincidence rate of the A3 model with the a1 model is output according to a preset rule.

The preset rule can be set according to actual conditions, for example, when the number of times of use reaches a certain number, the coincidence rate is divided into more points, when the number of times of use is less than the certain number, the coincidence rate is divided into less points, and the like, and the preset rule is not limited. This approach circumvents the situation where a certain AI model is used only a few times, but the match rate is also the highest, and is therefore determined to be the default AI scenario. The factor of using times is added, and the coincidence rate is more accurate.

The embodiment of the invention can dynamically record the modification log of the AI data output by the doctor to the AI model, and output the coincidence rate of the AI data output by each AI model based on the modification log; determining the AI model with the highest coincidence rate as a default AI scheme under the examination purpose, and automatically recommending the default AI scheme to the doctor when the doctor prepares to write an image structured report; according to the modification of the filling content of the structured report by the imaging doctor and the analysis of the purpose of patient examination, whether the AI model is suitable for the diagnosis of the case or not can be objectively judged, and the AI model with the highest default quality can be automatically pushed for use; the embodiment of the invention can also recommend the default AI scheme to the doctor, and simultaneously prompt the AI data output by all other AI models under the examination purpose and the corresponding coincidence rate to the doctor, so that the doctor can know the coincidence rate of the AI data output by each AI model under the same examination purpose, and the comparison between each AI model is convenient; the embodiment of the invention can also calculate the coincidence rate of the AI data output by the AI model corresponding to the default AI scheme after the doctor uses the default AI scheme to obtain a diagnosis conclusion, when the coincidence rate of the default AI scheme is lower than that of the AI data output by any AI model under the examination purpose, the coincidence rate is automatically prompted to the doctor so that the doctor can examine the data, even the report can be abandoned, and another more reasonable AI data output by the AI model can be directly called to rewrite the report, so that the improper AI model can be eliminated, the AI model with high quality can emerge, the selection judgment cost of a department is reduced, and the report writing efficiency and quality of the doctor are improved; the embodiment of the invention can also count the using times of each AI model in the preset threshold, output the coincidence rate of the AI data output by each AI model according to the preset rule based on the modification log and the using times of the AI model, and improve the accuracy of the coincidence rate, so that the recommended AI scheme is more reasonable and has stronger applicability.

Example two

Fig. 7 is a schematic structural diagram of a system for automatically recommending an AI scheme based on performance analysis of a structured report according to a second embodiment of the present invention, and as shown in fig. 7, the system includes: a recording module 10, a processing module 20 and a recommendation module 30, wherein,

the recording module 10 is connected with the processing module 20 and is used for recording modification logs of AI data output by different AI models with the same examination purpose;

based on the integration of the image AI model and the structured report of the relevant disease category, the output of the image AI model can be automatically filled in the relevant control in the image structured report, and the embodiment of the invention can judge whether the AI model is suitable for the diagnosis of the case according to the modification of the AI data by the diagnostician and the analysis of the examination purpose of the patient. Based on this analysis, embodiments of the invention may automatically select a default higher quality AI model in the background for recommendation, which the diagnostician may use.

The way in which different video AI models are evaluated by structured reporting is personalized. However, in any image AI model, the performance evaluation is related to the type/severity of the image findings for the purpose of inspection, thereby determining the performance difference between different image AI models in the two categories under different scenes for dynamic, automatic and reasonable selection.

An AI diagnostic model of lung nodules is exemplified. Some lung nodules have particularly high AI sensitivity and poor specificity, but the discovered nodules are better arranged according to the risk (the risk is evaluated according to analysis of size, shape and content texture); some lung nodules have less high AI sensitivity but better specificity. The reason for some patients to do breast CT is examination screening; some reasons are to judge whether there is disease in chest; some patients are followed up with past nodules, and so on.

Typically, if there are multiple AIs for the same purpose, the PACS system will push images of the patient to the AIs at the same time. The AIs each generate measurements and key images for the patient and send to a receive cache of the structured report server.

Fig. 8 is a schematic diagram of an interface of 3 AI diagnosis models displayed by the structured reporting system when a diagnostician opens a patient image and report in the system for automatically recommending an AI scheme based on the performance analysis of the structured report according to the second embodiment of the present invention; as shown in FIG. 8, when the diagnostician double-clicks the patient list to open the patient image and report the diagnosis, the structured reporting system informs the diagnostician that the results of the 3 AI models are available for selection, and that the physician can select one of the output data to compose the report. In the interface schematic diagram, data returned by the AI models of the three lung nodules are displayed in the "select external data", and the doctor manually selects the data according to the DICOM image of the patient.

Fig. 9 is a schematic interface diagram illustrating that AI data output by an AI model is filled in each control element of a structural report when a diagnostician selects one AI model for report composition in the system for automatically recommending an AI scheme based on performance analysis of a structural report according to the second embodiment of the present invention; as shown in fig. 9, when a diagnostician selects one of the AI diagnostic models, the AI data (measured values of lesions, key images, etc.) output by the diagnostician is automatically added to the corresponding controls of the structured report interface. For example, the volume of the nodes in the upper left Lung lobe is 111 cubic millimeters, and the Lung RADS given by the AI model is classified as class 2.

The processing module 20 is respectively connected to the recording module 10 and the recommending module 30, and is configured to output, based on the modification log, a coincidence rate of the AI data output by each AI model when the number of completed image structured reports for the inspection purpose reaches a preset threshold;

wherein modifying the log comprises: exam objectives, a list of lesions output by the AI model, physician confirmation of the list of lesions sent to the visual structured report, measurements, modified measurements, key images, adjusted key images, and diagnostic conclusions.

The preset threshold may be set at will according to requirements, for example, the number of completed image structured reports of lung nodule examination is set to 100 cases;

wherein, outputting the coincidence rate of the AI data output by each AI model comprises:

confirming a focus list sent to an image structured report by a doctor based on the focus list output by the AI model, and determining the probability of false positive focus;

determining a modification rate of the measurement values and the key images based on the measurement values, the modified measurement values, the key images, the adjusted key images, and the diagnostic conclusion;

and outputting the coincidence rate of the AI data output by each AI model according to the probability and/or the modification rate of the false positive focus.

For example, the number of completed image structured reports for a pulmonary nodule examination is set to 100, the examination purpose is a pulmonary nodule examination, 3 AI models corresponding to the examination purpose are used, 50 of the AI models are the a1 model, 30 of the AI models are the a2 model, 20 of the AI models are the A3 model, the probability of the false positive lesions of the a1 model is the average probability of the 50 cases, and the modification rate is the average modification rate of the 50 cases, and finally, the processing manner of the coincidence rate of the AI model, the coincidence rate of the a2 model, and the coincidence rate of the A3 model with the coincidence rate of the a1 model is output according to the average probability and the average modification rate.

In the invention, the calculation of the coincidence rate is automatically processed once when the preset threshold is met in the iterative processing of the coincidence rate, for example, the preset threshold is set as 100, that is, after 100 reports are reached, the system automatically recalculates the coincidence rate of the AI data output by each AI model under the checking purpose once, thereby reconfirming the default AI scheme again. The method can improve the accuracy of the default AI scheme, so that the system is more humanized.

Fig. 10 is a schematic diagram illustrating an interface for modifying AI data by a doctor in the system for automatically recommending an AI scheme based on the performance analysis of the structured report according to the second embodiment of the present invention; as shown in FIG. 10, the diagnostician modifies the AI data based on the DICOM image analysis of the patient.

The diagnostician reads the images, first checks through the list of lesions, selects certain lesions deemed reasonable, sends them to the structured report, and discards other lesion items found by the AI.

Fig. 11 is a schematic diagram illustrating an AI data interface returned by different AI models for the same inspection purpose in the system for automatically recommending an AI scheme based on the performance analysis of the structured report according to the second embodiment of the present invention; as shown in fig. 11, the physician clicks "send to report" in the interface after selecting a reasonable lesion.

Second, the doctor may correct the measurements output by the AI model, delete and supplement the key images, which is recorded by the structured report log system.

Third, the diagnostician completes the diagnostic analysis using the structured report, and composes a diagnostic decision. This diagnostic conclusion was also structured and labeled using RADLEX coding. For example, a decision "Lung-RADS is ranked 3" is given in the business of LUNG nodules.

In the case of lung nodules, after the report is complete, the system obtains the following records: examination goals, a list of lesions returned by the AI, a list of lesions that the physician confirms sent to the visual structured report, measurements, diagnostician-adjusted measurements, key images, diagnostician-adjusted key images, and a final diagnosis. Other AIs may be integrated with the structured report in a similar order, but the subsequent classification analysis logic may be changed, again without limitation.

The diagnostician may also review the returned results for different AI models, again taking the AI of the lung nodule as an example. If the patient is a health screener, then the assessment method for AI is the sequential rationality of the lesion list. That is, whether the order of the suspected lesions listed in the front is correct. Thus, the diagnostician can ensure that the lesions are analyzed for diagnostic value according to the AI submission sequence and that subsequent low-risk suspected lesion queues are discarded in time. If the patient is a febrile, coughing patient, the AI concern for the lung nodule is how much of the list of lesions it finds is selected by the physician to send into the report. Too many false positives are not a good output.

By analyzing the logic, when the AI model is loaded, the result loading of a certain AI model can be selected by default, and a plurality of AI results can be informed, and the performance differences of the AI results under different inspection purposes can be obtained.

And the recommending module 30 is connected with the processing module 20 and is used for determining the AI model with the highest coincidence rate as the default AI scheme under the examination purpose, and automatically recommending the default AI scheme to the doctor when the doctor prepares to write the image structured report.

Fig. 12 is an interface diagram illustrating automatic recommendation of a default AI scenario when a doctor opens a report of a certain patient in the system for automatically recommending an AI scenario based on performance analysis of a structured report according to the second embodiment of the present invention; as shown in fig. 12, if the doctor has a high match rate using XX lung nodule during the lung cancer screening examination, the XX lung nodule AI model is loaded by default.

The embodiment of the invention is provided with a recording module, a processing module and a recommending module, can dynamically record the modification log of AI data output by a doctor to the AI model, and outputs the coincidence rate of the AI data output by each AI model based on the modification log; determining the AI model with the highest coincidence rate as a default AI scheme under the examination purpose, and automatically recommending the default AI scheme to the doctor when the doctor prepares to write an image structured report; according to the modification of the filling content of the structured report by the imaging doctor and the analysis of the purpose of patient examination, whether the AI model is suitable for the diagnosis of the case or not can be objectively judged, and the AI model with the highest default quality can be automatically pushed for use.

EXAMPLE III

Fig. 13 is a schematic diagram of a system for automatically recommending an AI scheme based on the performance analysis of the structured report according to a third embodiment of the present invention; as shown in fig. 13, the processing module 20 further includes: a determination unit 202, a calculation unit 204, and an output unit 206, wherein,

a determining unit 202, connected to the output unit 206, for determining the probability of false positive lesions based on the lesion list output by the AI model and the lesion list confirmed by the doctor and sent to the image structured report;

a calculating unit 204 connected to the output unit 206 for determining a modification rate of the measurement value and the key image based on the measurement value, the modified measurement value, the key image, the adjusted key image, and the diagnosis conclusion;

an output unit 206, respectively connected to the determining unit 202 and the calculating unit 204, is used for outputting the coincidence rate of the AI data output by each AI model according to the probability and/or modification rate of false positive lesions.

Example four

Fig. 14 is a schematic diagram of a system for automatically recommending an AI scheme based on the performance analysis of the structured report according to a fourth embodiment of the present invention; as shown in fig. 14, the recommending module 30 further includes a prompting unit 302 for prompting the doctor about the AI data output by all other AI models for the examination purpose and the corresponding coincidence rate while recommending the default AI scheme to the doctor.

The prompting unit in the embodiment of the invention can recommend the default AI scheme to the doctor, and simultaneously prompt the doctor with the AI data output by all other AI models under the examination purpose and the corresponding coincidence rate, so that the doctor can know the coincidence rate of the AI data output by each AI model under the same examination purpose, and the comparison between each AI model is convenient.

EXAMPLE five

Fig. 15 is a schematic diagram of a system for automatically recommending an AI scheme based on the performance analysis of the structured report according to the fifth embodiment of the present invention; as shown in fig. 15, when the doctor uses the default AI protocol to draw a diagnosis, the system further includes: a calculation module 40 and a comparison module 50, wherein,

a calculating module 40, connected to the recording module 10, for calculating a coincidence rate of AI data output by the AI model corresponding to the default AI scheme, which is defined as a default AI scheme coincidence rate;

and the comparison module 50 is respectively connected with the calculation module 40 and the processing module 20, and is used for automatically prompting the doctor when the default AI scheme coincidence rate is lower than the coincidence rate of the AI data output by any AI model under the examination purpose.

When the default AI scheme coincidence rate is lower than the coincidence rate of the AI data output by any AI model under the examination purpose, automatically prompting the doctor. If the coincidence rate of the AI data output by one AI model is higher than the coincidence rate of the default AI scheme, the doctor is prompted, and the diagnostician can judge the accuracy of the measured value/key image when the default AI scheme is applied in the scene, so that the doctor can check the result or introduce a more reasonable AI model to rewrite the report.

The embodiment of the invention is provided with the calculation module and the comparison module, when a doctor uses a default AI scheme to obtain a diagnosis conclusion, the coincidence rate of AI data output by the AI model corresponding to the default AI scheme can be calculated, and when the coincidence rate of the default AI scheme is lower than that of the AI data output by any AI model under the examination purpose, the coincidence rate is automatically prompted to the doctor so that the doctor can examine the default AI scheme, even the report can be abandoned, and the AI data output by another more reasonable AI model can be directly called to rewrite the report, so that the improper AI model can be eliminated, the AI model with high quality can be revealed, the selection judgment cost of a department is reduced, and the report writing efficiency and quality of the doctor are improved.

EXAMPLE six

Fig. 16 is a schematic structural diagram of a system for automatically recommending an AI scheme based on performance analysis of a structured report according to a sixth embodiment of the present invention, and as shown in fig. 16, the processing module 20 further includes a statistics unit 208 connected to the output unit 206, for counting the number of times each AI model is used in a preset threshold, and outputting the compliance rate of the AI data output by each AI model according to a preset rule according to the probability and/or modification rate of false positive lesions and the number of times the AI model is used.

For example, the number of completed image structured reports for a pulmonary nodule examination is set to 100, the examination purpose is a pulmonary nodule examination, 3 AI models corresponding to the examination purpose are used, 50 of the AI models are an a1 model, 30 of the AI models are an a2 model, 20 of the AI models are an A3 model, the probability of false positive lesions of the a1 model is the average probability of the 50, and the modification rate is also the average modification rate of the 50, and finally, according to the average probability, the average modification rate and the number of uses, the processing mode of the coincidence rate of the AI model, the coincidence rate of the a2 model and the coincidence rate of the A3 model with the a1 model is output according to a preset rule.

The preset rule can be set according to actual conditions, for example, when the number of times of use reaches a certain number, the coincidence rate is divided into more points, when the number of times of use is less than the certain number, the coincidence rate is divided into less points, and the like, and the preset rule is not limited. This approach circumvents the situation where a certain AI model is used only a few times, but the match rate is also the highest, and is therefore determined to be the default AI scenario. The factor of using times is added, and the coincidence rate is more accurate.

The statistical unit in the embodiment of the invention can count the used times of each AI model in the preset threshold, based on the modification log and in combination with the used times of the AI model, output the coincidence rate of the AI data output by each AI model according to the preset rule, and improve the accuracy of the coincidence rate, so that the recommended AI scheme is more reasonable and has stronger applicability.

From the above description, it can be seen that the above-described embodiments of the present invention achieve the following technical effects: the invention can dynamically record the modification log of the AI data output by the doctor to the AI model, and based on the modification log, the coincidence rate of the AI data output by each AI model is output; determining the AI model with the highest coincidence rate as a default AI scheme under the examination purpose, and automatically recommending the default AI scheme to the doctor when the doctor prepares to write an image structured report; according to the modification of the filling content of the structured report by the imaging doctor and the analysis of the purpose of patient examination, whether the AI model is suitable for the diagnosis of the case or not can be objectively judged, and the AI model with the highest default quality can be automatically pushed for use; the invention can also recommend the default AI scheme to the doctor, and simultaneously prompt the AI data output by all other AI models under the examination purpose and the corresponding coincidence rate to the doctor, so that the doctor can know the coincidence rate of the AI data output by each AI model under the same examination purpose, and the comparison between each AI model is convenient; the method can also calculate the coincidence rate of the AI data output by the AI model corresponding to the default AI scheme after the doctor uses the default AI scheme to obtain a diagnosis conclusion, when the coincidence rate of the default AI scheme is lower than the coincidence rate of the AI data output by any AI model under the examination purpose, the method automatically prompts the doctor so as to facilitate the doctor to examine again, even can give up the report, directly calls another more reasonable AI data output by the AI model to rewrite the report, can eliminate the improper AI model, emerges the AI model with high quality, reduces the selection judgment cost of departments, and improves the efficiency and quality of the doctor in writing the report; the method can also count the using times of each AI model in the preset threshold, output the coincidence rate of the AI data output by each AI model according to the preset rule based on the modification log and the using times of the AI models, and improve the accuracy of the coincidence rate, so that the recommended AI scheme is more reasonable and has stronger applicability.

It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may alternatively be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, or fabricated separately as individual integrated circuit modules, or fabricated as a single integrated circuit module from multiple modules or steps. Thus, the present invention is not limited to any specific combination of hardware and software.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.