1. A method of constructing a fusion model, the method comprising:

constructing a static knowledge map according to remote sensing classification knowledge of a target crop group;

constructing a dynamic knowledge graph according to the static knowledge graph of each time point;

and fusing the dynamic knowledge map and the deep learning model to obtain a fusion model, wherein the fusion model is used for classifying the crop types of the target crop group.

2. The method of claim 1, wherein constructing a static knowledge-graph from remote sensing classification knowledge of a target crop population comprises:

acquiring the crop type of the target crop group in the remote sensing classification knowledge;

calculating to obtain the classification similarity between any two crop types in the crop types;

obtaining a relation subgraph corresponding to each crop type according to the classification similarity;

and constructing a static knowledge graph according to the plurality of relation subgraphs.

3. The method of claim 2, wherein the calculating a categorical similarity between any two of the crop types comprises:

calculating classification similarity between any two of the crop types based on a first expression, the first expression comprising:

wherein, assocB (c)_i，c_j) Representing the ith type and the jth type in the crop typesDegree of classification similarity of (C)_ijAnd the probability of dividing the ith type into the jth type is represented, and N represents the type number of the crop type.

4. The method of claim 1, wherein constructing a dynamic knowledge graph from the static knowledge graphs at the respective time points comprises:

determining corresponding time data according to the remote sensing classification knowledge;

calculating the time sequence similarity of the normalized vegetation indexes in each time point according to the time data;

and adding the time sequence similarity to the corresponding static knowledge graph to update the static knowledge graph at each time point to obtain a dynamic knowledge graph.

5. The method of claim 1, wherein fusing the dynamic knowledge-graph with a deep learning model to obtain a fused model comprises:

importing the dynamic knowledge graph in a deep learning model;

and training the deep learning model according to the knowledge graph so as to fuse the dynamic knowledge graph and the deep learning model to obtain a fusion model.

6. The method of claim 1, wherein prior to constructing the static knowledge-map from the remote sensing classification knowledge of the target crop population, the method further comprises:

receiving image data corresponding to a target crop group;

and combining the image data with the domain knowledge of the target crop group, and taking a combined result as remote sensing classification knowledge.

7. A method for using a fusion model, the method comprising:

the fusion model obtained in the fusion model construction method according to any one of claims 1 to 6, performing data aggregation to obtain aggregated information;

and outputting data according to the aggregation information, and determining a classification result of the crop types in the target crop group according to an output result.

8. The method of claim 7, wherein after determining the classification result of the crop type in the target crop group according to the output result, the method further comprises:

and drawing according to the classification result to obtain drawing data.

9. A fusion model construction apparatus, characterized in that the apparatus comprises:

the static module is used for constructing a static knowledge map according to the remote sensing classification knowledge of the target crop group;

the dynamic module is used for constructing a dynamic knowledge graph according to the static knowledge graph at each time point;

and the fusion module is used for fusing the dynamic knowledge map and the deep learning model to obtain a fusion model, wherein the fusion model is used for classifying the crop types of the target crop group.

10. A fusion model using apparatus, the apparatus comprising:

the aggregation module is used for carrying out data aggregation on the fusion model obtained in the fusion model construction method according to any one of claims 1 to 6 to obtain aggregation information;

and the output module is used for outputting data according to the aggregation information and determining the classification result of the crop types in the target crop group according to the output result.

11. An electronic device comprising a memory having stored therein program instructions and a processor that, when executed, performs the steps of the method of any of claims 1-8.

Background

With the rapid development of satellite technology, various methods for classifying and identifying crops by using remote sensing images appear, and the implementation of the traditional crop remote sensing classification method requires that high-resolution remote sensing images are adopted and a deep learning model is used for performing remote sensing classification on the types of crops. With the great development of the acquisition, storage and analysis technology of remote sensing data, the optical resolution and revisit frequency are greatly increased, the acquisition of time series remote sensing data with higher resolution becomes possible, and the research of applying a deep learning algorithm to carry out multi-temporal image analysis has appeared, but the research mostly treats the multi-temporal remote sensing data as independent multivariable data.

Therefore, in the prior art, when a deep learning model is used for remote sensing classification of crops, only information contained in an image is generally learned, and due to the fact that the deep learning lacks basic and important knowledge cognition and reasoning capabilities, the accuracy of classification results is low when the crops are classified and identified.

Disclosure of Invention

In view of this, an embodiment of the present application provides a fusion model construction method, a fusion model using device, and an electronic device, so as to solve the problem in the prior art that the accuracy is low when classifying crop types.

In a first aspect, an embodiment of the present application provides a fusion model building method, where the method includes:

constructing a static knowledge map according to remote sensing classification knowledge of a target crop group;

constructing a dynamic knowledge graph according to the static knowledge graph of each time point;

and fusing the dynamic knowledge map and the deep learning model to obtain a fusion model, wherein the fusion model is used for classifying the crop types of the target crop group.

In the implementation process, remote sensing classification knowledge of a target crop group for classifying the crop types is carried out according to needs, a static knowledge map corresponding to each time point is established, and a dynamic knowledge map is obtained by combining the static knowledge maps of the time points. The dynamic knowledge map and the deep learning model are fused to obtain the fusion model for classifying the crops of the target crop group, and the classification accuracy of the fusion model is effectively improved by combining the knowledge map with the time sequence on the basis of the deep learning model, so that the fusion model has cognitive and reasoning capabilities, and support is provided for the development of intelligent agriculture.

Optionally, the constructing a static knowledge map according to the remote sensing classification knowledge of the target crop group includes:

acquiring the crop type of the target crop group in the remote sensing classification knowledge;

calculating to obtain the classification similarity between any two crop types in the crop types;

obtaining a relation subgraph corresponding to each crop type according to the classification similarity;

and constructing a static knowledge graph according to the plurality of relation subgraphs.

In the implementation process, the relation subgraph of the classification similarity of each crop type in the target crop group is obtained through remote sensing classification knowledge, and the static knowledge map of the whole target crop group can be obtained according to a plurality of relation subgraphs. By acquiring the accuracy of the classification similarity calculation between any two crop types, the accuracy of each relation subgraph is increased, so that the accuracy of the static knowledge graph is increased, and the method is suitable for various conditions with different crop types.

Optionally, the calculating to obtain the classification similarity between any two of the crop types includes:

calculating classification similarity between any two of the crop types based on a first expression, the first expression comprising:

wherein, assocB (c)_i，c_j) Representing the classification similarity between the ith type and the jth type in the crop types, C_ijAnd the probability of dividing the ith type into the jth type is represented, and N represents the type number of the crop type.

In the implementation process, the classification similarity between any two crop types in the target crop group is calculated through the first expression, the expression and quantification of the entity relationship of the crop types in the target crop group can be carried out, and the pertinence and the effectiveness of the classification similarity are effectively improved.

Optionally, the constructing a dynamic knowledge graph according to the static knowledge graph at each time point includes:

determining corresponding time data according to the remote sensing classification knowledge;

calculating the time sequence similarity of the normalized vegetation indexes in each time point according to the time data;

and adding the time sequence similarity to the corresponding static knowledge graph to update the static knowledge graph at each time point to obtain a dynamic knowledge graph.

In the implementation process, the time sequence similarity in the time data of each time point is calculated, the corresponding time sequence similarity is added into the corresponding static knowledge map for updating, the static knowledge map can be dynamically changed, and the dynamic knowledge map corresponding to the remote sensing classification knowledge is obtained. The static knowledge graph changes, recurses or evolves along with the time, the knowledge graph is continuously updated, the evolution of the knowledge graph is carried out, the dynamism of the knowledge graph corresponding to the time is realized, and the precision, the accuracy and the real-time performance of the dynamic knowledge graph are effectively improved.

Optionally, the fusing the dynamic knowledge graph with the deep learning model to obtain a fused model includes:

importing the dynamic knowledge graph in a deep learning model;

and training the deep learning model according to the knowledge graph so as to fuse the dynamic knowledge graph and the deep learning model to obtain a fusion model.

In the implementation process, the dynamic knowledge graph can be substituted for training on the basis of the deep learning model through the combination of the deep learning model and the dynamic knowledge graph, so that the deep learning model and the dynamic knowledge graph are fused to obtain a fusion model. The precision and the accuracy of the deep learning model for classifying the crop types are effectively improved on the basis of remote sensing classification knowledge.

Optionally, before the constructing the static knowledge map according to the remote sensing classification knowledge of the target crop group, the method further includes:

receiving image data corresponding to a target crop group;

and combining the image data with the domain knowledge of the target crop group, and taking a combined result as remote sensing classification knowledge.

In the implementation process, on the basis of the image data corresponding to the scene of the target crop group, the data combination is carried out by combining the domain knowledge corresponding to the target crop group, so that the remote sensing classification knowledge corresponding to the target crop group can be obtained, and accurate data support is provided for the subsequent construction of the knowledge map according to the remote sensing classification knowledge.

In a second aspect, an embodiment of the present application further provides a fusion model using method, where the method includes:

performing data aggregation according to the fusion model obtained in any one of the fusion model construction methods to obtain aggregation information;

and outputting data according to the aggregation information, and determining a classification result of the crop types in the target crop group according to an output result.

In the implementation process, the fusion model constructed in the fusion model construction method is aggregated and output, so that the crop types of the target crop group can be accurately classified to obtain a corresponding classification result, and the fusion model can be correspondingly used in agriculture.

Optionally, after determining the classification result of the crop type in the target crop group according to the output result, the method further includes:

and drawing according to the classification result to obtain drawing data.

In the implementation process, the classification result is charted, so that the population information and the classification result of the target crop group can be correspondingly displayed, the population information and the classification result of the target crop group can be directly checked and analyzed by a user, the use experience of the fusion model is improved, and various user requirements are met.

In a third aspect, an embodiment of the present application further provides a fusion model building apparatus, where the apparatus includes:

the static module is used for constructing a static knowledge map according to the remote sensing classification knowledge of the target crop group;

the dynamic module is used for constructing a dynamic knowledge graph according to the static knowledge graph at each time point;

and the fusion module is used for fusing the dynamic knowledge map and the deep learning model to obtain a fusion model, wherein the fusion model is used for classifying the crop types of the target crop group.

In the implementation process, the static knowledge map is constructed through the static module according to the remote sensing classification knowledge of the target crop group, the dynamic knowledge map is constructed through the dynamic module based on the time information of the static knowledge map, and the dynamic knowledge map and the deep learning model are fused through the fusion module to obtain the fusion model. On the basis of a deep learning model, the knowledge graph with the time sequence is combined, so that the classification accuracy of the fusion model is effectively improved, the fusion model has cognition and reasoning capabilities, and support is provided for the development of intelligent agriculture.

In a fourth aspect, an embodiment of the present application further provides a fusion model using apparatus, where the apparatus includes:

the aggregation module is used for carrying out data aggregation according to the fusion model obtained in any one of the fusion model construction methods to obtain aggregation information;

and the output module is used for outputting data according to the aggregation information and determining the classification result of the crop types in the target crop group according to the output result.

In the implementation process, the aggregation module performs data aggregation on the fusion model to obtain aggregation information, and the output module outputs the aggregation information to obtain a classification result. The crop types of the target crop groups can be accurately classified to obtain corresponding classification results, so that the fusion model can be correspondingly used in agriculture.

In a fifth aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory and a processor, where the memory stores program instructions, and when the processor reads and executes the program instructions, the electronic device executes steps in any implementation manner of the fusion model building method and the fusion model using method described above.

In summary, the application provides a fusion model construction method, a fusion model using device and electronic equipment, wherein the construction of the knowledge graph is carried out according to the remote sensing classification knowledge of the crop groups classified by the crop types, and the accuracy of classifying the crop types by the fusion model can be improved by carrying out fusion training on the knowledge graph and the deep learning model, so that support is provided for the development of intelligent agriculture.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.

Fig. 1 is a schematic flow chart of a fusion model construction method provided in an embodiment of the present application;

fig. 2 is a detailed flowchart of step S1 according to an embodiment of the present disclosure;

fig. 3 is a detailed flowchart of step S2 according to an embodiment of the present disclosure;

fig. 4 is a schematic flowchart of a method for using a fusion model according to an embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of a fusion model building apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a fusion model using apparatus according to an embodiment of the present application.

Icon: 100-a fusion model construction device; 110-static module; 120-a dynamic module; 130-a fusion module; 200-fusion model using device; 210-an aggregation module; 220-output module.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without any creative effort belong to the protection scope of the embodiments of the present application.

The embodiment of the application provides a fusion model construction method and a fusion model using method, which are applied to a server, wherein the server can be an electronic device with a logic calculation function, such as a Personal Computer (PC), a tablet Personal Computer (PC), a smart phone, a Personal Digital Assistant (PDA) and the like. The method can construct the corresponding knowledge map according to the remote sensing classification knowledge of the target crop group, and fuse the knowledge map and the depth model so as to classify the crop types of the target crop group according to the fusion model and improve the accuracy and precision of classification.

Referring to fig. 1, fig. 1 is a schematic flow chart of a fusion model construction method according to an embodiment of the present application, including the following steps:

and step S1, constructing a static knowledge map according to the remote sensing classification knowledge of the target crop group.

The method comprises the steps of constructing a static Knowledge Graph (KG) according to remote sensing classification Knowledge, wherein the KG refers to a multi-relation Graph formed by entities and relations, and by applying theories and methods of subjects such as mathematics, graphics, information visualization technology, information science and the like and methods such as metrology introduction analysis, co-occurrence analysis and the like, and utilizing Knowledge Graph ideas in an image classification task, the relevance between categories can be regarded as one relation in the Knowledge Graph. The application range is expanded from semantic analysis to information extraction, a series of different graphs of the relation between the knowledge development process and the structure are displayed, knowledge resources and carriers thereof are described by using a visualization technology, and knowledge and the mutual relation among the knowledge resources, the carriers, the knowledge resources, the carriers, the knowledge resources, the information resources.

It should be noted that the Remote Sensing classification knowledge of the target crop group may include satellite Remote Sensing Image data (RS) acquired through a satellite and domain knowledge in the agricultural field. The remote sensing classification knowledge can analyze the response of different crops to remote sensing signals and acquire data such as spectra, textures, vegetation indexes and change characteristics of the different crops.

Optionally, before step S1, the method further comprises steps Sa1-Sa 2:

in step Sa1, image data corresponding to the target crop group is received.

The image data includes image characteristics of the target crop group, and the image data may be picture information such as films or photos recording electromagnetic waves of various ground features, such as satellite photos or aerial photos.

For example, the image data may be multi-temporal Sentinel-2 (Sentinel No. 2) satellite remote sensing image data, or may be image data of other satellites.

And step Sa2, combining the image data and the domain knowledge of the target crop group, and taking the combined result as remote sensing classification knowledge.

Alternatively, the domain knowledge may be a priori knowledge of the agricultural domain, such as data derived from agricultural experts, literature in text form, web pages in web structured data; the image features can comprise different crop image spectral feature observation tests and other classification prior knowledge, and the observation tests comprise unmanned aerial vehicle observation and near-earth camera observation. Illustratively, the manner in which domain knowledge is obtained may be the receipt of agricultural experts or literature in text form.

It is worth explaining that the remote sensing classification knowledge corresponding to the target crop group can be obtained by combining the field knowledge corresponding to the target crop group on the basis of the image data corresponding to the scene of the target crop group, and accurate data support is provided for the follow-up construction of the knowledge graph according to the remote sensing classification knowledge.

After the execution of step S1, the process proceeds to step S2.

And step S2, constructing a dynamic knowledge graph according to the static knowledge graph of each time point.

The remote sensing classification knowledge comprises time data, and different time points have corresponding static knowledge maps, so that the corresponding time data is added into each static knowledge map, and the static knowledge maps are updated to construct dynamic knowledge maps.

It is worth to be noted that the model frameworks of the static knowledge maps at each time point are the same, and the static knowledge maps are correspondingly filled according to the change of the target crop group due to the change of time, so as to obtain the static knowledge maps corresponding to each time point.

Illustratively, temporal knowledge evolution in a dynamic knowledge graph expresses that knowledge occurs, recurses or evolves over time through a model of the dynamic knowledge graph. For example, the growth state of a certain crop in 3,4 and 5 months can be included in the dynamic knowledge graph, the dynamic knowledge graph changes correspondingly according to the change of time along with the growth of the crop, and the accuracy and the real-time performance of the knowledge graph are increased according to the change of the time dimension.

After the execution of step S2, the process proceeds to step S3.

And step S3, fusing the dynamic knowledge map and the deep learning model to obtain a fusion model.

The fusion model is used for classifying the crop types of the target crop group, so that the accuracy, the precision and the calculation efficiency of crop classification are effectively improved, Deep Learning (DL) is a research direction in the field of Machine Learning (ML), Deep Learning is an internal rule and a representation level of Learning sample data, and information obtained in the Learning process is data such as characters, images and sounds. Deep learning enables recognition of data such as text, images, and sounds.

Optionally, the step S3 may further include steps S31-S32:

and step S31, importing the dynamic knowledge graph into a deep learning model.

The Deep learning model may be a Deep Belief Network (DBN) based on a Restricted Boltzmann Machine (RBM) or a Stacked Auto Encoder (SAE) based on an Auto Encoder (AE); a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), and the like.

And step S32, training the deep learning model according to the knowledge graph so as to fuse the dynamic knowledge graph and the deep learning model to obtain a fusion model.

Optionally, taking CNN as an example, when performing fusion training, image data in the remote sensing classification knowledge may be input into a CNN convolutional neural network, a COVN convolutional layer is input to perform convolution, a convolutional layer result is input into a full-link layer FLC (animation file format), all features are connected, in a model framework, a graph volume layer is added behind the full-link layer, and the graph volume includes class similarity information in a dynamic knowledge graph, so as to obtain a final fusion model.

It is worth to be noted that through the combination of the deep learning model and the dynamic knowledge map, the dynamic knowledge map can be substituted into the deep learning model for training, so that the deep learning model and the dynamic knowledge map are fused to obtain a fusion model. The precision and the accuracy rate of classifying the crop types according to the deep learning model on the basis of remote sensing classification knowledge are effectively improved.

In the embodiment shown in fig. 1, the dynamic knowledge map is fused with the deep learning model to obtain a fusion model for classifying the crops of the target crop group, and the classification accuracy of the fusion model can be effectively improved by combining the knowledge map with the time sequence on the basis of the deep learning model, so that the fusion model has the cognitive and reasoning capabilities, and supports the development of intelligent agriculture.

Referring to fig. 2, fig. 2 is a detailed flowchart of a step S1 according to an embodiment of the present disclosure, which includes the following steps:

and step S11, acquiring the crop type of the target crop group in the remote sensing classification knowledge.

The target crop group may include a plurality of different types of crop, such as wheat, rice, soybean, etc.

It is worth explaining that concept and entity extraction can be carried out on remote sensing classification knowledge, entity relation expression and quantification are carried out to obtain entity relations among crop types of the target crop group, and classification similarity among the crop types in the target crop group is calculated according to the entity relations. In the concept and entity extraction, the remote sensing classification knowledge can be modeled into a triple firstly, and the modeling mode is as follows:

KG＝{(s，r，o)|s，o∈E，r∈R}；

wherein, KG represents a static knowledge graph, E represents a set of entities, R represents a possible set of relationships in an ontology, s represents a subject, R represents a relationship, and o represents an object. In the construction of the knowledge graph, a main body s is a basic unit for the graph construction, and the reasonable construction of the relationship graph can enable the related information of the entity to be clearer and is beneficial to the classification of the crop types by a subsequent fusion model. And selecting all possible crop types in the remote sensing image as entity nodes, such as winter wheat, summer corn, spring corn, sorghum, soybean, rice, cotton and the like. Each crop can be accompanied by the representation of the spectral characteristics or the vegetation index characteristics of the remote sensing images, and the representation can be used as the attribute value of the crop entity to fill the attribute of the crop entity.

After the execution of step S11, the process proceeds to step S12.

And step S12, calculating the classification similarity between any two crop types in the crop types.

The prior knowledge of the image category error ratio can be adopted, for example, classification similarity is expressed by using a classification confusion matrix or an association matrix error and the like, and the cross error between the category pairs is calculated by the association matrix, so that the similarity between the categories is implied.

Optionally, the classification similarity between any two of the crop types may be calculated based on a first expression, the first expression including:

wherein, assocB (C)_i，c_j) Representing the classification similarity between the ith type and the jth type in the crop types, C_ijAnd the probability of dividing the ith type into the jth type is represented, and N represents the type number of the crop type.

After the execution of step S12, the process proceeds to step S13.

And step S13, acquiring a relation subgraph corresponding to each crop type according to the classification similarity.

The method comprises the steps of establishing entity relationship graphs of single crop types according to the classification similarity of association between entity relationships in a target crop group, and combining the entity relationship graphs of the single types into relationship subgraphs, wherein each relationship subgraph is a main class.

After the execution of step S13, the process proceeds to step S14.

And step S14, constructing a static knowledge graph according to the plurality of relation subgraphs.

And combining all the relation subgraphs to construct a complete static knowledge graph.

It is worth noting that if data of one class is often misclassified as another class, the two classes are considered to have a high similarity.

In the embodiment shown in fig. 2, the accuracy of the calculation of the classification similarity between any two crop types is obtained according to the entity relationship expression and the quantization by performing concept and entity extraction on the remote sensing classification knowledge, so that the accuracy of each relationship subgraph is increased, the accuracy of the static knowledge graph is increased, and the method is suitable for various conditions with different crop types.

Referring to fig. 3, fig. 3 is a detailed flowchart of a step S2 according to an embodiment of the present disclosure, which includes the following steps:

and step S21, determining corresponding time data according to the remote sensing classification knowledge.

Optionally, the triplet of the original static knowledge graph may be modified to obtain a quadruplet, where the quadruplet is as follows:

TKG＝{(s,r,o,t)|s,o∈E,r∈R,t∈T}；

wherein TKG represents a dynamic knowledge graph, T is time, and T is a set of all timestamps.

It is worth explaining that in the detection of the deep learning algorithm in the agricultural field of land utilization, coverage classification, scene classification and object detection, the multi-temporal remote sensing classification knowledge is used as independent multivariable data to be processed, the time dimension is ignored, the time and spectrum dimensions of the data are not fully utilized, and the classification effect is poor. The acquired time data of the remote sensing classification knowledge can capture the time mode knowledge of different expressions of different classes of objects, and basic and important cognitive and reasoning capabilities are added to the deep learning algorithm.

After the execution of step S21, the process proceeds to step S22.

And step S22, calculating the time sequence similarity of the normalized vegetation indexes in each time point according to the time data.

According to the Time knowledge evolution expression in the Dynamic knowledge map, on the basis of a model of the static knowledge map, because knowledge occurs, recurses or evolves along with the Time, NDVI (Normalized Difference Vegetation Index) morphological Time sequence similarity can be calculated through a DTW (Dynamic Time Warping) algorithm according to the reality of Time sequence remote sensing classification.

Optionally, the DTW distance, i.e. the dynamic time warping distance, is one of the main methods for time-series distance measurement based on dynamic programming, and its principle is as follows:

two time sequences S are set¹(t)＝{s₁ ¹,s₂ ¹,…,s_m ¹}，S²(t)＝{s¹²,s₂ ²,…,s_n ²And the lengths of the n-type metal oxide semiconductor are m and n respectively. The m x n matrixes A are constructed by sorting according to the time positions of the m x n matrixes A_m×_n. Matrix A_m×_nIs a distance between each element ofIn matrix A_m×_nIn this specification, a set of adjacent matrix elements is referred to as a curved path, where W is { W ═ W₁,w₂,…,w_kW is the k-th element of W_k＝(a_ij)_k. The two time series are formed into a matrix, the values in the matrix being the distances between the elements (the square root of the square of the difference between the corresponding elements in the series). Therefore, the DTW algorithm can be summarized as using the idea of dynamic programming to find an optimal path with minimum bending cost, that is:

wherein i is 2,3,4, …, m, j is 2,3, …, n, A_m×nMinimum accumulated value D (m, n) of the medium curved path. The matching of time sequence distance is carried out through the DTW distance, and the point-to-point alignment mode of the time sequence can be staggered, so that the delay of the growth conditions of the similar crops caused by the phenological or farming modes is solved, and the robustness is improved. And learning the DTW distance as similarity prior knowledge among the crop classes. Therefore, with the addition of remote sensing classification knowledge, the NDVI time sequence construction is updated, the model is continuously updated according to the time sequence similarity change of the constructed NDVI, and the knowledge evolution is realized.

It should be noted that software such as enii (The Environment for visualization Images, remote sensing image processing software) can be used to calculate and process The NDVI dynamic time warping distance DTW at each time point based on The python programming Environment and The above formula, so as to obtain The time sequence similarity of The NDVI morphology.

After the execution of step S22, the process proceeds to step S23.

Step S23, adding the time sequence similarity to the corresponding static knowledge graph to update the static knowledge graph at each time point to obtain a dynamic knowledge graph.

And adding the time sequence similarity of the NDVI forms into the corresponding static knowledge graph, namely adding the structural temporal dimension information into the static knowledge graph to complete the construction of the dynamic knowledge graph.

In the embodiment shown in fig. 3, the time sequence similarity in the time data of each time point is calculated, and the corresponding time sequence similarity is added to the corresponding static knowledge map for updating, so that the static knowledge map can be dynamically changed to obtain a dynamic knowledge map corresponding to the remote sensing classification knowledge. The static knowledge graph changes, recurses or evolves along with the time, the knowledge graph is continuously updated, the evolution of the knowledge graph is carried out, the dynamism of the knowledge graph corresponding to the time is realized, and the precision, the accuracy and the real-time performance of the dynamic knowledge graph are effectively improved.

Referring to fig. 4, fig. 4 is a schematic flow chart of a fusion model using method according to an embodiment of the present application, including the following steps:

and step S4, performing data aggregation according to the fusion model to obtain aggregation information.

The fusion model may be obtained by any one of the fusion model construction methods in the embodiments, the hidden data is expressed through data aggregation, the class embedded information is received, convolution is performed between classes, and neighborhood information is aggregated to obtain corresponding aggregated information.

After the execution of step S4, the process proceeds to step S5.

And step S5, outputting data according to the aggregation information, and determining the classification result of the crop types in the target crop group according to the output result.

Wherein the data output is performed according to the aggregation information, the output result may be new data and class embedding, and the classification result is calculated according to the output result, for example, the new data and class embedding is further used to calculate a class score, and the output result may be input into a Softmax (output layer excitation function) probability classifier to generate a final classification result.

Optionally, the classification result may be displayed in an expression form of an image, a table, or a text, the server may further send the classification result to a corresponding user terminal, for example, a work terminal of a worker such as a detector or a researcher, so that the worker can view and analyze the classification result, and the server may further receive feedback data submitted by the worker according to the classification result, so as to improve accuracy of the classification result and improve user experience.

Optionally, after step S5, the method may further include:

and step S6, drawing according to the classification result to obtain drawing data.

The drawing data can be used for drawing the crop types and the distribution positions of the target crop groups correspondingly so as to be checked and analyzed by a user, and can also be stored in a historical database so as to be used for workers to perform operations such as longitudinal analysis on the crop types of the target crop groups according to time dimension.

Optionally, by plotting the classification result, the population information and the classification result of the target crop group can be correspondingly displayed. The user can directly check and analyze the population information and the classification result of the target crop group, the use experience of the fusion model is improved, and various user requirements are met.

In the embodiment shown in fig. 4, by aggregating and outputting the fusion model constructed in the fusion model construction method, the crop types of the target crop group can be accurately classified to obtain a corresponding classification result, so as to realize corresponding use of the fusion model in agriculture.

Referring to fig. 5, fig. 5 is a schematic structural diagram of a fusion model building apparatus according to an embodiment of the present application, where the fusion model building apparatus 100 includes:

the static module 110 is used for constructing a static knowledge map according to the remote sensing classification knowledge of the target crop group;

a dynamic module 120, configured to construct a dynamic knowledge graph according to the static knowledge graph at each time point;

a fusion module 130, configured to fuse the dynamic knowledge map with a deep learning model to obtain a fusion model, where the fusion model is used to classify the crop types of the target crop group.

The static module 110 further includes a computation submodule and a map submodule;

the calculation submodule is used for acquiring the crop type of the target crop group in the remote sensing classification knowledge;

calculating to obtain the classification similarity between any two crop types in the crop types;

the map submodule is used for acquiring a corresponding relation subgraph of each crop type according to the classification similarity;

and obtaining a static knowledge graph according to the plurality of relation subgraphs.

The calculating submodule is further used for calculating the classification similarity between any two crop types in the crop types based on a first expression, and the first expression comprises:

wherein, assocB (C)_i，c_j) Representing the classification similarity of the ith type and the jth type in the crop types, and C belongs to R^N*NRepresenting a correlation matrix, C_ijAnd the probability of dividing the ith type into the jth type is represented, and N represents the type number of the crop type.

The dynamic module 120 further includes: a timing sub-module and an update sub-module;

the time sequence submodule is used for determining corresponding time data according to the remote sensing classification knowledge;

calculating the time sequence similarity of the normalized vegetation indexes in each time point according to the time data;

and the updating submodule is used for adding the time sequence similarity to the corresponding static knowledge graph so as to update the static knowledge graph at each time point to obtain a dynamic knowledge graph.

The fusion module 130 further includes: importing a submodule and a training submodule;

the import submodule is used for importing the dynamic knowledge graph in a deep learning model;

and the training submodule is used for training the deep learning model according to the knowledge graph so as to fuse the dynamic knowledge graph and the deep learning model to obtain a fusion model.

The fusion model construction apparatus 100 further includes: a combining module;

the combination module is used for receiving image data corresponding to the target crop group;

and combining the image data with the domain knowledge of the target crop group, and taking a combined result as remote sensing classification knowledge.

Because the principle of solving the problem in the fusion model construction apparatus in the embodiment of the present application is similar to that in the embodiment of the fusion model construction method, the implementation of the fusion model construction apparatus in the embodiment of the present application can refer to the description in the embodiment of the fusion model construction method, and repeated details are not repeated.

In the embodiment shown in fig. 5, the fusion model constructed by the modules can be combined with the knowledge graph with the time sequence on the basis of the deep learning model, so that the classification accuracy of the fusion model is effectively improved, the fusion model has the cognitive and reasoning capabilities, and the support is provided for the development of intelligent agriculture.

Referring to fig. 6, fig. 6 is a schematic structural diagram of a fusion model using apparatus 200 according to an embodiment of the present application, including: an aggregation module 210 and an output module 220.

An aggregation module 210, configured to perform data aggregation according to the fusion model obtained in any one of the fusion model construction methods in embodiment 1, to obtain aggregation information;

and the output module 220 is configured to output data according to the aggregation information, and determine a classification result of the crop types in the target crop group according to an output result.

The fusion model using apparatus 200 further includes: a charting module 230;

and the drawing module 230 is used for drawing according to the classification result to obtain drawing data.

Because the principle of solving the problem in the fusion model using apparatus in the embodiment of the present application is similar to that in the embodiment of the fusion model using method, the implementation of the fusion model using apparatus in the embodiment of the present application can refer to the description in the embodiment of the fusion model using method, and repeated details are not repeated.

In the embodiment shown in fig. 6, the fusion model is used by each module, so that the crop types of the target crop group can be accurately classified and displayed, and corresponding classification results and charting data are obtained, so as to realize corresponding use of the fusion model in agriculture.

The embodiment of the present application further provides an electronic device, which includes a memory and a processor, where the memory stores program instructions, and when the processor reads and runs the program instructions, the processor executes the steps in any one of the method for constructing the fusion model and the method for using the fusion model provided in the embodiment.

It should be understood that the electronic device may be a Personal Computer (PC), a tablet PC, a smart phone, a Personal Digital Assistant (PDA), or other electronic device having a logical computing function.

The embodiment of the application also provides a readable storage medium, wherein computer program instructions are stored in the readable storage medium, and the computer program instructions are read by a processor and run to execute the steps in the fusion model building method and the fusion model using method.

In summary, the application provides a fusion model construction method, a fusion model using device and electronic equipment, wherein the construction of the knowledge graph is carried out according to the remote sensing classification knowledge of the crop groups classified by the crop types, and the accuracy of classifying the crop types by the fusion model can be improved by carrying out fusion training on the knowledge graph and the deep learning model, so that support is provided for the development of intelligent agriculture.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. The apparatus embodiments described above are merely illustrative, and for example, the block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices according to various embodiments of the present application. In this regard, each block in the block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams, and combinations of blocks in the block diagrams, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Therefore, the present embodiment further provides a readable storage medium, in which computer program instructions are stored, and when the computer program instructions are read and executed by a processor, the computer program instructions perform the steps of any of the block data storage methods. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a RanDom Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.