1. A method of data processing, the method comprising:

acquiring a data query request, wherein the data query request is used for querying project associated data;

inquiring a graph database based on the data inquiry request to obtain an inquiry result and feed back the inquiry result; the graph database comprises nodes and relationship edges, wherein the nodes represent project information, and the relationship edges represent association data among different projects.

2. The method of claim 1, prior to the obtaining the data query request, further comprising:

constructing a graph database;

the process of constructing the graph database includes:

acquiring project associated data, wherein the project associated data is obtained based on the association relation among all projects;

and loading the project associated data based on a preset loading service, and storing the loaded project associated data to the graph database to complete the construction of the graph database.

3. The method of claim 2, wherein obtaining item association data comprises:

acquiring item description information of each item from a relational database, wherein the item description information at least comprises item attributes and item types;

determining project data with an incidence relation based on the project attributes and the project types;

and determining the project data with the association relationship as project association data.

4. The method of claim 1, wherein querying a graph database based on the data query request, obtaining query results and feeding back comprises:

extracting attribute information of the items to be inquired carried in the data inquiry request;

and reading the item associated data corresponding to the item attribute information to be inquired from the graph database and feeding back the item associated data based on the graph inquiry language.

5. The method according to claim 2, wherein when the project association data is online, the loading the project association data based on a preset loading service and storing the loaded project association data to the graph database includes:

loading project associated data in an online state based on a graph data service;

and storing the loaded project association data in an online state to the graph database.

6. The method according to claim 2, wherein when the project associated data is in an offline state, the loading the project associated data based on a preset loading service and storing the loaded project associated data to the graph database includes:

loading the project associated data in an off-line state based on a preset message service to obtain project associated data in a preset format;

and storing the project association data in the preset format to the graph database.

7. The method of claim 1, further comprising:

and accessing the project associated data loaded in the graph database through preset access service.

8. A data processing apparatus, characterized in that the apparatus comprises:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a data query request which is used for querying project associated data;

and the query unit is used for querying a graph database based on the data query request to obtain a query result and feeding back the query result, wherein the graph database comprises nodes and relationship edges, the nodes represent the item information, and the relationship edges represent the associated data among different items.

9. The apparatus of claim 8, further comprising: a building unit;

the construction unit is used for constructing a graph database;

the construction unit comprises:

the acquisition module is used for acquiring project associated data, and the project associated data is obtained based on the association relation among all projects;

and the loading module is used for loading the project associated data based on preset loading service, storing the loaded project associated data to the graph database and completing the construction of the graph database.

10. The apparatus of claim 8, wherein the obtaining module comprises:

the acquisition submodule is used for acquiring the item description information of each item from the relational database, and the item description information at least comprises item attributes and item types;

the first determining submodule is used for determining the project data with the incidence relation based on the project attributes and the project types;

and the second determining submodule is used for determining the project data with the association relationship as project association data.

Background

A relational database is a database that uses a relational model to organize data, and the relational database stores data in rows and columns, and a series of rows and columns of the relational database are called tables, and a group of tables constitutes the database.

In the prior art, table connection is required when performing association query on multi-layer associated item data of a relational database, and the table connection involves a large amount of Input/Output (IO) operations and memory consumption, thereby reducing the processing speed of performing the association query.

Therefore, the existing relational database is inefficient in inquiring the associated data of the items.

Disclosure of Invention

In view of this, the present application discloses a data processing method and apparatus, which aim to improve efficiency of querying item-related data.

In order to achieve the purpose, the technical scheme is as follows:

a first aspect of the present application discloses a data processing method, including:

acquiring a data query request, wherein the data query request is used for querying project associated data;

inquiring a graph database based on the data inquiry request to obtain an inquiry result and feed back the inquiry result; the graph database comprises nodes and relation edges, wherein the nodes represent project information, and the relation edges represent association data among different projects.

Preferably, before the query request for data, the method further includes:

constructing a graph database;

the process of constructing the graph database includes:

acquiring project association data, wherein the project association data are obtained based on the association relation among all projects;

and loading the project associated data based on a preset loading service, and storing the loaded project associated data to the graph database to complete the construction of the graph database.

Preferably, the acquiring of the project association data includes:

acquiring project description information of each project from a relational database, wherein the project description information at least comprises project attributes and project types;

determining project data with an incidence relation based on the project attributes and the project types;

and determining the project data with the association relationship as project association data.

Preferably, the querying a graph database based on the data query request to obtain and feedback a query result includes:

extracting attribute information of the items to be inquired carried in the data inquiry request;

and reading the item associated data corresponding to the item attribute information to be inquired from the graph database and feeding back the item associated data based on the graph inquiry language.

Preferably, when the project associated data is in an online state, the loading the project associated data based on a preset loading service, and storing the loaded project associated data in the graph database includes:

loading project associated data in an online state based on a graph data service;

and storing the loaded project association data in an online state to the graph database.

Preferably, when the project associated data is in an offline state, the loading the project associated data based on a preset loading service, and storing the loaded project associated data in the graph database includes:

loading the project associated data in an off-line state based on a preset message service to obtain project associated data in a preset format;

and storing the project association data in the preset format to the graph database.

Preferably, the method further comprises the following steps:

and accessing the project associated data loaded in the graph database through preset access service.

A second aspect of the present application discloses a data processing apparatus, the apparatus comprising:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a data query request which is used for querying item related data;

and the query unit is used for querying a graph database based on the data query request to obtain a query result and feeding back the query result, wherein the graph database comprises nodes and relationship edges, the nodes represent the item information, and the relationship edges represent the associated data among different items.

Preferably, the method further comprises the following steps: a building unit;

the construction unit is used for constructing a graph database;

the construction unit comprises:

the acquisition module is used for acquiring project associated data, and the project associated data is obtained based on the association relation among all projects;

and the loading module is used for loading the project associated data based on preset loading service, storing the loaded project associated data to the graph database and completing the construction of the graph database.

Preferably, the obtaining module includes:

the acquisition submodule is used for acquiring the item description information of each item from the relational database, and the item description information at least comprises item attributes and item types;

the first determining submodule is used for determining the project data with the association relation based on the project attributes and the project types;

and the second determining submodule is used for determining the project data with the association relationship as project association data.

According to the technical scheme, the data query request is obtained and used for querying item association data, the graph database is queried based on the data query request, the query result is obtained and fed back, the graph database comprises nodes and relationship edges, the nodes represent item information, and the relationship edges represent association data among different items. According to the scheme, when the graph database is queried through the data query request to obtain the project associated data, the data do not need to be obtained in a table connection mode involving a large number of IO operations and memory consumption, the project associated data corresponding to the data query request in the graph database only need to be queried through the data query request, and the project associated data can be efficiently queried through the graph database, so that reading of other non-relevant data caused by querying the project associated data is prevented, and the efficiency of querying the project associated data is improved. In addition, the graph database provides a visual graph display interface, so that the project association data display is more intuitive.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

FIG. 1 is a block diagram of a data processing system according to an embodiment of the present disclosure;

fig. 2 is a schematic flow chart of a data processing method disclosed in an embodiment of the present application;

FIG. 3 is a flowchart illustrating a process of obtaining project-related data disclosed in an embodiment of the present application;

FIG. 4 is a schematic flow chart illustrating query of a graph database based on a data query request to obtain a query result and feedback, according to an embodiment of the present application;

FIG. 5 is a flowchart illustrating a process of loading and storing project-related data into a graph database based on a predetermined loading service according to an embodiment of the present application;

FIG. 6 is a flowchart illustrating another process of loading and storing project-related data into a graph database based on a predetermined loading service according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

As can be seen from the background art, the conventional relational database is inefficient in querying the item association data.

In order to solve the problem, the embodiment of the application discloses a data processing method and a data processing device, when a data query request is obtained to query a graph database to obtain item associated data, the data is not required to be obtained in a table connection mode involving a large number of IO operations and memory consumption, only the item associated data corresponding to the data query request in the graph database is required to be queried through the data query request, and the graph database can be used for efficiently querying the item associated data, so that reading of other non-related data is prevented from being caused when the item associated data is queried, and the efficiency of querying the item associated data is improved. In addition, the graph database provides a visual graph display interface, so that the project association data can be displayed more intuitively.

The specific implementation is illustrated by the following examples.

The data processing method disclosed in the present application can be applied to a data processing system as shown in fig. 1, and referring to the architecture diagram of the data processing system shown in fig. 1, the data processing system includes a query computation module 11, an operation and maintenance module 12, a data loading module 13, and a graph database core module 14, and the graph database core module 14 includes a graph storage unit 15, a graph processing unit 16, an HA service unit 17, a metadata service unit 18, and a task scheduling unit 19.

The data processing system processes the project associated data as follows:

the graph processing unit 16 acquires item association data, which is obtained based on the association relationship between the items.

The data loading module 13 loads the project-related data based on a preset loading service.

The graph storage unit 15 stores the loaded project association data into the graph database, and completes the construction of the graph database.

The drawing processing unit 16 acquires item description information of each item from the relational database, the item description information including at least an item attribute and an item type.

The drawing processing unit 16 determines the item data in which the association relationship exists based on the item attribute and the item type, and determines the item data in the association relationship as the item association data.

If the project associated data is in the online state, the data loading module 13 performs batch loading/information-loading stream loading on the project associated data in the online state based on the graph data service.

If the project associated data is in the offline state, the data loading module 13 loads the project associated data in the offline state based on the preset message service, so as to obtain the project associated data in the preset format.

The map storage unit 15 stores the item association data in a preset format to the map database.

The data loading module 13 accesses the item associated data loaded in the graph database through a preset access service.

The map storage unit 15 stores the loaded item association data into the map database.

The query computation module 11 extracts the attribute information of the item to be queried carried in the data query request, and reads and feeds back the item association data corresponding to the attribute information of the item to be queried from the graph database based on the graph query language.

The operation and maintenance module 12 monitors the state of the service system, system configuration, installation information, upgrade information, operation information, a visual interface, and the like in real time.

The HA service unit 17 is responsible for processing functions of data fault tolerance, data consistency, uninterrupted service, and the like.

The metadata service unit 18 is used for describing the content, quality, representation, spatial reference, management and other features of the data set, and providing metadata service for the outside.

The graph storage sub-module stores the project association data and other unstructured data (data in the format of csv, xml, json, etc.) into the graph database.

The task scheduling unit 19 schedules the execution of tasks according to the order of events and the priority of the tasks.

In the embodiment of the application, when the graph database is queried by the data query request to obtain the item associated data, the data does not need to be obtained in a table connection mode involving a large number of IO operations and memory consumption, and the item associated data corresponding to the data query request in the graph database only needs to be queried by the data query request. In addition, the graph database provides a visual graph display interface, so that the project association data display is more visual.

Referring to fig. 2, a schematic flow chart of a data processing method disclosed in an embodiment of the present application is shown, where the data processing method is applied to the data processing system in fig. 1 in the above embodiment, and the data processing method mainly includes the following steps:

s201: and constructing a graph database, wherein the graph database comprises nodes and relationship edges, the nodes represent project information, and the relationship edges represent association data among different projects.

The graph database expresses the real world relationship in a graph mode, is direct and natural, and is easy to model; a large amount of data can be efficiently inserted, and the data can be qualified for more than one hundred million levels; relevant data can be efficiently inquired, relevant inquiry is pertinently optimized, such as on a storage model, a data structure, an inquiry algorithm and the like, and the inquiry of local data is prevented from causing the reading of all data; the query language aiming at graph retrieval is provided, and continuous development of correlation analysis services is facilitated; professional analysis algorithms and tools are provided, a data batch import tool is also provided, and a visual graph display interface is provided, so that the analysis result of the data is more visually displayed.

The process of constructing a graph database is as follows:

the method comprises the steps of firstly, acquiring project associated data, wherein the project associated data are obtained based on the association relation among projects, then loading the project associated data based on a preset loading service, storing the loaded project associated data to a graph database, and completing the construction of the graph database.

The process of acquiring the project association data is as follows:

firstly, acquiring item description information of each item from a relational database, wherein the item description information at least comprises an item attribute and an item type, then determining item data with an association relationship based on the item attribute and the item type, and finally determining the item data with the association relationship as item association data.

The relational database may be an oracle database, a mysql database, or the like.

The item description information includes item attributes, item types, item IDs, and the like.

And if the item attribute and the item type corresponding to the item are consistent with the item attribute and the item type corresponding to another item, determining that an association relationship exists between the item and the other item.

For convenience of understanding the association relationship between the items, the following description is given by way of example:

for example, the A project and the B project are determined, the project attribute of the A project comprises a creator system administrator, the project type of the A project comprises a government investment project, the project attribute of the B project comprises a creator system administrator, and the project type of the B project comprises a government investment project.

The pre-set loading service may be a graph data service, a kafka messaging service, etc. The preset loading service is not specifically limited in the present application.

The process of loading the project associated data based on the preset loading service and storing the loaded project associated data into the graph database is as follows:

and when the project associated data is in an online state, loading the project associated data in the online state based on the graph data service, and storing the loaded project associated data in the online state to the graph database.

And when the project associated data are in an offline state, loading the project associated data in the offline state based on the kafka message service to obtain project associated data in a preset format, and storing the project associated data in the preset format to a graph database.

The project associated data in the preset format may be project associated data in a csv format and project associated data in a json format, and the project associated data in the preset format is unstructured data.

Because the different types of item description data are stored in the relational database, and the item description information in the different types of items does not have too much association and organization, the item description information in the different types of items in the relational database is extracted and stored into the graph database, so that the association relationships of the different types of items are organized, and the item association data can be conveniently inquired.

And accessing the project associated data loaded in the graph database through preset access service.

The preset access service may be a REST access service or the like.

S201 is a preprocessing process of S202 to S203.

S202: and acquiring a data query request, wherein the data query request is used for querying the project associated data.

In S202, when a data query request sent by a client is received, item association data in the database is queried based on the data query request.

S203: inquiring a graph database based on the data inquiry request to obtain and feed back an inquiry result; the graph database comprises nodes and relation edges, wherein the nodes represent project information, and the relation edges represent association data among different projects.

Inquiring a graph database based on the data inquiry request, and obtaining and feeding back an inquiry result as follows:

firstly, extracting the attribute information of the item to be queried carried in the data query request, and then reading and feeding back the item associated data corresponding to the attribute information of the item to be queried from the graph database based on the graph query language.

And the query result, namely the item associated data corresponding to the attribute information of the item to be queried, is displayed in a digital network dynamic map mode.

The format of the query result may be json, xml, etc.

The graph database provides a visual graph display interface, namely the query result is displayed through the visual graph display interface, so that the project associated data is displayed more intuitively.

The graph query language may be a Cypher query language, a Gremlin query language, a SPARQL query language, or the like, and the determination of the specific graph query language is set by a technician according to an actual situation, which is not specifically limited in the present application.

The project association data query method and the project association data query device improve project association data query efficiency, achieve association among different types of projects, and association of various information such as archives, peripheral projects and historical projects caused by the projects, automatically associate and display in a digital mesh dynamic map mode, and achieve full association of project maps.

According to the method and the device for querying the project associated data, when the project associated data is queried, the project associated data does not need to be acquired in a table connection mode involving a large number of IO operations and memory consumption, only the loaded project associated data needs to be stored in the graph database, and since the graph database can be used for querying the project associated data efficiently, reading of other non-relevant data is prevented from being caused when the project associated data is queried, and therefore efficiency of querying the project associated data is improved. In addition, the graph database provides a visual graph display interface, so that the project association data display is more intuitive.

Referring to fig. 3, a process involved in acquiring the item-related data in S201 described above mainly includes the following steps:

s301: and acquiring item description information among the items from the relational database, wherein the item description information at least comprises item attributes and item types.

S302: and determining the project data with the association relation based on the project attributes and the project types.

S303: and determining the project data with the association relationship as project association data.

The execution principle of S301 to S303 is consistent with the execution principle of S201, and it can be referred to here, and is not described here again.

In the embodiment of the application, the item description information among all items is acquired from the relational database, the item data with the association relation is determined based on the item description information and the item attributes and the item types, the item data with the association relation is determined as the item association data, and the purpose of obtaining the item association data among different types of items is achieved.

Referring to fig. 4, a process involved in querying a graph database based on a data query request in the above step S203, obtaining a query result and feeding back the query result, mainly includes the following steps:

s401: and extracting the attribute information of the item to be inquired carried in the data inquiry request.

S402: and based on graph query language, reading item associated data corresponding to the attribute information of the item to be queried from the graph database and feeding back the item associated data.

The execution principle of S401-S402 is consistent with the execution principle of S203, and it can be referred to here, and is not described here again.

In the embodiment of the application, the purpose of reading and feeding back the item associated data corresponding to the item attribute information to be queried from the graph database is achieved by extracting the item attribute information to be queried carried in the data query request and based on the graph query language.

Referring to fig. 5, a process involved in S201 described above, when the project associated data is in an online state, loading the project associated data based on a preset loading service, and storing the loaded project associated data in a graph database, mainly includes the following steps:

s501: and loading project associated data in an online state based on the graph data service.

S502: and storing the loaded project association data in the online state to a graph database.

The execution principle of S501-S502 is consistent with the execution principle of S201, and it can be referred to here, and no further description is given here.

In the embodiment of the application, the graph data service is invoked to load the project associated data in the online state, and the loaded project associated data in the online state is stored in the graph database, so that the project associated data in the graph database can be conveniently inquired, and the efficiency of inquiring the project associated data is improved.

Referring to fig. 6, a process involved in S201 described above, when the project associated data is in an offline state, loading the project associated data based on a preset loading service, and storing the loaded project associated data in a graph database mainly includes the following steps:

s601: and loading the project associated data in the off-line state based on the preset message service to obtain the project associated data in the preset format.

S602: and storing the project association data in a preset format to a graph database.

The execution process and the execution principle of S601-S602 are consistent with the execution principle of S202, and may be referred to herein, and are not described again.

In the embodiment of the application, the project associated data in the online state is loaded by calling the graph data service, and the loaded project associated data in the online state is stored in the graph database, so that the project associated data in the graph database can be conveniently inquired, and the efficiency of inquiring the project associated data is improved.

Based on the data processing method disclosed in fig. 2 in the foregoing embodiment, the embodiment of the present application also discloses a data processing apparatus, and as shown in fig. 7, the data processing apparatus mainly includes an obtaining unit 701 and a querying unit 702.

An obtaining unit 701, configured to obtain item association data, where the item association data is obtained based on an association relationship between items.

A query unit 702, configured to query a graph database based on a data query request, obtain a query result, and feedback the query result; the graph database comprises nodes and relationship edges, wherein the nodes represent project information, and the relationship edges represent association data among different projects.

Further, the device also comprises a construction unit.

A construction unit for constructing a graph database.

A building unit comprising: the device comprises an acquisition module and a loading module.

And the acquisition module is used for acquiring the project associated data, and the project associated data is obtained based on the association relation among the projects.

And the loading module is used for loading the project associated data based on the preset loading service, storing the loaded project associated data to the graph database and completing the construction of the graph database.

Further, the obtaining module comprises an obtaining submodule, a first determining submodule and a second determining submodule.

And the acquisition submodule is used for acquiring the item description information of each item from the relational database, and the item description information at least comprises item attributes and item types.

And the first determining submodule is used for determining the project data with the association relation based on the project attributes and the project types.

And the second determining submodule is used for determining the project data with the association relationship as project association data.

Further, the query unit 702 includes an extraction module and a reading module.

And the extraction module is used for extracting the attribute information of the item to be inquired carried in the data inquiry request.

And the reading module is used for reading and feeding back the item associated data corresponding to the item attribute information to be queried from the graph database based on the graph query language.

Further, when the item association data is in an online state, the loading module includes a first loading submodule and a first storage module.

And the first loading submodule is used for loading the project associated data in the online state based on the graph data service.

And the first storage module is used for storing the loaded project associated data in the online state to the graph database.

Further, when the item association data is in the offline state, the loading module includes a second loading submodule and a second storage module.

And the second loading submodule is used for loading the project associated data in the off-line state based on the preset message service to obtain the project associated data in the preset format.

And the second storage module is used for storing the project association data in the preset format to the graph database.

Further, the system also comprises an access unit.

And the access unit is used for accessing the item associated data loaded in the graph database through a preset access service.

According to the method and the device for querying the project associated data, when the project associated data is queried, the project associated data does not need to be acquired in a table connection mode involving a large number of IO operations and memory consumption, only the loaded project associated data needs to be stored in the graph database, and since the graph database can be used for querying the project associated data efficiently, reading of other non-relevant data is prevented from being caused when the project associated data is queried, and therefore efficiency of querying the project associated data is improved. In addition, the graph database provides a visual graph display interface, so that the project association data display is more intuitive.

While, for purposes of simplicity of explanation, the foregoing method embodiments have been described as a series of acts or combination of acts, it will be appreciated by those skilled in the art that the present application is not limited by the illustrated ordering of acts, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.

It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the system-like embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The steps in the method of the embodiments of the present application may be sequentially adjusted, combined, and deleted according to actual needs.

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing is only a preferred embodiment of the present application and it should be noted that, for a person skilled in the art, several modifications and improvements can be made without departing from the principle of the present application, and these modifications and improvements should also be considered as the protection scope of the present application.