1. A data integration and distribution platform is characterized by comprising a data interface module, a data acquisition tool, a data processing module and a database;

the data interface module is mounted on the data integration platform, and the data acquisition tool is embedded in the data integration platform;

the data interface module is used for acquiring first data of an external application system;

the data acquisition tool is used for extracting second data of the database;

the data processing module is used for processing the first data and the second data and/or providing data service for the external application system through the data interface module;

the database is also used for storing the processed first data.

2. The platform of claim 1, wherein the data processing module comprises a message queue server, a message service unit, a gateway service unit, and a core service unit;

the data interface module sends the acquired first data to the core service unit sequentially through the message queue server and the message service unit; or the gateway service unit sends the information to the core service unit;

the message queue server is used for receiving the first data and sending the first data to the message service unit based on a queue mechanism;

the message service unit is used for processing the first data and then forwarding the first data to the core service unit according to the subscription message of the first data;

the gateway service unit is used for receiving the first data, processing the first data and then forwarding the first data to the core service unit;

and the core service unit is used for carrying out secondary processing on the received first data, returning the first data to the message queue server and storing the first data in the database.

3. The data integration and distribution platform according to claim 2, wherein the data interface module comprises an MQ data interface unit, an HTTP data interface unit and a DDS data interface unit; and the data processing module performs data interaction with the external application system through the MQ data interface unit, the HTTP data interface unit and/or the DDS data interface unit.

4. The data integration distribution platform of claim 3, wherein when the first data is collected through the MQ data interface unit, the data processing module is specifically configured to:

converting the first data into first JSON format data and sending the first JSON format data to the message queue server;

configuring a first JSON adapter in the message service unit, receiving and processing the first JSON format data, and forwarding the first JSON format data to the core service unit for secondary processing;

and returning the first JSON format data to the message queue server and storing the first JSON format data in the database.

5. The data integration and distribution platform of claim 4, wherein the data processing module is further configured to:

and receiving a first data service request of the external application system through the message queue server, and returning a first data service request result to the external application system.

6. The data integration and distribution platform of claim 3, wherein when the first data is collected through the HTTP data interface unit, the data processing module is specifically configured to:

converting the first data into second JSON format data and sending the second JSON format data to the message queue server;

configuring a second JSON adapter in the gateway service unit, receiving and processing the second JSON format data, and forwarding the second JSON format data to the core service unit for secondary processing;

and returning the second JSON format data to the message queue server and storing the second JSON format data in the database.

7. The data integration and distribution platform of claim 6, wherein the data processing module is further configured to:

receiving a second data service request of an external application system through the gateway service unit and forwarding the second data service request to the core service unit;

and calling corresponding data from the database through the core service unit according to the second data service request, and returning a second data request result to the external application system.

8. The data integration and distribution platform of claim 3, wherein when the first data is collected through the DDS data interface unit, the data processing module is specifically configured to:

configuring a DDS adapter, receiving the first data through the DDS adapter, and converting the first data into third JSON format data;

sending the third JSON format data to the message queue server;

configuring a third JSON adapter in the message service unit, receiving the third JSON format data, and forwarding the third JSON format data to the core service unit for secondary processing;

and returning the third JSON format data to the message queue server and storing the third JSON format data in the database.

9. The data integration and distribution platform of claim 1, wherein the data collection tool extracts the second data from the database by means of ETL.

10. A data integration and distribution system, comprising the data integration and distribution platform according to any one of claims 1 to 9, and further comprising a plurality of application systems.

Background

With the rapid development of information technology, various industries are dedicated to informatization construction, a large number of application information systems are constructed based on different business requirements, rapid development of the industries is effectively promoted, a large number of information islands are brought at the same time, and isomerization and complexity of the application information systems also bring more serious challenges to information management of enterprises.

In order to realize information intercommunication and data sharing among information systems and break through information isolated islands, a solution in the prior art is to modify interfaces of systems needing information intercommunication, and although the method can solve the problem of data sharing among information systems, related application systems need to be modified; in addition, the method can only meet the sharing requirement of specific data among systems, and has poor expandability and low flexibility.

Disclosure of Invention

The invention aims to provide a modular data display platform and a system, which are used for solving at least one technical problem in the prior art.

In order to achieve the purpose, the invention adopts the following technical scheme:

in a first aspect, the invention provides a data integration and distribution platform, which comprises a data interface module, a data acquisition tool, a data processing module and a database;

the data interface module is mounted on the data integration platform, and the data acquisition tool is embedded in the data integration platform;

the data interface module is used for acquiring first data of an external application system;

the data acquisition tool is used for extracting second data of the database;

the data processing module is used for processing the first data and the second data and/or providing data service for the external application system through the data interface module;

the database is also used for storing the processed first data.

In one possible design, the data processing module includes a message queue server, a message service unit, a gateway service unit, and a core service unit;

the data interface module sends the acquired first data to the core service unit sequentially through the message queue server and the message service unit; or the gateway service unit sends the information to the core service unit;

the message queue server is used for receiving the first data and sending the first data to the message service unit based on a queue mechanism;

the message service unit is used for processing the first data and then forwarding the first data to the core service unit according to the subscription message of the first data;

the gateway service unit is used for receiving the first data, processing the first data and then forwarding the first data to the core service unit;

and the core service unit is used for carrying out secondary processing on the received first data, returning the first data to the message queue server and storing the first data in the database.

In one possible design, the data interface module includes an MQ data interface unit, an HTTP data interface unit, and a DDS data interface unit; and the data processing module performs data interaction with the external application system through the MQ data interface unit, the HTTP data interface unit and/or the DDS data interface unit.

In one possible design, when the first data is collected through the MQ data interface unit, the data processing module is specifically configured to:

converting the first data into first JSON format data and sending the first JSON format data to the message queue server;

configuring a first JSON adapter in the message service unit, receiving and processing the first JSON format data, and forwarding the first JSON format data to the core service unit for secondary processing;

and returning the first JSON format data to the message queue server and storing the first JSON format data in the database.

In one possible design, the data processing module is further specifically configured to:

and receiving a first data service request of the external application system through the message queue server, and returning a first data service request result to the external application system.

In one possible design, when the first data is collected through the HTTP data interface unit, the data processing module is specifically configured to:

converting the first data into second JSON format data and sending the second JSON format data to the message queue server;

configuring a second JSON adapter in the gateway service unit, receiving and processing the second JSON format data, and forwarding the second JSON format data to the core service unit for secondary processing;

and returning the second JSON format data to the message queue server and storing the second JSON format data in the database.

In one possible design, the data processing module is further specifically configured to:

receiving a second data service request of an external application system through the gateway service unit and forwarding the second data service request to the core service unit;

and calling corresponding data from the database through the core service unit according to the second data service request, and returning a second data request result to the external application system.

In one possible design, when the first data is collected through the DDS data interface unit, the data processing module is specifically configured to:

configuring a DDS adapter, receiving the first data through the DDS adapter, and converting the first data into third JSON format data;

sending the third JSON format data to the message queue server;

configuring a third JSON adapter in the message service unit, receiving the third JSON format data, and forwarding the third JSON format data to the core service unit for secondary processing;

and returning the third JSON format data to the message queue server and storing the third JSON format data in the database.

In one possible design, the data collection tool extracts the second data from the database by way of ETL.

In a second aspect, the present invention provides a data integration and distribution system, which includes the data integration and distribution platform as described in any one of the possible designs of the first aspect, and further includes several application systems.

Has the advantages that:

the data interface module is mounted on the data integration platform, and the data acquisition tool is embedded in the data integration platform; acquiring first data of an external application system through a data interface module; extracting second data of the database through a data acquisition tool; the first data and the second data are processed through the data processing module, and/or data services are provided for an external application system through the data interface module; and storing the processed first data through a database. The invention can support various data interfaces including but not limited to HTTP interface, MQ interface and DSS interface, and data acquisition tool for database, so that the platform can configure, manage and monitor the data interface according to the service requirement, thereby realizing the data interaction with the external application system rapidly; the data interface module is arranged in a mounting mode, so that the data interface module can be added and deleted according to service requirements on the premise of not changing a main frame of the platform, and the expandability of the platform is improved.

Drawings

FIG. 1 is a block diagram of a data integration and distribution platform according to an embodiment of the present invention;

FIG. 2 is a diagram of the architecture deployment of a data integration and distribution platform according to an embodiment of the present invention;

fig. 3 is a block diagram illustrating a data interaction process with an external application system according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments in the present description, belong to the protection scope of the present invention.

Examples

As shown in fig. 1-3, in a first aspect, the present invention provides a data integration and distribution platform, which includes a data interface module, a data acquisition tool, a data processing module, and a database;

the data interface module is mounted on the data integration platform, and the data acquisition tool is embedded in the data integration platform; by mounting the data interface module on the data integration platform, the data interface can be configured, added or deleted according to the service requirement, and the universality and the expandability of the platform are ensured.

The data interface module is used for acquiring first data of an external application system;

the data interface module comprises a data interface in a general format and a self-defined data interface, wherein the general data interface can directly acquire data provided by the system through an interface json format configured in data maintenance management and configuration information of an application system; the self-defined data interface can generate a corresponding interface Adapter (Adapter) through an interface with extensible configuration provided in data maintenance management, and corresponding data acquisition can be carried out through the interface.

The data interface module comprises but is not limited to an MQ data interface unit, an HTTP data interface unit and a DDS data interface unit; the data processing module performs data interaction with the external application system through the MQ data interface unit, the HTTP data interface unit and/or the DDS data interface unit; correspondingly, the first Data includes, but is not limited to, MQ (Message Queue) Data, HTTP (Hypertext Transfer Prtcl) Data, and DDS (Data Distribution Service) Data.

It should be noted that the HTTP data interface unit is an asynchronous interface, and returns to a task ID calling end every time it is called by the platform, and whether the currently executed task is completed can be queried according to the task ID.

The HTTP data interface unit provides corresponding webservice for calling an external application system, and the application scenario is as follows: data needs to be actively acquired, and the next processing is carried out depending on the acquired data, namely, a data request and a response need to be synchronized; infrequent scenes are accessed or acquired by the data, such as initialized data, which is followed by infrequently updated data; the method is suitable for a data exchange short connection scene.

And the MQ interface unit provides a corresponding message queue interface mode and provides the message queue interface mode for an external application system to call. The application scene is as follows: real-time data, such as alarm push, real-time data of the acquisition equipment, and the like; data which can be processed asynchronously, because the application system has own data storage, the data of the integrated platform is basically the data which can be processed asynchronously; the method is suitable for a long connection scene of frequent data interaction; a scenario where notification needs to be applied.

When the data interface unit is used for collecting the DDS data, because the DDS data cannot cross a router (cross a network segment), a corresponding DDS gateway needs to be installed in the same network segment served by the DDS, and after the DDS data is actively subscribed, the data is transmitted to the integrated platform for unified processing and storage through a standard MQ interface form.

It should be noted that the external application systems include, but are not limited to, a portal application system, an operation and maintenance system, a political engineering system, a data presentation system, an enterprise management system, and the like, which are not specifically limited and are not described herein again.

The data acquisition tool is used for extracting second data of the database;

it should be noted that the data collection tool extracts the second data from the database in an ETL (Extract Transform Load) manner, where the second data is heterogeneous data stored in the database, and the second data includes but is not limited to: text files, CSV (Comma-Separated Values) files, JMS (Java Message Service) files, and WebService files.

Wherein, preferably, the database is a damming Database (DB).

In addition, it can be understood that, when an external application system cannot provide a data interface, the data integration and distribution platform may also obtain data of the external application system through the data acquisition tool.

The data processing module is used for processing the first data and the second data and/or providing data service for the external application system through the data interface module;

it should be noted that the data processing module includes, but is not limited to, a message queue server, a message service unit, a gateway service unit, and a core service unit. In addition, various third party components may be included, such as: a registry service component, a Redis data cache service component, a Zookeeper coordination component, a RockettMQ message service component, an Elastic-Job distributed scheduling component, and a Nginx load balancing component. Based on this, as shown in fig. 2, an architecture deployment diagram of the data integration and distribution platform in this embodiment is shown, where each component or unit is deployed in a cluster and networked through a local area network, and except for the dreams database cluster, other components or units may be deployed by using a virtual machine or a docker container.

The database is also used for storing the processed first data.

Based on the above disclosure, the present embodiment can support various data interfaces, including but not limited to an HTTP interface, an MQ interface, and a DSS interface, and a data acquisition tool for a database, so that a platform can configure, manage, and monitor the data interfaces according to service requirements, thereby rapidly implementing data interaction with an external application system; the data interface module is arranged in a mounting mode, so that the data interface module can be newly added and deleted according to service requirements on the premise of not changing a main frame of the platform, and the expandability of the platform is improved

As an optional implementation manner, the data processing module includes a message queue server, a message service unit, a gateway service unit, and a core service unit;

the data interface module sends the acquired first data to the core service unit sequentially through the message queue server and the message service unit; or the gateway service unit sends the information to the core service unit;

the message queue server is used for receiving the first data and sending the first data to the message service unit based on a queue mechanism;

the message service unit is used for processing the first data and then forwarding the first data to the core service unit according to the subscription message of the first data;

specifically, after receiving the first data, the message service unit performs, according to a subscription message of the first data, current limiting, resource allocation, reception, capacity expansion, and message forwarding on the first data.

The gateway service unit is used for receiving the first data, processing the first data and then forwarding the first data to the core service unit;

specifically, after receiving the first data, the gateway service unit performs current limiting, fusing, log management, black and white list query, authentication, and interface forwarding on the first data.

And the core service unit is used for carrying out secondary processing on the received first data, returning the first data to the message queue server and storing the first data in the database.

Specifically, after receiving the first data, the core service unit performs parsing, verification, cleaning, desensitization, merging, sorting, and data forwarding on the first data.

In addition, the core business service unit is also provided with a data pipeline and a WEB management system, and the core business service unit performs data interaction with the database through the data pipeline and performs configuration management on a data interface and parameters through the WEB management system.

In an optional implementation manner, when the first data is collected through the MQ data interface unit, the data processing module is specifically configured to:

converting the first data into first JSON format data and sending the first JSON format data to the message queue server;

it should be noted that, when the first data of the external application system is collected through the MQ data interface unit, the platform may provide a template number for each application system, where the template number describes system information of the access application system and description of the accessed parameters; the template number can be pre-configured in the background of the platform, so that the flow of parameter description information transmitted by a calling party is simplified.

It should be noted that the message queue server includes a subscription topic of an accessed application system, and the subscription topic includes an ID assigned to the application system.

Wherein the first JSON format data comprises a template number and a data set of an application system.

The JSON format comprises a general JSON format and a custom JSON format.

Configuring a first JSON adapter in the message service unit, receiving and processing the first JSON format data, and forwarding the first JSON format data to the core service unit for secondary processing;

the JSON adapter comprises a general JSON adapter and a custom JSON adapter.

Specifically, the message service unit performs current limiting, resource allocation, receiving, capacity expansion and message forwarding processing on the first JSON-format data; and the core service unit analyzes, verifies, cleans, desensitizes, merges and sorts the first JSON format data and forwards the data.

And returning the first JSON format data to the message queue server and storing the first JSON format data in the database.

In an optional implementation manner, the data processing module is further specifically configured to:

and receiving a first data service request of the external application system through the message queue server, and returning a first data service request result to the external application system.

It should be noted that, when an external application system initiates a first data service request to a platform, the first data service request carries a topic name, and the topic name may be specifically defined according to a service requirement; the first data service request is a JSON format request and comprises a service number and specific service data; after the message queue server receives the first data service request, if the message queue server has data matched with the first data service request, returning the corresponding data to the external application service system, otherwise, returning a prompt of request failure to the external application system; wherein the data corresponding to the first data service request includes but is not limited to: total record number, current page number, size of each page, total page number and specific service data.

In an optional implementation manner, when the first data is collected through the HTTP data interface unit, the data processing module is specifically configured to:

converting the first data into second JSON format data and sending the second JSON format data to the message queue server;

it should be noted that, when the HTTP data interface unit collects the first data of the external application system, the platform may provide a template number for each application system, where the template number describes system information of the access application system and description of the access parameter; the template number can be pre-configured in the background of the platform, so that the flow of parameter description information transmitted by a calling party is simplified.

Configuring a second JSON adapter in the gateway service unit, receiving and processing the second JSON format data, and forwarding the second JSON format data to the core service unit for secondary processing;

specifically, the gateway service unit performs current limiting, fusing, log management, black and white list query, authentication and interface forwarding on the first data; and the core service unit analyzes, verifies, cleans, desensitizes, merges and sorts the first JSON format data and forwards the data.

And returning the second JSON format data to the message queue server and storing the second JSON format data in the database.

In an optional implementation manner, the data processing module is further specifically configured to:

receiving a second data service request of an external application system through the gateway service unit and forwarding the second data service request to the core service unit;

and calling corresponding data from the database through the core service unit according to the second data service request, and returning a second data request result to the external application system.

When an external application system initiates a second data service request to a platform, the second data service request carries a service number, a field name, a start page, a size of each page, and a data filtering condition; when the gateway service unit receives the second data service request, the gateway service unit forwards the second data service request to the core service unit, and the core service unit calls data matched with the second data service request to the database according to the second data service request, if the database has the data which can be matched, the corresponding data is returned to the external application service system, otherwise, a prompt of request failure is returned to the external application system; wherein the data corresponding to the first data service request includes but is not limited to: total record number, current page number, size of each page, total page number and specific service data.

In an optional implementation manner, when the first data is acquired through the DDS data interface unit, the data processing module is specifically configured to:

configuring a DDS adapter, receiving the first data through the DDS adapter, and converting the first data into third JSON format data;

it should be noted that, a DDS adapter is written in the DDS adapter, each DDS adapter can subscribe to a plurality of DDS data, and the platform assigns a plurality of template numbers to the DDS adapter, each template number corresponding to one DDS data definition; and then, the DDS adapter converts the received first data into third JSON format data and sends the third JSON format data to the message queue server. How to subscribe, receive and analyze the first data can be realized by a JAVA tool library.

Sending the third JSON format data to the message queue server;

configuring a third JSON adapter in the message service unit, receiving the third JSON format data, and forwarding the third JSON format data to the core service unit for secondary processing;

specifically, the message service unit performs current limiting, resource allocation, receiving, capacity expansion and message forwarding processing on the first JSON-format data; and the core service unit analyzes, verifies, cleans, desensitizes, merges and sorts the first JSON format data and forwards the data.

And returning the third JSON format data to the message queue server and storing the third JSON format data in the database.

In a second aspect, the present invention provides a data integration and distribution system, which includes the data integration and distribution platform as described in any one of the possible designs of the first aspect, and further includes several application systems.

Finally, it should be noted that: the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.