1. A method for pushing data across data sources, comprising:

packaging data transmission requirements through JSON;

analyzing JSON to obtain query requests of different databases, querying data, and encapsulating the data into a data set;

and the data transmission module transmits the packaged data set to a corresponding interface in an HTTP network transmission mode.

2. The method for pushing data across data sources as claimed in claim 1, wherein a data producer, a data transmission rate and a data receiver are encapsulated in the JSON.

3. The method for pushing data across data sources according to claim 1, wherein the parsing JSON obtains query requests of different databases, queries data, and encapsulates data into a data set, including:

creating a link to a database;

inquiring data from a database according to the link according to the analyzed inquiry condition, and acquiring the data;

and encapsulating the acquired data into a plurality of formatted data sets.

4. The method for pushing data across data sources according to claim 1, wherein the data transmission module sends the encapsulated data sets to corresponding interfaces in a HTTP network transmission manner, and the method comprises:

creating an HTTP link according to the interface configuration of the other party;

and transmitting the encapsulated data set to an interface of the other party through an HTTP network protocol, wherein the basis for judging whether the data is successfully transmitted is as follows: when the transmission is successful, returning to success; when the transmission is unsuccessful, returning to failure, and repeatedly executing the steps of: and transmitting the packaged data set to an interface of the other party through an HTTP network protocol.

5. A data push apparatus across data sources, comprising:

the first packaging unit is used for packaging data transmission requirements through JSON;

the second packaging unit is used for analyzing JSON to obtain query requests of different databases, querying data and packaging the data into a data set;

and the transmission unit is used for transmitting the packaged data set to the corresponding interface by the data transmission module in an HTTP network transmission mode.

6. The apparatus of claim 5, wherein the JSON encapsulates a data producer, a data transmission rate, and a data receiver.

7. The data pushing apparatus across data sources according to claim 5, wherein the second encapsulating unit includes:

creating a link to a database;

inquiring data from a database according to the link according to the analyzed inquiry condition, and acquiring the data;

and encapsulating the acquired data into a plurality of formatted data sets.

8. The data pushing apparatus of claim 5, wherein the transmission unit comprises:

creating an HTTP link according to the interface configuration of the other party;

and transmitting the encapsulated data set to an interface of the other party through an HTTP network protocol, wherein the basis for judging whether the data is successfully transmitted is as follows: when the transmission is successful, returning to success; when the transmission is unsuccessful, returning to failure, and repeatedly executing the steps of: and transmitting the packaged data set to an interface of the other party through an HTTP network protocol.

Background

The data magnitude index of the modern internet industry increases, each company constructs own data warehouse storage data, and the analysis and application of the data become more diversified along with the development of enterprises. In order to avoid the high coupling of each department, a HTTP network transmission mode is usually adopted to extract data from a data warehouse and push the data to other departments, and different data requirements of each department require separate development of programs to realize data transmission.

Disclosure of Invention

In view of the above technical problems in the related art, the present invention provides a method and an apparatus for data push across data sources, which can overcome the above disadvantages in the prior art.

In order to achieve the technical purpose, the technical scheme of the invention is realized as follows:

a method of data push across data sources, the method comprising:

packaging data transmission requirements through JSON;

analyzing JSON to obtain query requests of different databases, querying data, and encapsulating the data into a data set;

and the data transmission module transmits the packaged data set to a corresponding interface in an HTTP network transmission mode.

Further, a data producer, a data transmission rate, and a data receiver are encapsulated in the JSON.

Further, the analyzing JSON obtains query requests of different databases, queries data, and encapsulates the data into a data set, including:

creating a link to a database;

inquiring data from a database according to the link according to the analyzed inquiry condition, and acquiring the data;

and encapsulating the acquired data into a plurality of formatted data sets.

Further, the data transmission module sends the encapsulated data set to a corresponding interface in a HTTP network transmission manner, including:

creating an HTTP link according to the interface configuration of the other party;

and transmitting the encapsulated data set to an interface of the other party through an HTTP network protocol, wherein the basis for judging whether the data is successfully transmitted is as follows: when the transmission is successful, returning to success; when the transmission is unsuccessful, returning to failure, and repeatedly executing the steps of: and transmitting the packaged data set to an interface of the other party through an HTTP network protocol.

In another aspect, the present invention provides a data pushing apparatus across data sources, the apparatus comprising:

the first packaging unit is used for packaging data transmission requirements through JSON;

the second packaging unit is used for analyzing JSON to obtain query requests of different databases, querying data and packaging the data into a data set;

and the transmission unit is used for transmitting the packaged data set to the corresponding interface by the data transmission module in an HTTP network transmission mode.

Further, a data producer, a data transmission rate, and a data receiver are encapsulated in the JSON.

Further, the second packaging unit includes:

creating a link to a database;

inquiring data from a database according to the link according to the analyzed inquiry condition, and acquiring the data;

and encapsulating the acquired data into a plurality of formatted data sets.

Further, the transmission unit includes:

creating an HTTP link according to the interface configuration of the other party;

and transmitting the encapsulated data set to an interface of the other party through an HTTP network protocol, wherein the basis for judging whether the data is successfully transmitted is as follows: when the transmission is successful, returning to success; when the transmission is unsuccessful, returning to failure, and repeatedly executing the steps of: and transmitting the packaged data set to an interface of the other party through an HTTP network protocol.

The invention has the beneficial effects that: according to the invention, through simple parameter configuration, data of different data sources can be inquired, and the speed and the failure retry times of HTTP network data transmission can be adjusted, so that a large amount of development and compilation work is avoided; the invention matches different database selections of different companies through the data source expandability of PrestoDB.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 illustrates a flow diagram of a method of data push across data sources, in accordance with an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a data pushing apparatus across data sources according to an embodiment of the present invention;

fig. 3 shows a schematic structural diagram of a data transmission module according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

As shown in fig. 1, a data pushing method across data sources includes:

packaging data transmission requirements through JSON;

analyzing JSON to obtain query requests of different databases, querying data, and encapsulating the data into a data set;

and the data transmission module transmits the packaged data set to a corresponding interface in an HTTP network transmission mode.

In some embodiments of the invention, a data producer, a data transmission rate and a data receiver are encapsulated in the JSON.

In some embodiments of the present invention, the parsing JSON to obtain query requests of different databases, query data, and encapsulate the data into a data set, includes:

creating a link to a database;

inquiring data from a database according to the link according to the analyzed inquiry condition, and acquiring the data;

and encapsulating the acquired data into a plurality of formatted data sets.

In some embodiments of the present invention, the sending, by the data transmission module, the encapsulated data set to the corresponding interface in a HTTP network transmission manner includes:

creating an HTTP link according to the interface configuration of the other party;

and transmitting the encapsulated data set to an interface of the other party through an HTTP network protocol, wherein the basis for judging whether the data is successfully transmitted is as follows: when the transmission is successful, returning to success; when the transmission is unsuccessful, returning to failure, and repeatedly executing the steps of: and transmitting the packaged data set to an interface of the other party through an HTTP network protocol.

As shown in FIG. 2, in another aspect, the present invention provides a data pushing apparatus across data sources, the apparatus comprising:

the first packaging unit is used for packaging data transmission requirements through JSON;

the second packaging unit is used for analyzing JSON to obtain query requests of different databases, querying data and packaging the data into a data set;

and the transmission unit is used for transmitting the packaged data set to the corresponding interface by the data transmission module in an HTTP network transmission mode.

In some embodiments of the invention, a data producer, a data transmission rate and a data receiver are encapsulated in the JSON.

In some embodiments of the invention, the second packaging unit comprises:

creating a link to a database;

inquiring data from a database according to the link according to the analyzed inquiry condition, and acquiring the data;

and encapsulating the acquired data into a plurality of formatted data sets.

In some embodiments of the present invention, the transmission unit includes:

creating an HTTP link according to the interface configuration of the other party;

and transmitting the encapsulated data set to an interface of the other party through an HTTP network protocol, wherein the basis for judging whether the data is successfully transmitted is as follows: when the transmission is successful, returning to success; when the transmission is unsuccessful, returning to failure, and repeatedly executing the steps of: and transmitting the packaged data set to an interface of the other party through an HTTP network protocol.

The invention comprises a JSON generation module, a JSON analysis module, a data transmission module and a tool using method, and is developed by using JAVA compiling.

The JSON generation module is used for describing three elements of data: data producer, data transmission rate, data receiver.

The jobName records the service name, the envName records the execution environment of the service name, and the batchSize records the data volume packaged in each batch in data transmission.

The reader is used for encapsulating database configuration, including database type (e.g. hive), database login name, password, query SQL, and the like. The database can be various data sources, such as hive, MySQL and all databases which can be queried by PrestoDB, and different query SQL is encapsulated according to different business requirements. PrestoDB is a memory computing framework that can link a wide variety of databases, and thus the present invention can extend data sources by relying on this property of PrestoDB.

The writer is used for encapsulating data receiver related configurations, such as an http interface url, access modes (get, post, etc.) of the http interface, data field alias configuration, the number of times of failure retries, and the like.

JSON parsing module

Encapsulating JSON conditions by a JAVA implementation class, wherein the following are required:

a. environment name (envName): represents the environment of the configuration, and different environments can be configured here, such as development environment, test environment, etc.;

b. task name (jobName): representing the task name;

c. batch size (batch size): indicating how many pieces of data can be sent at most per batch, e.g., 50;

d. data stream read (reader): this encapsulates the query conditions for the database;

e. data stream output (writer): this encapsulates the configuration information for the HTTP network protocol interface.

3. Data analysis module

This module is the execution module of the reader model. The method comprises the steps of creating a link of a database, wherein the database can be MySQL, HIVE, PRESTO, MONGO, REDIS and the like, executing query according to analyzed query conditions to obtain data, and packaging the queried data into a plurality of formatted data sets.

4. Data transmission module

As shown in fig. 3, after the data parsing module is executed, the present invention sends the encapsulated data set to the corresponding interface through the data transmission module in the HTTP network transmission manner.

a, creating an HTTP link according to the configuration of an interface of the other party;

b, transmitting the packaged data set to an interface of the other party through an HTTP network protocol;

c, judging whether the data is successfully transmitted, if so, returning to 'success', and if not, returning to 'failure', and executing a retry mechanism, namely, re-executing the step b.

5. Device using method

The device is developed by JAVA and runs on a virtual machine configured with JDK. May be performed by means of scheduling tools such as crontab, Azkaban, etc.

Before use, only the corresponding JSON, namely conf. The starting command is as follows:

java-jar database-to-http.jar conf.properties。

according to the invention, through simple parameter configuration, data of different data sources can be inquired, and the speed and the failure retry times of HTTP network data transmission can be adjusted, so that a large amount of development and compilation work is avoided; the invention matches different database selections of different companies through the data source expandability of PrestoDB.

Although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.