1. A data storage device across data sources, the data storage device comprising: the JSON configuration generation module, the data parser module, the data connector construction module and the data storage execution module are developed by using JAVA compiling; the data connector constructs modular extensibility.

2. The data storage device across data sources of claim 1, wherein the JSON configuration generation module is a JSON configuration generator, and wherein the service request is parsed by the JSON configuration producer to generate the corresponding JSON configuration for use by the entire tool.

3. The data storage device across data sources as claimed in claim 2, wherein the data parsing module is a data parser, the data parser parses data into corresponding data structures according to corresponding JSON configurations.

4. The data storage device across data sources as claimed in claim 3, wherein the data connector constructing module is a database connector, and the connection of the corresponding database is configured according to the JSON configuration and the data storage executing module is notified.

5. The data storage device of claim 1, wherein the data storage execution module is a data storage executor; the data storage executor executes data storage.

Background

Modern internet industry is more and more complementary, and when we need to access external company data, the data of the other party is usually acquired by adopting an HTTP network transmission mode and then stored in the database of the one party according to the data type. For example, when we access the high data, the data is acquired through the HTTP interface and stored in the Hive data warehouse of our party; when the data is accessed to the ink weather data, the data is acquired through an HTTP interface and is stored in the Redis/MongoDB of the client. Therefore, the storage of each database needs to be developed by a program to realize the data storage, and the development is time-consuming and labor-consuming.

Each time data is accessed and stored in the database, a program needs to be developed independently, and time and labor are wasted. When data needs are iterated, the program also needs to be iterated. The development technology has high threshold and is not easy to popularize. More and more data storage programs are not beneficial to the optimized management of internal codes by companies.

Disclosure of Invention

In view of the above, the present invention has been made to provide a network appointment capacity scheduling method that overcomes or at least partially solves the above problems.

According to one aspect of the present invention, there is provided a data storage across data sources, the data storage comprising: the JSON configuration generation module, the data parser module, the data connector construction module and the data storage execution module are developed by using JAVA compiling; the data connector constructs modular extensibility.

Optionally, the JSON configuration generating module is a JSON configuration generator, and the service request is analyzed by the JSON configuration producer to generate a corresponding JSON configuration for the entire tool to use.

Optionally, the data parsing module is a data parser, and the data parser parses data into a corresponding data structure according to the corresponding JSON configuration.

Optionally, the data connector constructing module is a database connector, and sets a connection of a corresponding database according to the JSON configuration and notifies the data storage executing module.

Optionally, the data storage execution module is a data storage executor; the data storage executor executes data storage.

The data storage device across data sources provided by the invention has the beneficial effects that: the data storage device includes: the JSON configuration generation module, the data parser module, the data connector construction module and the data storage execution module are developed by using JAVA compiling; the data connector construction module has expandability and provides a data storage device crossing data sources, corresponding database configuration is packaged through JSON, the JSON configuration is analyzed to obtain a data storage, and then data obtained through an HTTP interface is stored in different databases through the data storage.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a block diagram of a data storage device across data sources according to an embodiment of the present invention;

fig. 2 is a schematic diagram of data flow across data sources according to an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

The terms "comprises" and "comprising," and any variations thereof, in the present description and claims and drawings are intended to cover a non-exclusive inclusion, such as a list of steps or elements.

The technical solution of the present invention is further described in detail with reference to the accompanying drawings and embodiments.

As shown in fig. 1, a data storage device across data sources, the data storage device comprising: the JSON configuration generation module 1, the data parser module 2, the data connector construction module 3 and the data storage execution module 4 are developed by using JAVA compiling; the data connector constructs 3 module extensibility.

The JSON configuration generation module 1 is a JSON configuration generator, the service request is analyzed by the JSON configuration producer, and corresponding JSON configuration is generated and used by the whole tool. The data parser module 2 is a data parser, and the data parser parses data into a corresponding data structure according to the corresponding JSON configuration.

The data connector constructing module 3 is a database connector, sets the connection of the corresponding database according to JSON configuration and informs the data storage executing module 4. The data storage execution module 4 is a data storage executor; the data storage executor executes data storage.

A JSON configuration generator, which is largely divided into two parts:

data Reader: the url of the HTTP interface, a request mode (get or post), an HTTP analysis implementation class and analysis parameters are packaged;

data Writer: type refers to a database type, connect encapsulates database link information, and conf encapsulates data storage format related information.

A data parser, which generates an HTTP parser according to the Reader configuration in the JSON configuration generator, wherein the parser has the following 3 methods:

a. execution method execute: the purpose is to start the resolution to be performed,

b. verification method hasNext (): the purpose is to determine whether the data has been parsed,

c. acquiring next piece of data next (): and when the result obtained by the last step of checking method is that the data is not analyzed, executing the method to continue executing.

The analyzer realizes the analysis capability of data in different formats through user definition due to the characteristics of the interface, so that the analyzer has good expansibility and can be customized to meet different requirements.

The database connector structure: the database connector is a connector which is constructed corresponding to the database according to the Writer configuration in the JSON configuration generator, supports various databases such as MySQL, Hive, Redis, MongoDB and the like, and also supports dynamic expansion of the databases.

The connector receives the data sent by the data parser, triggers the data storage executor and stores the data in each database. The connector receiving data analyzer realizes the analysis capability of data in different formats through user definition due to the characteristics of the interface of the connector receiving data analyzer, so that the connector receiving data analyzer has good expansibility and can be customized to meet different requirements.

The data storage executor depends on each database connector, and the core function of the data storage executor is to store the data sent by the data parser to the connector into the corresponding database.

And the database connector receives the data sent by the data parser, triggers the data storage executor and stores the data into each database.

FIG. 2 shows that the data flow enters the database connector through the HTTP interface, the database connector is divided into a Hive connector, a MySQL connector, a Redis connector, a MongoDB connector and the like, and the data is stored in the corresponding database (Hive, MySQL, Redis, MongoDB) through the connector through the storage executor.

The device using method comprises the following steps: the device is developed by JAVA and runs on a virtual machine configured with JDK. May be performed by means of scheduling tools such as crontab, Azkaban, etc. Before use, only the corresponding JSON, namely conf. The starting command is as follows: java-jar database-to-http. jar conf.

Has the advantages that:

1. external data (third-party data) can be stored in different databases through simple JSON configuration, and a large amount of development and compilation work is avoided.

2. The data parser and the database connector have good expandability, and a new connector can be generated in a user-defined mode to realize a data storage function.

The above embodiments are provided to further explain the objects, technical solutions and advantages of the present invention in detail, it should be understood that the above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the scope of the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.