1. A database update message processing method is characterized by comprising the following steps:

acquiring a data updating message of a first database, wherein the data updating message carries a data table identifier;

inquiring the pre-configured configuration information to obtain an update identifier corresponding to the data table identifier;

judging whether a data updating message corresponding to the updating identification exists or not;

and when the data updating message corresponding to the updating identification exists, deleting the data updating message.

2. The database update message processing method according to claim 1, wherein the querying preconfigured configuration information to obtain the update identifier corresponding to the data table identifier includes:

inquiring whether the pre-configured configuration information has the data table identification or not;

when the data table identification exists, acquiring a data model identification corresponding to the data table identification in the pre-configured configuration information;

determining whether the data table corresponding to the data table identification is a main table according to pre-configured configuration information;

when the data table corresponding to the data table identification is a main table, generating an updating identification according to the data model identification and a main key value of the main table;

and when the data table corresponding to the data table identification is a slave table, generating an updating identification according to the data model identification and the external key value of the slave table.

3. The database update message processing method according to claim 2, wherein after determining whether the data update message corresponding to the update identifier already exists, the method further comprises:

and when the data updating message corresponding to the updating identification does not exist, establishing the corresponding relation between the data updating message and the updating identification.

4. The database update message processing method according to claim 3, wherein the establishing of the correspondence between the data update message and the update identifier includes:

obtaining the effective time of an updating identifier corresponding to the identifier of the data table in the pre-configured configuration information;

and establishing the corresponding relation between the data updating message and the updating identification according to the effective time of the updating identification.

5. A data synchronization method, characterized in that the data synchronization method comprises:

acquiring a data updating message obtained by the database updating message processing method according to any one of claims 1 to 4;

acquiring updating data from a first database according to the data updating message;

performing data conversion on the updated data according to the pre-configured configuration information to obtain target data;

and saving the target data to a second database.

6. The data synchronization method according to claim 5, wherein the performing data conversion on the update data according to the pre-configured configuration information to obtain target data comprises:

analyzing the updated data according to a pre-stored template to obtain target data; and/or

And performing data conversion on the updated data according to the pre-stored field corresponding relation to obtain target data.

7. A method for configuring database configuration information, the method comprising:

receiving a connection information configuration instruction for a first database, and configuring the connection information of the first database according to the connection information configuration instruction of the first database;

receiving a configuration instruction for the association relationship of each data table in the first database, and configuring each data table in the first database according to the configuration instruction for the association relationship to determine an update identifier corresponding to each data table identifier and valid time of the update identifier, where the update identifier and the valid time of the update identifier are used in the database update message processing method of any one of claims 1 to 4.

8. The method according to claim 7, wherein the configuring, according to the configuration instruction of the association relationship, each data table in the first database includes:

configuring data model identifications of the data models in the first database, effective time of update identifications corresponding to the data model identifications, and primary tables and secondary tables in the data models, and configuring primary key values of the primary tables and foreign key values of the secondary tables.

9. A method for configuring database configuration information, the method comprising:

receiving a connection information configuration instruction aiming at a second database, and configuring the connection information of the second database according to the connection information configuration instruction of the first database;

receiving a conversion rule configuration instruction aiming at the data in the second database and the data in the first database, and configuring a conversion rule of the data in the second database and the data in the first database according to the conversion rule configuration instruction; the transformation rules are for use in the data synchronization method of claim 5 or 6.

10. A database update message processing apparatus, characterized in that the database update message processing apparatus comprises:

the first update message acquisition module is used for acquiring a data update message of a first database, wherein the data update message carries a data table identifier;

the query module is used for querying the pre-configured configuration information to obtain the update identification corresponding to the data table identification;

the judging module is used for judging whether a data updating message corresponding to the updating identification exists or not;

and the deleting module is used for deleting the data updating message when the data updating message corresponding to the updating identification exists.

11. A data synchronization apparatus, characterized in that the data synchronization apparatus comprises:

a second update message acquisition module, configured to acquire the data update message obtained by the database update message processing apparatus according to claim 9;

the update data acquisition module is used for acquiring update data from a first database according to the data update message;

the conversion module is used for performing data conversion on the updated data according to the pre-configured configuration information to obtain target data;

and the storage module is used for storing the target data to a second database.

12. A database configuration information configuring apparatus, comprising:

the first receiving module is used for receiving a connection information configuration instruction aiming at a first database and configuring the connection information of the first database according to the connection information configuration instruction of the first database;

a second receiving module, configured to receive a configuration instruction of an association relationship for each data table in the first database, and configure each data table in the first database according to the configuration instruction of the association relationship, so as to determine an update identifier corresponding to each data table identifier and valid time of the update identifier, where the valid time of the update identifier and the update identifier is used in the database update message processing apparatus according to claim 8.

13. A database configuration information configuring apparatus, comprising:

a third receiving module, configured to receive a connection information configuration instruction for a second database, and configure connection information of the second database according to the connection information configuration instruction of the first database;

a fourth receiving module, configured to receive a conversion rule configuration instruction for data in the second database and data in the first database, and configure a conversion rule between the data in the second database and the data in the first database according to the conversion rule configuration instruction; the conversion rule is for use in the data synchronization device of claim 12.

14. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 4 or 5 to 6 or 7 to 8 or 9.

15. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 4 or 5 to 6 or 7 to 8 or 9.

Background

With the wider and wider online application, the data volume is increased suddenly, and the application scenes of massive data search and cross-data source combined query are more and more. The traditional relational database has relatively poor query performance and insufficient flexibility, and some novel extensible and high-performance databases are required to be used for supplement.

In the conventional technology, the data of the conventional relational database is synchronized to the novel database, so that the novel database is directly searched during searching.

However, when the messages are synchronized between the databases at present, the data in one database is updated and needs to be synchronized to another database in time, but the same data in one database may exist in different data tables, that is, one data model includes a plurality of associated data tables, when the data is updated, the plurality of data tables are updated together, so that one update of another database for the database may need to be synchronized for many times, which causes repeated and worthless processing and reduces system performance.

Disclosure of Invention

Therefore, in order to solve the above technical problems, it is necessary to provide an update message processing method, a data synchronization method, and a configuration information configuration method, which can avoid repeated operations caused by repeated data update messages, reduce resource utilization, and improve processing efficiency.

A database update message processing method comprises the following steps:

acquiring a data updating message of a first database, wherein the data updating message carries a data table identifier;

inquiring the pre-configured configuration information to obtain an update identifier corresponding to the data table identifier;

judging whether a data updating message corresponding to the updating identification exists or not;

and when the data updating message corresponding to the updating identification exists, deleting the data updating message.

In one embodiment, the querying the preconfigured configuration information to obtain the update identifier corresponding to the data table identifier includes:

inquiring whether the pre-configured configuration information has the data table identification or not;

when the data table identification exists, acquiring a data model identification corresponding to the data table identification in the pre-configured configuration information;

determining whether the data table corresponding to the data table identification is a main table according to pre-configured configuration information;

when the data table corresponding to the data table identification is a main table, generating an updating identification according to the data model identification and a main key value of the main table;

and when the data table corresponding to the data table identification is a slave table, generating an updating identification according to the data model identification and the external key value of the slave table.

In one embodiment, after the determining whether there is a data update message corresponding to the update identifier, the method further includes:

and when the data updating message corresponding to the updating identification does not exist, establishing the corresponding relation between the data updating message and the updating identification.

In one embodiment, the establishing the correspondence between the data update message and the update identifier includes:

obtaining the effective time of an updating identifier corresponding to the identifier of the data table in the pre-configured configuration information;

and establishing the corresponding relation between the data updating message and the updating identification according to the effective time of the updating identification.

A data synchronization method, the data synchronization method comprising:

acquiring a data updating message obtained by the database updating message processing method in any one of the embodiments;

acquiring updating data from a first database according to the data updating message;

performing data conversion on the updated data according to the pre-configured configuration information to obtain target data;

and saving the target data to a second database.

In one embodiment, the data conversion of the update data according to the preconfigured configuration information to obtain the target data includes:

analyzing the updated data according to a pre-stored template to obtain target data; and/or

And performing data conversion on the updated data according to the pre-stored field corresponding relation to obtain target data.

A method for configuring database configuration information, the method comprising:

receiving a connection information configuration instruction for a first database, and configuring the connection information of the first database according to the connection information configuration instruction of the first database;

receiving a configuration instruction for the association relationship of each data table in the first database, and configuring each data table in the first database according to the configuration instruction for the association relationship to determine an update identifier corresponding to each data table identifier and valid time of the update identifier, where the update identifier and the valid time of the update identifier are used in the database update message processing method in any one of the embodiments.

In one embodiment, the configuring, according to the configuration instruction of the association relationship, each data table in the first database includes:

configuring data model identifications of the data models in the first database, effective time of update identifications corresponding to the data model identifications, and primary tables and secondary tables in the data models, and configuring primary key values of the primary tables and foreign key values of the secondary tables.

A method for configuring database configuration information, the method comprising:

receiving a connection information configuration instruction aiming at a second database, and configuring the connection information of the second database according to the connection information configuration instruction of the first database;

receiving a conversion rule configuration instruction aiming at the data in the second database and the data in the first database, and configuring a conversion rule of the data in the second database and the data in the first database according to the conversion rule configuration instruction; the conversion rule is for use in the data synchronization method in any of the above embodiments.

A database update message processing apparatus, the database update message processing apparatus comprising:

the first update message acquisition module is used for acquiring a data update message of a first database, wherein the data update message carries a data table identifier;

the query module is used for querying the pre-configured configuration information to obtain the update identification corresponding to the data table identification;

the judging module is used for judging whether a data updating message corresponding to the updating identification exists or not;

and the deleting module is used for deleting the data updating message when the data updating message corresponding to the updating identification exists.

A data synchronization apparatus, the data synchronization apparatus comprising:

a second update message acquisition module, configured to acquire a data update message obtained by the database update message processing apparatus according to any of the embodiments;

the update data acquisition module is used for acquiring update data from a first database according to the data update message;

the conversion module is used for performing data conversion on the updated data according to the pre-configured configuration information to obtain target data;

and the storage module is used for storing the target data to a second database.

A database configuration information configuration apparatus, the database configuration information configuration apparatus comprising:

the first receiving module is used for receiving a connection information configuration instruction aiming at a first database and configuring the connection information of the first database according to the connection information configuration instruction of the first database;

a second receiving module, configured to receive a configuration instruction for an association relationship of each data table in the first database, and configure each data table in the first database according to the configuration instruction for the association relationship, so as to determine an update identifier corresponding to each data table identifier and valid time of the update identifier, where the valid time of the update identifier and the update identifier are used in the database update message processing apparatus in any one of the embodiments.

A database configuration information configuration apparatus, the database configuration information configuration apparatus comprising:

a third receiving module, configured to receive a connection information configuration instruction for a second database, and configure connection information of the second database according to the connection information configuration instruction of the first database;

a fourth receiving module, configured to receive a conversion rule configuration instruction for data in the second database and data in the first database, and configure a conversion rule between the data in the second database and the data in the first database according to the conversion rule configuration instruction; the conversion rule is for use in the data synchronization apparatus in any of the above embodiments.

A computer device comprising a memory storing a computer program and a processor implementing the steps of the method as described in any one of the above embodiments when the processor executes the computer program.

A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method as set forth in any one of the above embodiments.

According to the update message processing method, the data synchronization method and the configuration information configuration method, after the data update message of the first database is received, the update identifier corresponding to the data table identifier is inquired, whether the update identifier is valid or not is judged, if yes, the data update message is deleted, the situation that the same data is subjected to update processing for many times is avoided, and the processing efficiency is improved.

Drawings

FIG. 1 is a diagram of an application environment of an update message processing method, a data synchronization method, and a configuration information configuration method according to an embodiment;

FIG. 2 is a flow diagram illustrating a method for update message processing according to one embodiment;

FIG. 3 is a flow diagram illustrating a method for data synchronization in one embodiment;

FIG. 4 is a flowchart illustrating a method for configuring configuration information according to an embodiment;

FIG. 5 is a flow chart illustrating a method for configuring configuration information in accordance with another embodiment;

FIG. 6 is a schematic diagram of a data synchronization system for structurally diverse databases, in one embodiment;

FIG. 7 is a block diagram of an update message processing apparatus in one embodiment;

FIG. 8 is a block diagram showing the structure of a data synchronization apparatus according to an embodiment;

FIG. 9 is a block diagram of a configuration information configuring apparatus according to an embodiment;

FIG. 10 is a block diagram showing a configuration of a further configuration information configuring apparatus according to another embodiment;

FIG. 11 is a diagram illustrating an internal structure of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The update message processing method, the data synchronization method and the configuration information configuration method provided by the application can be applied to the application environment shown in fig. 1. Wherein the servers 102 are in communication with the first database 104 and the second database 106, respectively. The server 102 acquires a data update message of the first database 104, wherein the data update message carries a data table identifier; inquiring the pre-configured configuration information to obtain an update identifier corresponding to the data table identifier; judging whether a data updating message corresponding to the updating identification exists or not; and when the data updating message corresponding to the updating identification exists, deleting the data updating message. In addition, the server 102 acquires the update data from the first database 104 according to the integrated data update message; performing data conversion on the updated data according to the pre-configured configuration information to obtain target data; the target data is saved to the second database 106. The above implementation process depends on the preset configuration information of the server 102, which includes the connection information of the first database 104 and the second database 106, the processing rule of the data update message of the first database 104, and the data conversion rule of the second database 106 and the first database 104. Therefore, after the data updating message of the first database is received, the updating identification corresponding to the data table identification is inquired, whether the updating identification is effective or not is judged, if yes, the data updating message is deleted, the situation that the same data is updated for multiple times is avoided, and the processing efficiency is improved. The server 102 may be implemented as a stand-alone server or as a server cluster comprised of multiple servers. There are differences in the structure of the first database 104 and the second database 106, wherein optionally the first database 104 is a relational database such as MySQL (which is a relational database management system that stores data in different tables), and the second database 106 is a new database such as ES (elastic search, which is a distributed, highly extended, high real-time search and data analysis engine.

In one embodiment, as shown in fig. 2, a database update message processing method is provided, which is described by taking the method as an example applied to the server in fig. 1, and includes the following steps:

s202: and acquiring a data updating message of the first database, wherein the data updating message carries the data table identifier.

Specifically, the data update message of the first database is obtained by the server by monitoring the first database. Optionally, the server may monitor the first database by setting a monitoring module, store the monitored data update message, and then read the stored data update message by another thread for processing.

In practical application, a Canal (which is a tool for analyzing incremental logs based on a database, monitors changes of data and generates messages, and currently mainly supports MySQL database.) is used to monitor whether a first database has data update, if so, generate a data update message, store the generated data update message into a cluster, for example, into a kafka (which is a high-throughput distributed publish-subscribe message system) cluster, and a subsequent server acquires the data update message from the cluster through another thread, processes the data update message, obtains an integrated data update message, and stores the integrated data update message into another cluster, for example, into another kafka cluster.

Specifically, the data table identifier refers to a unique identifier of the data table in the first database, which may be represented by the name of the data table or by both the data model and the name of the data table.

It should be noted that the server is preconfigured with connection information of the first database, so that the server can connect to the corresponding first database according to the connection information of the first database, and can monitor data update messages of the first database, and further, the server is preconfigured with a library name, a database type, an IP, a port number, a user name, a password (encrypted storage), and the like, and the server is connected to the first database through the information, so that security of data in the first database is ensured.

S204: and querying the pre-configured configuration information to obtain an update identifier corresponding to the data table identifier.

Specifically, the configuration information is pre-configured in the server, and includes connection information of the first database, association configuration in the first database, and processing rule of the data update message of the first database. The association configuration refers to the number of the data model in the first database, the correspondence between the data model and the tables, the fields to be taken by each table, which table is the primary table, the association relationship (primary key, foreign key) between the tables, and the like. The processing rule of the data updating message is generated according to the association configuration, for example, the processing rule and the data model number are one object, and the processing delay time of the same group of data updating messages is set at the same time.

The server inquires the configuration information to obtain the data model identification corresponding to the data table identification, so that the updated data is uniquely determined according to the data model identification and the corresponding key, namely the change identification is determined according to the data model identification and the corresponding key.

S206: and judging whether a data updating message corresponding to the updating identification exists.

S208: and when the data updating message corresponding to the updating identification exists, deleting the data updating message.

Specifically, the server determines whether a data update message corresponding to the update identifier exists, and if the data update message currently exists, it indicates that the data change information is already stored, and deletes the current data update message to avoid duplication.

In one embodiment, when there is no data update message corresponding to the update identifier, a corresponding relationship between the data update message and the update identifier is established, that is, a new integrated data update message is generated, so as to lay a foundation for subsequent message processing.

In order to avoid deleting the message of the subsequent data re-change, in this embodiment, the valid time of the data update message, that is, the delay time in the configuration information, that is, the valid time of the data update message is set after the data update message is stored each time, and in the valid time, if the data update message that is repeated with the data update message exists, the data update message is deleted, and after the valid time passes, even if the data update message for the same data operation is received, the data update message is stored, so that the message of the subsequent data re-change is prevented from being deleted. The setting of the valid time may be manually preset, or may be set according to the data update time of the database, for example, updating one data, and if the time synchronized to the database is a, the delay time may be set according to a, so as to use the update of the one data as a data update message.

According to the update message processing method, after the data update message of the first database is received, the update identifier corresponding to the data table identifier is inquired, whether the update identifier is valid or not is judged, if yes, the data update message is deleted, namely the identifier already receives the data update message, the situation that the same data is subjected to update processing for many times is avoided, and the processing efficiency is improved.

In one embodiment, querying the preconfigured configuration information to obtain an update identifier corresponding to the data table identifier includes: inquiring whether the pre-configured configuration information has a data table identifier; when the data table identification exists, acquiring a data model identification corresponding to the data table identification in the pre-configured configuration information; determining whether the data table corresponding to the data table identification is a main table according to the pre-configured configuration information; when the data table corresponding to the data table identification is a main table, generating an updating identification according to the data model identification and a main key value of the main table; and when the data table corresponding to the data table identification is the slave table, generating an update identification according to the data model identification and the foreign key value of the slave table.

In one embodiment, the establishing the corresponding relationship between the data update message and the update identifier includes: obtaining the effective time of an update identifier corresponding to the identifier of the data table in the pre-configured configuration information; and establishing a corresponding relation between the data updating message and the updating identification according to the effective time of the updating identification.

Specifically, the data update message completely corresponds to the data change operation of the first database one to one, which is exemplified by: a client table customer, associated with phone table phone and address table address, and assuming that the client information is stored and a piece of data is inserted in 3 tables of the customer table phone and address table address, the server will generate 3 pieces of data update messages. The server does not need to repeatedly process each data updating message, and the data only needs to be synchronized once. Therefore, the server merges the data update messages to generate an integrated message: the server acquires and configures 'first database table association configuration' and corresponding 'processing delay time length', when the server acquires a data updating message, the server acquires a data table name of the data updating message, when the data indicates that a certain table in 'first database table association configuration' is matched, firstly, whether the table in the data updating message is a main table is judged, if the table is the main table, the main key name of the main table in the configuration is acquired and the main key value is acquired from the data updating message, if the table is not the main table, the foreign key name in the configuration is acquired and the foreign key value is acquired from the data updating message, then, a lock with a timing failure is established by taking the data model number and the main key value or the foreign key value as dimensions, the time of the lock failure is the 'processing delay time length' of the configuration, subsequent messages are directly abandoned if the subsequent messages collide with the lock according to the method, and the first data updating message is processed, other data updating messages in the processing delay time length are discarded later, and the same data updating message which is created by taking the data model number and the primary key value or the foreign key value as the dimension is the final data updating message.

In one embodiment, as shown in fig. 3, a data synchronization method is provided, which is described by taking the application of the method to the server in fig. 1 as an example, and includes the following steps:

s302: and acquiring the data updating message obtained by the database updating message processing method in any embodiment.

Specifically, the data update message may be obtained as described above, and is not described herein again.

S304: and acquiring the updating data from the first database according to the data updating message.

Specifically, the server acquires update data from the first database according to the data update message, for example, acquires connection information of the first database according to the configuration information, then connects the server to the first database, and acquires the update data according to the data update message after the server successfully connects the first database.

S306: and performing data conversion on the updated data according to the pre-configured configuration information to obtain target data.

S308: and saving the target data to a second database.

Specifically, the conversion of the updated data is performed to make the updated data meet the requirement of the second database, and the server converts the updated data according to a pre-configured data conversion rule to obtain the target data.

Optionally, performing data conversion on the update data according to the preconfigured configuration information to obtain the target data, including: analyzing the updated data according to a pre-stored template to obtain target data; and/or performing data conversion on the updated data according to the pre-stored field corresponding relation to obtain target data.

That is, the server may support two conversion methods, one according to the FreeMarker (a template engine, a general tool that is based on templates and data to be changed and used to generate output text (HTML web pages, e-mails, configuration files, source codes, etc.)) template parsing method, and the other according to the field correspondence and saved in the second database.

In the embodiment, the data updating messages are integrated, so that only one data updating message is generated for updating the same data, and only one operation is needed when the data of the second database is synchronized subsequently, thereby reducing the repetition times, reducing the resource occupation and improving the processing efficiency.

In one embodiment, as shown in fig. 4, a database configuration information configuration method is provided, which is described by taking the method as an example applied to the server in fig. 1, and includes the following steps:

s402: and receiving a connection information configuration instruction aiming at the first database, and configuring the connection information of the first database according to the connection information configuration instruction of the first database.

Specifically, this step is mainly to configure the connection information of the first database, wherein the server may provide a "database connection information" management page: first database connection information is managed. The connection configuration can be newly added by clicking the "newly added database connection" button. The initialization shows all the historically configured database connection information (database name, database type, user name), and can be edited by clicking on a historical one of the configurations.

The first database connection information may include a library name, a database type, an IP, a port number, a user name, a password (encrypted storage), and the like. The first database connection information is used for connecting the first database to inquire the latest data. Example (c): the database name A, the database type MySQL, the IP address 10.1.1.1, the port number 1521, the user name queryuser and the password 123.

S404: receiving a configuration instruction for the association relationship of each data table in the first database, and configuring each data table in the first database according to the configuration instruction for the association relationship to determine an update identifier corresponding to each data table identifier and valid time of the update identifier, where the update identifier and the valid time of the update identifier are used in the database update message processing method in any one of the embodiments.

Specifically, this step is mainly the management of the data model of the first database and the configuration of the processing rules for the data update messages.

For the management of the data model, the server can provide a data model management page, newly add the data model configuration by clicking a 'newly added data model' button, create a data model number, fill in description, and guide a user to complete the configuration of 'data acquisition', 'data conversion and synchronization'. The page initialization shows the business model information (data model number, description and the like), and the details can be viewed and edited by clicking a button.

Optionally, configuring, for a processing rule of the data update message, that is, configuring each data table in the first database according to the configuration instruction of the association relationship, includes: configuring data model identifications of the data models in the first database, effective time of update identifications corresponding to the data model identifications, a master table and a slave table in the data models, and configuring master key values of the master table and external key values of the slave table.

The server may provide a data acquisition configuration page to configure the association relationship of the first database table and the processing rule of the data update message, where the configuration rule includes the delay time.

The management of the data model may include a first database table association configuration, including but not limited to a data model number, tables that the data model contains, table correspondence for each table and database, which fields each table is to take, which table is a primary table, and associations (primary key, foreign key) between tables.

For example, the serial number of the configuration data model is RLT0000000001, a client table customer exists in a library A, and fields such as a client ID (main key), a name, a certificate type, a certificate number and the like exist; the related telephone table phone has a B library, and the B library has fields such as a main key ID, a client ID, a mobile phone number, input time and the like; the address table address is stored in a B library and comprises fields such as a main key ID, province, city, district, detailed address and the like; the main table is the customer, and the main key of the customer table customer with the customer ID corresponding to the external key is stored in the telephone table phone and the address table address.

In one embodiment, as shown in fig. 5, a database configuration information configuration method is provided, which is described by taking the method as an example applied to the server in fig. 1, and includes the following steps:

s502: and receiving a connection information configuration instruction aiming at the second database, and configuring the connection information of the second database according to the connection information configuration instruction of the first database.

Specifically, this step is mainly to configure the connection information of the second database, wherein the server may provide a "database connection information" management page: managing the second database connection information. The connection configuration can be newly added by clicking the "newly added database connection" button. The initialization shows all the historically configured database connection information (database name, database type, user name), and can be edited by clicking on a historical one of the configurations.

The second database connection information includes a library name, a database type, an IP, a port number, a user name, a password, and the like. The second database connection information is used to connect to the second database.

A second database data structure: the second database is created with a generally flattened data structure. Data model numbers, library names, table names and field information need to be set, and the field information comprises: field names, field types, index types, word segmenters, etc.

Example (c): creating a customer table in the ES, setting a field name as a text type, supporting an index, and using a participler IKAnalyzer; the phones and addresses are set as the type of nested.

S504: receiving a conversion rule configuration instruction aiming at the data in the second database and the data in the first database, and configuring a conversion rule of the data in the second database and the data in the first database according to the conversion rule configuration instruction; the conversion rule is for use in the data synchronization method in any of the above embodiments.

Specifically, a data conversion rule configuration page may be further provided in the server, so as to be used for the user to configure the data conversion rule.

Specifically, the data conversion rules are in one-to-one correspondence with the data model numbers, describing the correspondence between the fields in the tables in the first database and the fields in the second database. And two modes of Freemarker template analysis mode conversion and field corresponding relation conversion are supported.

Example (c): the NAME field in the CUSTOMER table in the library A (MySQL) corresponds to the NAME field in the CUSTOMER table in the library B (ES), and then the NAME field in the CUSTOMER table in the corresponding configuration library A (MySQL) corresponds to the NAME field in the CUSTOMER table in the library B (ES); and the CONTENT field of the A library PHONE table corresponds to the CONTENT field in the nested field PHONEs in the B library customer table, and the CONTENT field of the A library PHONE table is correspondingly configured to correspond to the nested field PHONEs in the B library customer table.

Specifically, referring to fig. 6, fig. 6 is a schematic diagram of a data synchronization system of databases with different structures in an embodiment.

In this embodiment, configuration information is first configured through the background management module, where the configuration information includes connection information of the first database, connection information of the second database, table association information of the first database, a data update message processing rule of the first database, structure information of the second database, and a data conversion rule of the first database and the second database.

Secondly, monitoring data change of a first database through the Canal, generating a data updating message, storing the data updating message into the kafka cluster, so that a message processing module reads the data updating message from the kafka cluster and processes the data updating message according to a data updating message processing rule, wherein the data updating message comprises inquiring whether a data table identifier in the data updating message exists or not, if the data table identifier does not exist, outputting an alarm, if the data table identifier exists, acquiring a data model identifier corresponding to the data table identifier and a corresponding main key or external key, and further reading a main key value or an external key value from the data updating message, wherein the main key and the external key are mainly determined according to whether a data table corresponding to the data table identifier is the main table or not, if the main table identifier exists, acquiring the main key, and if the main table identifier is not, acquiring the auxiliary key, so that the message processing module calculates an updating identifier corresponding to the data updating message, namely, the data model identifier and the key value, and judges whether the data updating message corresponding to the updating identifier exists at present, if so, the corresponding data updating message already exists, and the new data updating message is deleted, namely, the data updating message is integrated. If the update information does not exist, the data update information is stored in another kafka, wherein the information processing module can also obtain the delay time of the update identifier so as to set the effective time of the update identifier, and after the effective time is over, the data update information and the update identifier are deleted, and the subsequent data update information is continuously obtained.

In addition, the data processing module acquires the integrated data updating message from the kafka regularly or according to user triggering, acquires the updating data from the first database according to the data updating message, converts the updating data according to the data conversion rules of the first database and the second database to obtain target data, and stores the target data in the second database, so that the data can be inquired from the second database subsequently.

In the above embodiment, data synchronization work between different types of databases may be implemented, for example: MySQL data is synchronized into the ES. Only configuration (0 encoding) can convert the data into the required structure and save the structure in the target database, such as: and converting the data associated with the multiple tables into the data with a flat structure and saving the data in the ES. And finally, the number of times of synchronous processing of repeated data is reduced, and the system performance is improved.

It should be understood that, although the steps in the flowcharts of fig. 2 to 5 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2 to 5 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least some of the other steps or stages.

In one embodiment, as shown in fig. 7, there is provided a database update message processing apparatus including: a first update message obtaining module 701, an inquiring module 702, a judging module 703 and a deleting module 704, wherein:

a first update message obtaining module 701, configured to obtain a data update message of a first database, where the data update message carries a data table identifier;

a query module 702, configured to query preconfigured configuration information to obtain an update identifier corresponding to the data table identifier;

a determining module 703, configured to determine whether a data update message corresponding to the update identifier already exists;

a deleting module 704, configured to delete the data update message when the data update message corresponding to the update identifier already exists.

In one embodiment, the query module 702 includes:

the query unit is used for querying whether the pre-configured configuration information has a data table identifier or not;

the data model identification obtaining unit is used for obtaining a data model identification corresponding to the data table identification in the pre-configured configuration information when the data table identification exists;

the judging unit is used for determining whether the data table corresponding to the data table identification is a main table according to the pre-configured configuration information;

the updating identifier generating unit is used for generating an updating identifier according to the data model identifier and the primary key value of the primary table when the data table corresponding to the data table identifier is the primary table; and when the data table corresponding to the data table identification is the slave table, generating an update identification according to the data model identification and the foreign key value of the slave table.

In one embodiment, the database update message processing apparatus further includes:

and the establishing module is used for establishing the corresponding relation between the data updating message and the updating identification when the data updating message corresponding to the updating identification does not exist.

In one embodiment, the establishing module includes:

the time acquisition unit is used for acquiring the effective time of the update identifier corresponding to the data table identifier in the pre-configured configuration information;

and the establishing unit is used for establishing the corresponding relation between the data updating message and the updating identifier according to the effective time of the updating identifier.

In one embodiment, as shown in fig. 8, there is provided a data synchronization apparatus including: a second update message obtaining module 801, an update data obtaining module 802, a conversion module 803, and a storage module 804, wherein:

a second update message obtaining module 801, configured to obtain a data update message obtained by the database update message processing apparatus according to any of the embodiments described above;

an update data obtaining module 802, configured to obtain update data from a first database according to the data update message;

a conversion module 803, configured to perform data conversion on the update data according to the preconfigured configuration information to obtain target data;

and the storage module 804 is used for storing the target data to the second database.

In one embodiment, the conversion module 803 is further configured to analyze the updated data according to a pre-stored template to obtain target data; and/or performing data conversion on the updated data according to the pre-stored field corresponding relation to obtain target data.

In one embodiment, as shown in fig. 9, there is provided a database configuration information configuring apparatus including: a first receiving module 901 and a second receiving module 902, wherein:

a first receiving module 901, configured to receive a connection information configuration instruction for a first database, and configure connection information of the first database according to the connection information configuration instruction of the first database;

a second receiving module 902, configured to receive a configuration instruction of an association relation for each data table in the first database, and configure each data table in the first database according to the configuration instruction of the association relation, so as to determine an update identifier corresponding to each data table identifier and valid time of the update identifier, where the update identifier and the valid time of the update identifier are used in the database update message processing apparatus in any of the embodiments.

In one embodiment, the second receiving module 902 is further configured to configure a data model identifier of each data model in the first database, a valid time of an update identifier corresponding to the data model identifier, and a primary table and a secondary table in each data model, and configure a primary key value of the primary table and a foreign key value of the secondary table.

In one embodiment, as shown in fig. 10, there is provided a database configuration information configuring apparatus including: a third receiving module 1001 and a fourth receiving module 1002, wherein:

a third receiving module 1001, configured to receive a connection information configuration instruction for a second database, and configure connection information of the second database according to the connection information configuration instruction of the first database;

a fourth receiving module 1002, configured to receive a conversion rule configuration instruction for data in the second database and data in the first database, and configure a conversion rule between the data in the second database and the data in the first database according to the conversion rule configuration instruction; the conversion rule is for use in the data synchronization apparatus in any of the above embodiments.

For specific limitations of the update message processing apparatus, the data synchronization apparatus, and the configuration information configuration apparatus, reference may be made to the above limitations of the update message processing method, the data synchronization method, and the configuration information configuration method, which are not described herein again. The modules in the update message processing device, the data synchronization device and the configuration information configuration device may be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 11. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for caching data update messages. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an update message processing method, a data synchronization method, a configuration information configuration method.

Those skilled in the art will appreciate that the architecture shown in fig. 11 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: acquiring a data updating message of a first database, wherein the data updating message carries a data table identifier; inquiring the pre-configured configuration information to obtain an update identifier corresponding to the data table identifier; judging whether a data updating message corresponding to the updating identification exists or not; and when the data updating message corresponding to the updating identification exists, deleting the data updating message.

In one embodiment, querying preconfigured configuration information to obtain an update identification corresponding to a data table identification, as implemented by a processor executing a computer program, comprises: inquiring whether the pre-configured configuration information has a data table identifier; when the data table identification exists, acquiring a data model identification corresponding to the data table identification in the pre-configured configuration information; determining whether the data table corresponding to the data table identification is a main table according to the pre-configured configuration information; when the data table corresponding to the data table identification is a main table, generating an updating identification according to the data model identification and a main key value of the main table; and when the data table corresponding to the data table identification is the slave table, generating an update identification according to the data model identification and the foreign key value of the slave table.

In one embodiment, the determining whether the data update message corresponding to the update identifier already exists is further performed by a processor executing the computer program, and the determining further includes: and when the data updating message corresponding to the updating identifier does not exist, establishing the corresponding relation between the data updating message and the updating identifier.

In one embodiment, the establishing of the correspondence between the data update message and the update identifier, which is implemented when the processor executes the computer program, includes: obtaining the effective time of an update identifier corresponding to the identifier of the data table in the pre-configured configuration information; and establishing a corresponding relation between the data updating message and the updating identification according to the effective time of the updating identification.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: acquiring a data updating message obtained by the database updating message processing method in any one of the embodiments; acquiring updating data from a first database according to the data updating message; performing data conversion on the updated data according to the pre-configured configuration information to obtain target data; and saving the target data to a second database.

In one embodiment, the data conversion of the update data according to the pre-configured configuration information to obtain the target data, which is implemented when the processor executes the computer program, includes: analyzing the updated data according to a pre-stored template to obtain target data; and/or performing data conversion on the updated data according to the pre-stored field corresponding relation to obtain target data.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: receiving a connection information configuration instruction aiming at a first database, and configuring the connection information of the first database according to the connection information configuration instruction of the first database; receiving a configuration instruction for the association relationship of each data table in the first database, and configuring each data table in the first database according to the configuration instruction for the association relationship to determine an update identifier corresponding to each data table identifier and valid time of the update identifier, where the update identifier and the valid time of the update identifier are used in the database update message processing method in any one of the embodiments.

In one embodiment, the configuring, according to the association relation, of each data table in the first database by the configuration instruction implemented when the processor executes the computer program includes: configuring data model identifications of the data models in the first database, effective time of update identifications corresponding to the data model identifications, a master table and a slave table in the data models, and configuring master key values of the master table and external key values of the slave table.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: receiving a connection information configuration instruction aiming at a second database, and configuring the connection information of the second database according to the connection information configuration instruction of the first database; receiving a conversion rule configuration instruction aiming at the data in the second database and the data in the first database, and configuring a conversion rule of the data in the second database and the data in the first database according to the conversion rule configuration instruction; the conversion rule is for use in the data synchronization method in any of the above embodiments.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: acquiring a data updating message of a first database, wherein the data updating message carries a data table identifier; inquiring the pre-configured configuration information to obtain an update identifier corresponding to the data table identifier; judging whether a data updating message corresponding to the updating identification exists or not; and when the data updating message corresponding to the updating identification exists, deleting the data updating message.

In one embodiment, querying preconfigured configuration information to obtain an update identification corresponding to a data table identification, implemented when the computer program is executed by the processor, comprises: inquiring whether the pre-configured configuration information has a data table identifier; when the data table identification exists, acquiring a data model identification corresponding to the data table identification in the pre-configured configuration information; determining whether the data table corresponding to the data table identification is a main table according to the pre-configured configuration information; when the data table corresponding to the data table identification is a main table, generating an updating identification according to the data model identification and a main key value of the main table; and when the data table corresponding to the data table identification is the slave table, generating an update identification according to the data model identification and the foreign key value of the slave table.

In one embodiment, the determining whether the data update message corresponding to the update identification already exists, when performed by the processor, further comprises: and when the data updating message corresponding to the updating identifier does not exist, establishing the corresponding relation between the data updating message and the updating identifier.

In one embodiment, the establishing of the correspondence between the data update message and the update identifier, which is implemented when the computer program is executed by the processor, includes: obtaining the effective time of an update identifier corresponding to the identifier of the data table in the pre-configured configuration information; and establishing a corresponding relation between the data updating message and the updating identification according to the effective time of the updating identification.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: acquiring a data updating message obtained by the database updating message processing method in any one of the embodiments; acquiring updating data from a first database according to the data updating message; performing data conversion on the updated data according to the pre-configured configuration information to obtain target data; and saving the target data to a second database.

In one embodiment, the data conversion of the update data according to the pre-configured configuration information to obtain the target data, which is implemented when the computer program is executed by the processor, includes: analyzing the updated data according to a pre-stored template to obtain target data; and/or performing data conversion on the updated data according to the pre-stored field corresponding relation to obtain target data.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: receiving a connection information configuration instruction aiming at a first database, and configuring the connection information of the first database according to the connection information configuration instruction of the first database; receiving a configuration instruction for the association relationship of each data table in the first database, and configuring each data table in the first database according to the configuration instruction for the association relationship to determine an update identifier corresponding to each data table identifier and valid time of the update identifier, where the update identifier and the valid time of the update identifier are used in the database update message processing method in any one of the embodiments.

In one embodiment, the configuring of each data table in the first database according to the configuration instructions of the association relationship, which are implemented when the computer program is executed by the processor, includes: configuring data model identifications of the data models in the first database, effective time of update identifications corresponding to the data model identifications, a master table and a slave table in the data models, and configuring master key values of the master table and external key values of the slave table.

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: receiving a connection information configuration instruction aiming at a second database, and configuring the connection information of the second database according to the connection information configuration instruction of the first database; receiving a conversion rule configuration instruction aiming at the data in the second database and the data in the first database, and configuring a conversion rule of the data in the second database and the data in the first database according to the conversion rule configuration instruction; the conversion rule is for use in the data synchronization method in any of the above embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.