1. A method of data derivation, comprising:

configuring the matching relation between the source list and the target list and the data verification rule to generate a configuration file;

reading the configuration file, responding to a data selection instruction, and determining a target data range in a database;

screening out invalid data which do not accord with the data verification rule in the target data range;

treating the invalid data to ensure that all data in the target data range accord with the data verification rule to obtain target data;

executing a conversion instruction, and matching the format of the target data with the target list; and

and executing a derivation instruction to derive the target data matched with the target list.

2. The data derivation method according to claim 1, wherein the configuring of the matching relationship between the source table column and the target table column and the data verification rule to generate the configuration file comprises:

forming a template file according to the target system data;

screening out a target list according to the template file;

screening a source table column matched with the target table column from a source library, and configuring a matching relation between the source table column and the target table column; and

and configuring a data verification rule according to the target list.

3. The data derivation method of claim 1, wherein the target data range comprises: target data interval, target table list format, and attachment status.

4. The data export method of claim 1, wherein the remedying the invalid data comprises:

and correcting or deleting the invalid data.

5. The data export method of claim 3, prior to said executing an export instruction, further comprising:

storing the accessory data in the form of binary data according to the accessory state, and generating an accessory export instruction containing the accessory data.

6. The data export method of claim 5, after said executing an export instruction to export the target data that matches the target table column, further comprising:

the result of the data derivation is displayed,

wherein the data derivation result comprises: the target data and the accessory data that match the target table column.

7. A data export system, comprising:

the configuration module is used for configuring the matching relation between the source list and the target list and the data verification rule to generate a configuration file;

the range determining module is used for reading the configuration file, responding to a data selection instruction and determining a target data range in a database;

the cleaning module is used for screening out invalid data which do not accord with the data verification rule in the target data range;

the treatment module is used for treating the invalid data to ensure that all data in the target data range accord with the data verification rule to obtain target data;

the conversion module is used for executing a conversion instruction and matching the format of the target data with the target list; and

and the export module is used for executing an export instruction and exporting the target data matched with the target list.

8. The data export system of claim 7, wherein the configuration module executes in a manner comprising:

forming a template file according to the target system data;

screening out a target list according to the template file;

screening a source table column matched with the target table column from a source library, and configuring a matching relation between the source table column and the target table column; and

and configuring a data verification rule according to the target list.

9. The data derivation system of claim 7, wherein the target data range comprises: target data interval, target table list format, and attachment status.

10. The data export system of claim 9, further comprising:

and the accessory processing module is used for storing the accessory data in a binary data form according to the accessory state and generating an accessory export instruction containing the accessory data.

Background

Databases are an important foundation for information resource management in modern society. For convenience of data management and viewing, data in a Database is usually exported by a DBA (Database Administrator) to be a file of other format that accommodates different systems. In daily maintenance and management of the database, the main export mode is to export the data in the database by executing export instructions carried by the database.

However, the current data export mode is mainly from the perspective of DBA, and the degree of wedge with the current service is not high, so that the exported data cannot be directly applicable to a required system, and the exported data needs to be secondarily processed by a user, so that the threshold of data export of a database is increased, and the operation of a common user is inconvenient. The current data export mode can not process and screen the data, and only can realize uniform data export. In addition, the Data export method cannot export the Data stored in the database in the form of an attachment to a file of a DMP (Data Management Platform) at a time, which is not favorable for efficient Data transfer and utilization.

Disclosure of Invention

The present application provides a data export method and system that seeks to address or partially address at least one of the above-mentioned problems with the background or other deficiencies in the art.

The application provides a data export method, which comprises the following steps:

configuring the matching relation between the source list and the target list and the data verification rule to generate a configuration file;

reading the configuration file, responding to a data selection instruction, and determining a target data range in a database;

screening invalid data which do not accord with the data verification rule in a target data range;

processing the invalid data to ensure that all data in the target data range accord with the data verification rule to obtain target data;

executing a conversion instruction, and matching the format of the target data with the target list; and

and executing a derivation instruction to derive the target data matched with the target list.

In some embodiments, configuring the matching relationship between the source table column and the target table column and the data verification rule, and generating a configuration file includes:

forming a template file according to the target system data;

screening out a target list according to the template file;

screening a source list matched with the target list from a source library, and configuring the matching relation between the source list and the target list; and

and configuring a data verification rule according to the target list.

In some embodiments, the target data range includes: target data interval, target table list format, and attachment status.

In some embodiments, remedying invalid data comprises: and correcting or deleting the invalid data.

In some embodiments, prior to executing the export instruction, further comprising:

the accessory data is stored in the form of binary data according to the accessory status, and an accessory export instruction containing the accessory data is generated.

In some embodiments, after executing the export instruction to export the target data matching the target list, the method further includes:

the result of the data derivation is displayed,

wherein the data derivation result comprises: target data that matches the target table list, and accessory data.

The present application also proposes a data export system comprising: the device comprises a configuration module, a range determination module, a cleaning module, a treatment module, a conversion module and a derivation module. The configuration module is used for configuring the matching relation between the source list and the target list and the data verification rule to generate a configuration file. The range determination module is used for reading the configuration file and responding to the data selection instruction to determine the target data range in the database. And the cleaning module is used for screening out invalid data which do not accord with the data verification rule in the target data range. And the treatment module is used for treating the invalid data to ensure that all data in the target data range accord with the data verification rule to obtain the target data. The conversion module is used for executing the conversion instruction and matching the format of the target data with the target list. And the export module is used for executing the export instruction and exporting the target data matched with the target list.

In some embodiments, the configuration module performs a method comprising:

forming a template file according to the target system data;

screening out a target list according to the template file;

screening a source list matched with the target list from a source library, and configuring the matching relation between the source list and the target list; and

and configuring a data verification rule according to the target list.

In some embodiments, the target data range includes: target data interval, target table list format, and attachment status.

In some embodiments, further comprising:

and the accessory processing module is used for storing the accessory data in a binary data form according to the accessory state and generating an accessory export instruction containing the accessory data.

According to the technical scheme of the embodiment, at least one of the following advantages can be obtained.

According to the data export method and the data export system, after the configuration file is set, the user can export the required target data only by sending the data selection instruction, and the method and the system have universality and lower use threshold. The method and the device can also export the accessory data together, and ensure the integrity of the data. In addition, a data verification rule is configured, so that invalid data can be effectively treated and then exported, and the effectiveness and the usability of exported target data are guaranteed.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings in which:

FIG. 1 is a flow diagram of a data export method according to an exemplary embodiment of the present application; and

fig. 2 is a schematic structural diagram of a data export system according to an exemplary embodiment of the present application.

Detailed Description

For a better understanding of the present application, various aspects of the present application will be described in more detail with reference to the accompanying drawings. It should be understood that the detailed description is merely illustrative of exemplary embodiments of the present application and does not limit the scope of the present application in any way. Like reference numerals refer to like elements throughout the specification. The expression "and/or" includes any and all combinations of one or more of the associated listed items.

In the drawings, the size, dimension, and shape of elements have been slightly adjusted for convenience of explanation. The figures are purely diagrammatic and not drawn to scale. As used herein, the terms "approximately", "about" and the like are used as table-approximating terms and not as table-degree terms, and are intended to account for inherent deviations in measured or calculated values that would be recognized by one of ordinary skill in the art. In addition, in the present application, the order in which the processes of the respective steps are described does not necessarily indicate an order in which the processes occur in actual operation, unless explicitly defined otherwise or can be inferred from the context.

It will be further understood that terms such as "comprising," "including," "having," "including," and/or "containing," when used in this specification, are open-ended and not closed-ended, and specify the presence of stated features, elements, and/or components, but do not preclude the presence or addition of one or more other features, elements, components, and/or groups thereof. Furthermore, when a statement such as "at least one of" appears after a list of listed features, it modifies that entire list of features rather than just individual elements in the list. Furthermore, when describing embodiments of the present application, the use of "may" mean "one or more embodiments of the present application. Also, the term "exemplary" is intended to refer to an example or illustration.

Unless otherwise defined, all terms (including engineering and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

In addition, the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Fig. 1 is a flowchart of a data export method according to an exemplary embodiment of the present application.

As shown in fig. 1, the present application provides a data export method, which may include:

and configuring the matching relation between the source list and the target list and the data verification rule to generate a configuration file. And reading the configuration file, responding to the data selection instruction, and determining a target data range in the database. And screening out invalid data which do not accord with the data verification rule in the target data range. And (4) processing the invalid data to ensure that all data in the target data range accord with the data verification rule to obtain the target data. And executing the conversion instruction to match the format of the target data with the target list. And executing the export instruction to export the target data matched with the target list.

Step S1, configure the matching relationship between the source table list and the target table list and the data verification rule, and generate a configuration file.

Specifically, first, a template file is formed from target system data. The target system data may include Excel format, and the like. In the present application, the specific format type of the target system data is not limited. In the present application, the template file may be a DMP template file. Further, the template file is imported, and the target list is screened out according to the template file. Further, a source list matched with the target list is screened out from the source library, and the matching relation between the source list and the target list is configured. In addition, the data checking rule is configured according to the target list. And further obtaining a configuration file comprising the matching relation between the source list and the target list and the data verification rule.

Specifically, the configuration of the data verification rule according to the target table list includes the following processes:

firstly, identifying the matching intention (match entry) of a target list and a source list, automatically analyzing and identifying the format of target system data and the standard requirement on the data according to the target list, and further determining the matching intention of the target list and the source list. For example, if the target system data is structured script data having data fields defined by script tags and data contents in a predetermined format, and the source library is unstructured text data, it may be determined that the matching intent is the structuralization of the unstructured data, that is, the source data defined by the matching data fields is extracted from the text data in the source library, and after performing the formatting process, the source data is filled as the data contents in the data fields. The mapping lookup table may be pre-stored, and a plurality of matching intents corresponding to the target table columns and the source table columns are preset in the lookup table, and the matching intents are obtained by referring to the mapping lookup table according to the target table columns and the source table columns. Or setting a trained binary classifier, and obtaining the matching intents corresponding to the target list and the source list through binary classification according to the target list and the source list.

And secondly, analyzing the data verification rule slot based on the identified matching intention. According to the identified matching intention, a data verification rule slot set by the matching intention can be called. For example, the above-mentioned structured matching intention of the unstructured data, the data verification rule set for it may include verification of correspondence of data content and data field, data content format integrity verification, data content format normalization verification, and the like.

And then writing the matching intention and the data verification rule slot into a configuration file, and calling the configuration file in the subsequent steps so as to obtain the executed matching intention and the data verification rule.

Further, the user can export the target data and the attachment data by a task starting mode, and the specific task starting step is as follows.

Step S2, reading the configuration file, and determining the target data range in the database in response to the data selection instruction.

The user can issue a data selection instruction at the client, specifically, the data selection instruction comprises selection of a target data interval, a target list format and an attachment state. Furthermore, the data selection instruction of the user is combined with the matching relation between the source list and the target list in the configuration file and the configuration file of the data verification rule, and the target list format can be determined in the source library according to the data selection instruction of the user.

Further, invalid data in the target data interval can be screened out according to the data verification rule, and the specific screening steps are as follows.

In step S3, invalid data that does not comply with the data verification rule is screened out in the target data range.

And comparing all data in the target data interval with the data verification rule, screening out invalid data which do not accord with the data verification rule, and recording the invalid data. In addition, valid data is determined in the target data interval and stored in the temporary user for subsequent invocation.

Further, aiming at invalid data, the application treats the invalid data for use, and the specific treatment steps are as follows.

And step S4, the invalid data is treated to ensure that all the data in the target data range accord with the data verification rule, and the target data is obtained.

Specifically, the manner of remedying the invalid data may include deletion or modification. In the application, the treatment mode can be selected according to the specific state of the invalid data, and if the invalid data can not be corrected to accord with the data verification rule, the invalid data can be deleted to ensure that all data have availability and validity in the target data range.

The invention executes the control management of data verification and treatment according to the matching intention and the data verification rule slot defined in the matching file. For the invalid data correction in step S3, analyzing whether the invalid data correction can pass the check defined by the data check rule slot position according to the data check rule, if the invalid data correction can pass the check defined by a certain data check rule slot position, filling the slot position, and if the invalid data correction cannot pass the check defined by the data check rule slot position, blank the slot position; if after one round of verification analysis, it is judged that a certain data verification rule slot position of the current invalid data is still blank, and the current invalid data is still not successfully corrected, the corresponding next round of correction can be executed aiming at the blank data verification rule slot position; for example, if the data content and data field correspondence check and the data content format integrity check slot are successfully filled, but the data content format normalization check slot is still blank, the adjustment and correction of the data content format normalization are performed in a targeted manner, and then whether the slot is filled or not is judged; if after multiple rounds of correction, a blank data check rule slot still exists, the blank data check rule slot can be deleted.

In step S5, a conversion instruction is executed to match the format of the target data with the target list.

Specifically, the conversion instruction may be responded to by way of an sql (Structured Query Language) script. sql is a high-level non-procedural programming language, so different database systems with completely different underlying structures can use the same structured query language as an interface for data entry and management. When the format of the target data is matched with the target list through the sql, the data export efficiency can be improved, and meanwhile, the flexibility of data conversion is improved.

Further, according to the accessory state in the target data range, for example, if there is accessory data to be exported, the accessory data is first stored in the form of binary data, and then an accessory export instruction containing the accessory data is generated. The accessory data is exported together through the subsequent exporting step, so that the integrity of the data is ensured

In step S6, a derivation instruction is executed to derive target data matching the target list.

After the export of the target data or the accessory data matched with the target list is completed, the data export result can be displayed. Specifically, the data derivation results include: and the target data and the attachment data matched with the target list are used for improving the visibility of data export and facilitating the user to visually check the export result.

Of course, the exported target data and the accessory data can be downloaded according to the requirement so as to be used by the user.

According to the data export method, after the configuration file is set, the user can export the required target data only by sending the data selection instruction. The method and the device can also export the accessory data together, and ensure the integrity of the data. In addition, a data verification rule is configured, so that invalid data can be effectively treated and then exported, and the effectiveness and the usability of exported target data are guaranteed.

Fig. 2 is a schematic structural diagram of a data export system according to an exemplary embodiment of the present application.

As shown in fig. 2, the present application further provides a data export system, including:

the configuration module 1 is used for configuring the matching relationship between the source list and the target list and the data verification rule to generate a configuration file;

the range determining module 2 is used for reading the configuration file, responding to the data selection instruction and determining a target data range in the database;

the cleaning module 3 is used for screening out invalid data which do not accord with the data verification rule in a target data range;

the treatment module 4 is used for treating the invalid data to ensure that all data in the target data range accord with the data verification rule to obtain target data;

the conversion module 5 is used for executing a conversion instruction and matching the format of the target data with the target list; and

and the export module 6 is used for executing the export instruction and exporting the target data matched with the target list.

In some embodiments, the execution of the configuration module 1 includes: forming a template file according to the target system data; screening out a target list according to the template file; screening a source list matched with the target list from a source library, and configuring the matching relation between the source list and the target list; and configuring a data verification rule according to the target list.

In some embodiments, specifically, the configuration module 1 performs configuration of the data check rule according to the target table list, including the following processes: firstly, identifying the matching intention (match entry) of a target list and a source list, automatically analyzing and identifying the format of target system data and the standard requirement on the data according to the target list, and further determining the matching intention of the target list and the source list. For example, if the target system data is structured script data having data fields defined by script tags and data contents in a predetermined format, and the source library is unstructured text data, it may be determined that the matching intent is the structuralization of the unstructured data, that is, the source data defined by the matching data fields is extracted from the text data in the source library, and after performing the formatting process, the source data is filled as the data contents in the data fields. The mapping lookup table may be pre-stored, and a plurality of matching intents corresponding to the target table columns and the source table columns are preset in the lookup table, and the matching intents are obtained by referring to the mapping lookup table according to the target table columns and the source table columns. Or setting a trained binary classifier, and obtaining the matching intents corresponding to the target list and the source list through binary classification according to the target list and the source list. And secondly, analyzing the data verification rule slot based on the identified matching intention. According to the identified matching intention, a data verification rule slot set by the matching intention can be called. For example, the above-mentioned structured matching intention of the unstructured data, the data verification rule set for it may include verification of correspondence of data content and data field, data content format integrity verification, data content format normalization verification, and the like. And then writing the matching intention and the data verification rule slot into a configuration file, and calling the configuration file in the subsequent steps so as to obtain the executed matching intention and the data verification rule.

In some embodiments, the abatement module 4 performs control management of data verification and abatement according to the matching intention and the data verification rule slot defined in the matching file. For invalid data correction, whether the invalid data correction can pass the verification defined by the data verification rule slot position is analyzed according to a data verification rule, if the invalid data correction can pass the rule verification defined by a certain data verification rule slot position, the slot position is filled, and if the invalid data correction cannot pass the verification of the data verification rule slot position, the slot position is blank; if after one round of verification analysis, it is judged that a certain data verification rule slot position of the current invalid data is still blank, and the current invalid data is still not successfully corrected, the corresponding next round of correction can be executed aiming at the blank data verification rule slot position; for example, if the data content and data field correspondence check and the data content format integrity check slot are successfully filled, but the data content format normalization check slot is still blank, the adjustment and correction of the data content format normalization are performed in a targeted manner, and then whether the slot is filled or not is judged; if after multiple rounds of correction, a blank data check rule slot still exists, the blank data check rule slot can be deleted.

In some embodiments, the target data range includes: target data interval, target table list format, and attachment status.

In some embodiments, further comprising: and the accessory processing module is used for storing the accessory data in a binary data form according to the accessory state and generating an accessory export instruction containing the accessory data.

Since the data export system of the present application is proposed based on the data export method, the related modules are all used for implementing the steps of the method, and specific working principles and procedures are not repeated herein, and reference may be made to the contents in the data export method.

According to the data export system, after the configuration file is set, the user can export the required target data only by sending the data selection instruction, and the data export system has universality and is low in use threshold. The method and the device can also export the accessory data together, and ensure the integrity of the data. In addition, a data verification rule is configured, so that invalid data can be effectively treated and then exported, and the effectiveness and the usability of exported target data are guaranteed.

The objects, technical solutions and advantageous effects of the present invention are further described in detail with reference to the above-described embodiments. It should be understood that the above description is only a specific embodiment of the present invention, and is not intended to limit the present invention. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.