1. A method of data storage, the method comprising a plurality of servers, the method further comprising:

grouping the servers to obtain a plurality of server groups, and grouping data to be stored according to a preset rule to obtain a plurality of data groups to be stored;

determining a storage mapping relation between the data group to be stored and the server group, and storing the data group to be stored on the corresponding server group according to the storage mapping relation;

determining a load value of each server group, and determining a first server group and a second server group from all the server groups according to the load values;

determining first target data stored on a first server group according to a preset first condition, and determining second target data stored on a second server group according to a preset second condition;

and updating the storage mapping relation, migrating the first target data to the second server group and migrating the second target storage data to the first server group.

2. The method of claim 1, wherein determining the load value for each server group comprises:

for each server group, determining a load index value of each server in the server group;

determining the load value of each server according to the load index value and the corresponding weight;

and determining the load value of the server group according to the load value of each server.

3. The method of claim 2, wherein the load indicator value comprises one or more of: cpu utilization, memory utilization, and interface response time.

4. The method of claim 2, wherein determining the first server group and the second server group from all the server groups according to the load value comprises:

taking the server group with the load value larger than or equal to a preset threshold value in all the server groups as a first server group;

determining a load average value according to the load value of each server group;

and taking the server group with the load value smaller than the load average value in all the server groups as a second server group.

5. The method of claim 4, wherein determining the first target data stored on the first server group according to the preset first condition comprises:

determining the amount of the first target data;

when the number of the first server groups is 1, acquiring first data from the first server groups, wherein the number of the first data is equal to the number of the first target data;

when the number of the first server groups is larger than 1, determining a difference value ratio between the first server groups according to the difference value between the load value of each first server group and the load average value; acquiring first data from each first server group according to the difference ratio, and taking the first data as first target data, wherein the quantity ratio of the first data is equal to the difference ratio, and the sum of the quantities of the first data is equal to the quantity of the first target data;

determining, according to a preset second condition, second target data stored on the second server group includes:

determining the quantity of second target data, wherein the quantity of the second target data is equal to the quantity of the first target data;

when the number of the second server groups is 1, acquiring second data from the second server groups, wherein the number of the second data is equal to the number of the second target data;

when the number of the second server groups is larger than 1, determining a difference value ratio between the second server groups according to the difference value between the load value of each second server group and the load average value; and acquiring second data from each second server group according to the difference ratio, and taking the second data as second target data, wherein the quantity ratio of the second data is equal to the difference ratio, and the sum of the quantities of the second data is equal to the quantity of the second target data.

6. The method of claim 5, wherein the first data is determined according to the following process:

determining a number of accesses to data stored on a first server group;

sorting the data stored on the first server group according to the access times, wherein the larger the access times, the earlier the sorting of the corresponding data is;

at least one data in the data stored on the first server group in the top order is taken as a first data;

determining the second data according to the following process:

determining a number of accesses to data stored on the second server group;

sorting the data stored on the second server group according to the access times, wherein the smaller the access times, the earlier the sorting of the corresponding data is;

and taking at least one data in the data stored on the second server group in the top sequence as second data.

7. The method of claim 6, wherein updating the storage mapping relationship, migrating the first target data to the second server group, and migrating the second target storage data to the first server group comprises:

and migrating the first target data with the highest access times to the second server group with the lowest load value and migrating the second target data with the lowest access times to the first server group with the highest load value.

8. A data storage system, the system comprising a plurality of servers, the system further comprising:

the grouping module is used for grouping the servers to obtain a plurality of server groups, and grouping data to be stored according to a preset rule to obtain a plurality of data groups to be stored;

the storage module is used for determining a storage mapping relation between the data group to be stored and the server group and storing the data group to be stored on the corresponding server group according to the storage mapping relation;

the server group determination module is used for determining the load value of each server group and determining a first server group and a second server group from all the server groups according to the load value;

the target data determining module is used for determining first target data stored on the first server group according to a preset first condition and determining second target data stored on the second server group according to a preset second condition;

and the updating module is used for updating the storage mapping relation, migrating the first target data to the second server group and migrating the second target storage data to the first server group.

9. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.

Background

With the development of internet technology, data is growing explosively, and data storage technology is becoming more important for dealing with the situation of massive data, and in particular, the data storage has the requirements of large access amount, large data amount, high concurrency and the like. Currently, a common data storage system is a centralized data system, that is, data is stored on each server. Such storage systems consume large amounts of memory resources of the servers, and each server stores the same data, which is a waste of storage resources.

Disclosure of Invention

In view of this, embodiments of the present invention provide a data storage method and system, which can solve the problem that a server memory cannot store data in full under the condition of a large data volume, effectively relieve the memory pressure of the server, save server resources, and ensure reliability, security, and access efficiency of data storage; meanwhile, the data storage position can be automatically adjusted, the hot spot data and the access pressure are automatically dispersed, and load balancing is realized.

To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a data storage method, including a plurality of servers, the method further including:

grouping the servers to obtain a plurality of server groups, and grouping data to be stored according to a preset rule to obtain a plurality of data groups to be stored;

determining a storage mapping relation between the data group to be stored and the server group, and storing the data group to be stored on the corresponding server group according to the storage mapping relation;

determining a load value of each server group, and determining a first server group and a second server group from all the server groups according to the load values;

determining first target data stored on a first server group according to a preset first condition, and determining second target data stored on a second server group according to a preset second condition;

and updating the storage mapping relation, migrating the first target data to the second server group and migrating the second target storage data to the first server group.

Optionally, determining the load value of each server group includes: for each server group, determining a load index value of each server in the server group; determining the load value of each server according to the load index value and the corresponding weight; and determining the load value of the server group according to the load value of each server.

Optionally, the load indicator value comprises one or more of: cpu utilization, memory utilization, and interface response time.

Optionally, determining the first server group and the second server group from all the server groups according to the load value includes: taking the server group with the load value larger than or equal to a preset threshold value in all the server groups as a first server group; determining a load average value according to the load value of each server group; and taking the server group with the load value smaller than the load average value in all the server groups as a second server group.

Optionally, according to a preset first condition, determining the first target data stored on the first server group includes:

determining the amount of the first target data;

when the number of the first server groups is 1, acquiring first data from the first server groups, wherein the number of the first data is equal to the number of the first target data;

when the number of the first server groups is larger than 1, determining a difference value ratio between the first server groups according to the difference value between the load value of each first server group and the load average value; acquiring first data from each first server group according to the difference ratio, and taking the first data as first target data, wherein the quantity ratio of the first data is equal to the difference ratio, and the sum of the quantities of the first data is equal to the quantity of the first target data;

determining, according to a preset second condition, second target data stored on the second server group includes:

determining the quantity of second target data, wherein the quantity of the second target data is equal to the quantity of the first target data;

when the number of the second server groups is 1, acquiring second data from the second server groups, wherein the number of the second data is equal to the number of the second target data;

when the number of the second server groups is larger than 1, determining a difference value ratio between the second server groups according to the difference value between the load value of each second server group and the load average value; and acquiring second data from each second server group according to the difference ratio, and taking the second data as second target data, wherein the quantity ratio of the second data is equal to the difference ratio, and the sum of the quantities of the second data is equal to the quantity of the second target data.

Optionally, the first data is determined according to the following procedure:

determining a number of accesses to data stored on a first server group;

sorting the data stored on the first server group according to the access times, wherein the larger the access times, the earlier the sorting of the corresponding data is;

at least one data in the data stored on the first server group in the top order is taken as a first data;

determining the second data according to the following process:

determining a number of accesses to data stored on the second server group;

sorting the data stored on the second server group according to the access times, wherein the smaller the access times, the earlier the sorting of the corresponding data is;

and taking at least one data in the data stored on the second server group in the top sequence as second data.

Optionally, the updating the storage mapping relationship, migrating the first target data to the second server group, and migrating the second target storage data to the first server group includes:

and migrating the first target data with the highest access times to the second server group with the lowest load value and migrating the second target data with the lowest access times to the first server group with the highest load value.

To achieve the above object, according to another aspect of an embodiment of the present invention, there is provided a data storage system including a plurality of servers, the system further including:

the grouping module is used for grouping the servers to obtain a plurality of server groups, and grouping data to be stored according to a preset rule to obtain a plurality of data groups to be stored;

the storage module is used for determining a storage mapping relation between the data group to be stored and the server group and storing the data group to be stored on the corresponding server group according to the storage mapping relation;

the server group determination module is used for determining the load value of each server group and determining a first server group and a second server group from all the server groups according to the load value;

the target data determining module is used for determining first target data stored on the first server group according to a preset first condition and determining second target data stored on the second server group according to a preset second condition;

and the updating module is used for updating the storage mapping relation, migrating the first target data to the second server group and migrating the second target storage data to the first server group.

Optionally, the server group determination module is further configured to: for each server group, determining a load index value of each server in the server group; determining the load value of each server according to the load index value and the corresponding weight; and determining the load value of the server group according to the load value of each server.

Optionally, the load indicator value comprises one or more of: cpu utilization, memory utilization, and interface response time.

Optionally, the server group determination module is further configured to: taking the server group with the load value larger than or equal to a preset threshold value in all the server groups as a first server group; determining a load average value according to the load value of each server group; and taking the server group with the load value smaller than the load average value in all the server groups as a second server group.

Optionally, the target data determination module is further configured to:

determining the amount of the first target data;

when the number of the first server groups is 1, acquiring first data from the first server groups, wherein the number of the first data is equal to the number of the first target data;

when the number of the first server groups is larger than 1, determining a difference value ratio between the first server groups according to the difference value between the load value of each first server group and the load average value; acquiring first data from each first server group according to the difference ratio, and taking the first data as first target data, wherein the quantity ratio of the first data is equal to the difference ratio, and the sum of the quantities of the first data is equal to the quantity of the first target data;

determining the quantity of second target data, wherein the quantity of the second target data is equal to the quantity of the first target data;

when the number of the second server groups is 1, acquiring second data from the second server groups, wherein the number of the second data is equal to the number of the second target data;

when the number of the second server groups is larger than 1, determining a difference value ratio between the second server groups according to the difference value between the load value of each second server group and the load average value; and acquiring second data from each second server group according to the difference ratio, and taking the second data as second target data, wherein the quantity ratio of the second data is equal to the difference ratio, and the sum of the quantities of the second data is equal to the quantity of the second target data.

Optionally, the target data determination module is further configured to:

determining a number of accesses to data stored on a first server group;

sorting the data stored on the first server group according to the access times, wherein the larger the access times, the earlier the sorting of the corresponding data is;

at least one data in the data stored on the first server group in the top order is taken as a first data;

determining a number of accesses to data stored on the second server group;

sorting the data stored on the second server group according to the access times, wherein the smaller the access times, the earlier the sorting of the corresponding data is;

and taking at least one data in the data stored on the second server group in the top sequence as second data.

Optionally, the update module is further configured to: and migrating the first target data with the highest access times to the second server group with the lowest load value and migrating the second target data with the lowest access times to the first server group with the highest load value.

To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided an electronic apparatus including: one or more processors; a storage device, configured to store one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the data storage method according to the embodiment of the present invention.

To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program implementing a data storage method of an embodiment of the present invention when executed by a processor.

One embodiment of the above invention has the following advantages or benefits: grouping the servers to obtain a plurality of server groups, and grouping data to be stored according to a preset rule to obtain a plurality of data groups to be stored; the storage mapping relation between the data group to be stored and the server group is determined, and the data group to be stored is stored on the corresponding server group according to the storage mapping relation, so that the problem that the server memory cannot store data in full under the condition of large data volume can be solved, the memory pressure of the server is effectively relieved, the server resources are saved, and the reliability, the safety and the access efficiency of data storage are ensured; determining a first server group and a second server group from all server groups according to the load value by determining the load value of each server group; determining first target data stored on a first server group according to a preset first condition, and determining second target data stored on a second server group according to a preset second condition; and updating the storage mapping relation, migrating the first target data to the second server group and migrating the second target storage data to the first server group, automatically adjusting the data storage position, automatically dispersing the hot spot data and the access pressure, and realizing load balancing.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of the main flow of a data storage method of an embodiment of the present invention;

FIG. 2 is a schematic illustration of a sub-flow of a data storage method of an embodiment of the invention;

FIG. 3 is a schematic illustration of a sub-flow of a data storage method of an embodiment of the invention;

FIG. 4 is a schematic diagram of the major modules of a data storage system of an embodiment of the present invention;

FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 6 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a schematic diagram of a main flow of a data storage method according to an embodiment of the present invention, the data storage method including a plurality of servers. As shown in fig. 1, the method includes:

step S101: and grouping the servers to obtain a plurality of server groups, and grouping the data to be stored according to a preset rule to obtain a plurality of data groups to be stored.

In this embodiment, the number of server groups and the number of data groups to be stored may be flexibly set, and the present invention is not limited herein. It is worth mentioning that each server group includes at least two servers to store the same data to be stored on the at least two servers, so as to ensure that the data to be stored has backup. When the data to be stored is grouped, the grouping rule can be flexibly set according to the specific content of the data, and the invention is not limited herein. For example, if the data to be stored is data relating to a commodity, the data may be grouped according to the category to which the commodity belongs. In other alternative embodiments, a unique identifier (e.g., ID) may be named for the data to be stored, and then the data may be grouped according to the unique identifier, or may be grouped according to a Hash algorithm. The Hash algorithm (Hash algorithm) can convert a data into a flag, and the flag has a very close relationship with each byte of the source data.

As a specific example, assuming that there are 20 servers, it can be divided into 4 server groups: server group A, server group B, server group C and server group D. Each server group has 5 servers. Assuming that the data to be stored is commodity data of 160 categories, the data to be stored may be divided into 4 groups according to the categories: a data group to be stored T1, a data group to be stored T2, a data group to be stored T3, and a data group to be stored T4. Each data set to be stored contains 40 categories of commodity data.

Step S102: and determining a storage mapping relation between the data group to be stored and the server group, and storing the data group to be stored on the corresponding server group according to the storage mapping relation.

The storage mapping relation is used for representing the storage relation between the data group to be stored and the server group. When the data group to be stored is stored, all the data to be stored in the data group to be stored can be stored in each server in the corresponding server group, that is, if 5 servers exist in the corresponding server group, the same data group to be stored is stored in each server, so that the data can be guaranteed to have backup, and the access request can be dispersed to different servers in the subsequent data access, so as to reduce the access pressure.

Continuing with the above example as an example, the memory mapping relationship may be as shown in table 1 below:

table 1:

data set to be stored	Storage location
		Data set to be stored T1	Server group A
Data set to be stored T2	Server group B
		Data set to be stored T3	Server group C
Data set to be stored T4	Server group D

In this example, there are 5 servers in each server group, and one server may be set as a central server in the 5 servers, and the central server is responsible for distribution, storage and synchronization of the data group to be stored.

In a specific embodiment, after determining the storage mapping relationship, the storage mapping relationship may be recorded in a configuration file, that is, the IP of the data group to be stored and the server group corresponding to the data group to be stored are recorded in the configuration file, and the IP of the central server of each server group may also be recorded. More specifically, two tables may be set before the data is stored, wherein one table is used for recording the storage state of each server (this table may be referred to as table a), and the other table is used for recording the storage state of each data group to be stored (this table may be referred to as table B). The function of the table B is to manually intervene in the loading state of certain data to realize flexible control of individual data. When data storage begins, firstly, the content of the configuration file is obtained, the central server of each server group determines the data group which needs to be stored by the server group according to the content of the configuration file, then the data group which needs to be stored by the server group and has the state of 'not stored' in the table B is obtained, the data group is distributed to each server (including the central server) in the server group through the data interface, and then the state of the data group in the table B is updated to 'stored'. When storing data, the central server also needs to update the status in table a to "in storage" for the respective server. When the server storage is completed, the state of the corresponding server in table a is updated to "storage completed". If the states corresponding to all servers in the server group in table a are "storage completed", the state of the data group in the update table B is "completed".

The data set to be stored has already been stored on the corresponding server set, via steps S101 and S102. Then, data is randomly accessed, the load of each server group is different, and the access frequency of certain data on a certain server group is far higher than that of other data, namely, hot data occurs, so that the access pressure of the server group is too high, and the access pressure of other server groups is lower. Therefore, in steps S103-S105, the hot spot data needs to be distributed and the load of each server group needs to be equalized, and the specific process is as follows.

Step S103: and determining the load value of each server group, and determining a first server group and a second server group from all the server groups according to the load values.

As shown in fig. 2, the load value of each server group may be determined according to the following process:

step S201: for each server group, determining a load index value of each server in the server group; wherein the load indicator value comprises one or more of: cpu utilization, memory utilization, and interface response time. The load index values can be obtained by the existing method, and the invention is not described herein.

Step S202: determining the load value of each server according to the load index value and the corresponding weight; specifically, each load index value and the corresponding weight thereof may be multiplied, and the obtained products may be added to obtain the load value of the server.

Step S203: and determining the load value of the server group according to the load value of each server. Specifically, the load values of all servers in the server group may be added to obtain the load value of the server group.

After obtaining the load value of each server group, the first server group and the second server group may be determined from all the server groups according to the load value, and the specific process may include:

taking the server group with the load value larger than or equal to a preset threshold value in all the server groups as a first server group; the preset threshold may be flexibly set according to an application scenario, and the present invention is not limited herein. In this embodiment, when the load value of a certain server group is greater than the preset threshold, it indicates that the load capacity of the server group reaches a certain limit, and other server groups are required to share the load.

Determining a load average value according to the load value of each server group;

and taking the server group with the load value smaller than the load average value in all the server groups as a second server group.

For example, if the preset threshold is 5, the load value of the server group a is 6, the load value of the server group B is 4, the load value of the server group C is 8, and the load value of the server group D is 2, the average load value is 5, the first server group is the server group a and the server group C, and the second server group is the server group B and the server group D.

In this embodiment, the number of the first server group and the second server group may be one or more. When the number of the first server group and the second server group is one, respectively, the storage locations of the first target data and the second target data may be directly exchanged. When the number of the first server groups is multiple and the number of the second server groups is 1, or when the number of the second server groups is multiple and the number of the first server groups is 1, or when the number of the first server groups and the number of the second server groups are both multiple, the first target data and the second target data need to be distributed between the multiple first server groups and the multiple second server groups according to a certain rule. Specifically, as shown in step S104.

Step S104: according to a preset first condition, determining first target data stored on the first server group, and according to a preset second condition, determining second target data stored on the second server group.

Wherein, according to a preset first condition, the process of determining the first target data stored on the first server group comprises:

(1) the amount of the first target data is determined. In this embodiment, the number of the first target data can be flexibly set according to the total amount of data stored on each server and the application scenario. For example, if the data stored in each server is data related to 40 categories of commodities, the number of the first target data may be set to 40 × 10% to 4, that is, the first target data of 4 categories needs to be specified.

(2) When the number of the first server groups is 1, acquiring first data from the first server groups, wherein the number of the first data is equal to the number of the first target data. That is, when there is only one first server group, it is necessary to determine 4 categories of first data as first target data on the server group, wherein the process of determining the first data is described below.

(3) When the number of the first server groups is larger than 1, determining a difference value ratio between the first server groups according to the difference value between the load value of each first server group and the load average value; and acquiring first data from each first server group according to the difference ratio, and taking the first data as first target data, wherein the quantity ratio of the first data is equal to the difference ratio, and the sum of the quantities of the first data is equal to the quantity of the first target data.

Continuing with the above example as an example, the first server group and the first server group are the server group a and the server group C. The difference between the load value 6 of the server group a and the load average value 5 is 1, the difference between the load value 8 of the server group C and the load average value 5 is 3, the difference ratio between the first server groups (i.e., the difference ratio between the server group a and the server group C) is 1:3, and since the number of the first target data is 4, it is necessary to obtain 1 category of commodity data from the server group a as the first data, and obtain 3 categories of commodity data from the server group C as the first data.

Likewise, the process of determining the second target data stored on the second server group according to the preset second condition includes:

(1) determining the quantity of second target data, wherein the quantity of the second target data is equal to the quantity of the first target data. For example, if the number of the first target data is 4 categories of product data, the number of the second target data is also 4 categories of product data. The reason why the number of the second target data is equal to the number of the first target data in the present embodiment is to equalize the number of data stored in each server group after the storage mapping relationship is updated and the data storage location is reallocated.

(2) And when the number of the second server groups is 1, acquiring second data from the second server groups, wherein the number of the second data is equal to the number of the second target data. That is, when there is only one second server group, 4 categories of second data need to be determined as second target data on the server group, wherein the process of determining the second data is described below.

(3) When the number of the second server groups is larger than 1, determining a difference value ratio between the second server groups according to the difference value between the load value of each second server group and the load average value; and acquiring second data from each second server group according to the difference ratio, and taking the second data as second target data, wherein the quantity ratio of the second data is equal to the difference ratio, and the sum of the quantities of the second data is equal to the quantity of the second target data.

Continuing with the above example as an example, the second server group is server group B and server group D. If the difference between the load value 4 of the server group B and the load average value 5 is 1, the difference between the load value 2 of the server group D and the load average value 5 is 3, and the difference ratio between the second server groups (i.e., the difference ratio between the server group B and the server group D) is 1:3, it is necessary to acquire 1 category of commodity data from the server group B as the second data, and acquire 3 categories of commodity data from the server group D as the second data.

Further, as shown in fig. 3, the first data may be determined according to the following process:

step S301: determining a number of accesses to data stored on a first server group; for the same data, the data is stored on all servers in the corresponding server group, and therefore, the number of accesses of the data is equal to the sum of the number of accesses of the data on all servers in the server.

Step S302: sorting the data stored on the first server group according to the access times, wherein the larger the access times, the earlier the sorting of the corresponding data is;

step S303: and taking at least one data in the data stored on the first server group in the top sequence as a first data. Specifically, the data may be selected according to the number of the first data that needs to be acquired by each first server group determined above. Continuing to take the above example as an example, for the server group a, if commodity data of 1 category needs to be acquired from the server group as first target data, the first ordered data is taken as first data; for the server group C, 3 categories of commodity data need to be acquired from the server group as first target data, and data in the order of the first, second, and third are taken as first data.

Further, the second data may be determined according to the following procedure:

determining a number of accesses to data stored on the second server group; for the same data, the data is stored on all servers in the corresponding server group, and therefore, the number of accesses of the data is equal to the sum of the number of accesses of the data on all servers in the server.

Sorting the data stored on the second server group according to the access times, wherein the smaller the access times, the earlier the sorting of the corresponding data is;

and taking at least one data in the data stored on the second server group in the top sequence as second data. Specifically, the data may be selected according to the number of the second data that needs to be acquired by each second server group determined above. Continuing to take the above example as an example, for the server group B, if it is necessary to acquire commodity data of 1 category from the server group as second target data, the data sorted first is taken as second data; for the server group D, commodity data of 3 categories needs to be acquired from the server group as second target data, and data of the first, second, and third orders are taken as second data.

Step S105: updating the storage mapping relationship, migrating the first target data to the second server group, and migrating the second target storage data to the first server group.

In this embodiment, the first target data is data with a large number of accesses on the server group with the largest load, and may be referred to as hot spot data. The second target data is data with a smaller number of accesses on a server group with a smaller load. The storage positions of the server group and the server group are exchanged, so that the load of each server group can be balanced, the hot data can be dispersed, and the stability and the efficiency of data access can be improved.

Since there may be a plurality of first target data, second target data, first server groups, and second server groups, when updating the storage mapping relationship, a certain rule needs to be followed, specifically, the first target data with the highest access frequency is migrated to the second server group with the lowest load value, and the second target data with the lowest access frequency is migrated to the first server group with the highest load value. The first target data with the highest access times are migrated to the second server group with the lowest load value, so that the load value of the second server group can be improved, the second target data with the lowest access times are migrated to the first server group with the highest load value, the load value of the first server group can be reduced, hot spot data are dispersed, the load of each server group is balanced, and the data access stability and efficiency are improved.

According to the data storage method provided by the embodiment of the invention, the plurality of servers are grouped to obtain a plurality of server groups, and the data to be stored are grouped according to a preset rule to obtain a plurality of data groups to be stored; the storage mapping relation between the data group to be stored and the server group is determined, and the data group to be stored is stored on the corresponding server group according to the storage mapping relation, so that the problem that the server memory cannot store data in full under the condition of large data volume can be solved, the memory pressure of the server is effectively relieved, the server resources are saved, and the reliability, the safety and the access efficiency of data storage are ensured; determining a first server group and a second server group from all server groups according to the load value by determining the load value of each server group; determining first target data stored on a first server group according to a preset first condition, and determining second target data stored on a second server group according to a preset second condition; the storage mapping relation is updated so that the first target data are stored on the second server group and the second target data are stored on the first server group, the data storage position can be automatically adjusted, the hot spot data and the access pressure are automatically dispersed, and load balancing is achieved.

In order to make the data storage method according to the embodiment of the present invention more clear, the following example is used as an example for explanation.

(1) Assuming that there are 20 servers, the 20 servers are divided into 4 groups, each group of servers has 5 servers (hereinafter, the group of servers is referred to as a shard), and then one server is respectively arranged in each shard as a central server (which may also be referred to as a leader host), and the central server is responsible for data distribution, loading (i.e. storing) and synchronization of data storage task completion. The commodity data of 160 categories are required to be stored, the data to be stored are grouped according to the categories and are divided into 4 data groups, and each data group has 40 categories of commodity data.

(2) And setting a storage mapping relation between the data group to be stored and each fragment. Specifically, the commodity data of each category in the data group to be stored and the corresponding fragment information may be written into the configuration file, that is, the category of the data group to be stored 1, the IP of the server corresponding to the fragment 1, and the IP of the leader host are written into the configuration file. The database is provided with two tables, one for recording the future loading data state (table a) of each server, one server IP and one category ID per row. The other records the load status of the data for each category (table B).

(4) And the data storage task starts to run, the content of the configuration file is acquired, the leader host of each fragment starts to acquire the commodity data which needs to be loaded by the fragment and has the status of 'unloaded' in the table B, the commodity data are distributed to each server (including a local machine) in the fragment through an interface, and then the corresponding category status of the table B is updated to be 'loaded'. Each server stores the received category ID in a cached queue and then initiates multithreading, consuming the data in the queue.

(5) Before loading commodity data, an interface of a leader host is called first, and the state of a corresponding record in the update table A is informed to be 'loading'. Next, loading of the commodity data under each category ID is started. After the loading is completed, the leader host is informed that the state of the corresponding record in the update table a is "loading completed".

(6) And the leader host regularly scans the loading state of each server in the chip in the table A according to the category ID, and if the loading is completely finished, the state of the corresponding category ID in the updating table B is finished. The function of the table B is to manually intervene in the loading state of the commodity data of a certain classification so as to flexibly control the loading of the commodity data under individual classification.

(7) After the data to be stored is loaded through the steps (1) to (6), each fragment continuously monitors the access frequency of the data in the process of accessing the data, the number of times of accessing each category is calculated through a memory, and when the load capacity of the server of the fragment reaches a certain limit (namely when the load value of the server is greater than or equal to a preset threshold), each server in the fragment counts T category IDs with the highest access times (T is the number of the loaded categories of the server). Then, an interface of the leader host is called, and the leader host is synchronized.

As an example, the preset threshold is 7, and the load values of the segment 1, the segment 2, the segment 3, and the segment 4 are 6, 4, 8, and 2, respectively, then each server of the segment 3 arranges the access times of the commodity data of 40 categories stored thereon in a descending order, takes the 4 category IDs with the highest access times to send to the leader host, and the leader host merges the access times of the same category IDs to finally obtain the 4 category IDs with the highest access times.

(8) The method comprises the steps that the fragments with the load capacity reaching a certain limit inquire the load condition of each fragment by calling an interface of each fragment leader host, find out the fragments with the lowest M loads (M is the number of the fragments below the average value of the cluster load), calculate the difference value between the load value of the M fragments and the average value of the cluster load, calculate the number of the distribution categories (the categories with high priority distribution access times) of each fragment according to the difference value ratio among the M fragments, and then distribute the categories to the fragments by modifying a configuration file.

For example, the leader host of segment 3 queries the loads of segments 1, 2, and 4 through the interface, and calculates that the average load value is (6+4+8+ 2)/4-5, that is, segment 2 and segment 4 are the segments below the average cluster load value, and M-2. The difference ratio (5-4): 5-2) ═ 1:3 was calculated. Then slice 2 would obtain (1/4) × 4 ═ 1, and slice 4 would obtain (3/4) × 4 ═ 3. In the distribution, the data with the highest access times is preferentially distributed to the shard with the lowest load, that is, the data with the highest access times is preferentially distributed to the shard 4.

(9) And (4) the fragments which are allocated to the new classes are loaded according to the steps from (4) to (6). And simultaneously, the leader host of each fragment distributes the X categories (the value of X is equal to the number of the new categories distributed) with the lowest access times on each fragment to the high-load fragments in the step (7). And (4) carrying out the steps from (4) to (6) to load the data in the high-load slicing mode.

For example, segment 2 is assigned 1 category, so the leader host of segment 2 obtains a category with the lowest access frequency from 5 servers in its segment, and assigns the category to segment 3 through the leader host interface of segment 3. The fragment 4 also performs the same operation, finds out 3 categories with the lowest access times, and allocates the categories to the fragment 3 through the leader host interface of the fragment 3.

(10) And after the high-load fragment finishes loading the new data or after the low-load fragment finishes loading the new data, updating the storage mapping relation, and calling the leader host interfaces of the high-load fragment and the low-load fragment to delete the data.

(11) Therefore, the transfer of the hot spot data is completed, and the load of the fragments is reduced. Steps (7) - (10) may be performed periodically during server operation.

FIG. 4 is a schematic diagram of the main modules of a data storage system 400 according to an embodiment of the present invention, the system 400 comprising a plurality of servers. As shown in fig. 4, the data storage system 400 further includes:

a grouping module 401, configured to group the multiple servers to obtain multiple server groups, and group data to be stored according to a preset rule to obtain multiple data groups to be stored;

a storage module 402, configured to determine a storage mapping relationship between the to-be-stored data group and the server group, and store the to-be-stored data group on the corresponding server group according to the storage mapping relationship;

a server group determining module 403, configured to determine a load value of each server group, and determine a first server group and a second server group from all server groups according to the load value;

a target data determining module 404, configured to determine, according to a preset first condition, first target data stored on a first server group, and determine, according to a preset second condition, second target data stored on a second server group;

an updating module 405, configured to update the storage mapping relationship, migrate the first target data to the second server group, and migrate the second target storage data to the first server group.

Optionally, the server group determining module 403 is further configured to: for each server group, determining a load index value of each server in the server group; determining the load value of each server according to the load index value and the corresponding weight; and determining the load value of the server group according to the load value of each server.

Optionally, the load indicator value comprises one or more of: cpu utilization, memory utilization, and interface response time.

Optionally, the server group determining module 403 is further configured to: taking the server group with the load value larger than or equal to a preset threshold value in all the server groups as a first server group; determining a load average value according to the load value of each server group; and taking the server group with the load value smaller than the load average value in all the server groups as a second server group.

Optionally, the target data determining module 404 is further configured to:

determining the amount of the first target data;

when the number of the first server groups is 1, acquiring first data from the first server groups, wherein the number of the first data is equal to the number of the first target data;

when the number of the first server groups is larger than 1, determining a difference value ratio between the first server groups according to the difference value between the load value of each first server group and the load average value; acquiring first data from each first server group according to the difference ratio, and taking the first data as first target data, wherein the quantity ratio of the first data is equal to the difference ratio, and the sum of the quantities of the first data is equal to the quantity of the first target data;

determining the quantity of second target data, wherein the quantity of the second target data is equal to the quantity of the first target data;

when the number of the second server groups is 1, acquiring second data from the second server groups, wherein the number of the second data is equal to the number of the second target data;

when the number of the second server groups is larger than 1, determining a difference value ratio between the second server groups according to the difference value between the load value of each second server group and the load average value; and acquiring second data from each second server group according to the difference ratio, and taking the second data as second target data, wherein the quantity ratio of the second data is equal to the difference ratio, and the sum of the quantities of the second data is equal to the quantity of the second target data.

Optionally, the target data determining module 404 is further configured to:

determining a number of accesses to data stored on a first server group;

sorting the data stored on the first server group according to the access times, wherein the larger the access times, the earlier the sorting of the corresponding data is;

at least one data in the data stored on the first server group in the top order is taken as a first data;

determining a number of accesses to data stored on the second server group;

sorting the data stored on the second server group according to the access times, wherein the smaller the access times, the earlier the sorting of the corresponding data is;

and taking at least one data in the data stored on the second server group in the top sequence as second data.

Optionally, the update module 405 is further configured to: and migrating the first target data with the highest access times to the second server group with the lowest load value and migrating the second target data with the lowest access times to the first server group with the highest load value.

According to the data storage system provided by the embodiment of the invention, the plurality of servers are grouped to obtain a plurality of server groups, and the data to be stored are grouped according to a preset rule to obtain a plurality of data groups to be stored; the storage mapping relation between the data group to be stored and the server group is determined, and the data group to be stored is stored on the corresponding server group according to the storage mapping relation, so that the problem that the server memory cannot store data in full under the condition of large data volume can be solved, the memory pressure of the server is effectively relieved, the server resources are saved, and the reliability, the safety and the access efficiency of data storage are ensured; determining a first server group and a second server group from all server groups according to the load value by determining the load value of each server group; determining first target data stored on a first server group according to a preset first condition, and determining second target data stored on a second server group according to a preset second condition; and updating the storage mapping relation, migrating the first target data to the second server group and migrating the second target storage data to the first server group, automatically adjusting the data storage position, automatically dispersing the hot spot data and the access pressure, and realizing load balancing.

The data storage system can execute the method provided by the embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.

FIG. 5 illustrates an exemplary system architecture 500 of a data storage method or data storage system to which embodiments of the present invention may be applied.

As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 serves to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the terminal devices 501, 502, 503 to interact with a server 505 over a network 504 to receive or send messages or the like. The terminal devices 501, 502, 503 may have various communication client applications installed thereon, such as a shopping application, a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.

The terminal devices 501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 505 may be a server that provides various services, such as a background management server that supports shopping websites browsed by users using the terminal devices 501, 502, 503. The background management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (e.g., target push information and product information) to the terminal device.

It should be noted that the data storage method provided by the embodiment of the present invention is generally executed by the server 505, and accordingly, the data storage system is generally disposed in the server 505.

It should be understood that the number of terminal devices, networks, and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 6, a block diagram of a computer system 600 suitable for use with a terminal device implementing an embodiment of the invention is shown. The terminal device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 601.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a sending module, an obtaining module, a determining module, and a first processing module. The names of these modules do not in some cases constitute a limitation on the unit itself, and for example, the sending module may also be described as a "module that sends a picture acquisition request to a connected server".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise:

grouping a plurality of servers to obtain a plurality of server groups, and grouping data to be stored according to a preset rule to obtain a plurality of data groups to be stored;

determining a storage mapping relation between the data group to be stored and the server group, and storing the data group to be stored on the corresponding server group according to the storage mapping relation;

determining a load value of each server group, and determining a first server group and a second server group from all the server groups according to the load values;

determining first target data stored on a first server group according to a preset first condition, and determining second target data stored on a second server group according to a preset second condition;

updating the storage mapping relationship, migrating the first target data to the second server group, and migrating the second target storage data to the first server group.

According to the technical scheme of the embodiment of the invention, a plurality of server groups are obtained by grouping the plurality of servers, and the data to be stored are grouped according to a preset rule to obtain a plurality of data groups to be stored; the storage mapping relation between the data group to be stored and the server group is determined, and the data group to be stored is stored on the corresponding server group according to the storage mapping relation, so that the problem that the server memory cannot store data in full under the condition of large data volume can be solved, the memory pressure of the server is effectively relieved, the server resources are saved, and the reliability, the safety and the access efficiency of data storage are ensured; determining a first server group and a second server group from all server groups according to the load value by determining the load value of each server group; determining first target data stored on a first server group according to a preset first condition, and determining second target data stored on a second server group according to a preset second condition; and updating the storage mapping relation, migrating the first target data to the second server group and migrating the second target storage data to the first server group, automatically adjusting the data storage position, automatically dispersing the hot spot data and the access pressure, and realizing load balancing.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.