1. A platform-oriented automated assay receiving method, the method comprising:

determining whether the standard verification starting time before the target platform is upgraded is reached;

under the condition that the standard verification starting time before upgrading is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain test result data before upgrading;

determining whether the comparison verification starting time after the target platform is upgraded is reached;

under the condition that the upgraded comparison verification starting time is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain upgraded test result data;

and automatically comparing the test result data before upgrading with the test result after upgrading case by case, and generating an acceptance report according to the comparison result.

2. The method as claimed in claim 1, wherein before the step of pulling and executing the full set of acceptance cases of the target platform from the acceptance case library to obtain the test result data before upgrading, the method further comprises:

acquiring functional test cases and non-functional test cases related to the target platform from a plurality of target sources;

storing the obtained related functional test cases and non-functional test cases into an acceptance case library;

determining whether the cases in the acceptance case base change;

and under the condition that the change is determined, updating the cases in the test case library in an automatic increment aggregation mode or a stock updating mode.

3. The method of claim 2, wherein the target source comprises at least one of: the method comprises the following steps of function automatic test cases of platform framework test design of each version, batch automatic test cases for regression verification, platform automatic performance test cases on a performance test platform and platform reliability automatic test cases on a chaotic engineering platform, wherein the platform is customized on a batch test system.

4. The method of claim 1, wherein automatically comparing the test result data before the upgrade with the test result after the upgrade case by case comprises:

the comparison is carried out according to the following rules: the upgraded platform framework function success cases, the upgraded reliability effective cases and the batch and performance acceptance cases which are less in consumed time or less in consumed time increase and have the amplitude lower than the preset fault tolerance threshold value are judged to pass the acceptance; and judging that the upgraded platform framework fails to pass the functions, the upgraded reliability failure cases and batch and performance acceptance cases with the time consumption increasing amplitude exceeding a preset fault tolerance threshold value are not accepted.

5. The method of claim 1, wherein generating an acceptance report based on the comparison comprises:

according to the comparison result, carrying out statistics in different dimensions;

displaying the statistical result in sections in an html format;

wherein the sections comprise an overview section and an acceptance detail section, wherein the overview section comprises: the platform of this acceptance, the task name of this acceptance, the starting time of the acceptance task, the ending time of the acceptance task, the total number of the acceptance cases, the time-consuming fault-tolerant threshold value and the case total passing rate, and the acceptance detail section comprises: the number of cases of each acceptance content, the case passing rate and the case detail not passing.

6. A platform-oriented automated assay receiving device, comprising:

the first determining module is used for determining whether the standard verification starting time before the target platform is upgraded is reached;

the first execution module is used for pulling and executing a full set of acceptance cases of the target platform from the acceptance case library under the condition that the standard verification starting time before upgrading is determined to be reached, so as to obtain test result data before upgrading;

the second determination module is used for determining whether the comparison verification starting time after the target platform is upgraded is reached;

the second execution module is used for pulling and executing the acceptance case complete set of the target platform from the acceptance case library under the condition that the upgraded comparison verification starting time is determined to be reached, so as to obtain upgraded test result data;

and the comparison module is used for automatically comparing the test result data before the upgrade with the test result after the upgrade one by one and generating an acceptance report according to the comparison result.

7. The apparatus of claim 6, further comprising:

the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring functional test cases and non-functional test cases related to a target platform from a plurality of target sources before a test result data before upgrading is obtained by pulling and executing a full set of acceptance cases of the target platform from an acceptance case library;

the storage module is used for storing the obtained related functional test cases and non-functional test cases into an acceptance case library;

a third determining module, configured to determine whether a case in the acceptance case base changes;

and the updating module is used for updating the cases in the test case library in an automatic increment aggregation mode or a stock updating mode under the condition of determining the change.

8. The apparatus of claim 7, wherein the target source comprises at least one of: the method comprises the following steps of function automatic test cases of platform framework test design of each version, batch automatic test cases for regression verification, platform automatic performance test cases on a performance test platform and platform reliability automatic test cases on a chaotic engineering platform, wherein the platform is customized on a batch test system.

9. An electronic device comprising a processor and a memory for storing processor-executable instructions, the instructions when executed by the processor implementing the steps of the method of:

determining whether the standard verification starting time before the target platform is upgraded is reached;

under the condition that the standard verification starting time before upgrading is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain test result data before upgrading;

determining whether the comparison verification starting time after the target platform is upgraded is reached;

under the condition that the upgraded comparison verification starting time is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain upgraded test result data;

and automatically comparing the test result data before upgrading with the test result after upgrading case by case, and generating an acceptance report according to the comparison result.

10. A computer readable storage medium having stored thereon computer instructions which, when executed, implement the steps of a method comprising:

determining whether the standard verification starting time before the target platform is upgraded is reached;

under the condition that the standard verification starting time before upgrading is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain test result data before upgrading;

determining whether the comparison verification starting time after the target platform is upgraded is reached;

under the condition that the upgraded comparison verification starting time is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain upgraded test result data;

and automatically comparing the test result data before upgrading with the test result after upgrading case by case, and generating an acceptance report according to the comparison result.

Background

With the continuous development of financial business and the wide rise of digital demand, a large data platform built based on open source software ecology or third party company products has become a main means for calculating and processing scale data. In order to realize the continuous evolution of platform architecture and the emerging of functions, the platform must be upgraded, which results in the platform becoming more and more complex. In addition to the improvement in functionality brought about by a version upgrade, new defects may often accompany it. Therefore, the big data platform is used as a basic support of the whole big data system, and the quality needs to be guaranteed. The most common method is to perform comprehensive acceptance test on the upgraded platform based on the platform performance before upgrading so as to ensure that the stock function and the non-function of the platform are kept stable on the premise that the new characteristics meet the requirements.

The existing platform acceptance test generally comprises two steps: the first is benchmark verification before upgrading, and the second is comparison verification after upgrading. In the benchmark verification stage, function acceptance is firstly carried out, platform testers firstly complete stock function tests of the platform frame before upgrading, then the matched testers of each platform application complete batch tests of the respective application, and result data such as whether the reserved functions are normal, whether the batch execution is passed and the time consumed for batch execution are determined; and performing non-functional acceptance, respectively carrying out performance test and chaotic engineering by special testers from two dimensions of performance and reliability, and reserving comprehensive performance and reliability conditions of the platform before upgrading. In the comparison and verification stage, except that the upgraded incremental function test is added in the function acceptance of the platform framework, the rest of test contents are the same as those in the reference verification stage, and then the function and non-function result data before and after upgrading are compared to find out the difference, and an acceptance report is compiled to be used as a judgment basis for judging whether the upgrading is finally implemented in production.

However, the acceptance test method mainly has the following two problems in the practical application process:

1) in terms of quality, because large version upgrading of a platform is very easy to cause incompatibility to some special processing used before upgrading of platform applications on the platform, batch testing of the platform applications becomes a serious factor in function acceptance, and the application breadth, batch types and scene quantity covered by the platform application determine the risk of production batch interruption. However, the platform application batch test in the existing acceptance test method is highly dependent on testers of the applications, and often has the conditions that personnel can not be found due to change, the test period is difficult to match, the test data is not fixed, the scene coverage is incomplete, even the empty running batch is directly not carried out, so that an acceptance blind area exists, and the platform upgrading risk is uncontrollable;

2) in the aspect of efficiency, the existing acceptance test method relates to a plurality of roles such as platform side testers, application side testers, special testers and the like, the management, communication and coordination cost of an acceptance test responsible person is extremely high, each acceptance content needs to manually pay attention to whether the acceptance content is finished or not and actively informs the testers of the next content to continue, and once the acceptance content is informed or the test is not timely developed, the overall progress of the acceptance work is delayed; meanwhile, the retention of the standard verification result before upgrading, the comparison verification and the difference analysis after upgrading and the completion of the final acceptance report are all manual operations, the automation level is low, the overall efficiency of the acceptance work is not high, and the more complicated platform upgrading is, the more easily the on-line implementation of the production is influenced.

No effective solution has been proposed at present for how to perform high quality and high efficiency acceptance tests.

Disclosure of Invention

The application aims to provide a platform-oriented automatic test acceptance method and device, which can realize high-quality and high-efficiency acceptance test.

The application provides a platform-oriented automatic assay receiving method and a device, which are realized as follows:

a platform-oriented automated assay harvesting method, the method comprising:

determining whether the standard verification starting time before the target platform is upgraded is reached;

under the condition that the standard verification starting time before upgrading is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain test result data before upgrading;

determining whether the comparison verification starting time after the target platform is upgraded is reached;

under the condition that the upgraded comparison verification starting time is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain upgraded test result data;

and automatically comparing the test result data before upgrading with the test result after upgrading case by case, and generating an acceptance report according to the comparison result.

In one embodiment, before the step of pulling and executing the full set of acceptance cases of the target platform from the acceptance case library to obtain the test result data before upgrading, the method further includes:

acquiring functional test cases and non-functional test cases related to the target platform from a plurality of target sources;

storing the obtained related functional test cases and non-functional test cases into an acceptance case library;

determining whether the cases in the acceptance case base change;

and under the condition that the change is determined, updating the cases in the test case library in an automatic increment aggregation mode or a stock updating mode.

In one embodiment, the target source comprises at least one of: the method comprises the following steps of function automatic test cases of platform framework test design of each version, batch automatic test cases for regression verification, platform automatic performance test cases on a performance test platform and platform reliability automatic test cases on a chaotic engineering platform, wherein the platform is customized on a batch test system.

In one embodiment, the automatically comparing the test result data before the upgrade with the test result after the upgrade case by case includes:

the comparison is carried out according to the following rules: the upgraded platform framework function success cases, the upgraded reliability effective cases and the batch and performance acceptance cases which are less in consumed time or less in consumed time increase and have the amplitude lower than the preset fault tolerance threshold value are judged to pass the acceptance; and judging that the upgraded platform framework fails to pass the functions, the upgraded reliability failure cases and batch and performance acceptance cases with the time consumption increasing amplitude exceeding a preset fault tolerance threshold value are not accepted.

In one embodiment, generating an acceptance report based on the comparison results comprises:

according to the comparison result, carrying out statistics in different dimensions;

displaying the statistical result in sections in an html format;

wherein the sections comprise an overview section and an acceptance detail section, wherein the overview section comprises: the platform of this acceptance, the task name of this acceptance, the starting time of the acceptance task, the ending time of the acceptance task, the total number of the acceptance cases, the time-consuming fault-tolerant threshold value and the case total passing rate, and the acceptance detail section comprises: the number of cases of each acceptance content, the case passing rate and the case detail not passing.

A platform-oriented automated assay receiving device comprising:

the first determining module is used for determining whether the standard verification starting time before the target platform is upgraded is reached;

the first execution module is used for pulling and executing a full set of acceptance cases of the target platform from the acceptance case library under the condition that the standard verification starting time before upgrading is determined to be reached, so as to obtain test result data before upgrading;

the second determination module is used for determining whether the comparison verification starting time after the target platform is upgraded is reached;

the second execution module is used for pulling and executing the acceptance case complete set of the target platform from the acceptance case library under the condition that the upgraded comparison verification starting time is determined to be reached, so as to obtain upgraded test result data;

and the comparison module is used for automatically comparing the test result data before the upgrade with the test result after the upgrade one by one and generating an acceptance report according to the comparison result.

In one embodiment, the above apparatus further comprises:

the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring functional test cases and non-functional test cases related to a target platform from a plurality of target sources before a test result data before upgrading is obtained by pulling and executing a full set of acceptance cases of the target platform from an acceptance case library;

the storage module is used for storing the obtained related functional test cases and non-functional test cases into an acceptance case library;

a third determining module, configured to determine whether a case in the acceptance case base changes;

and the updating module is used for updating the cases in the test case library in an automatic increment aggregation mode or a stock updating mode under the condition of determining the change.

In one embodiment, the target source comprises at least one of: the method comprises the following steps of function automatic test cases of platform framework test design of each version, batch automatic test cases for regression verification, platform automatic performance test cases on a performance test platform and platform reliability automatic test cases on a chaotic engineering platform, wherein the platform is customized on a batch test system.

An electronic device comprising a processor and a memory for storing processor-executable instructions, the instructions when executed by the processor implementing the steps of the method of:

determining whether the standard verification starting time before the target platform is upgraded is reached;

under the condition that the standard verification starting time before upgrading is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain test result data before upgrading;

determining whether the comparison verification starting time after the target platform is upgraded is reached;

under the condition that the upgraded comparison verification starting time is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain upgraded test result data;

and automatically comparing the test result data before upgrading with the test result after upgrading case by case, and generating an acceptance report according to the comparison result.

A computer readable storage medium having stored thereon computer instructions which, when executed, implement the steps of a method comprising:

determining whether the standard verification starting time before the target platform is upgraded is reached;

under the condition that the standard verification starting time before upgrading is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain test result data before upgrading;

determining whether the comparison verification starting time after the target platform is upgraded is reached;

under the condition that the upgraded comparison verification starting time is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain upgraded test result data;

and automatically comparing the test result data before upgrading with the test result after upgrading case by case, and generating an acceptance report according to the comparison result.

The application provides a platform-oriented automatic test acceptance method, by setting benchmark verification starting time and comparing verification starting time, and setting a complete set of acceptance cases, so that case test results before upgrading and case test results after upgrading can be automatically triggered, and comparison of the test results before upgrading and the test results after upgrading can be automatically carried out, so that an acceptance report can be generated, the problems of low efficiency and accuracy of the existing acceptance test can be solved, and the technical effect of accurately and efficiently carrying out automatic acceptance test is achieved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort.

FIG. 1 is a flow chart of a method of one embodiment of a platform-oriented automated assay collection method provided herein;

FIG. 2 is a block diagram of one embodiment of a platform-oriented automated acceptance testing method provided herein;

FIG. 3 is a block diagram of a platform-oriented automated assay acceptance test system provided herein;

FIG. 4 is a block diagram of the structure of an acceptance case base module provided by the present application;

FIG. 5 is a block diagram of the structure of an acceptance scheduling module provided herein;

FIG. 6 is a block diagram of a data storage module provided herein;

FIG. 7 is a block diagram of a mass matching module provided herein;

FIG. 8 is a block diagram of the hardware architecture of an electronic device of a platform-oriented automated assay receiving method provided herein;

FIG. 9 is a block diagram of one embodiment of the platform-oriented automated assay receiving device provided herein.

Detailed Description

In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

FIG. 1 is a flow chart of a method of one embodiment of the platform-oriented automated assay collection method provided herein. Although the present application provides method operational steps or apparatus configurations as illustrated in the following examples or figures, more or fewer operational steps or modular units may be included in the methods or apparatus based on conventional or non-inventive efforts. In the case of steps or structures which do not logically have the necessary cause and effect relationship, the execution sequence of the steps or the module structure of the apparatus is not limited to the execution sequence or the module structure described in the embodiments and shown in the drawings of the present application. When the described method or module structure is applied in an actual device or end product, the method or module structure according to the embodiments or shown in the drawings can be executed sequentially or executed in parallel (for example, in a parallel processor or multi-thread processing environment, or even in a distributed processing environment).

Specifically, as shown in fig. 1, the above-mentioned platform-oriented automatic assay receiving method may include the following steps:

step 101: determining whether the standard verification starting time before the target platform is upgraded is reached;

step 102: under the condition that the standard verification starting time before upgrading is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain test result data before upgrading;

namely, a new acceptance test task can be established by an acceptance test responsible person, an acceptance environment is selected, the benchmark verification starting time before the platform is upgraded is preset, the acceptance scheduling module detects the fact that the benchmark verification time is up, and the cases in the acceptance case library are automatically replayed in the acceptance environment so as to obtain the test result data before the platform is upgraded.

Step 103: determining whether the comparison verification starting time after the target platform is upgraded is reached;

step 104: under the condition that the upgraded comparison verification starting time is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain upgraded test result data;

namely, after the platform is upgraded, the acceptance test responsible person can select an acceptance environment, the comparison verification starting time after the platform is upgraded is preset, the acceptance scheduling module detects that the comparison verification time is up, and the incremental and stock function acceptance cases of the platform frame dimension are automatically and sequentially played back in the acceptance environment, namely, the cases in which the acceptance case base is completely concentrated are automatically played back in the acceptance environment, so that the test result data before the upgrade is obtained.

Step 105: and automatically comparing the test result data before upgrading with the test result after upgrading case by case, and generating an acceptance report according to the comparison result.

The above mentioned acceptance case complete set of the target platform can be set by the following steps:

s1: acquiring functional test cases and non-functional test cases related to the target platform from a plurality of target sources;

s2: storing the obtained related functional test cases and non-functional test cases into an acceptance case library;

s3: determining whether the cases in the acceptance case base change;

s4: and under the condition that the change is determined, updating the cases in the test case library in an automatic increment aggregation mode or a stock updating mode.

That is, test cases, which may be functional or non-functional, are obtained from multiple sources and updated in real-time to ensure the integrity and validity of the case database. Wherein the target source may include, but is not limited to, at least one of: the method comprises the following steps of function automatic test cases of platform framework test design of each version, batch automatic test cases for regression verification, platform automatic performance test cases on a performance test platform and platform reliability automatic test cases on a chaotic engineering platform, wherein the platform is customized on a batch test system.

When the test results are compared, the comparison can be automatically performed according to the following rules: the upgraded platform framework function success cases, the upgraded reliability effective cases and the batch and performance acceptance cases which are less in consumed time or less in consumed time increase and have the amplitude lower than the preset fault tolerance threshold value are judged to pass the acceptance; and judging that the upgraded platform framework fails to pass the functions, the upgraded reliability failure cases and batch and performance acceptance cases with the time consumption increasing amplitude exceeding a preset fault tolerance threshold value are not accepted.

Namely, whether the upgraded platform meets the requirements or not is determined by taking reliability, time-consuming duration and the like as judgment targets.

For the comparison result, an acceptance report may be generated according to the comparison result, for example, statistics may be performed in different dimensions according to the comparison result; displaying the statistical result in sections in an html format; wherein the sections may include a summary section and an acceptance detail section, wherein the summary section may include: the platform of this acceptance, the task name of this acceptance, the starting time of the acceptance task, the ending time of the acceptance task, the total number of acceptance cases, the time-consuming fault-tolerant threshold value and the case total passing rate, and the acceptance detail sections can include: the number of cases of each acceptance content, the case passing rate and the case detail not passing.

The above method is described below with reference to a specific example, however, it should be noted that the specific example is only for better describing the present application and is not to be construed as limiting the present application.

In the embodiment, aiming at the defects of the existing acceptance test, a platform-oriented automatic acceptance test method is provided, and by dynamically constructing a unified acceptance case library, effective automatic test cases of the platform, platform application and special tests in daily monthly versions are automatically collected in real time and played back, so that the pertinence of the acceptance test is improved, and the quality of the acceptance test is ensured; by the task scheduling mechanism and the automatic quality comparison, unattended operation in the acceptance execution process and automatic generation of an acceptance report are realized, the acceptance communication cost is saved, and the acceptance test efficiency is improved.

Specifically, as shown in fig. 2, the method includes the following steps:

step S1, a new acceptance test task is established by an acceptance test responsible person, an acceptance environment is selected, and the reference verification starting time before the platform is upgraded is preset;

step S2, the acceptance scheduling module detects that the benchmark verification time is up, automatically plays back the stock function acceptance case of the platform frame dimension in the acceptance case base in the acceptance environment, and calls the data storage module to store the result data before upgrading to the database;

step S3, continuously and automatically playing back batch acceptance cases applied by the platform in the acceptance case library in an acceptance environment, and calling a data storage module to store the batch data before upgrading to a database;

step S4, connecting the performance testing platform, automatically playing back the non-functional acceptance case of the performance test in the acceptance case library in the acceptance environment, and calling the data storage module to store the performance data before upgrading to the database;

step S5, connecting the chaos testing platform, automatically playing back the non-functional acceptance case of the reliability test in the acceptance case base in the acceptance environment, and calling the data storage module to store the reliability data before upgrading to the database;

step S6, upgrading the platform;

step S7, the acceptance test responsible person selects the acceptance environment and presets the comparison verification starting time after the platform is upgraded;

step S8, the acceptance scheduling module detects that the comparison and verification time is up, automatically and sequentially plays back the increment of the platform frame dimension and the stock function acceptance case in the acceptance environment, and calls the data storage module to store the result data before upgrading to the database;

the batch acceptance and non-function acceptance part of the platform application repeats the steps S3-S5, and calls a data storage module to store the upgraded batch, performance and reliability data to a database;

step S9, the quality comparison module automatically compares the data before and after the case-level upgrade and generates an acceptance report;

and step S10, automatically pushing an acceptance report to an acceptance test responsible person.

For the above-mentioned test flow, mainly as shown in fig. 3, the following modules are involved:

1) the acceptance case library module 1: and subscribing related functional test cases and non-functional test cases of the platform through multiple channels, and automatically performing incremental collection or updating stock to a unified acceptance case library for playback of acceptance tests after the cases are changed. The source of cases, including but not limited to: the method comprises the following steps of function automatic test cases of platform framework test design of each version, batch automatic test cases for regression verification, platform automatic performance test cases on a performance test platform and platform reliability automatic test cases on a chaotic engineering platform, wherein the platform is customized on a batch test system.

2) And (3) an acceptance scheduling module 2: starting an acceptance task at a selected acceptance environment regularly according to a time point preset by an acceptance test responsible person, and executing acceptance in two steps for an acceptance case library: firstly, checking and accepting functions, namely, firstly, replaying platform frame function checking and accepting cases, and then, replaying platform application batch checking and accepting cases; and secondly, non-functional acceptance, namely connecting a performance test platform playback performance acceptance case first and then connecting a chaotic engineering platform playback reliability acceptance case. In the period, all playback actions are automatically executed in series without manual intervention, and the result data is pushed to the data storage module after the execution is finished.

3) The data storage module 3: after the acceptance task is started, automatically initializing an acceptance case set of the task obtained from the acceptance case library to the MySQL database, subsequently and synchronously receiving result data from an acceptance scheduling module after the case is played back, and automatically updating the corresponding case record. And when the test execution is finished completely, namely the test results of all the acceptance cases are updated, calling the quality comparison module to automatically carry out data comparison before and after upgrading.

4) A quality comparison module 4: and taking the acceptance task as a unit, respectively calling test result data before and after the platform is upgraded from the MySQL database, carrying out case-level automatic comparison, automatically generating an acceptance report in an html format, and automatically pushing the acceptance report to an acceptance test responsible person for production implementation decision so as to realize closed loop of the acceptance task.

As shown in fig. 4, the acceptance case library module 1 may include: a case change identification means 101 and a case maintenance means 102.

The case change recognition device 101 precisely recognizes the platform test cases in each case source channel through the platform identifier, automatically finds whether incremental cases are generated or storage cases are changed according to the last update time of the cases, and then transmits the corresponding information of case numbers, acceptance categories, case source channels and the like to the case maintenance device 102. The case maintenance device 102 stores the cases in a warehouse in real time and keeps the cases fresh according to the case change information received from the case change identification device 101. If the new case is the new case, executing an inserting operation; if the case is an inventory case, an update operation is performed. The case tables corresponding to different acceptance cases can be shown in table 1, table 2, table 3 and table 4, and the case maintenance device 102 is further configured to periodically scan the source channel of each case, find whether there is an invalid test case in the acceptance case base but which has been discarded in the case source channel, and if so, execute a deletion operation to ensure the validity of the acceptance case.

Table 1: platform frame function check and acceptance case list

Table 2: platform application batch acceptance case list

Table 3: performance test case table

Table 4: example table for reliability test

Fig. 5 is a schematic diagram of an acceptance scheduling module, which may include: a schedule executor 201 and a case executor 202.

The scheduling executor 201 is configured to start an acceptance task at regular time according to a preset verification time point, pull an acceptance case complete set of a platform to be accepted from an acceptance case library, and download the acceptance case complete set to the acceptance task initializing device 301 for task initialization. After initialization is completed, the case executor 202 is sequentially called to execute the acceptance cases in the acceptance case library according to the sequence of platform framework function acceptance, platform application batch acceptance, platform performance acceptance and platform reliability acceptance. Whether each item of acceptance content is completed or not is automatically judged according to an end mark returned by the case executor 202.

The case executor 202 is used for interacting with the test design system, the batch test system, the performance test platform and the chaotic engineering platform through interfaces, all acceptance cases are firstly sequenced according to case numbers, then the corresponding case source channels are directly connected one by one, test parameters are configured, test data are recovered, test scripts and expected result scripts are executed, and case execution results, execution time consumption and execution starting and stopping time which are reversely transmitted by each channel are downloaded to the result data recording device 302.

Fig. 6 is a schematic diagram of the data storage module 3, which may include: acceptance task initializing means 301 and result data recording means 302.

The acceptance task initializing device 301 is configured to convert an acceptance case complete set downloaded by the scheduling executor 201 into an SQL statement and insert the SQL statement into the MySQL database after the acceptance task is started, and initialize the task when four fields of an execution result, execution time consumption, execution start time, and execution end time are empty. The result data tables before and after upgrading have the same structure, but are stored separately for checking, which may be specifically shown in table 5.

Table 5: result data table of before/after upgrade acceptance case

And the result data recording device 302 is used for receiving the acceptance case result data downloaded by the case executor in real time, updating the pre-upgrade/post-acceptance case result data table in the MySQL according to the belonging acceptance rounds, and recording the execution result, the execution time consumption, the execution start time and the execution end time. When the test execution of the two rounds before and after upgrading is finished, namely the test results of all acceptance cases in the result data table of the before/after-upgrading acceptance case are updated, the result data comparison device 401 is called to automatically perform data comparison before and after upgrading.

Fig. 7 is a schematic diagram of the mass comparison module 4, which may include: result data comparing means 401, acceptance report generating means 402, and acceptance report pushing means 403.

The result data comparison device 401 is configured to respectively retrieve test result data before and after the platform is upgraded from the MySQL database, automatically compare the test result data case by case, and send the detail result to the acceptance report generation device 402. Wherein, the specific logic of comparison is as follows: the upgraded platform framework function success cases, the upgraded reliability effective cases and the batch and performance acceptance cases which are less in consumed time or less in consumed time increase and have the amplitude lower than the preset fault tolerance threshold value are judged to pass the acceptance; and judging that the upgraded platform framework functions do not pass through the cases, the reliability failure cases and the batch and performance acceptance cases with the time consumption increasing amplitude exceeding the preset fault-tolerant threshold value and fail to pass the acceptance.

The acceptance report generating device 402 is configured to perform statistics in different dimensions according to the detail result, and may display the following information in different sections in an html format: an acceptance summary section, which comprises a platform and a task name of the acceptance, the starting time and the ending time of the acceptance task, the total number of acceptance cases, a time-consuming fault-tolerant threshold value and a case total passing rate; the acceptance detail section comprises the number of cases of each acceptance content, the case passing rate, the case failing details and the like. Specific examples can be as shown in example 1 below. After the report file is generated, the acceptance report pushing device 403 is called to push the report file.

Sample 1: acceptance report sample

And the acceptance report pushing device 403 is configured to automatically use the generated html acceptance report as an attachment, push a mail according to a preset target person, and implement a decision for production and implement a closed loop of an acceptance task.

In the above example, a platform-oriented automatic acceptance testing framework and method are provided, aiming at a platform upgrading scene, through high automation, acceptance tasks are realized from case preparation, to task initiation, to case execution, to result comparison, and finally a report is generated and a closed loop is pushed to be unattended in the whole process, so that a large amount of management and communication cost is saved, and the acceptance efficiency is obviously improved compared with that of the existing acceptance testing method; meanwhile, in the aspect of acceptance quality, each platform applies the automatic batch test cases with effective band number in the process of multiplexing daily version test, so that wider application coverage and more comprehensive batch scene coverage are realized, and the test pertinence is obviously improved. Specifically, comparing the acceptance test method provided in this example with the existing acceptance test method, the following comparison results can be obtained as shown in table 6 below:

table 6: comparison of traditional acceptance testing with the inventive test framework

The method embodiments provided in the above embodiments of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking the electronic device as an example, fig. 8 is a hardware structure block diagram of the electronic device of the platform-oriented automatic assay receiving method provided in the present application. As shown in fig. 8, the electronic device 10 may comprise one or more (only one shown in the figure) processors 02 (the processors 02 may comprise, but are not limited to, a processing means such as a microprocessor MCU or a programmable logic device FPGA), a memory 04 for storing data, and a transmission module 06 for communication functions. It will be understood by those skilled in the art that the structure shown in fig. 8 is only an illustration and is not intended to limit the structure of the electronic device. For example, electronic device 10 may also include more or fewer components than shown in FIG. 8, or have a different configuration than shown in FIG. 8.

The memory 04 may be configured to store software programs and modules of application software, such as program instructions/modules corresponding to the platform-oriented automatic test receiving method in the embodiment of the present application, and the processor 02 executes various functional applications and data processing by running the software programs and modules stored in the memory 04, so as to implement the above-mentioned platform-oriented automatic test receiving method of the application software. The memory 04 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 04 may further include memory located remotely from the processor 02, which may be connected to the electronic device 10 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission module 06 is used for receiving or transmitting data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the electronic device 10. In one example, the transmission module 06 includes a Network adapter (NIC) that can be connected to other Network devices through a base station so as to communicate with the internet. In one example, the transmission module 06 may be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.

At the software level, the above-mentioned platform-oriented automated testing apparatus may be as shown in fig. 9, and may include:

a first determining module 901, configured to determine whether a benchmark verification start time before the target platform is upgraded is reached;

the first execution module 902 is configured to, when it is determined that the standard verification start time before the upgrade is reached, pull and execute a full set of acceptance cases of the target platform from the acceptance case library to obtain test result data before the upgrade;

a second determining module 903, configured to determine whether comparison verification start time after the target platform is upgraded is reached;

a second executing module 904, configured to pull and execute the full set of acceptance cases of the target platform from the acceptance case library to obtain upgraded test result data when it is determined that the upgraded comparison verification start time is reached;

the comparison module 905 is configured to automatically compare the test result data before the upgrade with the test result after the upgrade, case by case, and generate an acceptance report according to the comparison result.

In one embodiment, the above platform-oriented automated assay receiving device may further comprise: the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring functional test cases and non-functional test cases related to a target platform from a plurality of target sources before a test result data before upgrading is obtained by pulling and executing a full set of acceptance cases of the target platform from an acceptance case library; the storage module is used for storing the obtained related functional test cases and non-functional test cases into an acceptance case library; a third determining module, configured to determine whether a case in the acceptance case base changes; and the updating module is used for updating the cases in the test case library in an automatic increment aggregation mode or a stock updating mode under the condition of determining the change.

In one embodiment, the target source may include, but is not limited to, at least one of: the method comprises the following steps of function automatic test cases of platform framework test design of each version, batch automatic test cases for regression verification, platform automatic performance test cases on a performance test platform and platform reliability automatic test cases on a chaotic engineering platform, wherein the platform is customized on a batch test system.

In an embodiment, the comparison module 905 may specifically perform the comparison according to the following rules: the upgraded platform framework function success cases, the upgraded reliability effective cases and the batch and performance acceptance cases which are less in consumed time or less in consumed time increase and have the amplitude lower than the preset fault tolerance threshold value are judged to pass the acceptance; and judging that the upgraded platform framework fails to pass the functions, the upgraded reliability failure cases and batch and performance acceptance cases with the time consumption increasing amplitude exceeding a preset fault tolerance threshold value are not accepted.

In an embodiment, the comparison module 905 may specifically perform statistics in different dimensions according to a comparison result; displaying the statistical result in sections in an html format;

wherein the sections comprise an overview section and an acceptance detail section, wherein the overview section comprises: the platform of this acceptance, the task name of this acceptance, the starting time of the acceptance task, the ending time of the acceptance task, the total number of the acceptance cases, the time-consuming fault-tolerant threshold value and the case total passing rate, and the acceptance detail section comprises: the number of cases of each acceptance content, the case passing rate and the case detail not passing.

The embodiment of the present application further provides a specific implementation manner of an electronic device capable of implementing all steps in the platform-oriented automatic assay receiving method in the above embodiment, where the electronic device specifically includes the following contents: a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the processor is configured to call a computer program in the memory, and the processor implements all the steps of the platform-oriented automatic assay receiving method in the above embodiments when executing the computer program, for example, the processor implements the following steps when executing the computer program:

step 1: determining whether the standard verification starting time before the target platform is upgraded is reached;

step 2: under the condition that the standard verification starting time before upgrading is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain test result data before upgrading;

and step 3: determining whether the comparison verification starting time after the target platform is upgraded is reached;

and 4, step 4: under the condition that the upgraded comparison verification starting time is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain upgraded test result data;

and 5: and automatically comparing the test result data before upgrading with the test result after upgrading case by case, and generating an acceptance report according to the comparison result.

From the above description, it can be seen that, in the embodiment of the present application, by setting the reference verification start time and the comparison verification start time, and setting the full set of acceptance cases, the case test result before the upgrade and the case test result after the upgrade can be automatically triggered, and the comparison between the test result before the upgrade and the test result after the upgrade can be automatically performed, so as to generate an acceptance report, thereby solving the problem of low efficiency and accuracy of the existing acceptance test, and achieving the technical effect of accurately and efficiently performing the automatic acceptance test.

Embodiments of the present application further provide a computer-readable storage medium capable of implementing all steps in the platform-oriented automatic assay receiving method in the above embodiments, where the computer-readable storage medium stores thereon a computer program, and when the computer program is executed by a processor, the computer program implements all steps of the platform-oriented automatic assay receiving method in the above embodiments, for example, the processor implements the following steps when executing the computer program:

step 1: determining whether the standard verification starting time before the target platform is upgraded is reached;

step 2: under the condition that the standard verification starting time before upgrading is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain test result data before upgrading;

and step 3: determining whether the comparison verification starting time after the target platform is upgraded is reached;

and 4, step 4: under the condition that the upgraded comparison verification starting time is determined, pulling and executing a full set of acceptance cases of the target platform from an acceptance case library to obtain upgraded test result data;

and 5: and automatically comparing the test result data before upgrading with the test result after upgrading case by case, and generating an acceptance report according to the comparison result.

From the above description, it can be seen that, in the embodiment of the present application, by setting the reference verification start time and the comparison verification start time, and setting the full set of acceptance cases, the case test result before the upgrade and the case test result after the upgrade can be automatically triggered, and the comparison between the test result before the upgrade and the test result after the upgrade can be automatically performed, so as to generate an acceptance report, thereby solving the problem of low efficiency and accuracy of the existing acceptance test, and achieving the technical effect of accurately and efficiently performing the automatic acceptance test. The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the hardware + program class embodiment, since it is substantially similar to the method embodiment, the description is simple, and the relevant points can be referred to the partial description of the method embodiment.

The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

Although the present application provides method steps as described in an embodiment or flowchart, additional or fewer steps may be included based on conventional or non-inventive efforts. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an actual apparatus or client product executes, it may execute sequentially or in parallel (e.g., in the context of parallel processors or multi-threaded processing) according to the embodiments or methods shown in the figures.

The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

Although embodiments of the present description provide method steps as described in embodiments or flowcharts, more or fewer steps may be included based on conventional or non-inventive means. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an actual apparatus or end product executes, it may execute sequentially or in parallel (e.g., parallel processors or multi-threaded environments, or even distributed data processing environments) according to the method shown in the embodiment or the figures. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, article, or apparatus that comprises the recited elements is not excluded.

For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, in implementing the embodiments of the present description, the functions of each module may be implemented in one or more software and/or hardware, or a module implementing the same function may be implemented by a combination of multiple sub-modules or sub-units, and the like. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may therefore be considered as a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

The embodiments of this specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The described embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment. In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of an embodiment of the specification. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

The above description is only an example of the embodiments of the present disclosure, and is not intended to limit the embodiments of the present disclosure. Various modifications and variations to the embodiments described herein will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present specification should be included in the scope of the claims of the embodiments of the present specification.