1. A network structure search method, comprising:

receiving a network structure search request; the network structure search request comprises at least two resource constraint conditions;

sampling in a preset network structure library according to the at least two resource constraint conditions to obtain at least one network module;

selecting the network parameters of each network module in a preset parameter set according to the at least two resource constraint conditions;

generating a target network structure through the at least one network module and the network parameters of each network module;

and determining the target network structure as a search result of the network structure search request, and outputting the search result.

2. The method of claim 1, further comprising: generating the target network structure by adopting a pre-trained controller; wherein the controller is trained by:

acquiring at least two sample constraint conditions;

sampling in a preset network structure library according to the sample constraint condition to obtain a network structure;

determining performance parameters and computational complexity of the network structure;

comparing the performance parameters and the computation complexity with the sample constraint conditions respectively to obtain a score for evaluating the network structure;

and carrying out optimization training on the controller according to the scores to obtain the trained controller.

3. The method of claim 2, wherein the controller performs the search for the target network structure through a control function; wherein the control function is represented by the following formula:

α_T＝f(T；θ)；

wherein α represents any network structure; t represents eitherThe upper bound of the sample constraint condition, and the network structure satisfying the sample constraint condition T is represented as c (alpha) less than T; alpha is alpha_TA network structure representing an upper bound T of controller outputs that satisfies a sample constraint; c (—) represents a computation cost function; f (×) represents a mapping function to be learned, and θ represents a parameter included in the mapping function.

4. The method according to claim 2, wherein the sampling processing in a preset network structure library according to the sample constraint condition to obtain a network structure comprises:

sequentially selecting the depth, the width and the convolution kernel size of the network structure according to the sample constraint condition;

acquiring depth characters, width characters and convolution kernel size characters which respectively correspond to the depth, the width and the convolution kernel size;

connecting the depth character, the width character and the convolution kernel size character to form a character string;

and generating the network structure according to the character string.

5. The method of claim 3, wherein the objective function of the controller is represented by the formula:

wherein E is_T～τRepresenting the calculated model average performance of the controller for a plurality of sample constraints;_{π(·|T；θ)}a network fabric sampling strategy representing an upper bound T for any sample constraint; e_{α～π(·|T；θ)}The average performance of the structure calculated after a plurality of network structures are sampled according to the learned network structure sampling strategy is represented; r (alpha)_TI, T; w) represents any network structure a under the upper bound T of any sample constraint_TA score of (d); w represents a parameter in the score model; τ represents the distribution of T;t-tau represents the distribution of the values of T and obeys tau; s.t. indicates constrained.

6. The method of claim 5, wherein comparing the performance parameter and the computational complexity with the sample constraint to obtain a score for evaluating the network structure comprises:

and respectively comparing the performance parameters and the computation complexity with the sample constraint conditions through the objective function to obtain the score for evaluating the network structure.

7. The method of claim 6, wherein the optimally training the controller according to the scores to obtain a trained controller comprises:

according to the scores, maximizing the sample constraint conditions and the structure average performance of the network structure to obtain the maximized sample constraint conditions and the maximized structure average performance of the network structure;

and carrying out optimization training on the controller according to the sample constraint conditions after the maximization processing and the structure average performance of the network structure after the maximization processing to obtain the trained controller.

8. The method of claim 6, wherein comparing the performance parameter and the computational complexity with the sample constraint via the objective function to obtain the score for evaluating the network structure comprises:

classifying the at least two sample constraints to form at least one sample constraint class, wherein each sample constraint class comprises at least one sample constraint;

obtaining the target function corresponding to each sample constraint condition class;

inputting the sample constraint conditions in each sample constraint condition class into the corresponding objective function to obtain the sub-scores of each sample constraint condition class;

and fusing at least one sub-score corresponding to the at least one sample constraint condition class to obtain the score for evaluating the network structure.

9. The method of claim 5, wherein comparing the performance parameter and the computational complexity with the sample constraint to obtain a score for evaluating the network structure comprises: determining the score for evaluating the network structure using the score model;

the score model comprises at least two full-connection layers and at least two nonlinear transformation processing layers, and one nonlinear transformation processing layer is connected behind each full-connection layer.

10. The method of claim 9, wherein said determining said score for evaluating said network structure using said score model comprises:

network structure alpha of upper bound satisfying sample constraint condition output by the controller_TAnd an upper bound T of the sample constraints, input into the score model;

applying the network structure alpha through the full connection layer and the nonlinear transformation processing layer in the score model_TAnd sequentially carrying out at least one full-connection process and at least one nonlinear transformation process on the upper bound T of the sample constraint condition to obtain the score of the network structure.

11. The method of claim 9, further comprising:

when a plurality of network structures are obtained by the sampling processing, comparing every two network structures according to the following formula:

wherein, beta₁Denotes the first network structure, beta₂Representing a second network structure;representing an indication function, outputting 1 when the condition is true, and outputting 0 when the condition is false; acc [. X [ ]]Representing the accuracy of the network structure; c]Representing the computational complexity of the network structure; g (. beta.) of₁,β₂T) denotes β below the upper bound T of the sample constraints₁Is greater than beta₂A good comparison result;

sequencing the network structures according to the comparison result of every two network structures to form a network structure sequence;

and sequentially determining the score of each network structure according to the network structure sequence.

12. The method of claim 11, wherein said determining said score for each network structure in turn according to said sequence of network structures comprises:

determining an objective function of the score model, wherein the objective function of the score model is represented by the following formula:

wherein, L (w) represents a loss function when training for the parameter w in the score model; k represents the number of sample constraints; m represents the number of the plurality of network structures; phi denotes a hinge loss function; beta is a_iRepresents the ith network structure; beta is a_jRepresents the jth network structure; t is_kAn upper bound representing a kth sample constraint; r (beta)_i|T_k(ii) a w) represents the upper bound T in the sample constraints_kLower network structure beta_iA score of (d); r (beta)_j|T_k(ii) a w) in sample constraintsUpper bound T_kLower network structure beta_jA score of (d); g (. beta.) of_i,β_j,T_k) Represents an upper bound T on the sample constraint_kLower beta_iIs greater than beta_jA good comparison result;

minimizing the loss function L (w) to obtain a minimization result;

and controlling the size relationship between the reward values of any two network structures sampled by the controller according to the minimization processing result, wherein the size relationship is consistent with the sequence of the network structure sequence.

13. A network structure search apparatus, comprising:

the receiving module is used for receiving a network structure searching request; the network structure search request comprises at least two resource constraint conditions;

the sampling processing module is used for carrying out sampling processing in a preset network structure library according to the at least two resource constraint conditions to obtain at least one network module;

a selection module, configured to select a network parameter of each network module in a preset parameter set according to the at least two resource constraint conditions;

the generating module is used for generating a target network structure through the at least one network module and the network parameters of each network module;

and the determining module is used for determining the target network structure as the search result of the network structure search request and outputting the search result.

14. A network structure search device characterized by comprising:

a memory for storing executable instructions; a processor for implementing the network structure search method of any one of claims 1 to 12 when executing executable instructions stored in the memory.

15. A computer-readable storage medium having stored thereon executable instructions for causing a processor to execute the executable instructions to implement the network architecture search method of any one of claims 1 to 12.

Background

In recent years, the deep learning base based on the deep neural network has attracted extensive attention in academic and industrial fields, and has achieved breakthrough results in a plurality of application fields, including picture recognition, target detection, semantic segmentation, speech recognition, natural language processing, and the like. However, the network structure corresponding to the depth model often has the problems of large parameter quantity, high calculation complexity and the like, and resource constraint in practical application is difficult to satisfy. Finding the appropriate network structure under certain constraints therefore becomes an important issue.

The method for searching and generating the network structure in the related art only considers the network structure design under the condition of single resource constraint. However, in an actual scenario, a series of resource constraints are often required to be considered to adapt to a plurality of different application scenarios, and therefore, the method in the related art cannot implement simultaneous search under a plurality of resource constraints to generate a network structure, and the search efficiency for a plurality of resource constraints in the related art is low.

Disclosure of Invention

The embodiment of the application provides a network structure searching method, a network structure searching device, network structure searching equipment and a computer readable storage medium, and relates to the technical field of artificial intelligence. Because at least one network module is obtained by sampling according to the resource constraint conditions, the network parameters of each network module are selected according to the resource constraint conditions, and the target network structure is generated through the network modules and the network parameters obtained by sampling, the aim of simultaneously searching under the resource constraint conditions to generate the target network structure is achieved, and the aim of simultaneously searching according to the resource constraint conditions is achieved, so that the searching efficiency is improved.

The technical scheme of the embodiment of the application is realized as follows:

the embodiment of the application provides a network structure searching method, which comprises the following steps:

receiving a network structure search request; the network structure search request comprises at least two resource constraint conditions; sampling in a preset network structure library according to the at least two resource constraint conditions to obtain at least one network module; selecting the network parameters of each network module in a preset parameter set according to the at least two resource constraint conditions; generating a target network structure through the at least one network module and the network parameters of each network module; and determining the target network structure as a search result of the network structure search request, and outputting the search result.

An embodiment of the present application provides a network structure search apparatus, including: the receiving module is used for receiving a network structure searching request; the network structure search request comprises at least two resource constraint conditions; the sampling processing module is used for carrying out sampling processing in a preset network structure library according to the at least two resource constraint conditions to obtain at least one network module; a selection module, configured to select a network parameter of each network module in a preset parameter set according to the at least two resource constraint conditions; the generating module is used for generating a target network structure through the at least one network module and the network parameters of each network module; and the determining module is used for determining the target network structure as the search result of the network structure search request and outputting the search result.

In some embodiments, the apparatus further comprises: a processing module for generating the target network structure using a pre-trained controller; wherein the controller is trained by: acquiring at least two sample constraint conditions; sampling in a preset network structure library according to the sample constraint condition to obtain a network structure; determining performance parameters and computational complexity of the network structure; comparing the performance parameters and the computation complexity with the sample constraint conditions respectively to obtain a score for evaluating the network structure; and carrying out optimization training on the controller according to the scores to obtain the trained controller.

In some embodiments, the controller implements a search of the target network structure through a control function; wherein the control function is represented by the following formula: alpha is alpha_TF (T; θ); wherein α represents any network structure; t represents the upper bound of any sample constraint condition, and the network structure meeting the sample constraint condition T is represented as c (alpha) less than T; alpha is alpha_TA network structure representing an upper bound T of controller outputs that satisfies a sample constraint; c (—) represents a computation cost function; f (×) represents a mapping function to be learned, and θ represents a parameter included in the mapping function.

In some embodiments, the controller is further trained by: sequentially selecting the depth, the width and the convolution kernel size of the network structure according to the sample constraint condition; acquiring depth characters, width characters and convolution kernel size characters which respectively correspond to the depth, the width and the convolution kernel size; connecting the depth character, the width character and the convolution kernel size character to form a character string; and generating the network structure according to the character string.

In some embodiments, the objective function of the controller is represented by the following formula:

wherein, Ε_T～τRepresenting the calculated model average performance of the controller for a plurality of sample constraints; pi (· | T; theta) represents a network structure sampling strategy for an upper bound T of any sample constraint condition; e_{α～π(·|T；θ)}The average performance of the structure calculated after a plurality of network structures are sampled according to the learned network structure sampling strategy is represented; r (alpha)_TI, T; w) represents any network structure a under the upper bound T of any sample constraint_TA score of (d); w represents a parameter in the score model; τ represents the distribution of T; t &Tau represents the distribution of the value distribution of T obeying tau; s.t. indicates constrained.

In some embodiments, the controller is further trained by: and respectively comparing the performance parameters and the computation complexity with the sample constraint conditions through the objective function to obtain the score for evaluating the network structure.

In some embodiments, the controller is further trained by: according to the scores, maximizing the sample constraint conditions and the structure average performance of the network structure to obtain the maximized sample constraint conditions and the maximized structure average performance of the network structure; and carrying out optimization training on the controller according to the sample constraint conditions after the maximization processing and the structure average performance of the network structure after the maximization processing to obtain the trained controller.

In some embodiments, the controller is further trained by: classifying the at least two sample constraints to form at least one sample constraint class, wherein each sample constraint class comprises at least one sample constraint; obtaining the target function corresponding to each sample constraint condition class; inputting the sample constraint conditions in each sample constraint condition class into the corresponding objective function to obtain the sub-scores of each sample constraint condition class; and fusing at least one sub-score corresponding to the at least one sample constraint condition class to obtain the score for evaluating the network structure.

In some embodiments, the controller is further trained by: determining the score for evaluating the network structure using the score model; the score model comprises at least two full-connection layers and at least two nonlinear transformation processing layers, and one nonlinear transformation processing layer is connected behind each full-connection layer.

In some embodiments, the controller is further trained by: network structure alpha of upper bound satisfying sample constraint condition output by the controller_TAnd an upper bound T of the sample constraints, input into the score model; applying the network structure alpha through the full connection layer and the nonlinear transformation processing layer in the score model_TAnd sequentially carrying out at least one full-connection process and at least one nonlinear transformation process on the upper bound T of the sample constraint condition to obtain the score of the network structure.

In some embodiments, the apparatus further comprises: a comparing module, configured to compare every two network structures according to the following formula when the plurality of network structures are obtained through the sampling processing:

wherein, beta₁Denotes the first network structure, beta₂Representing a second network structure; [*]Representing an indication function, outputting 1 when the condition is true, and outputting 0 when the condition is false; acc [. X [ ]]Representing the accuracy of the network structure; c]Representing the computational complexity of the network structure; g (. beta.) of₁,β₂T) denotes β below the upper bound T of the sample constraints₁Is greater than beta₂A good comparison result; the sequencing module is used for sequencing the network structures according to the comparison result of every two network structures to form a network structure sequence; and the score determining module is used for sequentially determining the score of each network structure according to the network structure sequence.

In some embodiments, the score determination module is further to: determining an objective function of the score model, wherein the objective function of the score model is represented by the following formula:

wherein, L (w) represents a loss function when training for the parameter w in the score model; k represents the number of sample constraints; m represents the number of the plurality of network structures; phi denotes hinge lossA function; beta is a_iRepresents the ith network structure; beta is a_jRepresents the jth network structure; t is_kAn upper bound representing a kth sample constraint; r (beta)_i|T_k(ii) a w) represents the upper bound T in the sample constraints_kLower network structure beta_iA score of (d); r (beta)_j|T_k(ii) a w) represents the upper bound T in the sample constraints_kLower network structure beta_jA score of (d); g (. beta.) of_i,β_j,T_k) Represents an upper bound T on the sample constraint_kLower beta_iIs greater than beta_jA good comparison result; minimizing the loss function L (w) to obtain a minimization result; and controlling the size relationship between the reward values of any two network structures sampled by the controller according to the minimization processing result, wherein the size relationship is consistent with the sequence of the network structure sequence.

Embodiments of the present application provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor is configured to execute the computer instructions to implement the network structure searching method.

An embodiment of the present application provides a network structure search device, including: a memory for storing executable instructions; and the processor is used for realizing the network structure searching method when executing the executable instructions stored in the memory.

The embodiment of the present application provides a computer-readable storage medium, which stores executable instructions for causing a processor to execute the executable instructions to implement the network structure searching method described above.

The embodiment of the application has the following beneficial effects: the method comprises the steps of sampling in a preset network structure library according to a plurality of resource constraint conditions to obtain at least one network module, selecting network parameters of each network module in a preset parameter set according to the resource constraint conditions, and generating a target network structure according to the network modules and the network parameters obtained by sampling.

Drawings

FIG. 1A is a diagram of a NAS system framework in the related art;

fig. 1B is a network structure diagram of MobileNet in the related art;

FIG. 1C is a schematic diagram of a network structure search of a related art MnasNet taking into account computational resource constraints;

fig. 2 is a schematic diagram of an alternative architecture of a network structure search system provided in an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a server provided in an embodiment of the present application;

fig. 4 is an alternative flow chart of a network structure searching method provided in the embodiment of the present application;

FIG. 5 is a schematic flow chart diagram illustrating an alternative method for training a controller according to an embodiment of the present disclosure;

FIG. 6 is a schematic flow chart diagram illustrating an alternative method for training a controller according to an embodiment of the present disclosure;

FIG. 7 is a schematic flow chart diagram illustrating an alternative controller training method according to an embodiment of the present disclosure;

FIG. 8 is a schematic flow chart diagram illustrating an alternative method for training a controller according to an embodiment of the present disclosure;

FIG. 9 is a schematic flow chart diagram illustrating an alternative method for training a controller according to an embodiment of the present disclosure;

fig. 10 is a block diagram of a network structure search system provided in an embodiment of the present application;

FIG. 11 is a schematic diagram of LSTM generating a string representing a network structure according to an embodiment of the present application;

fig. 12 is a model structure diagram of a neural network structure evaluator provided in an embodiment of the present application;

fig. 13 is a schematic flow chart of an AutoML product according to an embodiment of the present application.

Detailed Description

In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the attached drawings, the described embodiments should not be considered as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of the present application belong. The terminology used in the embodiments of the present application is for the purpose of describing the embodiments of the present application only and is not intended to be limiting of the present application.

In order to better understand the network structure search method provided in the embodiment of the present application, first, a network structure search method in the related art is explained:

research efforts in the related art have focused on network structure search methods and network structure design methods for specific computing resource constraints. The neural network structure search method automatically generates a structure of a neural network using a Controller (Controller). The controller is optimized by using reinforcement learning techniques or other related techniques so that the controller can generate an optimal neural network structure.

The design of a network structure aiming at specific computing resource constraint refers to designing or searching a network structure meeting the resource constraint condition according to the computing resource constraint of a specific hardware platform.

The method in the related art mainly comprises the following steps: network Architecture Search (NAS) and network Architecture design for specific computing resource constraints.

In the network structure searching method, fig. 1A is a NAS system framework diagram in the related art, where the NAS system includes a controller 101 and a random initialization model 102, and the controller 101 may be a Long Short-Term Memory network (LSTM) based controller.

NAS generates the structure of a neural network by using an LSTM-based controller: for example, for a convolutional neural network, including the number of channels to be convolved, the height and width of the convolution kernel, etc.; for a recurrent neural network, connected nodes are included, as well as activation functions, among others. In this way, the specific network structure of a possible neural network and the probability of the controller generating the network structure can be derived. And training parameters of the model on a specific task by using the network structure to obtain the performance of the network structure. Finally, the accuracy of the generated network structure is calculated and verified through the random initialization model 102, the performance of the generated network structure is determined according to the accuracy, the performance (generally, accuracy) of the generated network structure and the probability of generating the network structure are fed back to the controller, and the performance of the controller 101 is optimized through a reinforcement learning technology, so that the controller 101 can generate a better network structure.

By repeating the above processes, the network structure generated by the controller 101 has better and better performance on specific tasks, so that the purpose of automatically generating the neural network structure by using a computer is achieved.

In the network structure design method aiming at the specific computing resource constraint, the current network structure design method aiming at the specific computing resource constraint mainly comprises a manually designed lightweight neural network and a network structure which meets the constraint and is automatically searched by a NAS (network attached storage). In order to obtain a lightweight network, a manual design method often depends on abundant manual design experience, and is difficult to apply to different tasks and scenes. For example, in the MobileNet, the calculation method and the convolution kernel size of each layer need to be carefully designed, as shown in fig. 1B, it is a network structure diagram of the MobileNet in the related art, and in the MobileNet network structure, at least a first convolution layer 120(conv _1 × 1), a separable convolution layer 121(sep _ conv _3 × 3), and a second convolution layer 122(conv _1 × 1) are included.

For the constrained network structure search method, the method in the related art can only consider a single calculation constraint. For example, in the mnsernt, a multi-objective reward function is constructed and a specific delay constraint is introduced to adjust a final reward value, as shown in fig. 1C, which is a network structure search diagram of the mnsernt in the related art in consideration of the constraint of computing resources, the controller 130 obtains a network structure by sampling, and sends the network structure to the model trainer 131, the model trainer 131 and the mobile device 132 perform multi-objective reward 133 according to the accuracy and delay of the network structure obtained by sampling by the controller 130, and send a reward value to the controller 130, thereby realizing that a network structure meeting the computing constraint is found by maximizing the reward value.

However, the methods in the related art have at least the following disadvantages: in the related art, most network structure search algorithms do not consider the calculation constraint (i.e. calculation resource constraint) of the actual scene, and are difficult to be directly applied to real application. The manually designed lightweight model is very dependent on the design experience of designers, the design process is tedious, and the method is difficult to be widely applied to various tasks and different scenes. For a network structure search method with computation constraints, when there are a series of different computation constraints, the existing method needs to be repeated for many times, so that a suitable network structure is found for each situation, and the process is very tedious and inefficient.

In order to solve at least one problem of a network structure searching method in the related art, the embodiment of the application provides a network structure searching method, a general network structure designer (namely a controller) is designed, any computing resource constraint is given, and a network structure conforming to the computing resource constraint is automatically given, so that the difficulty of designing an actual scene model is reduced.

The network structure searching method provided by the embodiment of the application comprises the steps of firstly, receiving a network structure searching request; the network structure search request comprises at least two resource constraint conditions; then, according to at least two resource constraint conditions, sampling processing is carried out in a preset network structure library to obtain at least one network module; selecting the network parameters of each network module in a preset parameter set according to at least two resource constraint conditions; finally, generating a target network structure through at least one network module and the network parameters of each network module; and determining the target network structure as a search result of the network structure search request, and outputting the search result. Therefore, as the sampling processing and selection are carried out in the preset network structure library aiming at the plurality of resource constraint conditions to obtain the final target network structure, the simultaneous searching under the plurality of resource constraint conditions is realized to generate the target network structure, and the simultaneous searching aiming at the plurality of resource constraint conditions is carried out, so that the searching efficiency is improved.

An exemplary application of the network structure search device according to the embodiment of the present application is described below, in one implementation, the network structure search device according to the embodiment of the present application may be implemented as any terminal such as a notebook computer, a tablet computer, a desktop computer, a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable game device), an intelligent robot, an intelligent video monitoring or traffic monitoring apparatus, and in another implementation, the network structure search device according to the embodiment of the present application may also be implemented as a server. Next, an exemplary application when the network configuration search device is implemented as a server will be explained.

Referring to fig. 2, fig. 2 is a schematic diagram of an alternative architecture of the network structure search system 10 according to the embodiment of the present application. In the embodiment of the present application, a target network structure is taken as an image recognition network as an example, in order to implement generation of an image recognition network for recognizing an image, a network structure search system 10 provided in the embodiment of the present application includes a terminal 100, a network 200, and a server 300, where the terminal 100 runs a network structure search application, a user may request generation of a specific type of target network structure (e.g., an image recognition network) on a client of the network structure search application, during implementation, the terminal 100 sends a network structure search request to the server 300 through the network 200, the network structure search request includes at least two resource constraints, the server 300, after receiving the network structure search request, performs sampling processing in a preset network structure library according to the at least two resource constraints in response to the network structure search request, obtaining at least one network module; selecting the network parameters of each network module in a preset parameter set according to at least two resource constraint conditions; generating a target network structure through at least one network module and the network parameters of each network module; and determines the target network structure as a search result of the network structure search request and transmits the search result to the terminal 100.

The network structure searching method provided by the embodiment of the application also relates to the technical field of artificial intelligence, and can be realized through a machine learning technology in the artificial intelligence technology. Machine Learning (ML) is a one-field multi-field cross subject, and relates to multiple subjects such as probability theory, statistics, approximation theory, convex analysis and algorithm complexity theory. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal education learning. In the embodiment of the application, the response to the network structure search request is realized through a machine learning technology so as to automatically search a target network structure, and the training and model optimization of the controller and the score model are realized.

Fig. 3 is a schematic structural diagram of a server 300 according to an embodiment of the present application, where the server 300 shown in fig. 3 includes: at least one processor 310, memory 350, at least one network interface 320, and a user interface 330. The various components in server 300 are coupled together by a bus system 340. It will be appreciated that the bus system 340 is used to enable communications among the components connected. The bus system 340 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are labeled as bus system 340 in fig. 3.

The Processor 310 may be an integrated circuit chip having Signal processing capabilities, such as a general purpose Processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like, wherein the general purpose Processor may be a microprocessor or any conventional Processor, or the like.

The user interface 330 includes one or more output devices 331, including one or more speakers and/or one or more visual display screens, that enable presentation of media content. The user interface 330 also includes one or more input devices 332, including user interface components to facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

The memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid state memory, hard disk drives, optical disk drives, and the like. Memory 350 optionally includes one or more storage devices physically located remote from processor 310. The memory 350 may include either volatile memory or nonvolatile memory, and may also include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read Only Memory (ROM), and the volatile Memory may be a Random Access Memory (RAM). The memory 350 described in embodiments herein is intended to comprise any suitable type of memory. In some embodiments, memory 350 is capable of storing data, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as exemplified below, to support various operations.

An operating system 351 including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

a network communication module 352 for communicating to other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 including: bluetooth, wireless compatibility authentication (WiFi), and Universal Serial Bus (USB), etc.;

an input processing module 353 for detecting one or more user inputs or interactions from one of the one or more input devices 332 and translating the detected inputs or interactions.

In some embodiments, the apparatus provided in this embodiment of the present application may be implemented in software, and fig. 3 illustrates a network structure searching apparatus 354 stored in the memory 350, where the network structure searching apparatus 354 may be a network structure searching apparatus in the server 300, and may be software in the form of programs and plug-ins, and includes the following software modules: the receiving module 3541, the sample processing module 3542, the selecting module 3543, the generating module 3544, and the determining module 3545 are logical and thus may be combined or further split in any combination depending on the functionality implemented. The functions of the respective modules will be explained below.

In other embodiments, the apparatus provided in the embodiments of the present Application may be implemented in hardware, and for example, the apparatus provided in the embodiments of the present Application may be a processor in the form of a hardware decoding processor, which is programmed to execute the network structure searching method provided in the embodiments of the present Application, for example, the processor in the form of the hardware decoding processor may be one or more Application Specific Integrated Circuits (ASICs), DSPs, Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field Programmable Gate Arrays (FPGAs), or other electronic components.

The network structure searching method provided by the embodiment of the present application will be described below with reference to an exemplary application and implementation of the server 300 provided by the embodiment of the present application. Referring to fig. 4, fig. 4 is an alternative flow chart of a network structure searching method provided in the embodiment of the present application, and will be described with reference to the steps shown in fig. 4.

Step S401, receiving a network structure search request; the network structure search request includes at least two resource constraints.

The network structure search request is used for requesting to search a network structure so as to search for a network structure meeting the at least two resource constraint conditions.

In the embodiment of the application, a user can send a network structure search request to a server through a terminal, a network structure search application runs on the terminal, or an application corresponding to a specific type of network structure runs on the terminal, and the user inputs a search requirement (namely a resource constraint condition) on the terminal, so that the terminal encapsulates the resource constraint condition in the network structure search request and sends the network structure search request to the server.

For example, the application scenario of the embodiment of the present application may be that a network structure search application is run on a terminal, and the network structure search application can automatically search a network structure required by a user, so that the user may input a plurality of resource constraint conditions on a client of the application, where the plurality of resource constraint conditions are a plurality of different constraint values set by different users with respect to a constraint condition of the same type, a plurality of different constraint values set by the same user with respect to a constraint condition of different types, or a plurality of different constraint values set by different users with respect to a constraint condition of a different type. After receiving the constraint value input by the user, the terminal encapsulates the constraint value as a resource constraint condition in a network structure search request, and sends the network structure search request to the server, so that the server responds to the network structure search request to search a target network structure meeting the constraint value input by the user.

For another example, in the application scenario of the embodiment of the present application, an image recognition application may be run on the terminal, and when the image recognition application performs image recognition, it needs to perform image recognition through a specific image recognition network, so that the target network structure for performing image recognition may also be searched by using the method of the embodiment of the present application. Of course, the application scenario in the embodiment of the present application is only exemplary, and besides the image recognition application is run on the terminal, the target network structure for performing image recognition needs to be searched, and the text processing application may also be run on the terminal, the target network structure for performing text processing needs to be searched, and the like.

Step S402, according to at least two resource constraint conditions, sampling processing is carried out in a preset network structure library to obtain at least one network module.

Here, the preset network structure library stores a plurality of network modules, the network modules are modules for forming network structures, the network modules corresponding to different network structures are different, and because target network structures required by different resource constraint conditions are different, target network structures obtained by searching with different resource constraint conditions are different. For example, the network module corresponding to the network structure for image recognition may include a convolution module and a pooling module, and the network module corresponding to the network structure for voice recognition may include a loop module.

In the embodiment of the application, the server performs sampling processing in a preset network structure library to obtain at least one network module meeting the at least two resource constraint conditions. It should be noted that the sampling processing refers to that the server randomly selects one or more points (i.e., network modules) in the preset network structure library based on the input resource constraint condition, that is, the server selects a point in the preset network structure library that the server itself considers to satisfy the resource constraint condition.

Step S403, selecting a network parameter of each network module from a preset parameter set according to at least two resource constraint conditions.

Here, the preset parameter set includes different network parameters, for example, for the depth of the network structure, different options such as 2 layers, 3 layers, and 4 layers are included; for the width of the network structure, different options including 3, 4 and 6 are included; different options for the convolution kernel size of the network structure are included, such as 3 × 3, 5 × 5, and 7 × 7. And the server selects any type of network parameters of each network module in a preset parameter set according to the resource constraint condition. It should be noted that, in the embodiment of the present application, when selecting the network parameter, a sampling processing mode may also be adopted, and the network parameter of each network module is obtained by sampling in the preset parameter set.

Step S404, generating a target network structure through at least one network module and the network parameters of each network module.

Here, all network modules are spliced according to the network module and the network parameter of each network module to form a target network structure, wherein each network module can be regarded as each layer of the target network structure, and the target network structure is formed by connecting a plurality of layers according to the network parameters.

Step S405, determines the target network structure as a search result of the network structure search request, and outputs the search result.

Here, after the server searches for the target network structure, the server uses the target network structure as a search result of a network structure search request sent by the terminal, and sends the target network structure to the terminal, so that the terminal performs corresponding processing according to the searched target network.

The method provided by the embodiment of the application can be applied to the field of data processing of any network structure, before data processing, the method can be used for searching the target network structure meeting the specific resource constraint condition, and then the searched target network structure is used for data processing, so that the effect of real-time data processing can be achieved without configuring the network structure in advance.

In the embodiment of the application, for a plurality of resource constraint conditions, sampling processing is performed in a preset network structure library to obtain at least one network module, for the plurality of resource constraint conditions, network parameters of each network module are selected from a preset parameter set, and a target network structure is generated through the network modules and the network parameters obtained through sampling.

In some embodiments, the target network structure may also be generated by using a pre-trained controller, i.e., the functions of the server are implemented by the pre-trained controller.

An embodiment of the present application provides a method for training a controller, and fig. 5 is an optional flowchart of the method for training a controller provided in the embodiment of the present application, and as shown in fig. 5, the method includes the following steps:

step S501, at least two sample constraints are obtained.

In the embodiment of the present application, before the controller is used to generate the target network structure, the controller needs to be trained in advance, so that the trained controller is used to search the target network structure. In some embodiments, the controller may be an LSTM network, and in embodiments of the present application, parameters in the LSTM network are trained.

And step S502, sampling in a preset network structure library according to the sample constraint condition to obtain a network structure.

Here, after the controller acquires the sample constraint condition, the controller performs sampling processing in the preset network structure library according to the sample constraint condition, where the sampling processing refers to that the controller randomly selects one or more points (i.e., network modules) in the preset network structure library based on the input sample constraint condition, that is, the controller selects a point in the preset network structure library that the controller considers itself to satisfy the resource constraint condition.

Step S503, determining the performance parameters and the computational complexity of the network structure. Here, the performance parameters include, but are not limited to, accuracy of the network structure, computation delay, and the like, which can characterize the performance of the network structure.

Step S504, comparing the performance parameters and the calculation complexity with sample constraint conditions respectively to obtain a score for evaluating the network structure.

Here, the score for evaluating the network structure may be a reward value, and a higher reward value indicates a better performance of the searched network structure, or a more accurate search result; the lower the reward value, the worse the performance of the searched network structure is indicated, or the less accurate the search result is.

In some embodiments, the searching for the target network structure may be implemented internally by the controller through a control function, i.e., the calculation internally by the controller is implemented through the control function, wherein the control function is represented by the following formula (1-1):

α_T＝f(T；θ) (1-1)；

wherein α represents any network structure; t represents the upper bound of any sample constraint condition, and the network structure meeting the sample constraint condition T is represented as c (alpha) less than T; alpha is alpha_TA network structure representing an upper bound T of controller outputs that satisfies a sample constraint; c (—) represents a computation cost function; f (×) represents a mapping function to be learned, and θ represents a parameter included in the mapping function.

In some embodiments, the objective function of the controller is represented by the following equation (1-2):

wherein E is_T～τRepresenting the calculated model average performance of the controller for a plurality of sample constraints; pi (· | T; theta) represents a network structure sampling strategy for an upper bound T of any sample constraint condition; e_{α～π(·|T；θ)}The average performance of the structure calculated after a plurality of network structures are sampled according to the learned network structure sampling strategy is represented; r (alpha)_TI, T; w) represents any network structure a under the upper bound T of any sample constraint_TA score of (d); w represents a parameter in the score model; τ represents the distribution of T; t-tau represents the distribution of the values of T and obeys tau; s.t. indicates constrained.

And S505, performing optimization training on the controller according to the score to obtain the trained controller.

In some embodiments, step S505 may be implemented by: step S5051, according to the score, performs maximization processing on the sample constraint condition and the structure average performance of the network structure, to obtain the sample constraint condition after the maximization processing and the structure average performance of the network structure after the maximization processing.

Step S5052, performing optimization training on the controller according to the sample constraint condition after the maximization processing and the structure average performance of the network structure after the maximization processing, to obtain a trained controller.

Here, the average performance for different sample constraints and different network structures may be maximized, and by maximizing the average performance, the model average performance of the controller trained based on the multiple sample constraints may be better, thereby improving the search performance of the controller as a whole, i.e., improving the average performance of the controller searching for the network structure.

According to the controller training method provided by the embodiment of the application, a plurality of sample constraint conditions are input when the controller is trained, and in the calculation process, the model average performance of the controller is calculated for the plurality of sample constraint conditions, that is, the plurality of sample constraint conditions are simultaneously considered through the average performance, so that the problem that when the plurality of sample constraint conditions exist, a plurality of searches are needed is avoided. That is, in the method according to the embodiment of the present application, by considering the model average performance of the controller under the multiple sample constraints, the multiple sample constraints can be considered at the same time, and the search efficiency of the network structure is improved.

Further, when the controller trained by the controller training method provided by the embodiment of the application is used for searching the network structure, the controller can simultaneously consider a plurality of input resource constraint conditions, so that the purpose of simultaneously searching under the resource constraint conditions to generate the target network structure is achieved, and the searching is simultaneously carried out aiming at the resource constraint conditions, so that the searching efficiency is improved.

Based on the objective function of the controller provided in fig. 5 and the above equation (1-2), fig. 6 is an alternative flow chart of the controller training method provided in the embodiment of the present application, as shown in fig. 6, and in some embodiments, step S504 may be implemented by the following steps:

step S601, comparing the performance parameters and the calculation complexity with sample constraint conditions respectively through an objective function to obtain a score for evaluating the network structure.

Alternatively, in other embodiments, step S504 may also be implemented by: step S602, a score model is adopted to determine a score for evaluating the network structure. Wherein, the score model comprises at least two fully-connected layers and at least two nonlinear transformation processing layers, and each fully-connected layer is connected with one nonlinear transformation processing layer, the number of the fully-connected layers is the same as that of the nonlinear transformation processing layers, and in some embodiments, the nonlinear transformation processing layers can be formed by ReLU functions.

In some embodiments, step S602 may be implemented by:

step S6021, outputting the upper bound network structure alpha meeting the sample constraint condition by the controller_TAnd an upper bound T of sample constraints, input into the score model.

Step S6022, the network structure alpha is processed by the full connection layer and the nonlinear transformation processing layer in the score model_TAnd an upper bound T of the sample constraint condition, and sequentially performing at least one full connection process and at least one nonlinear transformation process to obtain the score of the network structure.

Because each full-connection layer is connected with one nonlinear transformation processing layer, after each full-connection processing, the result after the full-connection processing is subjected to nonlinear transformation processing through the nonlinear transformation processing layer, and the steps are repeated in a circulating manner until each layer in the score model is processed, so that the score of the network structure is obtained.

In some embodiments, the training process of the score model may be based on the following rules: given any two network architectures a1 and a2 (where network architectures a1 and a2 are both models that the controller searches), a network architecture with better accuracy will have a better reward value (i.e., a higher score) if both network architectures meet the input resource constraints, and a network architecture with lower computational complexity will have a higher reward value if at least one of the two network architectures does not meet the input resource constraints.

Based on fig. 5, fig. 7 is an optional flowchart of the controller training method provided in the embodiment of the present application, and as shown in fig. 7, in some embodiments, step S502 may be implemented by:

and step S701, sequentially selecting the depth and width of the network structure and the size of a convolution kernel according to sample constraint conditions.

Here, the preset parameter set stores a plurality of parameters in different types in advance, for example, for the depth of the network structure, different options such as 2 layers, 3 layers, 4 layers, and the like may be stored; for the width of the network structure, different options such as 3, 4 and 6 can be stored; for the convolution kernel size of the network structure, different options of 3 × 3, 5 × 5, and 7 × 7 may be stored. After the controller receives the sample constraint, any one of depth, width and convolution kernel size can be sampled in the preset parameter set.

Step S702, acquiring depth characters, width characters and convolution kernel size characters corresponding to the depth, width and convolution kernel sizes respectively.

Here, in order to accurately represent different network structures, the network structures may be represented in a character form, that is, the network structures are represented in a serialized manner, and therefore, after the depth, the width and the convolution kernel size of the network structures are obtained, the depth, the width and the convolution kernel size of the network structures are first represented in a character form, that is, depth characters, width characters and convolution kernel size characters corresponding to the depth, the width and the convolution kernel size, respectively, are obtained, so as to represent the depth, the width and the convolution kernel size by using the depth characters, the width characters and the convolution kernel size characters.

In step S703, the depth character, the width character, and the convolution kernel size character are concatenated to form a character string. Here, the depth character, the width character, and the convolution kernel size character may be sequentially connected to form a character string.

Step S704, a network structure is generated according to the character string.

Here, the character string is directly converted to obtain a network structure. Therefore, the character strings are adopted to represent the network structures, so that different network structures can be accurately represented, and the problem that the searched network structures are inaccurate due to errors of search output in the searching process is solved.

Fig. 8 is an alternative flowchart of a controller training method provided in an embodiment of the present application, and as shown in fig. 8, in some embodiments, step S601 may be implemented by:

step S801, classifying at least two sample constraints to form at least one sample constraint class, where each sample constraint class includes at least one sample constraint.

In the embodiment of the present application, the sample constraint (or resource constraint) at least includes the following categories: a compute delay class, a floating point number compute quantity class, and a memory overhead class.

In the training process of the controller, when the controller receives a plurality of sample constraint conditions, the plurality of sample constraint conditions can be classified firstly, the sample constraint conditions of the same class are divided into the same sample constraint condition class, and each sample constraint condition class comprises at least one sample constraint condition; correspondingly, in the using process of the controller, when the controller receives a plurality of resource constraint conditions, the plurality of resource constraint conditions may also be classified first, and the resource constraint conditions of the same category are divided into the same resource constraint condition class, where each resource constraint condition class includes at least one resource constraint condition.

Step S802, an objective function corresponding to each sample constraint condition class is obtained.

Here, the sample constraint of each class corresponds to an objective function, for example, the sample constraint of the computation delay class may correspond to an objective function 1, the sample constraint of the floating point computation number class may correspond to an objective function 2, and the sample constraint of the memory overhead class may correspond to an objective function 3.

Step S803, inputting the sample constraint conditions in each sample constraint condition class into the corresponding objective function to obtain the sub-scores of each sample constraint condition class.

Here, each sample constraint condition class includes one or more sample constraint conditions, and the sample constraint conditions in each sample constraint condition class are input into the corresponding objective function for calculation, so as to obtain the sub-scores of the corresponding sample constraint condition class.

For example, the sample constraints 11 and 12 are included in the computation delay class, the sample constraints 21 and 22 are included in the floating-point number computation quantity class, and the sample constraints 31 and 32 are included in the memory overhead class.

Therefore, the sample constraints 11 and 12 can be input into the objective function 1, and the sub-scores 111 corresponding to the sample constraints 11 and 12 are calculated through the objective function 1, where the sub-scores 111 are used for evaluating the average performance of the network results finally searched by the controller under the condition of the computation delay;

the sample constraints 21 and 22 can also be input into the objective function 2, and the sub-scores 211 corresponding to the sample constraints 21 and 22 are calculated through the objective function 2, wherein the sub-scores 211 are used for evaluating the average performance of the network results finally searched by the controller under the floating point number calculation quantity condition;

the sample constraints 31 and 32 can also be input into the objective function 3, and a sub-score 311 corresponding to the sample constraints 31 and 32 is calculated by the objective function 3, where the sub-score 311 is used for evaluating the average performance of the network results finally searched by the controller under the memory overhead condition.

Step S804, at least one sub-score corresponding to at least one sample constraint condition class is fused to obtain a score for evaluating a network structure.

Here, the sub-scores corresponding to different sample constraint condition classes may be weighted and summed to obtain a score for evaluating the network structure. For example, the above sub-scores 111, 211, and 311 may be weighted and summed to obtain a final score.

Steps S801 to S804 are cases of sample constraints for a plurality of categories during the training of the controller. Of course, in the using process of the controller, the same method as that in steps S801 to S804 may be used for processing the resource constraint conditions of multiple categories, that is, when multiple resource constraint conditions are obtained and have different categories, the multiple resource constraint conditions may be classified to form at least one resource constraint condition category, where each resource constraint condition category includes at least one resource constraint condition; then, acquiring a target function corresponding to each resource constraint condition class; inputting the resource constraint conditions in each resource constraint condition class into a corresponding objective function to obtain the sub-scores of each resource constraint condition class; and finally, fusing at least one sub-score corresponding to at least one resource constraint condition class to obtain a score for evaluating the target network structure searched by the controller.

In some embodiments, when the controller performs the sampling process, a plurality of network structures may be sampled simultaneously, and therefore, a score of each network structure needs to be determined to determine the best network structure of the plurality of network structures or to determine a difference between each two network structures.

Based on fig. 5, fig. 9 is an alternative flowchart of the controller training method provided in the embodiment of the present application, and as shown in fig. 9, after step S502, the method further includes the following steps of determining a score of each network structure by:

in step S901, when a plurality of network structures are obtained by sampling processing, every two network structures are compared. Here, each two network structures may be compared using the following equations (1-3):

wherein, beta₁Denotes the first network structure, beta₂Representing a second network structure; [*]Representing an indication function, outputting 1 when the condition is true, and outputting 0 when the condition is false; acc [. X [ ]]Representing the accuracy of the network structure; c]Representing the computational complexity of the network structure; g (. beta.) of₁,β₂T) denotes β below the upper bound T of the sample constraints₁Is greater than beta₂Good comparison results.

In some embodiments, equations (1-3) may also be used to train the value model, i.e., the training of the value model is accomplished based on the rules defined by equations (1-3).

Step S902, according to the comparison result of every two network structures, sorting the plurality of network structures to form a network structure sequence.

Here, the network structure sequence may be formed by sorting the network results in order of good to bad.

Step S903, according to the network structure sequence, the score of each network structure is determined in sequence.

Here, the scores of the network structures in the network structure sequence are arranged in descending order.

In some embodiments, step S903 may be implemented by:

and step S9031, determining a target function of the score model.

Here, the objective function of the score model is expressed by the following formulas (1-4):

wherein, L (w) represents a loss function when training for the parameter w in the score model; k represents the number of sample constraints; m represents the number of the plurality of network structures; phi denotes a hinge loss function; beta is a_iRepresents the ith network structure; beta is a_jRepresents the jth network structure; t is_kAn upper bound representing a kth sample constraint; r (beta)_i|T_k(ii) a w) represents the upper bound T in the sample constraints_kLower network structure beta_iA score of (d); r (beta)_j|T_k(ii) a w) represents the upper bound T in the sample constraints_kLower network structure beta_jA score of (d); g (. beta.) of_i,β_j,T_k) Represents an upper bound T on the sample constraint_kLower beta_iIs greater than beta_jA good comparison result;

step S9032, the loss function L (w) is subjected to minimization processing, and a minimization processing result is obtained.

And step S9033, controlling the size relationship between the reward values of any two network structures sampled by the controller according to the minimization processing result, wherein the size relationship is consistent with the sequence of the network structure sequence.

Next, an exemplary application of the embodiment of the present application in a practical application scenario will be described.

The embodiment of the application provides a network structure searching method, and a series of resource constraints (namely resource constraint conditions or computing resource constraints, which can be parameters, computing delay and the like) are usually considered in an actual scene so as to adapt to various different application scenes. When the number of resource constraints is large, it becomes very cumbersome and difficult to design a network structure for each resource constraint individually. Therefore, how to simultaneously find a suitable network structure for a series of resource constraints is an urgent research problem to be solved.

The key problem to be solved by the embodiment of the application is as follows: a universal network structure designer (namely a controller) is designed, any resource constraint is given, a network structure conforming to the resource constraint is automatically given, and the difficulty of designing an actual scene model is reduced. In order to solve the problem, the embodiment of the application provides a network structure searching algorithm based on constraint perception, and the algorithm is applied to a controller.

Fig. 10 is a structural diagram of a network structure search system according to an embodiment of the present application, as shown in fig. 10, a resource constraint is input to a controller 1001, the controller 1001 is a constraint perception-based controller, the controller 1001 samples a network structure 1002, sends the sampled network structure 1002 to a constraint perception-based model evaluator 1003, and the constraint perception-based model evaluator 1003 determines a reward value 1004 (i.e., a score value) according to the input resource constraint and the sampled network structure, and updates the controller 1001 by the reward value 1004.

The key point of the embodiment of the application is that a network structure searching algorithm based on constraint perception is designed and trained, and a network structure meeting resource constraints is automatically designed aiming at any given resource constraints.

In some embodiments, the training process of the constraint awareness-based network structure search method (i.e., the network structure search method) of the embodiments of the present application is as follows:

1) inputting given resource constraint as input into a controller, and automatically sampling the resource constraint as a condition by the controller to a series of network structures meeting the resource constraint;

2) evaluating the performance parameters and the calculation complexity of the sampled network structure, and comparing the performance parameters and the calculation complexity with the considered resource constraint to obtain a reward value capable of evaluating the quality of the network structure;

3) the resulting reward values are used to guide the training of the controller so that the controller tends to find a network structure that meets the resource constraints and is good in performance.

It can be seen that the training method in the embodiment of the present application has two key points, which are how to design and train the controller to automatically find the network structure satisfying the resource constraint, and how to evaluate the network structure with an accurate reward value according to the specific resource constraint.

Regarding the design and training method of the controller model, for convenience of representation, α is intended to represent any network structure, and T represents an upper bound of a certain computational resource constraint, then the network structure satisfying the resource constraint can be represented as c (α) ≦ T. The network structure searcher with constrained sensing capability proposed by the embodiment of the application can be represented by the following formula (2-1):

α_T＝f(T；θ) (2-1)；

where f (×) represents a mapping function to be learned, and θ represents a parameter included in the mapping function.

In order to accurately represent different network structures, the network is to be represented in a serialization mode, and a one-to-one mapping relation between the network structures and character strings is constructed. In the embodiment of the application, any module in the network structure mainly comprises 3 variable variables of depth, width and convolution kernel size. The depth represents the number of network layers contained in the module, the width represents the number of channels of each network layer, and the convolution kernel size determines the convolution calculation mode and contains three optional convolution kernel sizes of 3 × 3, 5 × 5 and 7 × 7. To construct a controller that can search and generate a network structure, a character string representing the network structure is generated by constructing a long short term memory network (LSTM). As shown in fig. 11, the controller (i.e. LSTM 1101 in fig. 11) first determines the depth of the model, then determines the width and the corresponding convolution kernel size for each network layer, and finally concatenates all the characters corresponding to the choices together to obtain a complete character string representing the network structure.

In order to implement network structure search for different computational resource constraints, the average performance of the network structures found by the controller under different resource constraints is required to be sufficiently high, so the objective function can be expressed by the following equation (2-2):

wherein E is_T～τRepresents the model average performance of the controller calculated for a plurality of computational resource constraints (which may be referred to herein as sample constraints); π (· | T; θ) represents the network fabric sampling policy for an upper bound T of any computational resource constraint; e_{α～π(·|T；θ)}The average performance of the structure calculated after a plurality of network structures are sampled according to the learned network structure sampling strategy is represented; r (alpha)_TI, T; w) represents any network structure a under an upper bound T of any computational resource constraint_TThe prize value of (d); w represents parameters in a reward model (i.e., a point model or a neural network structure evaluator); τ represents the distribution of T; t-tau represents the distribution of the values of T and obeys tau; s.t. indicates constrained.

In the embodiment of the application, the performance of the controller for searching the network structure is improved by maximizing the average performance aiming at different computing resource constraints and different network structures by using a strategy gradient method based on reinforcement learning.

Design and training method for reward function in reward model, and aims to find suitable network nodes aiming at different computing resource constraintsThe comprehensive performance of different network structures under different constraints needs to be accurately evaluated, namely, an accurate reward value R (alpha) is obtained_TI, T; w). Since the same network structure should have different reward values under different constraints, a reward function sensitive to the calculation constraints is designed for this purpose. Because the function needs to obtain a final reward value through calculation according to the network structure alpha and the calculation resource constraint T, a multilayer perceptron model is designed to model a reward model, the model structure of the reward model is shown in FIG. 12 and is the model structure diagram of the neural network structure evaluator (namely the reward model) provided by the embodiment of the application, and the input of the reward model is the network structure alpha_TAnd a computational resource constraint T, the reward model including a plurality of fully connected layers (FCs) and a plurality of nonlinear transformation processing layers, wherein the nonlinear transformation processing layers may be nonlinear transformation processing layers composed of ReLU functions.

In some embodiments, it is first necessary to define whether the network structure is a good solution under certain computational resource constraints. A computational resource constraint T may be given, and a network structure with higher accuracy is better when the computational resource constraint is satisfied. Conversely, a network structure with computational complexity that exceeds the computational resource constraint or that has low accuracy should be a poor network structure.

Based on the above observations, two rules were designed to learn an accurate neural network structure evaluator (i.e., reward model). Here, any two network structures β can be specified₁And beta₂Then the rules of the design have the following cases: if both network structures meet the computational resource constraints, the network structure with the higher accuracy will have a higher reward value; alternatively, if at least one network fabric does not satisfy the computational resource constraints, a fabric with lower computational complexity will have a higher reward value.

The above rule can be formally expressed by the following equation (2-3):

wherein the content of the first and second substances,representing an indication function, outputting 1 when the condition is true, and outputting 0 when the condition is false; acc [. X [ ]]Representing the accuracy of the network structure; c]Representing the computational complexity of the network structure; g (. beta.) of₁,β₂T) denotes β below the upper bound T of the computational resource constraint₁Is greater than beta₂Good comparison result, i.e. beta₁Ratio of beta to beta₂Better outputs 1, otherwise outputs 0.

In order to obtain an accurate reward value, an appropriate objective function needs to be designed to train the neural network structure evaluator. Since the proposed rules are based on comparisons between network structures, the training process is to be modeled as a ranking problem for different network structures. In the embodiment of the application, M network structures are given, and model training of the neural network structure evaluator is guided through results of pairwise comparison, so that the magnitude relation of the reward values predicted by the neural network structure evaluator conforms to the proposed rule. To this end, given the K computational resource constraints, the objective function of the neural network structure evaluator is represented by the following equations (2-4):

wherein, L (w) represents a loss function when training for the parameter w in the score model; phi denotes a hinge loss function (hinge loss); beta is a_iRepresents the ith network structure; beta is a_jRepresents the jth network structure; t is_kRepresents an upper bound on the kth computational resource constraint (which may be referred to herein as a sample constraint); r (beta)_i|T_k(ii) a w) represents an upper bound T in the computational resource constraints_kLower network structure beta_iA score of (d); r (beta)_j|T_k(ii) a w) represents an upper bound T in the computational resource constraints_kLower network structure beta_jA score of (d); g (. beta.) of_i,β_j,T_k) Representing an upper bound T on a computing resource constraint_kLower beta_iIs greater than beta_jGood comparisonAnd (6) obtaining the result.

In the embodiment of the application, the magnitude relation between the reward values of any two network structures predicted by the neural network structure evaluator is kept consistent with the defined rule by minimizing the values of L (w), so that the accurate performance evaluation of the network structures aiming at different computing resource constraints is realized.

The following describes application scenarios and flows of the embodiments of the present application:

the method provided by the embodiment of the application can be applied to an automatic machine learning (AutoML) product and is responsible for automatically finding out the network structure meeting the constraint according to the computing resource constraint input by a user so as to meet the requirements of different application scenes and achieve a better 'automatic' user experience function.

Fig. 13 is a schematic flowchart of the method in an AutoML product according to the embodiment of the present application, and as shown in fig. 13, in an actual application scenario, the method in the embodiment of the present application may be applied to any neural network structure automatic search product considering computational resource constraints, a user only needs to input computational resource constraints 1301 required in a certain actual scenario, and the product automatically finds a network structure 1303 meeting the constraints through a built-in network structure searcher 1302 based on constraint perception and outputs the network structure, thereby greatly reducing the design cost of manually designing a network structure under different resource constraints. The embodiment of the application can find a network structure meeting the requirement aiming at any computing resource constraint provided by a user.

In the embodiment of the application, the product flow includes: 1) a user inputs an arbitrary computing resource constraint into the AutoML platform; 2) the AutoML platform can automatically find an optimal network structure aiming at the input computing resources; 3) and outputting the network structure for the user to use.

Table 1 shows a comparison between performances of network structures searched under different computation delay constraints by the method provided in the embodiment of the present application, and as can be seen from table 1, the constraint-aware-based network structure searcher provided in the embodiment of the present application can find a network structure meeting requirements for different computation delay constraints. Meanwhile, as the constraint becomes larger, the accuracy of the found network structure is gradually increased, and the general rule that the accuracy is increased when the model becomes larger is met.

TABLE 1

Computing delay constraints	80ms	110ms	140ms	170ms	200ms
						Accuracy of model classification	77.5％	78.4％	78.7％	79.0％	79.2％
Model computation delay	79ms	107ms	128ms	167ms	193ms

The beneficial effects brought by the technical scheme of the embodiment of the application can be summarized as follows: the problem that the existing network structure design method is independently designed for different computing resource constraints is solved; accurate evaluation of the network structure under the constraint of different computing resources is realized; the function of constrained search network structure for arbitrary computing resources is realized.

In some embodiments, in the training method of the controller provided in the above embodiments, the controller model may also be trained by maximizing the expectation of the reward value by replacing the strategy gradient with the near-end strategy Optimization (PPO).

Continuing with the exemplary structure of the network structure searching apparatus 354 implemented as a software module provided in the embodiments of the present application, in some embodiments, as shown in fig. 3, the software module stored in the network structure searching apparatus 354 of the memory 350 may be the network structure searching apparatus in the server 300, which includes:

the receiving module is used for receiving a network structure searching request; the network structure search request comprises at least two resource constraint conditions; the sampling processing module is used for carrying out sampling processing in a preset network structure library according to the at least two resource constraint conditions to obtain at least one network module; a selection module, configured to select a network parameter of each network module in a preset parameter set according to the at least two resource constraint conditions; the generating module is used for generating a target network structure through the at least one network module and the network parameters of each network module; and the determining module is used for determining the target network structure as the search result of the network structure search request and outputting the search result.

In some embodiments, the apparatus further comprises: a processing module for generating the target network structure using a pre-trained controller; wherein the controller is trained by the steps of: acquiring at least two sample constraint conditions; sampling in a preset network structure library according to the sample constraint condition to obtain a network structure; determining performance parameters and computational complexity of the network structure; comparing the performance parameters and the computation complexity with the sample constraint conditions respectively to obtain a score for evaluating the network structure; and carrying out optimization training on the controller according to the scores to obtain the trained controller.

In some embodiments, the controller implements a search of the target network structure through a control function; wherein the control function is represented by the following formula: alpha is alpha_TF (T; θ); wherein α represents any network structure; t represents the upper bound of any sample constraint condition, and the network structure meeting the sample constraint condition T is represented as c (alpha) less than T; alpha is alpha_TA network structure representing an upper bound T of controller outputs that satisfies a sample constraint; c (—) represents a computation cost function; f (×) represents a mapping function to be learned, and θ represents a parameter included in the mapping function.

In some embodiments, the controller is further trained by: sequentially selecting the depth, the width and the convolution kernel size of the network structure according to the sample constraint condition; acquiring depth characters, width characters and convolution kernel size characters which respectively correspond to the depth, the width and the convolution kernel size; connecting the depth character, the width character and the convolution kernel size character to form a character string; and generating the network structure according to the character string.

In some embodiments, the objective function of the controller is represented by the following formula:

wherein E is_T～τRepresenting the calculated model average performance of the controller for a plurality of sample constraints; pi (· | T; theta) represents a network structure sampling strategy for an upper bound T of any sample constraint condition; e_α～π(·|_T；θ)The average performance of the structure calculated after a plurality of network structures are sampled according to the learned network structure sampling strategy is represented; r (alpha)_TI, T; w) represents any network structure a under the upper bound T of any sample constraint_TA score of (d); w represents a parameter in the score model; τ represents the distribution of T; t-tau represents the distribution of the values of T and obeys tau; s.t. indicates constrained.

In some embodiments, the controller is further trained by: and respectively comparing the performance parameters and the computation complexity with the sample constraint conditions through the objective function to obtain the score for evaluating the network structure.

In some embodiments, the controller is further trained by: according to the scores, maximizing the sample constraint conditions and the structure average performance of the network structure to obtain the maximized sample constraint conditions and the maximized structure average performance of the network structure; and carrying out optimization training on the controller according to the sample constraint conditions after the maximization processing and the structure average performance of the network structure after the maximization processing to obtain the trained controller.

In some embodiments, the controller is further trained by: classifying the at least two sample constraints to form at least one sample constraint class, wherein each sample constraint class comprises at least one sample constraint; obtaining the target function corresponding to each sample constraint condition class; inputting the sample constraint conditions in each sample constraint condition class into the corresponding objective function to obtain the sub-scores of each sample constraint condition class; and fusing at least one sub-score corresponding to the at least one sample constraint condition class to obtain the score for evaluating the network structure.

In some embodiments, the controller is further trained by: determining the score for evaluating the network structure using the score model; the score model comprises at least two full-connection layers and at least two nonlinear transformation processing layers, and one nonlinear transformation processing layer is connected behind each full-connection layer.

In some embodiments, the controller is further trained by: network structure alpha of upper bound satisfying sample constraint condition output by the controller_TAnd an upper bound T of the sample constraints, input into the score model;applying the network structure alpha through the full connection layer and the nonlinear transformation processing layer in the score model_TAnd sequentially carrying out at least one full-connection process and at least one nonlinear transformation process on the upper bound T of the sample constraint condition to obtain the score of the network structure.

In some embodiments, the apparatus further comprises: a comparing module, configured to compare every two network structures according to the following formula when the plurality of network structures are obtained through the sampling processing:

wherein, beta₁Denotes the first network structure, beta₂Representing a second network structure;representing an indication function, outputting 1 when the condition is true, and outputting 0 when the condition is false; acc [. X [ ]]Representing the accuracy of the network structure; c]Representing the computational complexity of the network structure; g (. beta.) of₁,β₂T) denotes β below the upper bound T of the sample constraints₁Is greater than beta₂A good comparison result; the sequencing module is used for sequencing the network structures according to the comparison result of every two network structures to form a network structure sequence; and the score determining module is used for sequentially determining the score of each network structure according to the network structure sequence.

In some embodiments, the score determination module is further to: determining an objective function of the score model, wherein the objective function of the score model is represented by the following formula:

wherein, L (w) represents a loss function when training for the parameter w in the score model; k represents the number of sample constraints; m represents saidA number of the plurality of network structures; phi denotes a hinge loss function; beta is a_iRepresents the ith network structure; beta is a_jRepresents the jth network structure; t is_kAn upper bound representing a kth sample constraint; r (beta)_i|T_k(ii) a w) represents the upper bound T in the sample constraints_kLower network structure beta_iA score of (d); r (beta)_j|T_k(ii) a w) represents the upper bound T in the sample constraints_kLower network structure beta_jA score of (d); g (. beta.) of_i,β_j,T_k) Represents an upper bound T on the sample constraint_kLower beta_iIs greater than beta_jA good comparison result; minimizing the loss function L (w) to obtain a minimization result; and controlling the size relationship between the reward values of any two network structures sampled by the controller according to the minimization processing result, wherein the size relationship is consistent with the sequence of the network structure sequence.

It should be noted that the description of the apparatus in the embodiment of the present application is similar to the description of the method embodiment, and has similar beneficial effects to the method embodiment, and therefore, the description is not repeated. For technical details not disclosed in the embodiments of the apparatus, reference is made to the description of the embodiments of the method of the present application for understanding.

Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the embodiment of the present application.

Embodiments of the present application provide a storage medium having stored therein executable instructions, which when executed by a processor, will cause the processor to perform a method provided by embodiments of the present application, for example, the method as illustrated in fig. 4.

In some embodiments, the storage medium may be a computer-readable storage medium, such as a Ferroelectric Random Access Memory (FRAM), a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), a charged Erasable Programmable Read Only Memory (EEPROM), a flash Memory, a magnetic surface Memory, an optical disc, or a Compact disc Read Only Memory (CD-ROM), among other memories; or may be various devices including one or any combination of the above memories.

In some embodiments, executable instructions may be written in any form of programming language (including compiled or interpreted languages), in the form of programs, software modules, scripts or code, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

By way of example, executable instructions may correspond, but do not necessarily have to correspond, to files in a file system, and may be stored in a portion of a file that holds other programs or data, such as in one or more scripts in a hypertext Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). By way of example, executable instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.

The above description is only an example of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.