1. A control method of a space robot mechanical arm is characterized in that an image collected by a space robot base camera is obtained, and a reward function used in a mechanical arm control process is set; constructing a mechanical arm control strategy network, a state action value network and a state value network, inputting images to the control strategy network, outputting action information to control a mechanical arm, and interactively accumulating interactive information pairs for multiple times to form an experience pool; and constructing a target function required by training each network according to maximum entropy reinforcement learning, training each network parameter by using a practical gradient descent algorithm through interactive information in an experience pool to obtain a trained mechanical arm control strategy network, inputting image information, and outputting action information to control the mechanical arm.

2. A method of controlling a robotic arm of a space robot as claimed in claim 1, characterized in that the method comprises the steps of:

(1) according to the control of a Markov decision process modeling mechanical arm, obtaining an image collected by a space robot base camera, and defining the image at the moment t as state information s_tA matrix of the form W x H x 3, wherein 3 dimensions comprise channels of three RGB colors, the image of each dimension comprising W x H pixels, wherein W is the width of the image and H is the height of the image; the angular velocity a of the space robot joint_tAs the motion information, t represents a sampling time;

(2) setting a reward function r for use in robotic arm control_t(s_t,a_t) Completing the modeling of the Markov decision process; reward function r_t(s_t,a_t) The expression of (a) is as follows:

r_t(a_t,a_t)＝-[β₁d²+ln(β₂d²+ε_d)+β₃||a_t-a_t-1||²]

wherein p is_eFor the end position of the robot arm of the space robot, p_tThe target position of the robot mechanical arm in the target space is d is Euclidean distance, and d is | p_e-p_t‖，β₁＝10^-3，β₂＝1，β₃＝10^-1，ε_dThe function of (a) is to prevent the ln function from generating singularity and epsilon_dWhen the value is 0.001, superscript T is matrix transposition;

(3) constructing a mechanical arm control strategy networkWhereinParameters representing a robot arm control strategy network to the robot arm control strategy networkInput image s_tTo obtain motion information angular velocity a_tThe method specifically comprises the following steps:

(3-1) robot arm control strategy networkIs a convolutional neural network, the convolutional kernel weight w of which₁Is a G_w1*L_w1*M_w1Matrix, wherein G_w1Is the number of channels, L, of the convolution kernel_w1For the width of the convolution kernel, M_w1Is the height of the convolution kernel;

(3-2) robot arm control strategy networkThe second layer of the system is a batch normalization layer, the batch normalization layer respectively normalizes the multiple feature layers obtained in the previous layer, and the quantity of normalization parameters is in positive linear correlation with the number of layers of the feature layers;

(3-3) mechanical arm control strategy networkThe third layer of (2) is a maximum value pooling layer, and a filter P of the maximum value pooling layer₁A matrix of 2 x 2;

(3-4) mechanical arm control strategy networkThe fourth layer of (2) is a convolutional neural network, the convolutional kernel weight W of which₂Is a G_w2*L_w2*M_w2Matrix, wherein G_w2Is the number of channels, L, of the convolution kernel_w2For the width of the convolution kernel, M_w2Is the height of the convolution kernel;

(3-5) mechanical arm control strategy networkThe fifth layer of the system is a batch normalization layer, the batch normalization layer respectively normalizes the multiple feature layers in the previous layer, and the quantity of normalization parameters is in positive linear correlation with the number of layers of the feature layers;

(3-6) mechanical arm control strategy networkThe sixth layer of (2) is a maximum pooling layer, a filter P of the maximum pooling layer₂Matrix of 2 x 2

(3-7) robot arm control strategy networkThe seventh layer of (2) is a convolutional neural network, the convolutional kernel weight W of which₃Is a G_w3*L_w3*M_w3Matrix, wherein G_w3Is the number of channels, L, of the convolution kernel_w3For the width of the convolution kernel, M_w3Is the height of the convolution kernel;

(3-8) mechanical arm control strategy networkThe eighth layer of (1) is a batch normalization layer, which respectively normalizes the multiple feature layers in the previous layer, the number of normalization parameters and the featuresThe number of layers is relevant;

(3-9) robot arm control strategy networkThe ninth layer of (2) is a maximum value pooling layer, a filter P of the maximum value pooling layer₃A matrix of 2 x 2;

(3-10) robot arm control strategy networkThe tenth layer of the neural network is a fully-connected neural network, and the number of input neurons is the number F of the flattened characteristic output by the previous layer and output by the feature layer₉And the number of output neurons is F₁₀Neuron weight is W₁₀；

(3-11) robot arm control strategy networkThe eleventh layer of (2) is a fully-connected neural network, and the number of input neurons is F output by the last layer₁₀And the number of output neurons is F₁₁Neuron weight is W₁₁；

(3-12) robot arm control strategy networkThe twelfth layer of the neural network is a fully connected neural network, and the number of input neurons is F output by the previous layer₁₁The number of output neurons is the mean value of Gaussian distributionSum varianceThe weight of the neuron is W₁₂；

(3-13) obtaining a mechanical arm control strategy network according to the steps (3-1) - (3-12)

(3-14) to the robot arm control strategy network of step (3-13)Inputting the RGB three-channel image s collected in the step (1)_tMechanical arm control strategy networkOutputting the mean value mu of the obtained Gaussian distribution_tSum variance Σ_tMean value μ_tSum variance Σ_tAngular velocity a combined into space robot joint_tProbability distribution ofObtaining the angular velocity a of the mechanical arm joint through sampling_t；

(4) Constructing a mechanical arm state value network V according to the Markov decision process principle_ψWherein psi represents a parameter of the arm state value network by inputting the image s_tObtain a status value v_tThe method specifically comprises the following steps:

(4-1) repeating the step (3-1) -the step (3-11) and constructing a mechanical arm state value network V_ψThe first layer to the eleventh layer of (1);

(4-2) mechanical arm state value network V_ψThe twelfth layer of the neural network is a fully connected neural network, and the number of input neurons is F output by the previous layer₁₁Output neuron-state cost function v_tNeuron weight is W₁₂；

(4-3) obtaining a mechanical arm state value network V according to the step (4-1) to the step (4-2)_ψ；

(5) Constructing a mechanical arm state action value network Q according to the Markov decision process principle_θWherein θ represents a parameter of the robot status motion value network, and θ represents a parameter of the robot status motion value network_θInput image s_tAnd joint angular velocity a_tIn the form ofValue of state motion q_tThe mapping relationship of (2) specifically comprises the following steps:

(5-1) repeating the steps (3-1) to (3-10) and constructing a mechanical arm state action value network Q_θThe first to tenth layers of network structure;

(5-2) mechanical arm state action value network Q_θThe eleventh layer of (2) is a fully-connected neural network, and the number of input neurons is F output by the last layer₁₀And joint angular velocity a of space robot_tThe number of the combined neurons is F₁₁Neuron weight is W₁₁；

(5-3) mechanical arm state action value network Q_θThe twelfth layer of the neural network is a fully connected neural network, and the number of input neurons is F output by the previous layer₁₁The output neuron is a state action value function q_tNeuron weight is W₁₂；

(5-4) obtaining a mechanical arm state action value network Q according to the step (5-1) to the step (5-3)_θ；

(6) To step (3) mechanical arm control strategy networkInputting the image s collected at the sampling time t in the step (1)_tThe output is the joint angular velocity a of the mechanical arm_tThe angular velocity a of the joint_tThe output is transmitted to a proportional-derivative controller C, and the proportional-derivative controller C outputs the torque to obtain joint torque so as to realize the control of the robot; image s acquired at sampling time t_tAnd joint tracking desired joint angular velocity a_tInputting the reward function in the step (2) to obtain a reward value r_tAnd an image s at time t +1 is obtained_t+1Obtaining the interactive information pair E at the time t_t＝<s_t,a_t,r_t,s_t+1>；

(7) Traversing all images s acquired within a T period_t＝1:TRepeating the step (6) to obtain a plurality of groups of interactive information pairs, wherein the plurality of groups of interactive information pairs form an experience playback pool D and respectively provide values to the mechanical arm state in the step (4)Network V_ψAnd (5) a mechanical arm state action value network Q_θInputting images s collected at different sampling moments_tObtaining a status value v_ψ(s_t) Sum state action value q_t(s_t,a_t)；

(8) And (3) establishing an optimization target according to the maximum entropy reinforcement learning, so that the accumulated reward and the entropy of the strategy are maximized:

wherein the content of the first and second substances,representing the entropy of the information;

(9) network Q for minimizing Bellman residual errors to action value of mechanical arm state_θTraining the parameters to obtain a mechanical arm state action value network Q_θAt the optimum parameter J_QExpression of (θ):

(10) value network V for state of mechanical arm by minimizing square loss value_ψObtaining a mechanical arm state value network V by parameter training_ψOptimum parameter J of_V(ψ) expression:

(11) training the strategy function by minimizing the expected relative entropy divergence to obtain the optimized parameter of the strategy functionExpression:

(12) training the mechanical arm control strategy network of the step (3) to the step (5) by using the training target obtained in the step (9) to the step (11) and a gradient descent methodState action value network Q_θAnd status value network V_ψCompleting the training of the network;

(13) real-time acquisition of image s obtained by camera mounted on space robot base_tImages s to be acquired in real time_tMechanical arm control strategy network input to step (12)Outputting to obtain the angular velocity a of the mechanical arm joint at the sampling time t_tAnd the control on the space robot mechanical arm is realized, and a track planning task in a T time period is realized.

Background

Trajectory planning is the most common task performed by space robots and has been studied extensively. The Generalized Jacobian Matrix (GJM) of the space manipulator can realize the trajectory planning of the robot arm under the condition of not influencing the basic posture of the robot. However, in some cases, there may be singular points in the GJM, which will limit the feasible space for kinematic planning by the GJM inverse method. When the space robot performs path planning, a position of a dynamic singular point exists in a path, and limited joint speed cannot be realized at the position, so that the length of the planned path is increased. Therefore, the traditional space robot trajectory planning scheme mainly solves the problem of singular solution existing in the solution. In recent years, some methods using intelligent optimization solve the problem of the dynamic singularity of the space robot. For example, Wu et al, implemented the task of planning the trajectory of a single target point of a dual-arm space robot using DDPG algorithm, see Wu, Yun-Hua, et al, relationship learning guiding objective-arm trajectory planning for a free-flowing space robot [ J ]. Aerospace Science and technology, 2020, 98: 105657.

however, the pose acquisition of the target still needs to be realized by a separately designed controller, and the model-based and modularized design has certain disadvantages. First, limited modeling details limit the accuracy of the model, and modeling errors and constraints can reduce the control effect. Secondly, the design of the modular controller is very laborious, requiring manual adjustment of the control parameters of each module. Therefore, for the free floating space robot control, end-to-end model-free reinforcement learning is adopted, namely, the controller is directly learned from original image pixels, the problems of singular solution and modeling errors existing in the traditional method can be well solved, perception and decision are unified into one method, and the design of a modularized controller is avoided.

Disclosure of Invention

The invention aims to provide a control method of a space robot mechanical arm, which improves the existing free floating space robot control method to achieve the purpose that a space robot catches space garbage and invalid satellites.

The invention provides a control method of a space robot mechanical arm, which comprises the steps of firstly obtaining an image collected by a space robot base camera, and setting a reward function used in a mechanical arm control process; constructing a mechanical arm control strategy network, a state action value network and a state value network, inputting images to the control strategy network, outputting action information to control a mechanical arm, and interactively accumulating interactive information pairs for multiple times to form an experience pool; and constructing a target function required by training each network according to maximum entropy reinforcement learning, training each network parameter by using a practical gradient descent algorithm through interactive information in an experience pool to obtain a trained mechanical arm control strategy network, inputting image information, and outputting action information to control the mechanical arm.

The invention provides a control method of a space robot mechanical arm, which has the characteristics and advantages that:

according to the control method of the space robot mechanical arm, end-to-end model-free reinforcement learning is adopted, namely, the controller is directly learned from original image pixels, the problems of singular solution and modeling errors existing in the traditional method can be well solved, perception and decision are unified into one method, and modular controller design is avoided. End-to-end model-free reinforcement learning can well solve the problems of singular solution and modeling error in the traditional method, perception and decision are unified into one method, and the problem of manual parameter adjustment in the design process of a modular controller is avoided.

Drawings

Fig. 1 is a flow chart of a control method of a space robot manipulator according to the present invention.

Detailed Description

The invention provides a control method of a space robot mechanical arm, which comprises the steps of firstly obtaining an image collected by a space robot base camera, and setting a reward function used in a mechanical arm control process; constructing a mechanical arm control strategy network, a state action value network and a state value network, inputting images to the control strategy network, outputting action information to control a mechanical arm, and interactively accumulating interactive information pairs for multiple times to form an experience pool; and constructing a target function required by training each network according to maximum entropy reinforcement learning, training each network parameter by using a practical gradient descent algorithm through interactive information in an experience pool to obtain a trained mechanical arm control strategy network, inputting image information, and outputting action information to control the mechanical arm.

The flow chart of the mechanical arm control method of the space robot is shown in fig. 1, and the specific steps are as follows:

(1) according to the control of a Markov decision process modeling mechanical arm, obtaining an image collected by a space robot base camera, and defining the image at the moment t as state information s_tA matrix of the form W x H x 3, wherein 3 dimensions comprise channels of three RGB colors, the image of each dimension comprising W x H pixels, wherein W is the width of the image and H is the height of the image; the angular velocity a of the space robot joint_tAs the motion information, where t represents a sampling time;

(2) setting a reward function r for use in robotic arm control_t(s_t，a_t) Completing the modeling of the Markov decision process; reward function r_t(s_t，a_t) The expression of (a) is as follows:

r_t(s_t，a_t)＝-[β₁d²+ln(β₂d²+∈_d)+β₃||a_t-a_t-1||²]

wherein p is_eFor the end position of the robot arm of the space robot, p_tIs the target position of the robot mechanical arm in the object space, d isEuclidean distance, d | | | p_e-p_t||，β₁＝10^-3，β₂＝1，β₃＝10^-1，∈_dWhen the Ln function is equal to 0.001, the singularity of the ln function is prevented, and the superscript T is matrix transposition; adding-ln (beta) to the reward function₂d²+∈_d) This is to encourage that a smaller distance d will result in a higher reward, thereby improving accuracy. Furthermore, - β when the end effector is a long distance to the target capture point₁d²The reward is influenced more greatly, the mechanical arm can perform actions with larger amplitude, and the change of the reward value is not too violent, thereby being beneficial to fully exploring. And-beta₃||a_t-a_t-1||²Is a penalty term introduced to make the control curve smoother. Last itemThe aim is to reduce the moment output by the mechanical arm as much as possible, so that the interference on the base can be reduced.

(3) Constructing a mechanical arm control strategy networkWhereinParameters representing a robot arm control strategy network to the robot arm control strategy networkInput image s_tTo obtain motion information angular velocity a_tThe method specifically comprises the following steps:

(3-1) robot arm control strategy networkIs a convolutional neural network, the convolutional kernel weight W of which₁Is a G_w1*L_w1*M_w1Matrix, wherein G_w1Is the number of channels, L, of the convolution kernel_w1Is the convolution kernelWidth of (M)_w1Is the height of the convolution kernel;

(3-2) robot arm control strategy networkThe second layer of the system is a batch normalization layer, the batch normalization layer respectively normalizes the multiple feature layers obtained in the previous layer, and the quantity of normalization parameters is in positive linear correlation with the number of layers of the feature layers;

(3-3) mechanical arm control strategy networkThe third layer of (2) is a maximum value pooling layer, and a filter P of the maximum value pooling layer₁A matrix of 2 x 2;

(3-4) mechanical arm control strategy networkThe fourth layer of (2) is a convolutional neural network, the convolutional kernel weight W of which₂Is a G_w2*L_w2*M_w2Matrix, wherein G_w2Is the number of channels, L, of the convolution kernel_w2For the width of the convolution kernel, M_w2Is the height of the convolution kernel;

(3-5) mechanical arm control strategy networkThe fifth layer of the system is a batch normalization layer, the batch normalization layer respectively normalizes the multiple feature layers in the previous layer, and the quantity of normalization parameters is in positive linear correlation with the number of layers of the feature layers;

(3-6) mechanical arm control strategy networkThe sixth layer of (2) is a max-pooling layer, the filter P2 of which is a 2 x 2 matrix

(3-7) robot arm control strategy networkThe seventh layer of (2) is a convolutional neural network, the convolutional kernel weight W of which₃Is a G_w3*L_w3*M_w3Matrix, wherein G_w3Is the number of channels, L, of the convolution kernel_w3For the width of the convolution kernel, M_w3Is the height of the convolution kernel;

(3-8) mechanical arm control strategy networkThe eighth layer of the system is a batch normalization layer, the batch normalization layer respectively normalizes the multiple feature layers in the previous layer, and the quantity of normalization parameters is related to the number of layers of the feature layers;

(3-9) robot arm control strategy networkThe ninth layer of (2) is a maximum value pooling layer, a filter P of the maximum value pooling layer₃A matrix of 2 x 2;

(3-10) robot arm control strategy networkThe tenth layer of the neural network is a fully-connected neural network, and the number of input neurons is the number F of the flattened characteristic output by the previous layer and output by the feature layer₉And the number of output neurons is F₁₀Neuron weight is W₁₀；

(3-11) robot arm control strategy networkThe eleventh layer of (2) is a fully-connected neural network, and the number of input neurons is F output by the last layer₁₀And the number of output neurons is F₁₁Neuron weight is W₁₁；

(3-12) robot arm control strategy networkThe twelfth layer of (a) is a fully connected neural network, input neuronsF with the number of outputs of the previous layer₁₁The number of output neurons is the mean value of Gaussian distributionSum varianceThe weight of the neuron is W₁₂；

(3-13) obtaining a mechanical arm control strategy network according to the steps (3-1) - (3-12)

(3-14) to the robot arm control strategy network of step (3-13)Inputting the RGB three-channel image s collected in the step (1)_tMechanical arm control strategy networkOutputting the mean value mu of the obtained Gaussian distribution_tSum variance Σ_tMean value μ_tSum variance Σ_tAngular velocity a combined into space robot joint_tProbability distribution ofObtaining the angular velocity a of the mechanical arm joint through sampling_t；

(4) Constructing a mechanical arm state value network V according to the Markov decision process principle_ψWherein psi represents a parameter of the arm state value network by inputting the image s_tObtain a status value v_tThe method specifically comprises the following steps:

(4-1) repeating the step (3-1) -the step (3-11) and constructing a mechanical arm state value network V_ψThe first layer to the eleventh layer of (1);

(4-2) mechanical arm state value network V_ψThe twelfth layer of the neural network is a fully connected neural network, and the number of input neurons is the last oneF of layer output₁₁Output neuron-state cost function v_tNeuron weight is W₁₂；

(4-3) obtaining a mechanical arm state value network V according to the step (4-1) to the step (4-2)_ψ；

(5) Constructing a mechanical arm state action value network Q according to the Markov decision process principle_θWherein θ represents a parameter of the robot status motion value network, and θ represents a parameter of the robot status motion value network_θInput image s_tAnd joint angular velocity a_tObtaining a state action value q_tThe mapping relationship of (2) specifically comprises the following steps:

(5-1) repeating the steps (3-1) to (3-10) and constructing a mechanical arm state action value network Q_θThe first to tenth layers of network structure;

(5-2) mechanical arm state action value network Q_θThe eleventh layer of (2) is a fully-connected neural network, and the number of input neurons is F output by the last layer₁₀And joint angular velocity a of space robot_tThe number of the combined neurons is F₁₁Neuron weight is W₁₁；

(5-3) mechanical arm state action value network Q_θThe twelfth layer of the neural network is a fully connected neural network, and the number of input neurons is F output by the previous layer₁₁The output neuron is a state action value function q_tNeuron weight is W₁₂；

(5-4) obtaining a mechanical arm state action value network Q according to the step (5-1) to the step (5-3)_θ；

(6) To step (3) mechanical arm control strategy networkInputting the image s collected at the sampling time t in the step (1)_tThe output is the joint angular velocity a of the mechanical arm_tThe angular velocity a of the joint_tThe output is transmitted to a proportional-derivative controller C, and the proportional-derivative controller C outputs the torque to obtain joint torque so as to realize the control of the robot; when samplingImage s collected at moment t_tAnd joint tracking desired joint angular velocity a_tInputting the reward function in the step (2) to obtain a reward value r_tAnd an image s at time t +1 is obtained_t+1Obtaining the interactive information pair E at the time t_t＝<s_t，a_t，r_t，s_t+1>；

(7) Traversing all images s acquired within a T period_t＝1：TAnd (4) repeating the step (6) to obtain a plurality of groups of interactive information pairs, wherein the plurality of groups of interactive information pairs form an experience playback pool D and respectively go to the mechanical arm state value network V in the step (4)_ψAnd (5) a mechanical arm state action value network Q_θInputting images s collected at different sampling moments_tObtaining a status value v_ψ(s_t) Sum state action value q_t(s_t，a_t)；

(8) And (3) establishing an optimization target according to the maximum entropy reinforcement learning, so that the accumulated reward and the entropy of the strategy are maximized:

wherein the content of the first and second substances,representing the entropy of the information; the maximum information entropy is used for improving the exploration capability of the algorithm and enabling the learned strategy to generate larger randomness. The degree of randomness α of learning represents the degree of randomness of the maximization of the strategy in learning. In general, by introducing entropy for the policy, and ultimately achieving a higher entropy value, this will enable the agent to explore the environment more extensively.

(9) According to the principle of strategy iteration, under the maximum entropy reinforcement learning framework, strategy learning is alternately carried out through strategy evaluation and strategy improvement. Network Q for minimizing Bellman residual errors to action value of mechanical arm state_θTraining the parameters to obtain a mechanical arm state action value network Q_θAt the optimum parameter J_QExpression of (θ):

(10) value network V for state of mechanical arm by minimizing square loss value_ψObtaining a mechanical arm state value network V by parameter training_ψOptimum parameter J of_V(ψ) expression:

(11) training the strategy function by minimizing the expected relative entropy divergence to obtain the optimized parameter of the strategy functionExpression:

(12) training the mechanical arm control strategy network of the step (3) to the step (5) by using the training target obtained in the step (9) to the step (11) and a gradient descent methodState action value network Q_θAnd status value network V_ψCompleting the training of the network;

(13) real-time acquisition of image s obtained by camera mounted on space robot base_tImages s to be acquired in real time_tMechanical arm control strategy network input to step (12)Outputting to obtain the angular velocity a of the mechanical arm joint at the sampling time t_tAnd the control on the space robot mechanical arm is realized, and a track planning task in a T time period is realized.