1. A multi-purpose composite single-cell micro-manipulation system, comprising:

two-degree-of-freedom operating platform: used for placing a culture dish;

three-degree-of-freedom manipulator: the capillary needle is used for installing a capillary needle for adsorption or injection and is arranged on one side of the two-degree-of-freedom operating platform;

five-degree-of-freedom manipulator: the capillary needle is used for installing a capillary needle for adsorption or injection and is arranged on the other side of the two-degree-of-freedom operation platform;

telecentric binocular camera: the method is used for obtaining three-degree-of-freedom manipulator and five-degree-of-freedom manipulator capillary needle point images;

inverting the microscopic camera: for obtaining images of the capillary needles and the cells in the culture dish;

the upper computer: the device is used for acquiring images of the needle point of the capillary needle and images of cells in the capillary needle and the culture dish, processing the images to obtain control quantity, and respectively driving the two-degree-of-freedom operation platform, the three-degree-of-freedom manipulator and the five-degree-of-freedom manipulator to finish cell injection operation.

2. The multi-purpose composite unicell micro-manipulation system according to claim 1, wherein said five-degree-of-freedom manipulator comprises a three-degree-of-freedom translational micropositioner and two rotational joints.

3. The multi-purpose composite single-cell micro-manipulation system of claim 1, further comprising an LED light source for providing illumination from a telecentric binocular camera.

4. The multi-purpose composite single-cell micro-manipulation system of any one of claims 1 to 3, wherein the illumination of the inverted microscope camera is provided using a light source and a condenser lens.

5. A control method based on the multi-purpose composite single-cell micro-operation system of any one of claims 1 to 4, characterized by comprising the following steps:

placing a culture dish on a two-degree-of-freedom operation platform, installing a capillary needle for adsorption on a three-degree-of-freedom manipulator, and installing a capillary needle for injection on a five-degree-of-freedom manipulator;

the capillary needle tips of the two manipulators are enabled to be in the visual field of the binocular telecentric camera through manual adjustment, the binocular telecentric camera shoots capillary needle tip images, and the images are input into an upper computer to finish the alignment of the capillary needles on the two sides;

using an inverted microscope camera to shoot images of the capillary needle and the cells, inputting the images into an upper computer, acquiring the relative pose between the capillary needle and the cells, and operating the three-degree-of-freedom manipulator to enable the adsorbed capillary needle to be close to the cells and adsorb the cells, and moving the adsorbed cells to an injection position;

and (3) shooting images of the capillary needle and the cell by using an inverted microscope camera, acquiring the relative positions of the capillary needle and the cell injection point on the images, and controlling a five-degree-of-freedom manipulator to move the capillary needle by using an upper computer to complete cell injection.

6. The control method according to claim 5, wherein the culture dish is placed on the two-degree-of-freedom operation platform, the adsorption capillary needle is installed on the three-degree-of-freedom manipulator, and the injection capillary needle is installed on the five-degree-of-freedom manipulator, and the method further comprises a calibration step, so that the telecentric binocular camera and the inverted microscope camera use the same reference coordinate system and are the same as the reference coordinate system of all movements of the cell injection system based on the telecentric binocular vision and the inverted microscope vision during the working process.

7. The control method according to claim 5, wherein the capillary needle tips of the two manipulators are within the visual field of the binocular telecentric camera through manual adjustment, the binocular center camera shoots the capillary needle tip image and inputs the capillary needle tip image into an upper computer to complete the alignment of the two capillary needles, and the method specifically comprises the following steps:

the telecentric binocular camera shoots images of the capillary needles on two sides, and the upper computer processes the images and extracts the image characteristics of the capillary needles;

the upper computer obtains the relative poses of the axes of the two capillary needles and the needle point points through image characteristics, further converts the relative poses into motion parameters of the three-degree-of-freedom manipulator and the five-degree-of-freedom manipulator, and sends motion instructions to the corresponding manipulators to enable the capillary needles to generate spatial motion until the axes of the two capillary needles are aligned, so that the posture-based visual servo control is formed.

8. The control method according to claim 5, characterized in that the upper computer processes the images taken by the inverted microscope camera, in particular:

processing the image, wherein the processing comprises threshold segmentation and edge detection, and further determining a proper part of the cell for injecting the medicine;

in the process of cell adsorption of the capillary needle, a gap between the cell and the contact position of the capillary needle is identified, the displacement of the needle point in the next step is provided, and after the needle point of the capillary needle is attached to the surface of the cell, the adsorption tightness degree is judged according to the deformation degree of the cell surface contour.

9. The control method according to claim 5, wherein the telecentric camera is composed of a camera and a telecentric lens.

10. The method for controlling according to claim 8, wherein the suitable site is in particular to ensure that the capillary needle can be inserted into the organelle without damaging the cells.

Background

Cell microinjection is the most basic, most commonly used technique in bioengineering. The technology can be simply summarized by using a glass capillary needle with the diameter of about 10 microns to penetrate through a cell membrane and directly inject exogenous substances into cells. The main characteristics are as follows: 1) the damage to cells is small; 2) the injection amount of the exogenous substances is controllable; 3) the application range is wide, and the application range comprises the functions of injection fertilization in cytoplasm, drug verification and the like. The existing cell injection platform is mostly based on a general biological microscope and comprises two three-degree-of-freedom manipulators which can respectively control the adsorption of cells and a capillary needle for injection. In addition, a cell microinjection platform using a specially designed cell placement vessel instead of a capillary needle for adsorption and only one manipulator is very popular.

Automation of cell microinjection platforms, however, has long been a challenge in the art. The biological microscope as a basis can only provide one viewing angle, so that the operator (or computer) can not observe the alignment between the two capillary needles from other angles. The two capillary needles are not aligned effectively, and then the capillary needles slide on the surface of the cell, or the capillary needles contact the cell at a wrong angle, so that the cell is separated from the adsorption needle. This dilemma can be summarized as the lack of constraint imposed by monocular microscopic vision, which fails to solve the depth information of the capillary needle, resulting in erroneous estimation of the relative position between needles and cells. For the dual manipulator system, the common method is to utilize the characteristic of small depth of field of the biological microscope to confirm the relative depth of the two side capillary needles by the focusing degree of the capillary needles. The disadvantage of this method is that the depth of field of the microscope directly becomes the error interval of the depth. While for systems with dedicated placement vessels there are processing methods similar to indirect measurements: 1) the control system firstly moves the needle along the direction of the optical axis of the camera until the needle tip touches the bottom of the culture dish and deforms; 2) at the moment of deformation, the needle point moves forwards along the needle point axis in the image, and the control system can determine that the current position is the reference depth of motion control by observing the time of deformation of the needle point; 3) at the moment, the control system records the matched position feedback in the operator and records the position feedback as a reference position; 4) when the manipulator controls the capillary needle to move in space, the control system reads feedback information of the manipulator at any time, and the feedback information is compared with a reference position, and a comparison result can be regarded as the current position (including the depth) of the needle point. The greatest disadvantage of this method is the physical damage to the needle tip. However, both of the above two methods can only obtain the relative depth between the point and the point, and the problem of posture adjustment of the capillary needle cannot be actually solved. In order to completely solve the alignment problem of the capillary needle, it is necessary to provide a cell microinjection system having a multi-vision manipulator with more than three degrees of freedom.

Disclosure of Invention

In order to overcome the defects and shortcomings of the prior art, the invention provides a multi-eye composite single-cell micro-operation system and a control method.

The invention adopts the following technical scheme:

a multi-purpose composite single-cell micro-manipulation system, comprising:

two-degree-of-freedom operating platform: used for placing a culture dish;

three-degree-of-freedom manipulator: the capillary needle is used for installing a capillary needle for adsorption or injection and is arranged on one side of the two-degree-of-freedom operating platform;

five-degree-of-freedom manipulator: the capillary needle is used for installing a capillary needle for adsorption or injection and is arranged on the other side of the two-degree-of-freedom operation platform;

telecentric binocular camera: the method is used for obtaining three-degree-of-freedom manipulator and five-degree-of-freedom manipulator capillary needle point images;

inverting the microscopic camera: for obtaining images of the capillary needles and the cells in the culture dish;

the upper computer: the device is used for acquiring images of the needle point of the capillary needle and images of cells in the capillary needle and the culture dish, processing the images to obtain control quantity, and respectively driving the two-degree-of-freedom operation platform, the three-degree-of-freedom manipulator and the five-degree-of-freedom manipulator to finish cell injection operation.

Furthermore, the five-degree-of-freedom manipulator comprises a three-degree-of-freedom translational micropositioner and two rotary joints.

Further, the binocular video camera further comprises an LED light source used for providing illumination of the telecentric binocular camera.

Further, the illumination of the inverted microscope camera is provided using a light source and a condenser lens.

A control method based on a multi-purpose compound single-cell micro-operation system comprises the following steps:

placing a culture dish on a two-degree-of-freedom operation platform, installing a capillary needle for adsorption on a three-degree-of-freedom manipulator, and installing a capillary needle for injection on a five-degree-of-freedom manipulator;

the capillary needle tips of the two manipulators are enabled to be in the visual field of the binocular telecentric camera through manual adjustment, the binocular telecentric camera shoots capillary needle tip images, and the images are input into an upper computer to finish the alignment of the capillary needles on the two sides;

using an inverted microscope camera to shoot images of the capillary needle and the cells, inputting the images into an upper computer, acquiring the relative pose between the capillary needle and the cells, and operating the three-degree-of-freedom manipulator to enable the adsorbed capillary needle to be close to the cells and adsorb the cells, and moving the adsorbed cells to an injection position;

and (3) shooting images of the capillary needle and the cell by using an inverted microscope camera, acquiring the relative positions of the capillary needle and the cell injection point on the images, and controlling a five-degree-of-freedom manipulator to move the capillary needle by using an upper computer to complete cell injection.

Further, a culture dish is placed on the two-degree-of-freedom operation platform, a capillary needle for adsorption is installed on the three-degree-of-freedom manipulator, and a calibration step is further included after the capillary needle for injection is installed on the five-degree-of-freedom manipulator, so that the telecentric binocular camera and the inverted microscopic camera use the same reference coordinate system which is the same as the reference coordinate system of all movements of the cell injection system based on the telecentric binocular vision and the inverted microscopic vision in the working process.

Further, make the capillary needle point of two manipulators in the visual field of binocular telecentric camera through manual adjustment, the binocular centre of a circle camera shoots capillary needle point image, inputs upper computer, accomplishes the alignment of both sides capillary needle, specifically does:

the telecentric binocular camera shoots images of the capillary needles on two sides, and the upper computer processes the images and extracts the image characteristics of the capillary needles;

the upper computer obtains the relative poses of the axes of the two capillary needles and the needle point points through image characteristics, further converts the relative poses into motion parameters of the three-degree-of-freedom manipulator and the five-degree-of-freedom manipulator, and sends motion instructions to the corresponding manipulators to enable the capillary needles to generate spatial motion until the axes of the two capillary needles are aligned, so that the posture-based visual servo control is formed.

Further, the upper computer processes the image shot by the inverted microscope camera, specifically:

processing the image, wherein the processing comprises threshold segmentation and edge detection, and further determining a proper part of the cell for injecting the medicine;

in the process of cell adsorption of the capillary needle, a gap between the cell and the contact position of the capillary needle is identified, the displacement of the needle point in the next step is provided, and after the needle point of the capillary needle is attached to the surface of the cell, the adsorption tightness degree is judged according to the deformation degree of the cell surface contour.

Further, the telecentric camera is composed of a camera and a telecentric lens.

Further, the suitable site is in particular to ensure that the capillary needle is able to be inserted into the organelle without damaging the cells.

The invention has the beneficial effects that:

1) the invention adopts binocular telecentric vision to carry out space attitude identification and pinpoint location on the capillary needle. The telecentric camera belongs to an affine camera, can overcome the phenomenon of 'big-small-big-near' in the depth of field of the camera, and is more beneficial to measurement. Compared with a common microscope, the telecentric camera used in the invention has larger field of view and depth of field, and can still accurately observe even if the needle point rotates or translates in a small range.

2) The invention adopts a mixed visual servo control method based on gesture and image, which is respectively used for the alignment of the capillary needle and the movement of the capillary needle in the process of cell puncture. The control form can fully utilize the advantages of binocular telecentric vision and inverted microscopic vision, and solves the defect that a common cell injection platform lacks depth information.

Drawings

Fig. 1 is a schematic structural view of the present invention.

Detailed Description

The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited to these examples.

Examples

As shown in fig. 1, a multi-view composite single-cell micro-manipulation system, specifically a micro-manipulation system based on telecentric binocular vision and inverted monocular microscopic vision, mainly includes a two-degree-of-freedom manipulation platform 104, a three-degree-of-freedom manipulator 102, a five-degree-of-freedom manipulator 105, a telecentric binocular camera 106, an inverted microscopic camera 103, and an upper computer 101. The upper computer is respectively connected with the three-degree-of-freedom manipulator, the five-degree-of-freedom manipulator, the telecentric binocular camera 106 and the inverted microscope camera 103.

The two-degree-of-freedom operating platform comprises: fixed on the supporting structure, for placing and moving a culture dish for loading cells as an operation object.

The three-degree-of-freedom manipulator comprises: the capillary needle is used for installing a capillary needle for adsorption or injection and is arranged on one side of the two-degree-of-freedom operating platform; in this embodiment, an adsorption capillary needle is installed.

The five-degree-of-freedom manipulator comprises: the capillary needle for installing adsorption or injection is arranged on the other side of the two-degree-of-freedom operation platform, and the capillary needle for injection is installed in the embodiment.

The telecentric binocular camera is fixed on the supporting structure and used for shooting an attitude picture of the needle point of the capillary needle and transmitting the attitude picture to an upper computer through a cable and an image acquisition card, and illumination during working is provided by an LED light source.

The inverted microscope camera is used for shooting pictures of cells and the needle point of the capillary needle and transmitting the pictures to an upper computer through a cable and an image acquisition card, and the light of a working device is provided by a light source and a condenser lens for the inverted microscope.

In this embodiment, the telecentric binocular camera includes two 1394 cameras produced by AVT and a telecentric lens produced by Moritex.

The inverted microscope camera in this embodiment includes a Cameralink camera, produced by Basler, a tube and a flat field, long working distance achromatic biomicroscopic objective.

Furthermore, the five-degree-of-freedom manipulator comprises a three-degree-of-freedom translational micropositioner and two rotary joints; the five-degree-of-freedom manipulator can receive an instruction from the upper computer, so that the three-degree-of-freedom translational micropositioner and the two rotary joints move to control the posture and the position of the capillary needle tip; the five-degree-of-freedom manipulator has a manual macro-movement function with three translational degrees of freedom, and the macro-movement degree of freedom cannot be controlled by an upper computer. The three-degree-of-freedom manipulator can receive an instruction from the upper computer to generate movement so as to complete the position control of the capillary needle point. The two manipulators are arranged on two sides of the two-freedom-degree translation platform.

The structure of the system after being built is shown in figure 1, a binocular telecentric camera forms an angle of approximately 90 degrees, the binocular telecentric camera and an inverted microscopic camera are respectively positioned at the upper side and the lower side of a cell operation space (where a cell culture dish is placed), a three-degree-of-freedom manipulator provided with an adsorption capillary needle and a five-degree-of-freedom manipulator provided with an injection capillary needle are respectively positioned at the left side and the right side of the cell operation space, and the axes of the two capillary needles can be aligned. The mounting positions of the cameras, the manipulator and the capillary needle must be such that the capillary needle can appear simultaneously in most of the field of view of the three cameras during the movement of the manipulator to be controlled.

The cable connections between the upper computer 101 and the three-degree-of-freedom manipulation platform 102, the inverted microscope camera 103, the two-degree-of-freedom manipulation platform 104, the five-degree-of-freedom manipulation platform 105 and the binocular telecentric camera system 106 are omitted in fig. 1. In addition, the light source that the vision system usually needs to be configured with is also omitted.

In the embodiment, the depth information of the capillary needle space characteristics is extracted through the image shot by the binocular telecentric camera, and the extraction of the posture and the position of the capillary needle is completed; the upper computer obtains the relative position of the tip point of the capillary needle and the specified point of the cell of the operation object on the image through the inverted microscope camera; the upper computer calculates the spatial movement of the capillary needle to be completed by calculating the posture and the position of the capillary needle, so as to complete the alignment of the capillary needle and the injection of the cells, and the method comprises the following steps:

1) calibrating the telecentric binocular camera and the inverted microscope camera by using a calibration plate;

2) placing a culture dish on a two-degree-of-freedom operation platform, installing a capillary needle for adsorption on a three-degree-of-freedom manipulator, and installing a capillary needle for injection on a five-degree-of-freedom manipulator;

3) after the installation is finished, the needle point of the capillary needle is simultaneously appeared in two fields of view of the binocular telecentric camera through manual adjustment;

4) shooting a capillary needle tip image by using a telecentric binocular camera, and restoring the posture of the capillary needle by using an upper computer to complete the alignment of the capillary needles on two sides;

5) shooting cells in the capillary needle and the culture dish by using a telecentric binocular camera, reducing the relative pose between the capillary needle and the cells by using an upper computer, controlling the capillary needle to be close to the cells, adsorbing the cells, and moving the cells to an injection position;

6) shooting images of the capillary needle and the cell by using an inverted microscope camera, acquiring the relative positions of the capillary needle and the cell injection point on the images, and controlling the capillary needle to complete cell injection by using an upper computer;

further, in the step (1), the specific operation process is as follows:

11) using two view fields of a telecentric binocular camera to respectively shoot images of the calibration plate with different postures and positions;

12) placing a calibration plate on a two-degree-of-freedom translation platform to enable a coordinate system of the calibration plate to be overlapped with a reference coordinate system of the system;

13) shooting a picture of the calibration plate by using a telecentric binocular camera, processing the image obtained in the step by using an upper computer, extracting control points, completing calibration of the telecentric binocular camera, and obtaining internal parameters and external parameters of the telecentric binocular camera;

14) keeping the calibration plate still, shooting an image of the calibration plate by using an inverted microscope camera, processing the image by an upper computer, completing calibration of the inverted microscope camera, obtaining internal parameters and external parameters of the inverted microscope camera, and solving to obtain the relative poses among the telecentric binocular camera, the inverted microscope camera and a reference coordinate system;

15) and storing the calibration result in the upper computer.

Further, in the process of controlling the capillary needle to finish alignment, the binocular telecentricity camera always keeps shooting, the upper computer continuously extracts the posture of the capillary needle through an image shot by the binocular telecentricity camera, and the motion parameters of the manipulator are calculated. The step 4) comprises the following steps:

41) using a telecentric binocular camera to shoot images of the two sides of the capillary needles, processing the images by an upper computer, and extracting image features of the capillary needles;

42) restoring the relative poses of the axes of the two sides of the capillary needles and the needle point by using the internal parameters and the external parameters of the telecentric binocular camera and the image characteristics obtained in the step 41) by the upper computer;

43) the upper computer converts the obtained relative pose into motion parameters of the three-degree-of-freedom manipulator and the five-degree-of-freedom manipulator, so that a motion instruction is sent to the manipulators, and the capillary needle generates spatial motion;

44) and repeating the steps 41) to 43) until the axes of the two side capillary needles are aligned, and forming the visual servo control based on the gesture.

After the upper computer acquires the images of the inverted microscope camera and the binocular telecentric camera, image processing is carried out, wherein the image processing comprises the following steps: and (3) carrying out image segmentation based on methods such as edge detection or threshold processing, separating and respectively extracting image features to obtain cells and capillary needles.

The image features comprise ORB features, SIFT features or edge features and the like, are matched with the target image in features, calculate the error between the ORB features and the SIFT features, and provide information for visual servo.

The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.