1. A payment method, applied to a local identification device, comprising:

determining an RGB image containing a target face and a target image containing the target face; the target image includes: the target image and the RGB image are images meeting the requirement of acquisition time interval;

performing living body detection on a payment object corresponding to the target face through the RGB image and the target image; and performing face recognition on the target face through the RGB image and the target image;

and under the condition that the payment object is detected to be a living body object and the target face identification is passed, sending a payment request to a payment server, wherein the payment server is used for responding to the payment request and executing payment operation for the payment object.

2. The method of claim 1, wherein the face recognition of the target face from the RGB image and the target image comprises:

determining payment environment information at the current moment;

determining an image to be recognized matched with the payment environment information from the RGB image and the target image;

and carrying out face recognition on the target face based on the image to be recognized.

3. The method of claim 2, wherein the determining the payment environment information at the current time comprises:

determining a target environment parameter, wherein the target environment parameter comprises at least one of: the illumination intensity, the payment distance between the local identification device and the payment object, and the complexity of the environment in which the payment object is located;

and determining the payment environment information according to the target environment parameters.

4. The method according to claim 2 or 3, wherein the determining, from the RGB image and the target image, an image to be recognized that matches the payment environment information comprises:

detecting the illumination intensity of the payment environment at the current moment;

determining the RGB image as the image to be recognized under the condition that the illumination intensity meets the preset intensity requirement;

and under the condition that the illumination intensity does not meet the preset intensity requirement, determining the infrared IR image as the image to be identified.

5. The method according to any one of claims 2 to 4, wherein the face recognition of the target face based on the image to be recognized comprises:

extracting facial features of the image to be recognized to obtain a first facial feature;

searching the first facial feature in a local facial feature library, and determining that the face recognition of the target face is successful if the first facial feature is searched.

6. The method of claim 5, further comprising:

sending a search request to a payment server under the condition that the first facial feature is not searched in the local facial feature library, wherein the search request is used for requesting the payment server to search the first facial feature in a cloud facial feature library;

and determining that the face recognition is successful in the case of receiving a confirmation instruction returned by the payment server for the search request.

7. The method according to any one of claims 1 to 6, wherein the live body detection of the payment object corresponding to the target face through the RGB image and the target image comprises:

cropping a first image containing the target face in the RGB image and cropping a second image containing the target face in the target image;

live body detection of the payment object is performed based on the first image and the second image.

8. The method of claim 7, wherein the in-vivo detection of the payment object based on the first image and the second image comprises:

inputting the first image and the second image into a living body detection model for processing so as to detect the living body of the payment object.

9. The method according to claim 7 or 8, wherein said cropping a first image containing said target face in said RGB image comprises:

performing face detection on the target face in the RGB image to obtain a first detection result, where the first detection result includes: face frames and/or face key points;

cropping a first image including the target face in the RGB image based on the first detection result.

10. The method according to any one of claims 7 to 9, wherein said cropping a second image containing the target face in the target image comprises:

acquiring a first camera shooting parameter of a first camera shooting device for acquiring the RGB image, and acquiring a second camera shooting parameter of a second camera shooting device for acquiring the target image;

determining a mapping relation between the RGB image and the target image based on the first imaging parameter and the second imaging parameter;

determining a first mapping position of a face frame of the target face in the target image based on the mapping relation, and cutting a second image containing the target face in the target image based on the first mapping position; or, determining a second mapping position of each face key point of the target face in the target image based on the mapping relation, and cutting a second image containing the target face in the target image based on the second mapping position.

11. The method of claim 1, wherein determining the RGB image containing the target face and the target image containing the target face comprises:

acquiring a first video stream, carrying out face detection on video frames in the first video stream, and detecting to obtain a first video frame containing a face;

calculating the face quality of the face contained in the first video frame;

determining the RGB image from the first video frame if the face quality meets a quality requirement;

and acquiring a second video stream, and determining the target image from the second video stream.

12. The method of claim 11, wherein the calculating the face quality of the face contained in the first video frame comprises:

performing face detection on the first video frame to obtain a face detection result, wherein the face detection result includes at least one of the following: face key points, face ambiguity, face pose and face detection confidence;

and carrying out data analysis on the face detection result to obtain the face quality.

13. The method of claim 11, wherein determining the RGB image from the first video frame upon determining that the face quality meets a quality requirement comprises:

under the condition that the first video frame comprises a plurality of face parts, determining a face frame of each face part to obtain a plurality of face frames;

and taking the face image which is selected from the frame with the largest face frame in the plurality of face frames and contains the target face as the RGB image.

14. The method of claim 11, further comprising:

generating a target adjusting signal when the face quality of the face contained in the plurality of first video frames is continuously detected not to meet the quality requirement, wherein the target adjusting signal is used for adjusting at least one target parameter: paying the illumination intensity of the environment at the current moment, and acquiring the pose of a first camera of the first video stream;

adjusting the target parameter through the target adjusting signal;

after the target parameter is adjusted, the first video stream is obtained again; and carrying out face detection on the video frame in the newly acquired first video stream.

15. The method of claim 1, further comprising:

counting the successful payment times of the payment object in a first preset time period;

and under the condition that the payment success times meet a first preset time requirement, storing the facial features of the payment object in a local facial feature library.

16. The method of claim 15, wherein storing the facial features of the payment object in a local facial feature library in the case that the payment success number meets a first preset number requirement comprises:

under the condition that the successful payment times meet a first preset time requirement, acquiring historical payment information of the payment object;

determining the activity frequency of the payment position of the payment object at the current moment according to the historical payment information;

judging whether the activity frequency meets a frequency requirement or not;

storing facial features of the payment object in a local facial feature library if the activity frequency meets the frequency requirement.

17. The method according to any one of claims 1 to 16, further comprising:

acquiring a target payment object of which the payment success times in a second preset time period do not meet the requirement of the first preset times;

searching the facial features of the target payment object in a local facial feature library to obtain second facial features;

adding a target feature label to the second facial feature, wherein the target feature label is used for indicating that the second facial feature is a facial feature to be deleted.

18. The method according to any one of claims 1 to 17, further comprising:

counting the target times of face recognition passing of the payment object in a non-living body state under the condition that the payment object is determined not to be a living body object and the face recognition passing is determined;

and sending alarm prompt information to the payment object under the condition that the target times meet a second preset time requirement.

19. A payment device, comprising:

a determination unit configured to determine an RGB image containing a target face and a target image containing the target face; the target image includes: the system comprises an infrared IR image and a depth image, wherein the target image and the RGB image are images meeting the requirement of an acquisition time interval;

a living body detection unit for performing living body detection on the payment object corresponding to the target face through the RGB image and the target image;

a face recognition unit for performing face recognition on the target face through an RGB image and the target image;

and the payment request sending unit is used for sending a payment request to a payment server under the condition that the payment object is detected to be a living body object and the target face identification is passed, and the payment server is used for responding to the payment request and executing payment operation for the payment object.

20. A local identification device, comprising: the device comprises a first camera device, a second camera device and a controller; the first camera device and the second camera device are in communication connection with the controller, and the controller is in communication connection with the payment server through an upper computer;

the first camera device is configured to acquire a first video stream;

the second camera device is configured to acquire a second video stream;

the controller is configured to determine an RGB image containing a target face according to the first video stream and determine a target image containing a target face according to the second video stream; the target image includes: the target image and the RGB image are images meeting the requirement of acquisition time interval; and performing living body detection on a payment object corresponding to the target face through the RGB image and the target image; and performing face recognition on the target face through the RGB image and the target image; and sending a payment request to the payment server through the upper computer under the condition that the payment object is detected to be a living object and the target face is identified to pass, wherein the payment server is used for responding to the payment request and executing payment operation for the payment object.

21. A face payment system, comprising: a payment server, a host computer and a local identification device as claimed in any one of claims 1 to 18, wherein the host computer is adapted to enable a communication connection between the payment server and the local identification device.

22. An electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating over the bus when the electronic device is operating, the machine-readable instructions when executed by the processor performing a payment method as claimed in any one of claims 1 to 18.

23. A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, performs a payment method as claimed in any one of claims 1 to 18.

Background

In order to improve the convenience of payment, face-brushing payment is about to become a new payment mode in real life. Compared with the traditional cash payment and electronic payment means, the face-brushing payment does not need to carry any physical object, and the overall shopping experience of the user can be improved.

However, payment involves property safety issues, so face-brushing payment needs to guarantee face recognition accuracy. The face brushing person and the payment account belong to the same person, so that other people are prevented from impersonating to use the account of other people for payment, and various extreme and complex scenes can be faced at the same time.

At present, the accuracy rate of face recognition is low in the existing face payment mode under the extreme and complex payment environment. Therefore, in order to improve the security of face payment, a face payment scheme capable of meeting various complex payment environments is urgently needed.

Disclosure of Invention

The embodiment of the disclosure at least provides a payment method, a payment device, local identification equipment, a face payment system and face payment equipment.

In a first aspect, an embodiment of the present disclosure provides a payment method applied to a local identification device, including: determining an RGB image containing a target face and a target image containing the target face; the target image includes: the target image and the RGB image are images meeting the requirement of acquisition time interval; performing living body detection on a payment object corresponding to the target face through the RGB image and the target image; and performing face recognition on the target face through the RGB image and the target image; and under the condition that the payment object is detected to be a living body object and the target face identification is passed, sending a payment request to a payment server, wherein the payment server is used for responding to the payment request and executing payment operation for the payment object.

In the embodiment of the disclosure, after the RGB image and the target image containing the target face are determined, the living body detection is performed on the payment object through the RGB image and the target image, so that the accuracy of the living body detection can be improved, and meanwhile, the face recognition precision can be improved through the face recognition performed on the target face through the RGB image and the target image, so that the payment method can be applied to various payment scenes. Particularly, for a complex payment scene, the living body detection result and the face recognition result with high accuracy can still be obtained by adopting the technical scheme provided by the disclosure, so that the payment safety of the payment object is ensured.

In a possible embodiment, the face recognition of the target face through the RGB image and the target image includes: determining payment environment information at the current moment; determining an image to be recognized matched with the payment environment information from the RGB image and the target image; and carrying out face recognition on the target face based on the image to be recognized.

In the above embodiment, the accuracy of face recognition by different types of images is not necessarily the same, depending on the payment environment. For example, in a scene where the payment environment is dim or the payment environment is confused, face recognition through an RGB image may reduce the accuracy of face recognition. Therefore, the image to be recognized matched with the payment environment information is determined, and the face recognition is carried out according to the image to be recognized, so that the accuracy of face recognition can be improved, and the safety of face payment is ensured.

In a possible embodiment, the determining the payment environment information at the current time includes: determining a target environment parameter, wherein the target environment parameter comprises at least one of: the illumination intensity, the payment distance between the local identification device and the payment object, and the complexity of the environment in which the payment object is located; and determining the payment environment information according to the target environment parameters.

In the above embodiment, the payment environment information is determined by using a plurality of different target environment parameters, and a plurality of environment parameters affecting payment operation can be considered, so that the payment scheme disclosed by the invention can be applied to any complex payment scene, and the application range of the technical scheme disclosed by the invention is expanded.

In a possible embodiment, the determining, in the RGB image and the target image, an image to be recognized that matches the payment environment information includes: detecting the illumination intensity of the payment environment at the current moment; determining the RGB image as the image to be recognized under the condition that the illumination intensity meets the preset intensity requirement; and under the condition that the illumination intensity does not meet the preset intensity requirement, determining the infrared IR image as the image to be identified.

In the above embodiment, the illumination intensity of the payment environment at the current time may be detected by a sensor provided on the local identification device, and the illumination intensity of the payment environment at the current time may be determined by performing image processing on the RGB image. Detecting the illumination intensity of the payment environment, and using the illumination intensity as payment environment information; and the face recognition mode of the image to be recognized matched with the illumination intensity is determined in the RGB image and the target image, so that the accuracy of the face recognition can be improved.

In a possible embodiment, the face recognition of the target face based on the image to be recognized includes: extracting facial features of the image to be recognized to obtain a first facial feature; searching the first facial feature in a local facial feature library, and determining that the face recognition of the target face is successful if the first facial feature is searched.

In a possible embodiment, the method further comprises: sending a search request to a payment server under the condition that the first facial feature is not searched in the local facial feature library, wherein the search request is used for requesting the payment server to search the first facial feature in a cloud facial feature library; and determining that the face recognition is successful in the case of receiving a confirmation instruction returned by the payment server for the search request.

In the embodiment of the present disclosure, the comparison of the facial features may be performed in a local facial feature library, and the comparison of the facial features may also be requested from a payment server through a network. However, because the installation environments of the local recognition devices have large differences, when the communication performance of the network corresponding to the installation environment of the local recognition device is poor, the rate of facial feature comparison is seriously affected, and thus the payment operation of the payment object is seriously affected. Based on this, through the manner described in the above embodiment, firstly, a manner of performing facial feature comparison in the local facial feature library is set, and when the facial feature comparison in the local facial feature library fails, a manner of performing facial feature comparison in the payment server is used, so that an interaction link between the local recognition device and the payment server can be omitted, and even in the case of a poor network, the facial comparison operation can still be performed, thereby further improving the efficiency of facial comparison.

In one possible embodiment, the live body detection of the payment object corresponding to the target face through the RGB image and the target image includes: cropping a first image containing the target face in the RGB image and cropping a second image containing the target face in the target image; live body detection of the payment object is performed based on the first image and the second image.

In one possible embodiment, the live detection of the payment object based on the first image and the second image includes: and inputting the first image and the second image into a living body detection model for processing to obtain the living body detection of the payment object.

When the living body detection is performed on the payment object, the living body detection can be performed on the RGB image through the living body detection model, however, the limitation of the living body detection based on the single frame RGB image is too large, and therefore the accuracy of the human face living body detection cannot be ensured. Based on this, this disclosed technical scheme carries out the live body detection through RGB image and target image pair payment object, can synthesize the fusion identification technology of multiframe to carry out the live body detection to promote the live body and detect the precision.

In a possible embodiment, said cropping a first image containing said target face in said RGB image comprises: performing face detection on the target face in the RGB image to obtain a first detection result, where the first detection result includes: face frames and/or face key points; cropping a first image including the target face in the RGB image based on the first detection result.

In the above embodiment, the first image is obtained by cropping the RGB image, and the first image is used to perform the living body detection and the face recognition, so that unnecessary information in the RGB image can be filtered out, thereby improving the detection accuracy of the living body detection and the recognition accuracy of the face recognition.

In a possible embodiment, said cropping, in said target image, a second image containing said target face comprises: acquiring a first camera shooting parameter of a first camera shooting device for acquiring the RGB image, and acquiring a second camera shooting parameter of a second camera shooting device for acquiring the target image; determining a mapping relation between the RGB image and the target image based on the first imaging parameter and the second imaging parameter; determining a first mapping position of a face frame of the target face in the target image based on the mapping relation, and cutting a second image containing the target face in the target image based on the first mapping position; or, determining a second mapping position of each face key point of the target face in the target image based on the mapping relation, and cutting a second image containing the target face in the target image based on the second mapping position.

In the above embodiment, by determining the mapping relationship, a first mapping position of a face frame of the target face in the target image may be determined, or a second mapping position of each face key point of the target face in the target image may be determined. After the first mapping position or the second mapping position is determined, the position of the target face in the target image can be accurately determined by cutting out the second image containing the target face from the target image according to the first mapping position or the second mapping position, and the detection precision of the living body detection and the recognition precision of the face recognition can be improved when the living body detection and the face recognition are carried out according to the second image and the first image.

In one possible embodiment, the determining the RGB image containing the target face and the target image containing the target face includes: acquiring a first video stream, carrying out face detection on video frames in the first video stream, and detecting to obtain a first video frame containing a face; calculating the face quality of the face contained in the first video frame; determining the RGB image from the first video frame if the face quality meets a quality requirement; and acquiring a second video stream, and determining the target image from the second video stream.

In the above embodiment, after the first video frame with the face quality meeting the quality requirement is screened out from the first video stream, when the living body detection and the face recognition are performed through the first video frame, the detection precision of the living body detection can be improved, and the recognition precision of the face recognition can be improved, so that the payment method can be applied to various scenes to ensure the payment safety of the payment object.

In a possible implementation, the calculating the face quality of the face contained in the first video frame includes: performing face detection on the first video frame to obtain a face detection result, wherein the face detection result includes at least one of the following: face key points, face ambiguity, face pose and face detection confidence; and carrying out data analysis on the face detection result to obtain the face quality.

In the above embodiment, the accuracy of the face quality can be improved by determining the face quality of the face contained in the first video frame according to at least one of the face key point, the face ambiguity, the face pose, and the face detection confidence.

In a possible embodiment, determining the RGB image from the first video frame in the case that it is determined that the face quality meets a quality requirement includes: under the condition that the first video frame comprises a plurality of face parts, determining a face frame of each face part to obtain a plurality of face frames; and taking the face image which is selected from the frame with the largest face frame in the plurality of face frames and contains the target face as the RGB image.

In the above embodiment, the face image of the target face framed by the largest face frame among the plurality of face frames is used as the RGB image, so that the payment time of the payment process can be shortened to improve the payment efficiency.

In a possible embodiment, the method further comprises: generating a target adjusting signal when the face quality of the face contained in the plurality of first video frames is continuously detected not to meet the quality requirement, wherein the target adjusting signal is used for adjusting at least one target parameter: paying the illumination intensity of the environment at the current moment, and acquiring the pose of a first camera of the first video stream; adjusting the target parameter through the target adjusting signal; after the target parameter is adjusted, the first video stream is obtained again; and carrying out face detection on the video frame in the newly acquired first video stream.

In the above embodiment, the target adjustment signal adjusts the light intensity and the pose of the first imaging device, so that the face quality of the face image included in the first video frame can be improved, and the accuracy of the living body detection and the accuracy of the face recognition can be improved.

In a possible embodiment, the method further comprises: counting the successful payment times of the payment object in a first preset time period; and under the condition that the payment success times meet a first preset time requirement, storing the facial features of the payment object in a local facial feature library.

Due to the fact that the payment operation is of a certain locality, the important group covered by a convenience store is the surrounding group. At this time, the user characteristic information (facial features) which is frequently paid by face brushing can be synchronized to the local identification device according to the successful payment times of the payment object, and then the user performs characteristic comparison on the local identification device preferentially when brushing the face, so that the comparison efficiency is improved, and the computing resources of the local identification device are fully utilized.

In a possible embodiment, in the case that the payment success number meets a first preset number requirement, storing the facial features of the payment object in a local facial feature library includes: under the condition that the successful payment times meet a first preset time requirement, acquiring historical payment information of the payment object; determining the activity frequency of the payment position of the payment object at the current moment according to the historical payment information; judging whether the activity frequency meets a frequency requirement or not; storing facial features of the payment object in a local facial feature library if the activity frequency meets the frequency requirement.

In the above embodiment, by storing the facial features of the payment object in the local facial feature library when the activity frequency meets the frequency requirement, the payment object paying more frequently at the payment position can be accurately determined from the plurality of payment objects, so that the comparison efficiency is improved.

In a possible embodiment, the method further comprises: acquiring a target payment object of which the payment success times in a second preset time period do not meet the requirement of the first preset times; searching the facial features of the target payment object in a local facial feature library to obtain second facial features; adding a target feature label to the second facial feature, wherein the target feature label is used for indicating that the second facial feature is a facial feature to be deleted.

In the embodiment of the present disclosure, in order to release the local storage resource of the local recognition device, a corresponding target feature tag may be further set for a second facial feature that is not frequently used in the local facial feature library, so as to indicate that the second facial feature is a facial feature to be deleted through the target feature tag. Then, the local identification device can detect the target feature tag periodically, so as to delete the facial feature to be deleted, and at the same time, the local identification device can instruct the user to delete the facial feature to be deleted manually through the target feature tag.

In a possible embodiment, the method further comprises: counting the target times of face recognition passing of the payment object in a non-living body state under the condition that the payment object is determined not to be a living body object and the face recognition passing is determined; and sending alarm prompt information to the payment object under the condition that the target times meet a second preset time requirement.

In the above embodiment, the number of times of the target passing the face recognition of the payment object in the non-living state is counted, so that the security of the payment operation can be improved, and the property security of the payment object can be ensured.

In a second aspect, an embodiment of the present disclosure further provides a payment apparatus, including: a determination unit configured to determine an RGB image containing a target face and a target image containing the target face; the target image includes: the system comprises an infrared IR image and a depth image, wherein the target image and the RGB image are images meeting the requirement of an acquisition time interval; a living body detection unit for performing living body detection on the payment object corresponding to the target face through the RGB image and the target image; and performing face recognition on the target face through the RGB image and the target image; and the payment request sending unit is used for sending a payment request to a payment server under the condition that the payment object is detected to be a living body object and the target face identification is passed, and the payment server is used for responding to the payment request and executing payment operation for the payment object.

In a third aspect, an embodiment of the present disclosure further provides a local identification device, including: the device comprises a first camera device, a second camera device and a controller; the first camera device and the second camera device are in communication connection with the controller, and the controller is in communication connection with the payment server through an upper computer; the first camera device is configured to acquire a first video stream; the second camera device is configured to acquire a second video stream; the controller is configured to determine an RGB image containing a target face according to the first video stream and determine a target image containing a target face according to the second video stream; the target image includes: the target image and the RGB image are images meeting the requirement of acquisition time interval; and performing living body detection on a payment object corresponding to the target face through the RGB image and the target image; and performing face recognition on the target face through the RGB image and the target image; and sending a payment request to the payment server through the upper computer under the condition that the payment object is detected to be a living object and the target face is identified to pass, wherein the payment server is used for responding to the payment request and executing payment operation for the payment object.

In a fourth aspect, an embodiment of the present disclosure further provides a face payment system, including: the local identification device comprises a payment server and an upper computer, wherein the upper computer is used for realizing communication connection between the payment server and the local identification device.

In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, including: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating via the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the steps of the first aspect described above, or any possible implementation of the first aspect.

In a sixth aspect, this disclosed embodiment further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program, when executed by a processor, performs the steps in the first aspect described above or any one of the possible implementation manners of the first aspect.

In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly described below, and the drawings herein incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the technical solutions of the present disclosure. It is appreciated that the following drawings depict only certain embodiments of the disclosure and are therefore not to be considered limiting of its scope, for those skilled in the art will be able to derive additional related drawings therefrom without the benefit of the inventive faculty.

Fig. 1 shows a schematic structural diagram of a local identification device provided in an embodiment of the present disclosure;

fig. 2 is a schematic structural diagram of another local identification device provided in the embodiment of the present disclosure;

fig. 3 is a schematic structural diagram illustrating a control panel of a local identification device according to an embodiment of the present disclosure;

FIG. 4 illustrates a flow chart of a payment method provided by an embodiment of the present disclosure;

FIG. 5 illustrates a timing diagram of a payment method provided by an embodiment of the present disclosure;

fig. 6 shows a schematic diagram of a payment device provided by an embodiment of the present disclosure;

fig. 7 is a schematic diagram illustrating a face recognition system provided by an embodiment of the present disclosure;

fig. 8 shows a schematic diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure, presented in the figures, is not intended to limit the scope of the claimed disclosure, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the disclosure without making creative efforts, shall fall within the protection scope of the disclosure.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

The term "and/or" herein merely describes an associative relationship, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Research shows that the existing face payment mode has low accuracy rate of face recognition under extreme and complex payment environments. Therefore, in order to improve the security of face payment, a face payment scheme capable of meeting various complex payment environments is urgently needed.

Based on the above research, in the embodiment of the present disclosure, after the RGB image and the target image containing the target face are determined, the living body detection is performed on the payment object through the RGB image and the target image, so that the accuracy of the living body detection can be improved, and meanwhile, the face recognition precision can be improved through the face recognition performed on the target face through the RGB image and the target image, so that the payment method can be applied to various payment scenes. Particularly, for a complex payment scene, the living body detection result and the face recognition result with high accuracy can still be obtained by adopting the technical scheme provided by the disclosure, so that the payment safety of the payment object is ensured.

For the convenience of understanding the present embodiment, a detailed description will be given to a local identification device disclosed in the embodiments of the present disclosure.

Referring to fig. 1, a schematic structural diagram of a local identification device provided in an embodiment of the present disclosure is shown, where the local identification device includes: a first imaging device 11, a second imaging device 12, and a controller 13.

As shown in fig. 1, the first camera 11 and the second camera 12 are both in communication connection with the controller 13, and the controller 13 is in communication connection with the payment server through an upper computer.

A first camera 11, which may also be referred to as an RGB camera, is configured to capture a first video stream.

A second camera 12 configured to capture a second video stream.

A controller 13 configured to determine an RGB image containing a target face from the first video stream and determine a target image containing a target face from the second video stream; the target image includes: the target image and the RGB image are images meeting the requirement of acquisition time interval; and performing living body detection on a payment object corresponding to the target face through the RGB image and the target image; and performing face recognition on the target face through the RGB image and the target image; and sending a payment request to the payment server through the upper computer under the condition that the payment object is detected to be a living object and the target face is identified to pass, wherein the payment server is used for responding to the payment request and executing payment operation for the payment object.

In this disclosure, the payment server may be a bank server, and the upper computer may be a cash register. The local identification equipment is in communication connection with the upper computer through the controller, and the upper computer can establish communication connection with the bank server. That is, the upper computer is a device for transmitting data instructions in the middle.

Here, the target image and the RGB image satisfying the acquisition time interval requirement may be understood as: the acquisition time of the target image and the acquisition time of the RGB image are the same, or the time interval between the acquisition time of the target image and the acquisition time of the RGB image is less than a preset time interval, for example, 10 milliseconds.

As shown in fig. 1, the second image pickup device 12 includes: an IR camera 121.

An IR camera 121 configured to capture a second video stream containing infrared IR images.

In the disclosed embodiment, after the infrared IR image is acquired, the distance may be calculated based on 3D structured light calculations, generating a depth image.

In the embodiment of the present disclosure, a depth camera may also be provided to acquire a depth image by the depth camera.

The first camera 11 and the second camera 12 may be mounted in the vicinity of the device where the object payment system is located by a fixing means. Here, it may be possible to set the mounting positions of the first imaging device 11 and the second imaging device 12 on the fixing device to be adjustable. For example, it may be set to automatically adjust the mounting positions of the first camera 11 and the second camera 12 on the fixing device according to the height of the payment object.

Besides the adjustable installation positions, the first camera device 11 and the second camera device 12 can also be set to have adjustable illumination angles of the first camera device 11 and the second camera device 12 so as to meet more complex use scenes.

In the present embodiment, the number of the first image pickup devices 11 may be set to at least one, and for example, a plurality of first image pickup devices 11 may be provided. For example, an active first camera device and a standby first camera device may be provided, and when the active first camera device works normally, the standby first camera device is not turned on; when the main first camera device works abnormally, the standby first camera device is started. A plurality of second image pickup devices 12 may also be provided. For example, an active second camera device and a standby second camera device may be provided, and when the active second camera device works normally, the standby second camera device is not turned on; and when the main second camera shooting device works abnormally, the standby second camera shooting device is started.

According to the above description, the local identification device provided by the technical scheme of the disclosure can be applied to any payment scene, and meanwhile, the existing payment device can be utilized, so that the complete payment operation is relatively convenient and rapid.

In the embodiment of the disclosure, after the RGB image and the target image containing the target face are determined, the living body detection is performed on the payment object through the RGB image and the target image, so that the accuracy of the living body detection can be improved, and meanwhile, the face recognition precision can be improved through the face recognition performed on the target face through the RGB image and the target image, so that the payment method can be applied to various payment scenes. Particularly, for a complex payment scene, the living body detection result and the face recognition result with high accuracy can still be obtained by adopting the technical scheme provided by the disclosure, so that the payment safety of the payment object is ensured.

Fig. 2 is a schematic structural diagram of another alternative local identification device. As shown in fig. 2, the local recognition apparatus includes: a controller and a single chip system SOC (System on chip) which are in communication connection with an upper computer; the single-chip system SOC comprises a first image pickup device and a second image pickup device.

As shown in fig. 2, the controller includes: preview controller, device controller, software development kit symphony SDK, and AI controller.

The system on a chip SOC comprises: a camera service-camera server; a media service-multimedia server; an AI service-AI server; an embedded neural Network Processor (NPU); HAL semantic storage model; RGB cameras, IR cameras and depth cameras.

The preview controller is used to control the preview status of the preview video, for example, the preview resolution of each video frame in the preview video, the preview size of each video frame, the preview color of each video frame, and other status information. For example, as shown in fig. 3, the user may control the preview state of the preview video by manipulating a first control button of a preview controller shown in the target payment system.

The equipment controller is used for controlling the working states of the first camera device and the second camera device. Here, the operation state includes at least one of: on state, off state, altitude information, illumination angle, etc. For example, as shown in fig. 3, the user may control the operation state by manipulating the second control button of the device controller presented in the target payment system.

The AI controller is used for controlling detection parameters, wherein the detection parameters comprise: parameters of face detection, parameters of living body detection, and parameters of face recognition. Here, controlling the detection parameter may be understood as updating the detection parameter.

The RGB camera (first camera) is used to capture a video including the RGB image. The IR camera (second camera) is used to capture video containing the infrared IR image described above. The depth camera (second camera) is used to capture video containing the depth image.

The camera server is used for controlling the first camera device and the second camera device so as to realize communication connection between the controller and the camera devices (RGB camera device, IR camera and depth camera). The multimedia server is used for controlling the coding and decoding operations of the video streams collected by the first camera device and the second camera device. The AI server is used for providing face detection service, face recognition service and living body detection service. The HAL semantic storage model is used to store corresponding process data.

For the convenience of understanding of the present embodiment, a payment method disclosed in the embodiments of the present disclosure will be described in detail below. Referring to fig. 4, a flowchart of a payment method provided in the embodiment of the present disclosure is shown, where the method includes steps S401 to S405, where:

s401: determining an RGB image containing a target face and a target image containing the target face; the target image includes: infrared IR images and/or depth images, the target image and the RGB image being images that meet the acquisition time interval requirements.

In the disclosed embodiment, the target image includes the following cases:

the first condition is as follows: an infrared IR image; case two: infrared IR images and depth images; case three: a depth image.

The infrared IR image and the depth image corresponding to each condition are images meeting the requirement of the acquisition time interval between the infrared IR image and the RGB image.

Here, the RGB image may perform steps S403 and S405 described below with the target image in either case.

Here, the target image and the RGB image satisfying the acquisition time interval requirement may be understood as: the acquisition time of the target image and the acquisition time of the RGB image are the same, or the time interval between the acquisition time of the target image and the acquisition time of the RGB image is less than a preset time interval, for example, 10 milliseconds.

S403: performing living body detection on a payment object corresponding to the target face through the RGB image and the target image; and performing face recognition on the target face through the RGB image and the target image.

In the embodiment of the present disclosure, the face recognition may be performed on the target face at the same time when the living body detection is performed on the payment object. By setting a mode of simultaneously carrying out living body detection and face identification, the payment time of the payment process can be shortened, and the payment efficiency of the payment process is improved.

S405: and under the condition that the payment object is detected to be a living body object and the target face identification is passed, sending a payment request to a payment server, wherein the payment server is used for responding to the payment request and executing payment operation for the payment object.

In an optional embodiment, in the case that the payment object is detected not to be a living object and/or the target face recognition fails, a prompt message of payment failure is returned.

And returning prompt information of payment failure when at least one of the detection result and the recognition result fails according to the detection result of the living body detection and the recognition result of the face recognition.

Here, the local identification device may return a prompt message of payment failure to the background payment server, so that the background payment server records the failed payment operation. Besides, prompt information of payment failure can be displayed for the payment object on the payment interface.

In the embodiment of the present disclosure, before the payment object performs the payment operation by using the payment method described in the technical solution of the present disclosure, a user registration operation needs to be performed.

Here, the user may perform a registration operation through an APP matched with the above-described local recognition device. When performing the registration operation, the face of the user needs to be scanned so as to obtain the face image of the user, so as to determine the facial features of the user according to the face image. After the face image is acquired, corresponding account information can be bound for the face image, and the account information includes at least one of the following: and identity information such as bank accounts, contact information, names and the like for executing payment operation.

In the embodiment of the disclosure, in the case that the living body detection of the payment object passes and the face recognition passes, account information bound for the payment object in advance may be acquired, and the local recognition device may send the account information to the bank server, so that the bank server performs a payment operation, for example, a deduction operation, based on the account information.

In the embodiment of the disclosure, after the RGB image and the target image containing the target face are determined, the living body detection is performed on the payment object through the RGB image and the target image, so that the accuracy of the living body detection can be improved, and meanwhile, the face recognition precision can be improved through the face recognition performed on the target face through the RGB image and the target image, so that the payment method can be applied to various payment scenes. Particularly, for a complex payment scene, the living body detection result and the face recognition result with high accuracy can still be obtained by adopting the technical scheme provided by the disclosure, so that the payment safety of the payment object is ensured.

In an optional embodiment, in step S101, determining an RGB image including a target face and a target image including the target face specifically includes the following steps:

step S1011, acquiring a first video stream, and performing face detection on a video frame in the first video stream to obtain a first video frame containing a face through detection;

step S1012, calculating the face quality of the face contained in the first video frame;

step S1013, under the condition that the face quality meets the quality requirement, determining the RGB image according to the first video frame;

step S1014, acquiring a second video stream, and determining the target image from the second video stream.

In the embodiment of the present disclosure, a first video stream may be captured by a first camera device as shown in fig. 1, where video frames in the first video stream are RGB image frames. For each RGB image frame, face detection may be performed on the RGB image frame to detect whether a specified face (e.g., a human face) is included in the RGB image frame. If the RGB image frame is detected to contain the designated face, the RGB image frame is determined to be the first video frame.

At this time, the face quality of the face contained in the first video frame may be calculated, and it may be determined whether the face quality satisfies a quality requirement. For example, it may be determined whether the face quality is greater than or equal to a quality threshold, and if yes, it is determined that the quality requirement is satisfied, at this time, the RGB image may be determined from the first video frame.

In an embodiment of the present disclosure, calculating the face quality of the face included in the first video frame specifically includes the following steps:

(1) and performing face detection on the first video frame to obtain a face detection result, wherein the face detection result comprises at least one of the following: face key points, face ambiguity, face pose and face detection confidence;

(2) and carrying out data analysis on the face detection result to obtain the face quality.

First, face detection is performed on a first video frame, and the above-described face detection result is obtained. The face key points may include the following key points: facial contour keypoints, keypoints of eyes, keypoints of nose, keypoints of mouth, and keypoints of eyebrows. The face ambiguity is used for representing the definition of an image area where the target face is located in the first video frame. The face pose may comprise at least one of: the included angle between the target face part and the horizontal plane, the included angle between the plane where the target face part is located and the imaging plane of the first camera device, the sight line angle of eyes and the like. The face detection confidence is used to characterize the probability that the first video frame contains a face.

After the face detection result is obtained, the face quality may be determined based on the face detection result.

In an optional embodiment, a weight value may be correspondingly assigned to each sub-result in the face detection result, and the sum of the weight values corresponding to all the sub-results is 1. Then, each sub-result and the weight value are subjected to weighted summation calculation, and the obtained calculation result is used as the face quality.

In the above embodiment, the accuracy of the face quality can be improved by determining the face quality of the face contained in the first video frame according to at least one of the face key point, the face ambiguity, the face pose, and the face detection confidence.

In the disclosed embodiment, after the RGB image is determined as described above, a second video stream captured by the IR camera and/or the depth camera may be acquired. Then, a second video frame which is the same as the acquisition time of the first video frame is determined in the second video stream, or a second video frame which meets the requirement of the acquisition time interval of the first video frame is determined. And then determining the determined second video frame as the target image.

If the target image contains the infrared IR image, a second video stream captured by the IR camera may be acquired, and a second video frame that is the same as the capture time of the first video frame is determined in the second video stream, or a second video frame whose capture time interval with the first video frame meets the requirement is determined, and the second video frame is determined as the infrared IR image.

If the target image contains the depth image, a second video stream shot by the depth camera can be obtained, a second video frame which is the same as the acquisition moment of the first video frame is determined in the second video stream, or a second video frame which meets the requirement of the acquisition time interval of the first video frame is determined, and the second video frame is determined to be the depth image.

In the above embodiment, after the first video frame with the face quality meeting the quality requirement is screened out from the first video stream, when the living body detection and the face recognition are performed through the first video frame, the detection precision of the living body detection can be improved, and the recognition precision of the face recognition can be improved, so that the payment method can be applied to various scenes to ensure the payment safety of the payment object.

In an optional implementation manner, in the step S1011, determining the RGB image according to the first video frame specifically includes the following processes:

firstly, under the condition that a plurality of face parts are contained in the first video frame, determining a face frame of each face part to obtain a plurality of face frames;

and secondly, taking the face image of the target face selected from the largest face frame in the plurality of face frames as the RGB image.

If the payment environment at the current time is complex, for example, the number of customers in the store at the current time is large, or the number of people queuing for payment is large, a situation that the video frame of the first video stream contains multiple faces may occur. At this time, in order to avoid erroneous payment using face information of other payment objects, a target face may be determined from among a plurality of faces included in the first video frame so as to perform a payment operation using information about a payment object corresponding to the target face.

Here, a face frame of each face may be determined, and then, a face image of a target face selected from the face frames with the largest size among the plurality of face frames may be taken as an RGB image.

It is understood that after the payment operation is performed on the payment object corresponding to the target face framed by the maximum face frame, the payment success information may be sent to the payment object.

In the embodiment of the present disclosure, the method may further include the following steps:

determining a complete face frame containing a complete face from the plurality of face frames; carrying out face identification on the face in the complete face frame to obtain an identification result; and determining a successful face frame with successfully recognized face in the complete face frame according to the recognition result, and recording object information of an object to which the face framed by the successful face frame belongs, for example, label information, name information and other related information of the object to which the face belongs.

Through the processing mode, when the payment operation is executed through the wrong payment object, the selected wrong payment object can be determined timely and accurately, so that the property safety of the user is ensured.

In the above embodiment, the face image of the target face framed by the largest face frame among the plurality of face frames is used as the RGB image, so that the payment time of the payment process can be shortened to improve the payment efficiency. Meanwhile, by recording the identity information of the object corresponding to the face framed by the face frame containing the complete face in the plurality of face frames, the object with the payment error can be quickly and accurately determined under the condition that the target face is selected incorrectly, so that the safety of the payment operation is further improved.

In an optional embodiment, in a case that the first video frame includes a plurality of face parts, a selection operation of the payment object on the plurality of face parts may be further detected, and based on the selection operation, the face part selected by the payment object is determined to be a target face part, and an image including the target face part is cut out in the first video frame to be an RGB image.

Here, in order to avoid that the user selects the face of another user for face payment, verification information may be generated and sent to the user to whom the target face selected by the user belongs, for example, short message verification. And under the condition that the verification information input by the payment object is detected to be correct, determining that the face selected by the payment object is the face of the payment object.

In an alternative embodiment, the method further comprises the steps of:

(1) and generating a target adjusting signal when the face quality of the face contained in the plurality of first video frames is continuously detected not to meet the quality requirement, wherein the target adjusting signal is used for adjusting at least one target parameter: paying the illumination intensity of the environment at the current moment, and acquiring the pose of a first camera of the first video stream;

(2) adjusting the target parameter through the target adjusting signal;

(3) after the target parameter is adjusted, the first video stream is obtained again; and carrying out face detection on the video frame in the newly acquired first video stream.

If the payment environment at the current time does not meet the condition, the quality of the face contained in the first video frame may not meet the quality requirement. Based on this, it may be provided that the target adjustment signal is generated in a case where it is detected that the face quality of the face contained in the consecutive plurality of first video frames does not satisfy the quality requirement. Here, the target adjustment signal is used to adjust the illumination intensity of the payment environment at the current time and/or to adjust the pose of the first camera capturing the first video stream.

Specifically, an illumination device with adjustable illumination intensity may be set in the local identification device, and at this time, the illumination intensity of the illumination device may be adjusted by the target adjustment signal, so as to adjust the illumination intensity of the payment environment at the current time.

Here, in addition to adjusting the illumination intensity, the pose of the first image pickup apparatus, for example, information such as the lens orientation of the first image pickup apparatus, may be adjusted by the target adjustment signal.

In the above embodiment, the target adjustment signal adjusts the light intensity and the pose of the first imaging device, so that the face quality of the face image included in the first video frame can be improved, and the accuracy of the living body detection and the accuracy of the face recognition can be improved.

In an optional implementation manner, in step S103, performing face recognition on the target face through the RGB image and the target image specifically includes the following steps:

step S11, determining the payment environment information at the current moment;

in an optional embodiment, determining the payment environment information at the current time includes:

(1) determining a target environment parameter, wherein the target environment parameter comprises at least one of the following: the illumination intensity, the payment distance between the local identification device and the payment object, and the complexity of the environment in which the payment object is located;

(2) and determining the payment environment information according to the target environment parameters.

In embodiments of the present disclosure, after the target environmental parameters are determined, a score for each target environmental parameter may be determined. For example, the score for each target environmental parameter may be determined by calculating a ratio between the target environmental parameter and the standard environmental parameter.

After the score is calculated, a weight value which is allocated to each target environment parameter in advance can be obtained; then, the score and the weight value are weighted and summed to obtain payment environment information.

The payment environment information is determined through various different target environment parameters, and various environment parameters influencing payment operation can be considered, so that the payment scheme disclosed by the invention can be suitable for any complex payment scene, and the application range of the technical scheme disclosed by the invention is expanded.

Step S12, determining an image to be recognized that matches payment environment information in the RGB image and the target image.

Here, a threshold value may be set for the payment environment information, for example, when the payment environment information is greater than the threshold value a, the RGB image may be selected as the image to be recognized, and for example, when the payment environment information is less than or equal to the threshold value a, the target image may be selected as the image to be recognized.

Step S13, performing face recognition on the target face based on the image to be recognized.

In the above embodiment, the accuracy of face recognition by different types of images is not necessarily the same, depending on the payment environment. For example, in a scene where the payment environment is dim or the payment environment is confused, face recognition through an RGB image may reduce the accuracy of face recognition. Therefore, the image to be recognized matched with the payment environment information is determined, so that the face recognition is carried out according to the image to be recognized, the accuracy of the face recognition can be improved, and the safety of face payment is ensured.

In an optional embodiment, in the case that the payment environment information includes the illumination intensity, the step S12, determining the image to be recognized that matches the payment environment information from the RGB image and the target image, may further include the following steps:

detecting the illumination intensity of the payment environment at the current moment; determining the RGB image as the image to be recognized under the condition that the illumination intensity meets the preset intensity requirement; and under the condition that the illumination intensity does not meet the preset intensity requirement, determining the infrared IR image as the image to be identified.

Here, the illumination intensity of the payment environment at the current time may be determined by detecting the illumination intensity of the first video frame (or detecting the illumination intensity of other video frames adjacent to the first video frame).

In the implementation of the present disclosure, the illumination detection may be performed on the first video frame (or other adjacent video frames in the first video frame) through an identification model of the illumination intensity built in the local identification device, so as to obtain the illumination intensity of the payment environment at the current time.

In another alternative embodiment, a sensor may be built in the local identification device, so as to detect the illumination intensity of the payment environment where the local identification device is located in real time through the sensor.

After the illumination intensity is determined, it can be determined whether the current input illumination is good or not according to the illumination intensity. And if the illumination is good, determining that the illumination intensity meets the preset intensity requirement, and extracting the facial features by adopting the RGB image. If the illumination is judged to be insufficient, the illumination intensity is determined not to meet the preset intensity requirement, at the moment, the infrared IR image can be selected to be used for facial feature extraction, after the features are extracted, the face features are searched in a local facial feature library of local recognition equipment, and the identity of the face brushing object is determined.

In the above embodiment, the illumination intensity of the payment environment at the current time may be detected by a sensor provided on the local identification device, and the illumination intensity of the payment environment at the current time may be determined by performing image processing on the RGB image. Detecting the illumination intensity of the payment environment, and using the illumination intensity as payment environment information; and the face recognition mode of the image to be recognized matched with the illumination intensity is determined in the RGB image and the target image, so that the accuracy of the face recognition can be improved.

In an optional embodiment, the step S13, performing face recognition on the target face based on the image to be recognized, includes the following steps:

step S131, extracting facial features of an image to be recognized to obtain a first facial feature;

step S132, searching the first facial feature in a local facial feature library, and determining that the face recognition of the target face is successful under the condition that the first facial feature is searched.

Specifically, in the embodiment of the present disclosure, facial feature extraction may be performed on an image to be recognized through a feature extraction network, so as to obtain a first facial feature. Then, the local facial feature library is searched for the first facial feature, and in the case where the first facial feature is searched for, it is determined that the face recognition of the target face is successful.

In the embodiment of the disclosure, when the first facial feature is not searched in the local facial feature library, sending a search request to a payment server, where the search request is used to request the payment server to search the cloud facial feature library for the first facial feature; and determining that the face recognition is successful under the condition that a confirmation instruction returned by the payment server for the search request is received.

Thus, in searching for a first facial feature, the first facial feature may first be searched in a local facial feature library; and under the condition that the first facial feature is not searched, sending a search request to a payment server so that the payment server searches the first facial feature, and at the moment, searching the first facial feature in a cloud facial feature library by the payment server. The payment server, in the event that the first facial feature is searched, may return a confirmation instruction to the local recognition device to confirm detection of the first facial feature.

In the embodiment of the present disclosure, the comparison of the facial features may be performed in a local facial feature library, and the comparison of the facial features may also be requested from a payment server through a network. However, because the installation environments of the local recognition devices have large differences, when the communication performance of the network corresponding to the installation environment of the local recognition device is poor, the rate of facial feature comparison is seriously affected, and thus the payment operation of the payment object is seriously affected. Based on this, through the manner described in the above embodiment, firstly, a manner of performing facial feature comparison in the local facial feature library is set, and when the facial feature comparison in the local facial feature library fails, a manner of performing facial feature comparison in the payment server is used, so that an interaction link between the local recognition device and the payment server can be omitted, and even in the case of a poor network, the facial comparison operation can still be performed, thereby further improving the efficiency of facial comparison.

In an optional embodiment, in the step S103, performing living body detection on the payment object corresponding to the target face through the RGB image and the target image specifically includes the following steps:

a step S21 of cropping a first image containing the target face in the RGB image, and cropping a second image containing the target face in the target image;

step S22, performing liveness detection on the payment object based on the first image and the second image.

In the disclosed embodiment, a first image containing the target face may be cropped in the RGB image, and a second image containing the target face may be cropped in the infrared IR image and/or the depth image.

Here, if the sizes of the first image and the second image are not the same, the sizes of the first image and the second image may be adjusted to be the same.

Then, the first image and the second image are input into a living body detection model to be processed, and living body detection on the payment object is obtained.

When the living body detection is performed on the payment object, the living body detection can be performed on the RGB image through the living body detection model, however, the limitation of the living body detection based on the single frame RGB image is too large, and therefore the accuracy of the human face living body detection cannot be ensured. Based on this, this disclosed technical scheme carries out the live body detection through RGB image and target image pair payment object, can synthesize the fusion identification technology of multiframe to carry out the live body detection to promote the live body and detect the precision.

In an optional embodiment, in step S21, the cropping the first image containing the target face from the RGB image specifically includes the following steps:

(1) and performing face detection on the target face in the RGB image to obtain a first detection result, wherein the first detection result comprises: face frames and/or face key points;

(2) and cutting out a first image containing the target face in the RGB image based on the first detection result.

In the embodiment of the present disclosure, a face detection model may be used to perform face detection on a target face included in the RGB image, so as to obtain a face frame and/or a face key point.

Here, the face frame may be coordinates of respective vertices of the face frame. The face key points may be: key points of the face contour, key points of the eyes, key points of the nose, key points of the mouth, and key points of the eyebrows.

After the first detection result is determined, the first image can be cut out from the RGB image according to the first detection result.

In the above embodiment, by cropping the first image to perform the living body detection and the face recognition by the first image, unnecessary information in the RGB image can be filtered out, thereby improving the detection accuracy of the living body detection and the recognition accuracy of the face recognition.

In an optional implementation manner, in step S21, the clipping the second image including the target face in the target image specifically includes the following steps:

(1) acquiring a first camera shooting parameter of a first camera shooting device for acquiring the RGB image, and acquiring a second camera shooting parameter of a second camera shooting device for acquiring the target image;

(2) determining a mapping relation between the RGB image and the target image based on the first image pickup parameter and the second image pickup parameter;

(3) determining a first mapping position of a face frame of the target face in the target image based on the mapping relation, and cutting a second image containing the target face in the target image based on the first mapping position; or, determining a second mapping position of each face key point of the target face in the target image based on the mapping relation, and cutting a second image containing the target face in the target image based on the second mapping position.

In the embodiment of the present disclosure, a mapping relationship between each pixel point in the RGB image and the target image may be determined based on the image capturing parameters of the first image capturing device and the second image capturing device, where the mapping relationship may be understood as: and (4) mapping positions of pixel points A in the RGB image in the target image.

Specifically, a target mapping matrix may be determined according to the image capturing parameters of the first image capturing device and the second image capturing device, and the pixel points in the RGB image and the mapping positions in the target image may be determined by the target mapping matrix.

Accordingly, after the target mapping matrix is determined, the position information of the target face in the target image may be determined based on the target mapping matrix, and the second image containing the target face may be cropped in the target image based on the position information.

In an alternative embodiment, based on the target mapping matrix, the position information of the target face in the target image is determined, and based on the position information, the second image containing the target face is clipped in the target image, and the specific process is described as follows:

and acquiring a face frame obtained after face detection is carried out on the target face in the RGB image.

And determining the position information of the face frame, and then determining a first mapping position of the face frame in the target image according to the determined target mapping matrix.

After the first mapping position is determined, the position information of the target face in the target image can be determined based on the first mapping position, and then a second image containing the target face is cut out from the target image based on the position information.

In another alternative embodiment, based on the target mapping matrix, the position information of the target face in the target image is determined, and based on the position information, the second image containing the target face is clipped in the target image, and the specific process is described as follows:

and acquiring face key points obtained after face detection is carried out on the target face in the RGB image.

And determining the position information of each face key point, and then determining a second mapping position of the face key point in the target image according to the determined target mapping matrix.

After the second mapping position is determined, the position information of the target face in the target image can be determined based on the second mapping position, and a second image containing the target face can be cut out from the target image based on the position information.

In the above embodiment, by determining the mapping relationship, a first mapping position of a face frame of the target face in the target image may be determined, or a second mapping position of each face key point of the target face in the target image may be determined. After the first mapping position or the second mapping position is determined, the position of the target face in the target image can be accurately determined by cutting out the second image containing the target face from the target image according to the first mapping position or the second mapping position, and the detection precision of the living body detection and the recognition precision of the face recognition can be improved when the living body detection and the face recognition are carried out according to the second image and the first image.

In an alternative embodiment, the method further comprises the steps of:

(1) counting the successful payment times of the payment object in a first preset time period;

(2) and under the condition that the payment success times meet a first preset time requirement, storing the facial features of the payment object in a local facial feature library.

In the embodiment of the disclosure, the payment success times of each payment object in the first preset time period may be counted. If the payment success times are larger than the preset times, determining that the payment success times meet the first preset times requirement, and at this time, storing the facial features of the payment object in a local facial feature library.

In particular, facial features recognized by the local recognition device may be stored in a local facial feature library. In addition, the local identification device may request facial features of the payment device from the payment server to store the requested facial features in a local facial feature library.

Here, the first preset time period may be set to be one week, one half month, one month, or 2 months, and the specific duration of the first preset time period is not specifically limited by the present disclosure, which is subject to practical circumstances.

Due to the fact that the payment operation is of a certain locality, the important group covered by a convenience store is the surrounding group. At this time, the user characteristic information (facial features) which is frequently paid by face brushing can be synchronized to the local identification device according to the successful payment times of the payment object, and then the user performs characteristic comparison on the local identification device preferentially when brushing the face, so that the comparison efficiency is improved, and the computing resources of the local identification device are fully utilized.

In an alternative embodiment, step (2) above: under the condition that the payment success times meet a first preset time requirement, storing the facial features of the payment object in a local facial feature library, wherein the method comprises the following steps:

(1) acquiring historical payment information of the payment object under the condition that the successful payment times meet a first preset time requirement;

(2) determining the activity frequency of the payment position of the payment object at the current moment according to the historical payment information;

(3) judging whether the activity frequency meets the frequency requirement or not;

(4) and storing the facial features of the payment object in a local facial feature library under the condition that the activity frequency meets a frequency requirement.

In the embodiment of the disclosure, in the case that the payment success number of each payment object at any one payment position meets the first preset number requirement, the historical payment information of the payment object can be further acquired.

Here, the historical payment information may be the number of payments of the payment object at the payment location at the past time, or may also be the number of payments of the payment object within the range of the target area where the payment location is located at the past time.

If the payment times are determined to be larger than or equal to a certain threshold value, the activity frequency of the payment object at the payment position is determined to meet the frequency requirement, and at this time, the facial features of the payment object can be stored in the local facial feature library.

Here, the target area range may be understood as a circular area having a fixed distance as a radius with the current payment position as a center.

In the above embodiment, by storing the facial features of the payment object in the local facial feature library when the activity frequency meets the frequency requirement, the payment object paying more frequently at the payment position can be accurately determined from the plurality of payment objects, so that the comparison efficiency is improved.

In an alternative embodiment, the method further comprises the steps of:

(1) acquiring a target payment object of which the payment success frequency in a second preset time period does not meet the requirement of the first preset frequency;

(2) searching the facial features of the target payment object in a local facial feature library to obtain second facial features;

(3) and adding a target feature label to the second facial feature, wherein the target feature label is used for indicating that the second facial feature is a facial feature to be deleted.

In the embodiment of the present disclosure, in order to release the local storage resource of the local recognition device, a corresponding target feature tag may be further set for a second facial feature that is not frequently used in the local facial feature library, so as to indicate that the second facial feature is a facial feature to be deleted through the target feature tag. Then, the local identification device can detect the target feature tag periodically, so as to delete the facial feature to be deleted, and at the same time, the local identification device can instruct the user to delete the facial feature to be deleted manually through the target feature tag.

For example, for a certain payment object, the payment object does not perform payment operation in the store for a long time due to moving or changing work, and at this time, a target feature tag may be set in the local facial feature library for the second facial feature.

When the local identification device regularly cleans the data in the local facial feature library, the facial features provided with the target feature tags can be preferentially deleted, so that local storage resources of the local identification device are released.

In an alternative embodiment, the method further comprises the steps of:

(1) counting the target times of face recognition passing of the payment object in a non-living body state under the condition that the payment object is determined not to be a living body object and the face recognition passing is determined;

(2) and sending alarm prompt information to the payment object under the condition that the target times meet a second preset time requirement.

In the embodiment of the present disclosure, if it is determined that the payment object is not a living body according to the living body detection result, however, in the case where it is determined that the recognition is successful through the face recognition result, it may be preliminarily determined that another payment object performs the payment operation through a photograph of the current payment object or another non-living body object.

Therefore, in order to ensure the property security of the current payment object, the number of times of the target passing the face recognition of the payment object in the non-living body state can be counted, that is: the payment object performs a face payment operation in a non-living body state, and the number of times face recognition passes.

And if the target times are greater than or equal to the specified times threshold value, determining that the target times meet the second preset times requirement, and at the moment, sending alarm prompt information to the payment. Here, the specified number threshold may be set to 3 times, and other numbers may be set, and the present disclosure does not specifically limit this.

In the above embodiment, the number of times of the target passing the face recognition of the payment object in the non-living state is counted, so that the security of the payment operation can be improved, and the property security of the payment object can be ensured.

In an alternative embodiment, the method further comprises the steps of:

(1) determining purchasing behavior data of the payment object based on the goods type paid by the payment object after corresponding payment operation is performed on the payment object;

(2) estimating basic attribute information of the payment object based on the RGB image;

(3) determining purchasing characteristic information of the payment object based on the basic attribute information and the purchasing behavior data;

(4) and generating a goods placement strategy of a target store based on the purchase characteristic information, wherein the target store is a store associated with the payment operation.

In the embodiment of the present disclosure, after the payment operation is successfully performed, purchasing behavior data of the payment object may be further determined according to an item type of an item paid by the payment object at the current time, where the purchasing behavior data includes at least one of: the number of the purchased goods, the types of the purchased goods, the shelf positions of the shelves where the purchased goods are located and the like.

After that, basic attribute information of the payment object, for example, attribute information of height, age, gender, etc., may also be estimated from the RGB image.

After obtaining the basic attribute information and the purchasing behavior data, the basic attribute information and the purchasing behavior data can be combined to obtain the purchasing characteristic information of the payment object. After the purchase characteristic information of the payment object is obtained, the goods placing strategy of the target store can be generated by combining the purchase characteristic information of other payment objects, so that the goods placed by the target store can meet the shopping habits of users, and different shopping requirements of different users can be met.

In an alternative embodiment, the method further comprises the steps of:

and under the condition that the face is not detected within a third preset time period, controlling a first camera device and a second camera device to be in a dormant state, wherein the first camera device is a device for collecting the RGB image, and the second camera device is a device for collecting the target image.

By controlling the processing mode that the first camera device and the second camera device are in the dormant state under the condition that the face is not detected in the third preset time period, the local storage resource of the local identification equipment can be saved, and meanwhile, the consumption of electric energy can be reduced.

Referring to fig. 5, a flowchart of a payment method provided in the embodiment of the present disclosure is shown in fig. 5:

first, a "first frame map" is received, where the first frame map includes: the first frame RGB image, the first frame infrared IR image and the first frame depth image.

From "receiving the first frame image", the face detection is performed on the first frame RGB image for 20 ms. In the case where the inclusion of a human face is detected, the face quality of the human face image included in the first frame RGB image is detected over a time of 5 ms. In a case where it is determined that the face quality satisfies the quality requirement, the face feature of the first frame RGB image (or the first frame infrared IR image) is extracted over a time of 25 ms. When the facial features are extracted, the living body detection can be carried out through the first frame RGB image, the first frame infrared IR image and the first frame depth image. After the face features are extracted, feature comparison can be performed on the face features in 25ms, so that a face recognition result is determined according to a comparison result.

And under the condition that the living body detection is determined to pass and the face recognition passes, corresponding payment operation can be executed for the payment object.

The process of "receiving the second frame map", "receiving the third frame map", and "receiving the fourth frame map" in fig. 5 is the same as the process of "receiving the first frame map", and is not repeated here.

As can be known from the time sequence diagram shown in fig. 5, the payment method provided by the technical scheme of the present disclosure can complete the payment operation within 0.08 second, thereby shortening the time for executing the corresponding payment operation for the payment object and improving the payment efficiency.

It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.

Based on the same inventive concept, a payment device corresponding to the payment method is also provided in the embodiments of the present disclosure, and as the principle of solving the problem of the device in the embodiments of the present disclosure is similar to the payment method described above in the embodiments of the present disclosure, the implementation of the device may refer to the implementation of the method, and repeated details are not repeated.

Referring to fig. 6, a schematic diagram of a payment apparatus provided in an embodiment of the present disclosure, the apparatus includes: a determination unit 61, a living body detection unit 62, a face recognition unit 63, a payment request transmission unit 64; wherein the content of the first and second substances,

a determination unit 61 for determining an RGB image containing a target face and a target image containing the target face; the target image includes: the system comprises an infrared IR image and a depth image, wherein the target image and the RGB image are images meeting the requirement of an acquisition time interval;

a live body detection unit 62 for live body detecting the payment object corresponding to the target face through the RGB image and the target image;

a face recognition unit 63 for performing face recognition on the target face through the RGB image and the target image;

a payment request sending unit 64 configured to send a payment request to a payment server in a case where it is detected that the payment object is a living body object and the target face identification is passed, the payment server being configured to perform a payment operation for the payment object in response to the payment request.

In a possible implementation, the face recognition unit 63 is further configured to: determining payment environment information at the current moment; determining an image to be recognized matched with the payment environment information from the RGB image and the target image; and carrying out face recognition on the target face based on the image to be recognized.

In a possible implementation, the face recognition unit 63 is further configured to: determining payment environment information at the current moment, including: determining a target environment parameter, wherein the target environment parameter comprises at least one of: the illumination intensity, the payment distance between the local identification device and the payment object, and the complexity of the environment in which the payment object is located; and determining the payment environment information according to the target environment parameters.

In a possible implementation, the face recognition unit 63 is further configured to: detecting the illumination intensity of the payment environment at the current moment; determining the RGB image as the image to be recognized under the condition that the illumination intensity meets the preset intensity requirement; and under the condition that the illumination intensity does not meet the preset intensity requirement, determining the infrared IR image as the image to be identified.

In a possible implementation, the face recognition unit 63 is further configured to: extracting facial features of the image to be recognized to obtain a first facial feature; searching the first facial feature in a local facial feature library, and determining that the face recognition of the target face is successful if the first facial feature is searched.

In one possible embodiment, the apparatus is further configured to: sending a search request to a payment server under the condition that the first facial feature is not searched in the local facial feature library, wherein the search request is used for requesting the payment server to search the first facial feature in a cloud facial feature library; and determining that the face recognition is successful in the case of receiving a confirmation instruction returned by the payment server for the search request.

In one possible embodiment, the living body detecting unit 62 is configured to: cropping a first image containing the target face in the RGB image and cropping a second image containing the target face in the target image; live body detection of the payment object is performed based on the first image and the second image.

In one possible embodiment, the living body detecting unit 62 is further configured to: inputting the first image and the second image into a living body detection model for processing so as to detect the living body of the payment object.

In one possible embodiment, the living body detecting unit 62 is further configured to: performing face detection on the target face in the RGB image to obtain a first detection result, where the first detection result includes: face frames and/or face key points; cropping a first image including the target face in the RGB image based on the first detection result.

In one possible embodiment, the living body detecting unit 62 is further configured to: acquiring a first camera shooting parameter of a first camera shooting device for acquiring the RGB image, and acquiring a second camera shooting parameter of a second camera shooting device for acquiring the target image; determining a mapping relation between the RGB image and the target image based on the first imaging parameter and the second imaging parameter; determining a first mapping position of a face frame of the target face in the target image based on the mapping relation, and cutting a second image containing the target face in the target image based on the first mapping position; or, determining a second mapping position of each face key point of the target face in the target image based on the mapping relation, and cutting a second image containing the target face in the target image based on the second mapping position.

In a possible implementation, the determining unit 61 is further configured to: acquiring a first video stream, carrying out face detection on video frames in the first video stream, and detecting to obtain a first video frame containing a face; calculating the face quality of the face contained in the first video frame; determining the RGB image from the first video frame if the face quality meets a quality requirement; and acquiring a second video stream, and determining the target image from the second video stream.

In a possible implementation, the determining unit 61 is further configured to: performing face detection on the first video frame to obtain a face detection result, wherein the face detection result includes at least one of the following: face key points, face ambiguity, face pose and face detection confidence; and carrying out data analysis on the face detection result to obtain the face quality.

In a possible implementation, the determining unit 61 is further configured to: under the condition that the first video frame comprises a plurality of face parts, determining a face frame of each face part to obtain a plurality of face frames; and taking the face image which is selected from the frame with the largest face frame in the plurality of face frames and contains the target face as the RGB image.

In one possible embodiment, the apparatus is further configured to: generating a target adjusting signal when the face quality of the face contained in the plurality of first video frames is continuously detected not to meet the quality requirement, wherein the target adjusting signal is used for adjusting at least one target parameter: paying the illumination intensity of the environment at the current moment, and acquiring the pose of a first camera of the first video stream; adjusting the target parameter through the target adjusting signal; after the target parameter is adjusted, the first video stream is obtained again; and carrying out face detection on the video frame in the newly acquired first video stream.

In one possible embodiment, the apparatus is further configured to: counting the successful payment times of the payment object in a first preset time period; and under the condition that the payment success times meet a first preset time requirement, storing the facial features of the payment object in a local facial feature library.

In one possible embodiment, the apparatus is further configured to: under the condition that the successful payment times meet a first preset time requirement, acquiring historical payment information of the payment object; determining the activity frequency of the payment position of the payment object at the current moment according to the historical payment information; judging whether the activity frequency meets a frequency requirement or not; storing facial features of the payment object in a local facial feature library if the activity frequency meets the frequency requirement.

In one possible embodiment, the apparatus is further configured to: acquiring a target payment object of which the payment success times in a second preset time period do not meet the requirement of the first preset times; searching the facial features of the target payment object in a local facial feature library to obtain second facial features; adding a target feature label to the second facial feature, wherein the target feature label is used for indicating that the second facial feature is a facial feature to be deleted.

In one possible embodiment, the apparatus is further configured to: counting the target times of face recognition passing of the payment object in a non-living body state under the condition that the payment object is determined not to be a living body object and the face recognition passing is determined; and sending alarm prompt information to the payment object under the condition that the target times meet a second preset time requirement.

The description of the processing flow of each module in the device and the interaction flow between the modules may refer to the related description in the above method embodiments, and will not be described in detail here.

Referring to fig. 7, which is a schematic diagram of a face payment system provided in an embodiment of the present disclosure, as shown in fig. 7, the face payment system includes: a payment server 100, a local identification device 200 and a host computer 300. Here, the local recognition device may be the local recognition device described in the above embodiment.

The local recognition apparatus 200 includes: the payment system comprises a first camera device, a second camera device and a controller, wherein the controller is in communication connection with a payment server through an upper computer.

The face payment system is described below by taking a self-service payment system as an example.

In the embodiment of the present disclosure, the self-service payment system may be set in an environment with a good payment environment, for example, may be set in an environment with sufficient illumination intensity.

Firstly, a first camera device in the self-service payment system collects a first video stream, and carries out face detection on video frames in the first video stream to obtain a first video frame containing a face through detection. Then, the face quality of the face contained in the first video frame can be calculated through a built-in algorithm of the first camera. In case the face quality meets the quality requirement, the RGB image is determined from the first video frame, e.g. the image containing the face may be cropped in the first video frame as the RGB image.

Thereafter, a target image may be determined in a second video stream captured by a second camera of the self-service payment system. The first camera device and the second camera device synchronously acquire video streams.

After the target image is determined, live body detection can be performed on the payment object corresponding to the face based on the target image and the RGB image, and face recognition can be performed on the face.

Under the condition that the living body detection is passed and the face recognition is passed, the controller can send a payment request to the payment server through the upper computer. And the payment server responds to the payment request after acquiring the payment request and executes payment operation for the payment object.

Under the condition that the living body detection is passed and the face recognition is not passed, the controller can send a search request to the payment server through the upper computer, wherein the search request is used for requesting the payment server to recognize the face. The payment server can return a confirmation instruction to the local identification equipment by the controller under the condition that the face identification is successful so as to determine that the face identification is successful; otherwise, recognition failure information may be returned to the local recognition device to determine that face recognition failed. Under the condition that the face recognition is successful, the payment server can directly execute payment operation for the payment object; the payment operation may also be performed for the payment object in case a payment request of the local identification device is detected.

Corresponding to the payment method in fig. 1, an embodiment of the present disclosure further provides an electronic device 800, as shown in fig. 8, which is a schematic structural diagram of the electronic device 800 provided in an embodiment of the present disclosure, and includes:

a processor 81, a memory 82, and a bus 83; the memory 82 is used for storing execution instructions and includes a memory 821 and an external memory 822; the memory 821 herein is also referred to as an internal memory, and is configured to temporarily store operation data in the processor 81 and data exchanged with the external memory 822 such as a hard disk, the processor 81 exchanges data with the external memory 822 through the memory 821, and when the electronic device 800 operates, the processor 81 communicates with the memory 82 through the bus 83, so that the processor 81 executes the following instructions:

determining an RGB image containing a target face and a target image containing the target face; the target image includes: the target image and the RGB image are images meeting the requirement of acquisition time interval;

performing living body detection on a payment object corresponding to the target face through the RGB image and the target image; and performing face recognition on the target face through the RGB image and the target image;

and under the condition that the payment object is detected to be a living body object and the target face identification is passed, sending a payment request to a payment server, wherein the payment server is used for responding to the payment request and executing payment operation for the payment object.

The disclosed embodiments also provide a computer-readable storage medium having a computer program stored thereon, where the computer program is executed by a processor to perform the steps of the payment method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

The embodiments of the present disclosure also provide a computer program product, where the computer program product carries a program code, and instructions included in the program code may be used to execute the steps of the payment method described in the foregoing method embodiments, which may be referred to specifically for the foregoing method embodiments, and are not described herein again.

The computer program product may be implemented by hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing an electronic device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Finally, it should be noted that: the above-mentioned embodiments are merely specific embodiments of the present disclosure, which are used for illustrating the technical solutions of the present disclosure and not for limiting the same, and the scope of the present disclosure is not limited thereto, and although the present disclosure is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive of the technical solutions described in the foregoing embodiments or equivalent technical features thereof within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present disclosure, and should be construed as being included therein. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.