1. A noise processing method for real-time audio stream playback, comprising the steps of:

s1: sequentially storing the received original audio data packets into a cache queue for storing network data for caching;

s2: sequentially combining a plurality of adjacent original audio data packets in the buffer queue to obtain a combined data packet containing a plurality of original audio data packets;

s3: carrying out frequency conversion on the original audio data in the combined data packet through a frequency conversion algorithm to obtain a converted combined data packet;

s4: splitting the converted combined data packet according to the length of the original audio data packet to obtain a plurality of converted audio data packets;

s5: discarding the split last audio data packet;

s6: and storing the converted and reserved audio data packet into an audio data buffer queue to be played for playing.

2. The noise processing method for real-time audio stream playback according to claim 1,

in the step S2, sequentially combining a plurality of adjacent original audio data packets in the buffer queue, specifically, combining a first original audio data packet in the buffer queue with another adjacent original audio data packet to obtain a combined data packet; or

In step S2, a plurality of adjacent original audio data packets in the buffer queue are sequentially combined, specifically, a first original audio data packet in the buffer queue is combined with more than one original audio data packets sequentially adjacent to the first original audio data packet in the buffer queue to obtain a combined data packet.

3. The noise processing method for real-time audio stream playback according to claim 1,

the noise processing method further includes: at step S5, the last audio packet after splitting is discarded, and the original audio packet before conversion corresponding to the retained audio packet is deleted from the buffering queue.

4. The noise processing method for real-time audio stream playback according to claim 1,

when the plurality of adjacent original audio data packets in the buffer queue are sequentially combined in step S2, if the number of the original audio data packets participating in the combination in the last buffer queue is not enough to meet the number of the combination requirement, the remaining original audio data packets are discarded.

5. The noise processing method for real-time audio stream playing according to claim 1, wherein the noise processing method is executed by a browser end;

before the step S1, the noise processing method further includes:

the browser end establishes a long connection with the audio streaming server and receives the audio data packets which are pushed by the audio streaming server at constant time intervals through the long connection and have constant duration.

6. The noise processing method for real-time audio stream playback according to claim 5,

and the server pushes a 20ms long audio data packet to the browser end every 20 ms.

7. A noise processing system for real-time audio stream playing, comprising an audio stream server side and a browser side, wherein the browser side is configured to receive audio data packets of constant duration pushed by the audio stream server at constant time intervals, and to execute the noise processing method for real-time audio stream playing of any one of claims 1 to 4 based on the received audio data packets.

8. The noise processing system for real-time audio stream playback of claim 7, wherein the browser end comprises:

a transmission module that establishes a long connection with the audio streaming server and thus receives audio data packets from the audio streaming server;

the network data storage module is used for caching the received audio data packet into a network audio data caching queue;

the frequency conversion module is used for sequentially combining a plurality of adjacent audio data packets, performing frequency conversion on the combined audio data packets, splitting the converted audio data packets, discarding the last converted audio data packet, storing the rest audio data packets into a queue to be played, and deleting the corresponding audio data packets in the cache queue;

the audio storage module stores the converted audio data packets in a queue to be played;

and the audio playing module consumes the audio data packets of the queue to be played and plays the audio.

9. A computer-readable storage medium having stored therein a plurality of program codes, characterized in that,

the program code is adapted to be loaded and run by a processor to perform the noise processing method for real-time audio stream playback of any of claims 1-4.

10. A control device comprising a processor and a memory, said memory being adapted to store a plurality of program codes,

the program code is adapted to be loaded and run by the processor to perform the noise processing method for real-time audio stream playback of any of claims 1-4.

Background

In the security industry, scenes of playing audios and videos in real time or playing back the audios and videos are often used. In this scenario, the real-time requirements for the audio or video stream are very high. This also determines that in this scenario, too much data cannot be cached to ensure the fluency experience of playback. However, in this mode, when we play real-time audio stream data at the browser end, it is found that the playing effect of the audio stream is not good, and the following two reasons are mainly found through analysis:

firstly, the sampling frequency of the audio data is different between the audio data collected by the terminal and the audio data required by the browser playing. The sampling frequency of audio data collected by the terminal is 8000 times/second, but the sampling frequency of audio data required by the browser end is 48000 times/second, so that a process of converting source audio data into target audio data exists when the browser end plays a real-time audio stream. However, the problem of excessively modifying audio data exists in the current browser built-in response frequency conversion algorithm when performing frequency conversion on audio data, and the main expression is that the algorithm performs tone reduction processing on the last few bytes of audio data during frequency conversion. This is a major cause of noise when playing real-time audio stream data.

Secondly, in the process that the audio streaming server sends the audio data collected by the terminal to the browser end through the transmission system package, a time interval exists between network data packages. For example, when the audio streaming server sends the audio data to the browser end, the audio streaming server divides the audio data and encapsulates the divided audio data into a plurality of audio data packets with 20ms duration containing the audio data, and sends one data packet to the browser end every 20 ms. After the browser receives the data packets, if the frequency conversion is performed on the audio data in each received audio data packet in sequence through a frequency conversion algorithm, a large amount of noise is generated at this time, and the noise is quite obvious when a large amount of converted audio data is continuously played. Meanwhile, the problem of unsmooth audio playing at the browser end can be caused by network jitter and other reasons in the transmission process of the audio data packet.

Therefore, how to solve the problem of noise generated during the real-time audio stream playing process and ensure the fluency of audio stream playing is a problem to be solved urgently at present.

Disclosure of Invention

The present invention is directed to solving the above-mentioned problems, i.e., the problem of noise generation during the playing of real-time audio streams in the prior art.

In a first aspect, the present invention provides a noise processing method for real-time audio stream playing, the method comprising the steps of:

firstly, the browser end sequentially stores original audio data packets received from a transmission system into a cache queue for storing network data for caching. The original audio data packet contains original audio data.

And then sequentially combining a plurality of adjacent original audio data packets in the buffer queue to obtain a combined data packet containing a plurality of original audio data packets.

And carrying out frequency conversion on the original audio data in the combined data packet through a frequency conversion algorithm to obtain a converted combined data packet. The converted combined packet contains audio data converted from a combination of a plurality of original audio packets.

And splitting an audio data combined data packet obtained by combining and converting a plurality of original audio data packets to obtain an audio data packet containing converted audio data.

And then discarding the last audio data packet after splitting, so that the audio data in the remaining audio data packets are the audio data which is not subjected to the tone reduction processing by the frequency conversion algorithm.

And storing the converted audio data which are not subjected to the frequency conversion algorithm tone reduction processing into a cache queue of the audio data to be played by the browser.

And the browser acquires the converted audio data from the audio data cache queue to be played in sequence to play. Because the played audio data are the audio data after frequency conversion without the reduction processing of the browser frequency conversion algorithm, noise is not generated during playing.

In the above noise processing method for real-time audio stream playing, when a combined data packet including a plurality of converted audio data is split and the last audio data packet after the split is discarded, the last original audio data packet which is cached in the network audio data cache queue and has been subjected to the frequency conversion algorithm conversion processing is retained in the cache queue, and other original audio data packets which have been subjected to the frequency conversion algorithm conversion processing and played in the cache queue are discarded. And then combining the last original audio data packet which is reserved and is just subjected to the frequency conversion algorithm conversion processing with the adjacent subsequent original audio data packet to form a new combined data packet containing a plurality of original audio data. And then, carrying out subsequent audio conversion processing and playing of the converted audio stream data on the newly combined data packet according to the processing method until the playing of the real-time audio stream is finished.

After practice, the noise problem caused by audio conversion is perfectly solved after the audio stream is processed according to the noise processing method.

It should be noted that, due to the particularity of network transmission of audio data, the audio data packet inevitably has problems of network delay and jitter during transmission, and these transmission problems may cause the playing process to be unsmooth. The audio streaming server sends an audio data packet with the duration of 20ms every 20ms, the frequency conversion operation is started after 2-5 audio data packets are cached at the browser end, the converted audio data are sent to the cache queue of the audio data to be played by the browser to be played in sequence for playing, and the played audio data are removed from the cache queue of the audio data to be played by the browser, so that the problem of unsmooth playing caused by network delay and jitter is solved.

Through the two improvements, the problem of noise in the real-time audio stream playing process is solved, and meanwhile, the smoothness of audio stream playing after conversion is guaranteed.

In a second aspect, the present invention provides a system for noise processing method based on real-time audio stream playing, the system including an audio stream server and a browser end, the browser end including:

a transmission module that establishes a long connection with the audio streaming server and thus receives audio data packets from the audio streaming server;

the network data storage module is used for caching the received audio data packet into a network audio data caching queue;

the frequency conversion module is used for sequentially combining a plurality of adjacent audio data packets, performing frequency conversion on the combined audio data packets, splitting the converted audio data packets, discarding the last converted audio data packet, storing the rest audio data packets into a queue to be played, and deleting the corresponding audio data packets in the cache queue;

the audio storage module stores the converted audio data packets in a queue to be played;

and the audio playing module consumes the audio data packets of the queue to be played and plays the audio.

In a third aspect, the present invention also provides a computer readable storage medium storing a plurality of program codes, the program codes being adapted to be loaded and executed by a processor to perform the noise processing method for real-time audio stream playback of the first aspect.

In a fourth aspect, the present invention also provides a control device comprising a processor and a memory device, said memory device being adapted to store a plurality of program codes, said program codes being adapted to be loaded and run by said processor to perform the noise processing method for real-time audio stream playback of the aforementioned first aspect.

The beneficial technical effects are as follows:

the invention solves the problem of noise caused by excessive modification when the frequency conversion algorithm of the browser performs frequency conversion on the real-time audio stream data, so that noise cannot be generated due to frequency conversion in a scene of continuously playing the audio stream in real time.

The invention optimizes the frequency conversion algorithm without modifying the original audio data, thereby ensuring the reducibility of the audio data.

According to the invention, the frequency conversion operation is carried out after the plurality of audio data packets are cached at the browser end, and then the converted audio data are sent to the cache queue of the audio data to be played of the browser to be played in sequence, so that the fluency of the converted audio stream playing is ensured.

Drawings

Preferred embodiments of the present invention are described below with reference to the accompanying drawings, in which:

FIG. 1 is a flow chart illustrating the main steps of a noise processing method for real-time audio stream playback according to an embodiment of the present application;

FIG. 2 is a block diagram of a noise processing system for real-time audio stream playback according to an embodiment of the present application;

fig. 3 is a schematic diagram illustrating a noise processing method for real-time audio stream playback according to the present application.

Detailed Description

Some embodiments of the invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only for explaining the technical principle of the present invention, and are not intended to limit the scope of the present invention.

In the description of the present invention, a "module" or "processor" may include hardware, software, or a combination of both. A module may comprise hardware circuitry, various suitable sensors, communication ports, memory, software components such as program code, or a combination of software and hardware. The processor may be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and/or signal processing functionality. The processor may be implemented in software, hardware, or a combination thereof. Non-transitory computer readable storage media include any suitable medium that can store program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random-access memory, and the like. The term "a and/or B" denotes all possible combinations of a and B, such as a alone, B alone or a and B. The term "at least one A or B" or "at least one of A and B" means similar to "A and/or B" and may include only A, only B, or both A and B. The singular forms "a", "an" and "the" may include the plural forms as well.

In a first aspect, the present invention provides a noise processing method for real-time audio stream playback. Referring to fig. 1, fig. 1 is a flowchart illustrating main steps of a noise processing method for playing a real-time audio stream according to an embodiment of the present disclosure. As shown in fig. 1, the method of the present invention comprises the steps of:

step S1: sequentially storing the received original audio data packets into a cache queue for storing network data for caching;

when the real-time audio stream playing is started, a connection between the server and the browser end is established first, that is, a long link between the browser end and the audio stream server is established. After the long connection with the audio streaming server is established, the audio streaming server pushes the data packets with the constant time length to the browser side at the constant time interval. For example, a general audio streaming server pushes a 20ms long audio data packet to the browser end every 20 ms.

After receiving an original audio data packet sent by an audio streaming server through a transmission system, a browser end sequentially stores the received original audio data packet into a cache queue of the browser end for caching; and when the number of the original audio data packets cached in the cache queue reaches more than 2, the browser end starts to perform frequency conversion on the received original audio data.

Step S2: sequentially combining a plurality of adjacent original audio data packets in the buffer queue to obtain a combined data packet containing a plurality of original audio data packets;

the following describes in detail the process that the browser starts to perform frequency conversion on the original audio data in the received original audio data packet after the number of the audio data packets cached in the cache queue reaches more than two in combination with the schematic diagram of fig. 3.

As shown in fig. 3, there are two original audio data packets, i.e., an a data packet and a B data packet, in the buffer queue of the current original audio data. Two data packets are adjacent to each other in the data buffer queue and located in the first and second of the queue. The method of the invention acquires the first two original audio data packets of the queue from the data buffer queue at one time, and then splices and combines the two original audio data packets to form a combined audio data packet containing the two original audio data packets, namely the A + B data packet shown in figure 3.

It should be noted that, when the real-time audio stream is played, if the number of the remaining original audio packets is not enough to meet the requirement of combining the packets, the remaining original audio packets that cannot be combined are discarded. For example, as shown in this embodiment, if only one original audio data packet is left at last, and at least two original audio data packets are needed for the combined data packet, the last remaining original audio data packet is deleted at this time. Since the discarded original audio packets that cannot be combined are small and not large in number, even if the last remaining original audio packets are discarded, the playing of the audio stream is not affected.

Step S3: carrying out frequency conversion on the original audio data in the combined data packet through a frequency conversion algorithm to obtain a converted combined data packet;

and sending the combined data packet to a browser for frequency conversion, and performing frequency conversion on the audio data in the combined data packet by the browser through a frequency conversion algorithm (such as a response algorithm) to obtain the audio data after the frequency conversion. Namely, the combined data packet is subjected to frequency conversion by the browser to obtain the frequency-converted A + B data packet. The frequency-converted a + B data packet includes frequency-converted a + B combined audio data.

Step S4: splitting the converted combined data packet according to the length of the original audio data packet to obtain a plurality of converted audio data packets;

specifically, the a + B combined audio packet obtained after the frequency conversion is split into a frequency-converted a packet and a frequency-converted B packet. A. And the two frequency-converted data packets B respectively contain the audio data after the frequency conversion of the data packet A and the audio data after the frequency conversion of the data packet B.

Step S5: abandoning the last split audio data packet, and simultaneously deleting the original audio data packet before conversion corresponding to the retained audio data packet from the buffer queue;

specifically, the B packet obtained by splitting after the frequency conversion is discarded, and only the a packet obtained by splitting after the frequency conversion is retained. The split B data packet contains a tone reduction processing part caused by frequency conversion, so that noise is avoided during playing, the split B data packet is discarded, and only the split A data packet obtained after the frequency conversion is reserved. Although the audio data in the reserved a data packet is processed by the frequency conversion algorithm, the audio data obtained after the frequency conversion is the audio data that is not subjected to the modulation reduction processing by the browser frequency conversion algorithm, so that noise is not generated during playing.

And simultaneously deleting original audio data packets before conversion, namely A data packets, corresponding to the audio data reserved after conversion from the buffer queue.

And then the last original audio data packet which is reserved in the buffer queue and is just subjected to the conversion processing by the frequency conversion algorithm, namely the B data packet, is combined with the adjacent next original audio data packet to form a new combined data packet containing two original audio data. And then, carrying out subsequent audio conversion processing on the newly combined data packet according to the processing method.

Step S6: and storing the converted and reserved audio data packet into an audio data buffer queue to be played for playing.

Specifically, the audio data after conversion and without audio conversion algorithm tone reduction processing, that is, the audio data a after frequency conversion algorithm conversion in the data packet a is stored in the cache queue of the audio data to be played by the browser, and is played by the browser. And after the browser plays, deleting the played audio data from the audio data cache queue to be played. Namely, after playing, deleting the audio data A from the audio data cache queue to be played. And playing the audio stream data after the frequency conversion according to the method until the real-time audio stream playing is finished.

It should be noted that, in the present scheme, the audio data in the multiple original audio data packets are combined into combined data by using a splicing combination method, and then the frequency conversion processing is performed on the spliced combined data, so as to avoid the tone reduction processing of the last several bytes by the frequency conversion algorithm, thereby avoiding the noise problem of audio stream playing. The solution is not limited to the combination of only two original packets as described in the above specific embodiments. According to the specific situation, the solution can also combine more than two original data packets, and the specific processing method is not described here.

In addition, equivalent changes or substitutions can be made on the related technical features by a person skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can be within the protection scope of the invention.

In a second aspect, the present invention provides a noise processing system based on real-time audio stream playing, and a specific embodiment of the noise processing system is shown in fig. 2, where the system mainly includes: an audio stream browser end and a service end, namely an audio stream server.

It should be noted that the noise processing system based on real-time audio stream playing adopts the Brower/Server architecture.

As shown in fig. 2, the server mainly includes: an audio stream processing module 11 and a transmission module 12. The audio stream processing module 11 sends original audio stream data to the browser, the transmission module 12 establishes a long connection with the browser, and the service sends an original audio data packet of the audio stream to the browser through the long connection.

The browser end mainly comprises:

a transmission module 21 which establishes a long connection with the audio streaming server and thus receives audio data packets from the audio streaming server.

And the network data storage module 22 is used for buffering the received audio data packets into a network audio data buffering queue.

And the frequency conversion module 23 sequentially combines a plurality of adjacent audio data packets, performs frequency conversion on the combined audio data packets, and splits the converted audio data packets to obtain converted audio data packets. And discarding the last converted audio data packet, storing the remaining converted audio data packets into a queue to be played, and deleting the corresponding audio data packets in the buffer queue.

And the audio storage module 24 is used for storing the converted audio data in a queue to be played.

And an audio playing module 25, which consumes the audio data of the queue to be played for audio playing.

Specifically, when the real-time audio stream playback is started, the browser side first creates a connection between the server and the browser side, that is, a long link between the browser side and the audio stream server, together with the transmission module 12 of the server, via the transmission module 21. After the long connection with the audio streaming server is established, the audio streaming server pushes the data packets with the constant time length to the browser side at the constant time interval. For example, a general audio streaming server pushes a 20ms long audio data packet to the browser end every 20 ms.

After receiving the original audio data packets sent by the audio stream processing module 11 in the audio stream server through the transmission module 21, the browser end sequentially stores the received original audio data packets in a cache queue of the browser end network data storage module 22 for caching; and when the number of the original audio data packets cached in the cache queue reaches more than 2, the browser starts to perform frequency conversion on the received original audio data.

When two or more original audio data packets are in the buffer queue of the original audio data, the first adjacent data packets in the data buffer queue are combined into a combined data packet. The frequency conversion module 23 obtains a plurality of original audio data packets from the data buffer queue at a time, and then splices and combines the obtained plurality of original audio data packets to form a combined audio data packet including a plurality of original audio data packets.

It should be noted that, when the real-time audio stream is played, if the number of the remaining original audio packets is not enough to meet the requirement of combining the packets, the remaining original audio packets that cannot be combined are discarded.

The combined data packet is subjected to frequency conversion by the frequency conversion module 23 to obtain an audio data packet after frequency conversion.

And splitting the combined audio data packet obtained after the frequency conversion into a plurality of audio data packets after the frequency conversion. Each of the frequency-converted audio data packets contains frequency-converted audio data.

And discarding the last data packet obtained by splitting after the frequency conversion, and only keeping the data packets after the other frequency conversions. Although the audio data in the reserved data packet is processed by the frequency conversion algorithm, the audio data obtained after conversion and contained in the reserved data packet is the audio data which is not subjected to the frequency conversion of the browser frequency conversion algorithm and subjected to the tone reduction processing, so that noise is not generated during playing.

And simultaneously deleting the original audio data packet before conversion corresponding to the retained audio data after conversion from the buffering queue.

And then, the last original audio data packet which is reserved in the buffer queue and is just subjected to the conversion processing by the frequency conversion algorithm is combined with a plurality of adjacent original audio data packets, and the combined data packet is combined into a new combined data packet containing a plurality of original audio data. And then carrying out subsequent audio conversion processing on the newly combined data packet by a frequency conversion method.

And storing the converted audio data which is not subjected to the tone reduction processing by the audio conversion algorithm into an audio data cache queue to be played of the audio storage module 24, playing the audio data by the browser through the audio playing module 25, and deleting the played audio data from the audio data cache queue to be played. And playing the audio stream data after the frequency conversion according to the method until the real-time audio stream playing is finished.

In a third aspect, the present invention also provides a computer-readable storage medium storing a plurality of program codes, the program codes being adapted to be loaded and executed by a processor to perform the noise processing method based on real-time audio stream playing of the first aspect.

In a fourth aspect, the present invention also provides a processing apparatus comprising a processor and a storage device, said storage device being adapted to store a plurality of program codes, said program codes being adapted to be loaded and run by said processor to perform the method for noise processing based on real-time audio stream playback of the aforementioned first aspect.

So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can fall into the protection scope of the invention.